On Nasqueron, deployments occur on a dedicated deployment server, distinct from the development servers, both for configuration-as-code and infrastructure-as-code.

Terraform state files, terraform.tfstate, are stored on our deployment server, and can contain both credentials and personal information: plaintext secrets or accounts metadata injected by providers.

To protect this information, we can take two complementary approaches:

• Avoid storing any credential, but accept losing the state and requiring the provider to be compatible with this approach ; for example, the ovh/ovh provider doesn’t seem to support ephemeral blocks.
• Encrypt the state

OpenTofu can help here: it offers an out-of-the-box back-end fot sotring state in OpenBao. However, the OpenTofu registry doesn’t currently compile every providers for FreeBSD¹.

So what do we do with Terraform?

On our FreeBSD-based operations server, we wanted to ensure that, if a disk were compromised, our state files would remain mathematically inaccessible. That’s encryption at rest.

My first instinct was to use PEFS (Private Encrypted File System), a convenient stacked cryptographic filesystem native to FreeBSD.

PEFS allows to encrypt directories on top of a normal filesystem, regardless of whether it’s ZFS or UFS. Operations can be performed by users without requiring root access².

That’s the solution we selected to encrypt the Terraform states, as of September 2026.

Bonus: with a wrapper around PEFS, Terraform/OpenTofu, and Vault, we can mount the PEFS directory in cleartext only for the duration of the Terraform/OpenTofu command, then immediately unmount it afterward. In other words: pefs mount → terraform/tofu <command> → pefs unmount.

Our Terraform architecture

So, we split the architecture using ZFS and PEFS:

• We create a first ZFS dataset /opt/terraform.enc, mounted via PEFS using AES-256-XTS³ at /opt/terraform/encrypted. This is only used for the .tfstate files.
• We create a separate, unencrypted ZFS dataset for the providers files, optimized with lz4 compression and a 128K record size, fine for medium-to-large Go binaries. Let’s call that our ZFS workbench.

By exporting TF_DATA_DIR to the ZFS workbench, Terraform downloads, caches, and executes providers natively. By configuring the Terraform backend "local" block to point to the PEFS directory, the sensitive state file is written directly into the encrypted layer.

The encrypted directory can also be used directly as the backup source, so the data is already encrypted before it leaves the server. The backup operates on the underlying encrypted ZFS dataset, not the mounted plain text view.

 At rest (sealed)                          Unlocked (tf run in progress)
 ───────────────────────────────           ─────────────────────────────────────────
 /opt/terraform.enc  (ZFS, zstd)           /opt/terraform/encrypted  (PEFS, XTS)
 ├── .pefs.db   encrypted key chain        ├── tf-states/$project/terraform.tfstate
 └── ████████   ciphertext of              └── …
     ████████   your tfstate
     │                                       /opt/terraform/tf-data/$project  (ZFS, lz4)
     │ restic backup                         ├── providers/…   (executables)
     ▼                                       └── modules/…
 Backup store, encrypted client-side
 

Automating the crypto: the tf wrapper

Manually mounting PEFS, fetching the passphrase from HashiCorp Vault, and running Terraform is a recipe for forgotten mounts and leaked keys. To solve this, I wrote a POSIX shell wrapper (/usr/local/bin/tf) that handles the entire lifecycle.

When an operator runs tf apply, the script:

1. Verifies the user is in the ops or deploy group.
2. Mounts the PEFS volume as the deploy user.
3. Streams the PEFS passphrase directly from Vault into pefs addkey via a Unix pipe (the password never touches the disk, not even in /tmp).
4. Exports TF_DATA_DIR to the ZFS workbench dataset.
5. Executes Terraform or OpenTofu⁴
6. Uses a shell trap to automatically unmount the PEFS volume when Terraform exits, whether it succeeds, fails, or is interrupted.

The most interesting part of the wrapper is the zero-disk-IO secret injection:

vault kv get -field=password "$VAULT_KV_PATH" | sudo -u deploy pefs addkey -a aes256-xts -c -j - "$PEFS_MOUNT"

The wrapper also has explicit tf mount and tf unmount commands for those rare moments when an operator needs to perform manual state surgery or move files between projects.

The result is a highly secure pipeline, fully deployed by Salt. Our public Git repository contains only declarative code. Our ZFS workbench dataset handles the heavy lifting of Go binaries. And the confidential information lies safely inside an AES-256-encrypted safe, unmounted and inaccessible the moment the Terraform run completes.

Could we store everything in PEFS?

In addition to the state, Terraform creates .terraform directories containing a local copy of the provider Go executable code.

For me, these are two different problems:

• State files need strict confidentiality → encryption is the correct solution.
• Provider caches need good I/O performance and should not clutter the repository source → a dedicated ZFS dataset.

But as we’re a team, we explored the possibility of encrypting the whole /opt/terraform directory in PEFS.

Let’s give it a try: I encrypted the full directory and pointed Terraform at it, both for state and for the providers cache. After moving the .terraform directories and .tfstate files on it, I ran terraform init and KABOOM!

Terraform complained the executable format of the Go executable was invalid⁵. Worst, I couldn’t reproduce the issue: when copying an executable file manually, it worked fine.

Both Terraform and OpenTofu seem to rely heavily on two things:

• Atomic rename() system calls (the Unix mv command), which are not supported by PEFS (you have to cp first, then rm).
• Memory mapping (mmap) with execution flags (PROT_EXEC) to run the binaries.

So, it’s best to keep TF_DATA_DIR outside PEFS.

Sometimes, the best security architecture isn’t about finding a single perfect tool, but about knowing exactly where your tools’ limits are—and building a bridge between them.

Links

• Agora documentation on our Terraform / OpenTofu workflows
• tf wrapper
• Salt configuration for directories and ZFS datasets
• Full implementation commit

Notes

1. If you want to contribute to Nasqueron Operations, you can solve this issue by implementing an automated build toolchain to build the providers we use for FreeBSD amd64. ↩︎
2. For PEFS and FUSE operations, the sysctl vfs.usermount must be set to 1 to allow non-root users to perform the mount. That’s also the same setting that’s used on desktop to allow to mount a USB key. ↩︎
3. Default algorithm is AES-128-XTS. Check your CPU supports AES-NI instructions for crypto-acceleration before switching to AES-256-XTS. If not, the default will provide a better performance. ↩︎
4. That wasn’t the main goal of the wrapper, but as we have one, it automatically picks the right executable, opentofu by default, terraform is in an allowlist. ↩︎
5. That error can more usually occur when Go compilation flags are tweaked, see for example this thread. ↩︎