{"id":1011,"date":"2026-09-06T22:55:11","date_gmt":"2026-09-06T20:55:11","guid":{"rendered":"https:\/\/www.dereckson.be\/blog\/?p=1011"},"modified":"2026-09-06T22:55:11","modified_gmt":"2026-09-06T20:55:11","slug":"encrypt-terraform-state-with-pefs-on-freebsd","status":"publish","type":"post","link":"https:\/\/www.dereckson.be\/blog\/2026\/09\/06\/encrypt-terraform-state-with-pefs-on-freebsd\/","title":{"rendered":"Encrypt Terraform state with PEFS on FreeBSD"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">On Nasqueron, deployments occur on a dedicated deployment server, distinct from the development servers, both for configuration-as-code and infrastructure-as-code.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Terraform state files, <code>terraform.tfstate<\/code>, are stored on our deployment server, and can contain both credentials and personal information: plaintext secrets or accounts metadata injected by providers.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To protect this information, we can take two complementary approaches:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Avoid storing any credential, but accept losing the state and requiring the provider to be compatible with this approach ; for example, the <code>ovh\/ovh<\/code> provider doesn&#8217;t seem to support <a href=\"https:\/\/developer.hashicorp.com\/terraform\/language\/block\/ephemeral\">ephemeral blocks<\/a>. <\/li>\n\n\n\n<li>Encrypt the state<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">OpenTofu can help here: it offers an out-of-the-box back-end fot sotring state in OpenBao. However, the OpenTofu registry doesn&#8217;t currently compile every providers for FreeBSD<sup data-fn=\"62d7b693-7334-475b-a098-5c2bf3498d40\" class=\"fn\"><a href=\"#62d7b693-7334-475b-a098-5c2bf3498d40\" id=\"62d7b693-7334-475b-a098-5c2bf3498d40-link\">1<\/a><\/sup>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So what do we do with Terraform? <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">On our FreeBSD-based operations server, we wanted to ensure that, if a disk were compromised, our state files would remain mathematically inaccessible. That&#8217;s encryption at rest.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">My first instinct was to use <strong><a href=\"https:\/\/wiki.freebsd.org\/PEFS\">PEFS<\/a><\/strong> (Private Encrypted File System), a convenient stacked cryptographic filesystem native to FreeBSD.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">PEFS allows to encrypt directories on top of a normal filesystem, regardless of whether it&#8217;s ZFS or UFS. Operations can be performed by users without requiring root access<sup data-fn=\"826fe37b-3075-4685-ad37-db51dbc72562\" class=\"fn\"><a href=\"#826fe37b-3075-4685-ad37-db51dbc72562\" id=\"826fe37b-3075-4685-ad37-db51dbc72562-link\">2<\/a><\/sup>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That&#8217;s the solution we selected to encrypt the Terraform states, as of September 2026.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Bonus:<\/strong> with a wrapper around PEFS, Terraform\/OpenTofu, and Vault, we can mount the PEFS directory in cleartext only for the duration of the Terraform\/OpenTofu command, then immediately unmount it afterward. In other words: <code>pefs mount \u2192 terraform\/tofu &lt;command> \u2192 pefs unmount<\/code>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Our Terraform architecture<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">So, we split the architecture using ZFS and PEFS:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>We create a first ZFS dataset <code>\/opt\/terraform.enc<\/code>, mounted via PEFS using AES-256-XTS<sup data-fn=\"0423ed78-fadd-4605-a14e-1fc74fe55d38\" class=\"fn\"><a href=\"#0423ed78-fadd-4605-a14e-1fc74fe55d38\" id=\"0423ed78-fadd-4605-a14e-1fc74fe55d38-link\">3<\/a><\/sup> at <code>\/opt\/terraform\/encrypted<\/code>. This is <em>only<\/em> used for the <code>.tfstate<\/code> files.<\/li>\n\n\n\n<li>We create a separate, unencrypted ZFS dataset for the providers files, optimized with <code>lz4<\/code> compression and a <code>128K<\/code> record size, fine for medium-to-large Go binaries. Let&#8217;s call that our ZFS workbench.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">By exporting <code>TF_DATA_DIR<\/code> to the ZFS workbench, Terraform downloads, caches, and executes providers natively. By configuring the Terraform <code>backend \"local\"<\/code> block to point to the PEFS directory, the sensitive state file is written directly into the encrypted layer.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The encrypted directory can also be used directly as the backup source, so the data is already encrypted before it leaves the server. The backup operates on the underlying encrypted ZFS dataset, not the mounted plain text view.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\n At rest (sealed)                          Unlocked (tf run in progress)\n \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500           \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n \/opt\/terraform.enc  (ZFS, zstd)           \/opt\/terraform\/encrypted  (PEFS, XTS)\n \u251c\u2500\u2500 .pefs.db   encrypted key chain        \u251c\u2500\u2500 tf-states\/$project\/terraform.tfstate\n \u2514\u2500\u2500 \u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588   ciphertext of              \u2514\u2500\u2500 \u2026\n     \u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588   your tfstate\n     \u2502                                       \/opt\/terraform\/tf-data\/$project  (ZFS, lz4)\n     \u2502 restic backup                         \u251c\u2500\u2500 providers\/\u2026   (executables)\n     \u25bc                                       \u2514\u2500\u2500 modules\/\u2026\n Backup store, encrypted client-side\n \n<\/pre><\/div>\n\n\n<h2 class=\"wp-block-heading\">Automating the crypto: the <code>tf<\/code> wrapper<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Manually mounting PEFS, fetching the passphrase from HashiCorp Vault, and running Terraform is a recipe for forgotten mounts and leaked keys. To solve this, I wrote a POSIX shell wrapper (<code>\/usr\/local\/bin\/tf<\/code>) that handles the entire lifecycle.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">When an operator runs <code>tf apply<\/code>, the script:<\/p>\n\n\n\n<ol start=\"1\" class=\"wp-block-list\">\n<li>Verifies the user is in the <code>ops<\/code> or <code>deploy<\/code> group.<\/li>\n\n\n\n<li>Mounts the PEFS volume as the <code>deploy<\/code> user.<\/li>\n\n\n\n<li>Streams the PEFS passphrase directly from Vault into <code>pefs addkey<\/code> via a Unix pipe (the password never touches the disk, not even in <code>\/tmp<\/code>).<\/li>\n\n\n\n<li>Exports <code>TF_DATA_DIR<\/code> to the ZFS workbench dataset.<\/li>\n\n\n\n<li>Executes Terraform or OpenTofu<sup data-fn=\"c139a0c7-be3a-44bf-a4dc-ab87468b0ce5\" class=\"fn\"><a href=\"#c139a0c7-be3a-44bf-a4dc-ab87468b0ce5\" id=\"c139a0c7-be3a-44bf-a4dc-ab87468b0ce5-link\">4<\/a><\/sup><\/li>\n\n\n\n<li>Uses a shell <code>trap<\/code> to automatically unmount the PEFS volume when Terraform exits, whether it succeeds, fails, or is interrupted.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">The most interesting part of the wrapper is the <strong>zero-disk-IO secret injection<\/strong>:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><code>vault kv get -field=password \"$VAULT_KV_PATH\" | sudo -u deploy pefs addkey -a aes256-xts -c -j - \"$PEFS_MOUNT\"<\/code><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The wrapper also has explicit <code>tf mount<\/code> and <code>tf unmount<\/code> commands for those rare moments when an operator needs to perform manual state surgery or move files between projects.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The result is a highly secure pipeline, fully deployed by Salt. Our public Git repository contains only declarative code. Our ZFS workbench dataset handles the heavy lifting of Go binaries. And the confidential information lies safely inside an AES-256-encrypted safe, unmounted and inaccessible the moment the Terraform run completes.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Could we store everything in PEFS?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">In addition to the state, Terraform creates <code>.terraform<\/code> directories containing a local copy of the provider Go executable code.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For me, these are two different problems:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>State files need strict confidentiality \u2192 encryption is the correct solution.<\/li>\n\n\n\n<li>Provider caches need good I\/O performance and should not clutter the repository source \u2192 a dedicated ZFS dataset.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">But as we\u2019re a team, we explored the possibility of encrypting the whole <code>\/opt\/terraform<\/code> directory in PEFS.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Let&#8217;s give it a try: I encrypted the full directory and pointed Terraform at it, both for state and for the providers cache. After moving the <code>.terraform<\/code> directories and <code>.tfstate<\/code> files on it, I ran <code>terraform init<\/code> and KABOOM!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Terraform complained the executable format of the Go executable was invalid<sup data-fn=\"e2868daa-7e8f-434e-9572-42585ba60a39\" class=\"fn\"><a href=\"#e2868daa-7e8f-434e-9572-42585ba60a39\" id=\"e2868daa-7e8f-434e-9572-42585ba60a39-link\">5<\/a><\/sup>. Worst, I couldn&#8217;t reproduce the issue: when copying an executable file manually, it worked fine.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Both Terraform and OpenTofu seem to rely heavily on two things:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Atomic <code>rename()<\/code> system calls (the Unix <code>mv<\/code> command), which are not supported by PEFS (you have to <code>cp<\/code> first, then <code>rm<\/code>).<\/li>\n\n\n\n<li>Memory mapping (<code>mmap<\/code>) with execution flags (<code>PROT_EXEC<\/code>) to run the binaries.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">So, it\u2019s best to keep <code>TF_DATA_DIR<\/code> outside PEFS.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Sometimes, the best security architecture isn&#8217;t about finding a single perfect tool, but about knowing exactly where your tools&#8217; limits are\u2014and building a bridge between them.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Links<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/agora.nasqueron.org\/Operations_grimoire\/Deploy_with_Terraform\">Agora documentation<\/a> on our Terraform \/ OpenTofu workflows<\/li>\n\n\n\n<li><a href=\"https:\/\/devcentral.nasqueron.org\/source\/operations\/browse\/main\/roles\/salt-primary\/opentofu\/files\/tf.sh\">tf wrapper<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/devcentral.nasqueron.org\/source\/operations\/browse\/main\/roles\/salt-primary\/opentofu\/config.sls\">Salt configuration<\/a> for directories and ZFS datasets<\/li>\n\n\n\n<li><a href=\"https:\/\/devcentral.nasqueron.org\/rOPS9a17e6aa663bcd8148394eacaab6338a3e1d9f50\">Full implementation commit<\/a> <\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Notes<\/h2>\n\n\n<ol class=\"wp-block-footnotes\"><li id=\"62d7b693-7334-475b-a098-5c2bf3498d40\">If you want to contribute to Nasqueron Operations, you can solve this issue by implementing an automated build toolchain to build the providers we use for FreeBSD amd64.  <a href=\"#62d7b693-7334-475b-a098-5c2bf3498d40-link\" aria-label=\"Jump to footnote reference 1\">\u21a9\ufe0e<\/a><\/li><li id=\"826fe37b-3075-4685-ad37-db51dbc72562\">For PEFS and FUSE operations, the sysctl <code>vfs.usermount<\/code> must be set to 1 to allow non-root users to perform the mount. That&#8217;s also the same setting that&#8217;s used on desktop to allow to mount a USB key. <a href=\"#826fe37b-3075-4685-ad37-db51dbc72562-link\" aria-label=\"Jump to footnote reference 2\">\u21a9\ufe0e<\/a><\/li><li id=\"0423ed78-fadd-4605-a14e-1fc74fe55d38\">Default algorithm is AES-128-XTS. Check your CPU supports AES-NI instructions for crypto-acceleration before switching to AES-256-XTS. If not, the default will provide a better performance. <a href=\"#0423ed78-fadd-4605-a14e-1fc74fe55d38-link\" aria-label=\"Jump to footnote reference 3\">\u21a9\ufe0e<\/a><\/li><li id=\"c139a0c7-be3a-44bf-a4dc-ab87468b0ce5\">That wasn&#8217;t the main goal of the wrapper, but as we have one, it automatically picks the right executable, <code>opentofu<\/code> by default, <code>terraform<\/code> is in an allowlist. <a href=\"#c139a0c7-be3a-44bf-a4dc-ab87468b0ce5-link\" aria-label=\"Jump to footnote reference 4\">\u21a9\ufe0e<\/a><\/li><li id=\"e2868daa-7e8f-434e-9572-42585ba60a39\">That error can more usually occur when Go compilation flags are tweaked, <a href=\"https:\/\/discuss.hashicorp.com\/t\/using-custom-provider-gives-exec-format-error-error\/58741\">see for example this thread<\/a>.  <a href=\"#e2868daa-7e8f-434e-9572-42585ba60a39-link\" aria-label=\"Jump to footnote reference 5\">\u21a9\ufe0e<\/a><\/li><\/ol>","protected":false},"excerpt":{"rendered":"<p>On Nasqueron, deployments occur on a dedicated deployment server, distinct from the development servers, both for configuration-as-code and infrastructure-as-code. Terraform state files, terraform.tfstate, are stored on our deployment server, and can contain both credentials and personal information: plaintext secrets or accounts metadata injected by providers. To protect this information, we can take two complementary approaches: [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":1014,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"[{\"content\":\"If you want to contribute to Nasqueron Operations, you can solve this issue by implementing an automated build toolchain to build the providers we use for FreeBSD amd64. \",\"id\":\"62d7b693-7334-475b-a098-5c2bf3498d40\"},{\"content\":\"For PEFS and FUSE operations, the sysctl <code>vfs.usermount<\/code> must be set to 1 to allow non-root users to perform the mount. That's also the same setting that's used on desktop to allow to mount a USB key.\",\"id\":\"826fe37b-3075-4685-ad37-db51dbc72562\"},{\"content\":\"Default algorithm is AES-128-XTS. Check your CPU supports AES-NI instructions for crypto-acceleration before switching to AES-256-XTS. If not, the default will provide a better performance.\",\"id\":\"0423ed78-fadd-4605-a14e-1fc74fe55d38\"},{\"content\":\"That wasn't the main goal of the wrapper, but as we have one, it automatically picks the right executable, <code>opentofu<\/code> by default, <code>terraform<\/code> is in an allowlist.\",\"id\":\"c139a0c7-be3a-44bf-a4dc-ab87468b0ce5\"},{\"content\":\"That error can more usually occur when Go compilation flags are tweaked, <a href=\\\"https:\/\/discuss.hashicorp.com\/t\/using-custom-provider-gives-exec-format-error-error\/58741\\\">see for example this thread<\/a>. \",\"id\":\"e2868daa-7e8f-434e-9572-42585ba60a39\"}]"},"categories":[284,50],"tags":[9,65,407,405],"class_list":["post-1011","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-nasqueron","category-sysadmin","tag-freebsd","tag-nasqueron","tag-pefs","tag-terraform"],"_links":{"self":[{"href":"https:\/\/www.dereckson.be\/blog\/wp-json\/wp\/v2\/posts\/1011","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dereckson.be\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dereckson.be\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dereckson.be\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dereckson.be\/blog\/wp-json\/wp\/v2\/comments?post=1011"}],"version-history":[{"count":15,"href":"https:\/\/www.dereckson.be\/blog\/wp-json\/wp\/v2\/posts\/1011\/revisions"}],"predecessor-version":[{"id":1028,"href":"https:\/\/www.dereckson.be\/blog\/wp-json\/wp\/v2\/posts\/1011\/revisions\/1028"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.dereckson.be\/blog\/wp-json\/wp\/v2\/media\/1014"}],"wp:attachment":[{"href":"https:\/\/www.dereckson.be\/blog\/wp-json\/wp\/v2\/media?parent=1011"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dereckson.be\/blog\/wp-json\/wp\/v2\/categories?post=1011"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dereckson.be\/blog\/wp-json\/wp\/v2\/tags?post=1011"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}