On our first tests: 37,707 Vault resources, 121 stacks, no failures

November 2026

Platform teams ask us one question before any other: will it hold up at our size?

A realistic Vault estate in a large organisation is not one Terraform stack. It is a hundred or more Kubernetes clusters, each with its own auth mount and hundreds of roles. Next to them sits a platform layer of KV stores, AppRole, JWT, certificate auth, an SSH CA and a PKI. The KV stores hold thousands of secrets that no one manages with Terraform, and no deployment tool is allowed to touch them. Then one day someone changes the shared module that every cluster uses.

So we built that estate and ran Nojam through it.

What we built

• 120 Kubernetes auth mounts, one Terraform stack each, all built from one shared module. Each mount has:
  • 300 roles, each bound to 10 namespaces and 10 service accounts;
  • 10 policies.
• One platform stack holds:
  • 10 KV v2 stores;
  • AppRole, JWT and TLS certificate auth;
  • an SSH CA and a PKI root CA, with 50 roles each.
• 20,000 secrets written straight into the KV stores through Vault's API, outside Terraform, the way application teams actually write them.

That is 37,707 Terraform resources in 121 stacks. Every change went through Nojam's normal flow:

1. a pull request;
2. a plan for each affected stack, summarised on the pull request;
3. approval by a second person;
4. an apply of exactly the approved plan.

What we did to it

1. Bootstrap. All 121 stacks created from nothing.
2. The change everyone fears. One line added to the shared module, so all 36,000 roles in all 120 mounts are updated in one pull request.
3. An ordinary day. One team adds five roles to its own mount.
4. A platform change with live data. The KV stores' settings change while the 20,000 secrets sit in them.

After each phase the test checked the exact number of stacks planned and resources changed. Anything planned that shouldn't have been, or anything left behind, fails the test.

How it went

• Every phase changed exactly what it should, and nothing else. The module change planned 120 stacks and updated 36,000 roles. The team's change planned one stack. The platform change updated the 10 KV stores, and all 20,000 secrets were still there afterwards.

• Nojam stayed small. With 37,707 resources under management:

  • the server's memory stayed under 16 MB;
  • its database stayed under 70 MB;
  • Vault used about 1 GB.

  The time goes into OpenTofu and Vault, where it belongs.

• A team's change doesn't slow down as the estate grows. Five roles in one mount took 17 seconds in a 20-mount estate and 24 seconds in a 120-mount one. Each stack plans only its own state.

And with more than one job at a time?

We repeated the 20-mount version on a small server with two dedicated CPUs, running one, two and then four jobs at once:

The lesson is plain:

• Dedicated CPUs mattered most. The same work took 5 min 57 s in our first run, on one shared CPU, and 2 min 34 s here.
• Running jobs side by side helped up to the number of CPUs, and then stopped helping. On that box, OpenTofu, Vault and the database were all competing for the same two CPUs.
• Every run was still exact and failure-free.

Capacity grows with CPUs. Give each runner host as many slots as it has CPUs, add hosts as you need, and keep Vault on its own machines. Nojam itself used under 9 MB of memory throughout.

What the test caught

The first, smaller run (20 mounts) found a real bug. When one change touched many stacks, the summary Nojam posts on the pull request grew past GitHub's 65,536-character comment limit. GitHub refused it, and the pull request kept showing an out-of-date status table. An approval in Nojam is always tied to the exact plan, so nothing wrong could be applied. But reviewers deserve a page that is current.

We fixed it before the full run. The summary now always keeps:

• the status of every stack;
• every destructive change;
• the approval instructions.

The full detail for stacks that don't fit points to the Nojam UI. At 120 stacks the largest summary was 60 KB, and GitHub accepted every one.

The test also surfaced three Vault provider configurations that never settle and show a change on every plan:

• a PKI TTL written as "720h", which Vault returns in seconds;
• a JWT public key with a trailing newline;
• keys generated inside the configuration.

They are mistakes in Terraform code, not in Nojam, but real teams make them. They are now in our documentation.

The fine print

We would rather you hear the conditions from us:

• A deliberately small machine. The main run used one CPU for everything: Nojam, PostgreSQL, Vault and OpenTofu. One runner ran one job at a time. The hour-long phases are 120 jobs back to back. More runners run them side by side.
• A test Vault. Vault ran in development mode, in memory, and GitHub was simulated. A production Vault with persistent storage and audit devices answers more slowly, so expect longer plans.
• No speed-up of Terraform itself. Nojam doesn't make a single plan faster. It plans only the stacks a change affects, applies the approved plan without planning again, and makes it safe to split state so that plans stay small.

These are our first tests at this size, and we'll publish the next ones the same way.