BLOGS

Sharing One Hyperdisk Read-Only Across Many VMs on GKE

Summary

Many security, search and analytics systems serve data that never changes once it is written: index generations, model weights, nightly snapshots. The cheapest way to put that data in front of many servers on Google Cloud has long been a single Persistent Disk attached read-only to every reader at once. Google's fourth-generation machines (N4, C4 and relatives) do not support Persistent Disk, and their general-purpose replacement, Hyperdisk Balanced, cannot be attached read-only at all. This article explains the problem, compares the alternatives, and walks through a five-step solution that keeps the single-shared-disk design on Hyperdisk Balanced: attach the disk read-write to every reader, and have each reader enforce read-only itself, at the block layer and in the mount, before dropping the privilege it needed to do so. It covers the mechanics that are not documented anywhere, the cost at list prices, the failure cases to test, and the one guarantee you give up. It closes with how we run this at Stairwell.

Definitions

Immutable data is data that is never modified after it is written. It can be replaced, as a whole, by a newer version, but no byte of a published version ever changes. A nightly search index, a set of model weights, a compressed log archive and a database backup are all immutable. A live database table is not.

Why it matters. Immutable data is unusually cheap to serve. Almost every hard problem in serving data comes from writes: cache invalidation, replication lag, partial writes to recover from, locks between readers and writers. When nothing writes, a fleet of machines can read the same bytes through the same disk with no coordination at all, and the storage can be paid for once instead of once per replica.

Who should care. Anyone who runs several copies of a service that reads a large dataset which is rebuilt rather than edited: platform and infrastructure engineers, SREs, and anyone paying a cloud bill where disk capacity is multiplied by replica count. If your readers each copy the dataset onto their own disk, or you pay for a network filesystem to avoid the copies, this is for you.

Persistent Disk (PD) is Google Compute Engine's original block storage, in pd-standard, pd-balanced, pd-ssd and pd-extreme types. A PD can be attached to one VM read-write, or to many VMs read-only, and the hypervisor refuses writes on a read-only attachment.

Hyperdisk is Google's newer block storage, built on a different backend, and the only block storage the fourth-generation machine series can attach. Its capacity, IOPS and throughput are provisioned independently of one another. It comes in several types (Balanced, Throughput, Extreme, Balanced High Availability, and ML), and they differ in how many VMs can attach one disk and in which modes.

Read-only multi-attach is the ability to attach one disk to many VMs at once with every attachment read-only. In Kubernetes it surfaces as a PersistentVolume with the ReadOnlyMany access mode and volumeMode: Filesystem.

The problem

Why one shared disk is the right shape

Consider the options for getting 3 TB of immutable index files in front of a handful of server replicas.

  • Read straight from object storage on every lookup. Simple but slow, and not slow in a way that can be compared fairly. A lookup against a local disk is an in-memory index probe and one seek on an SSD. Opening and ranging a file in object storage costs tens of milliseconds before the first byte arrives, plus a per-request charge, and the data blocks of a multi-terabyte corpus do not fit in a cache that would hide it.
  • Load everything into RAM. Fast and ruinously expensive at these sizes.
  • Give every replica its own disk and copy the data onto it. This works, and it is what most teams do. But every replica pays for its own copy of the capacity, every new version means every replica copies again, and a copy of 3 TB takes the better part of an hour.
  • A network filesystem such as Filestore. Removes the copies but costs about two and a half times what a disk does per gigabyte, with a 1 TiB minimum per share.
  • Copy the data onto one disk, once, and attach that disk to every reader at the same time. Google Compute Engine has supported this for years: a Persistent Disk can be attached read-only to several VMs at once, ten for pd-balanced and about a hundred for pd-ssd. Each reader mounts the same ext4 filesystem. The first reader starts serving seconds after the attachment completes, not an hour after a copy starts, and the capacity is paid for once however many replicas there are.

The last option beats the rest. Publishing a new version is a new disk and a rolling restart. The old disk stays whole until the last reader lets go, so rollback is the same operation run backwards. There is no file server to run, no replication protocol and no per-replica copy pipeline. In Kubernetes terms it is a ReadOnlyMany PersistentVolume with volumeMode: Filesystem, which the GKE persistent-disk driver turns into a READ_ONLY attachment on each node.

It also matters who enforces the read-only. With a READ_ONLY attachment, the hypervisor refuses writes. A pod that gets compromised on a reader node, an operator who types mount -o remount,rw by reflex, a filesystem that decides on its own to replay a journal: none of them can change the bytes another node is reading. The readers do not have to trust each other, and the data does not have to trust the readers. No filesystem gives you that on its own.

What Hyperdisk changed

Google's fourth-generation machine series (N4, N4D, C4 and their relatives) do not attach Persistent Disk at all. They attach Hyperdisk, which is a different product with different rules, and Google's new capacity is fourth generation. The pools you can still fill with the older series are not getting any bigger, and nobody wants a serving tier pinned to hardware that is on its way out.

Hyperdisk splits what was one product into five. Throughput attaches to a single VM. Extreme shares among as many as 16 VMs in multi-writer mode. Balanced High Availability is Balanced replicated across two zones. Hyperdisk ML, built for serving model weights, is the one type that keeps a read-only multi-attach mode: it attaches to hundreds of VMs, the hypervisor enforces the read-only, GKE hands it to pods as an ordinary ReadOnlyMany filesystem, and since April 2026 it attaches to the same N4 and C4 series as Balanced.

Hyperdisk Balanced, the general-purpose type, attaches to one VM or to as many as eight, and in both modes the attachment is read-write. Ask for a read-only attachment and GCE says, in so many words:

googleapi: Error 400: hyperdisk-balanced disk type cannot be attached in read-only mode.

That error is what a ReadOnlyMany filesystem volume produces the instant kubelet tries to attach it. The shape the whole single-shared-disk design is built on does not exist on the hardware everyone is moving to.

Why not just use Hyperdisk ML

Hyperdisk ML would spare you everything in the rest of this article, and if your fleet is a few big disks, it is the right answer: use it and stop reading here. The catch is its pricing when the fleet is many small disks. Capacity costs the same as Balanced, but throughput is provisioned and billed separately, with a floor of 400 MiB/s per volume: $48 a month on every disk up to about 3.3 TiB. For one large, hot dataset that is noise. For a dozen datasets, most under a terabyte, each with two disks alive during a version switch, the floor adds about two-thirds to the capacity bill, for throughput none of them uses. A fleet that already runs on Hyperdisk Balanced also gains a third disk type used only by the serving tier.

What we wanted was the same single shared disk, at Hyperdisk Balanced prices, on the new hardware.

The solution

The multi-VM mode Hyperdisk Balanced does have is called READ_WRITE_MANY, and it exists for clustered filesystems and databases that coordinate their own writes. ext4 is not one of those: two nodes mounting the same ext4 read-write would destroy it in seconds, and the GKE driver knows that, so it only hands a multi-attached Hyperdisk to a pod as a raw block device, never as a mounted filesystem.

So the idea is: let the disk be attached read-write, and make every reader treat it as read-only anyway. Each server pod receives the raw device, flags it read-only at the block layer for the whole node, mounts it itself with the two ext4 options that guarantee the mount will never write, verifies that the kernel agrees, and then throws away the privilege it needed to do the mount before a single request is served. The write-once, attach-many lifecycle stays exactly as it was. What moves is the enforcement of "read-only": from the hypervisor to the mount flags, the block-layer flag, and the discipline of never creating a second writer.

Whether that trade is acceptable is covered below. First the mechanics, since several of them are not documented anywhere we could find.

The solution in five steps

Step 1: Fill the disk once, through a single-attach claim

A new version starts life as an ordinary single-attach Hyperdisk Balanced. Give it to one filler Job through a plain ReadWriteOnce filesystem claim, so kubelet formats and mounts it the way it formats any disk. The Job copies the data from object storage and runs sync. When the Job finishes, kubelet unmounts the filesystem, and that unmount is what marks the ext4 journal clean.

Provision the disk high for the fill (the copy can drive far more IOPS than serving ever will) and plan to relax it later; see Step 5 for the rule that constrains when.

If fillers run on Spot capacity, they will get preempted. A filler killed mid-copy just fails its Job, and the controller fills the same disk again. The dangerous case is a filler killed after the Job succeeded but before kubelet's unmount ran. The data is all there, but the journal says the filesystem was never unmounted. Step 4 is where that matters.

Step 2: Switch the disk to READ_WRITE_MANY, then publish the volume

Wait until GCE reports no VM holding the disk, then flip its access mode:

gcloud compute disks update <disk> --zone us-central1-a --access-mode READ_WRITE_MANY

GCE refuses that update while anything is attached, which turns it into a useful commit point: nothing can become a multi-reader until the writer has let go. Only after the mode reads back as READ_WRITE_MANY should the controller create the shared PersistentVolume for the serving pods. Create it earlier and the first reader attaches, the second fails, and the Deployment sits half-rolled.

Treat this switch as the point of no return: a disk that has been published is never attached writable again. If it needs to change, fill a new one.

Step 3: Hand each reader the raw block device

The serving PersistentVolume is a static one pointing at the disk by name:

spec:
  accessModes: [ReadWriteMany]
  volumeMode: Block
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: pd.csi.storage.gke.io
    volumeHandle: projects/PROJECT/zones/us-central1-a/disks/DISK
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - {key: topology.gke.io/zone, operator: In, values: [us-central1-a]}

Two things matter here. csi.readOnly has to stay unset, because true makes kubelet ask for a READ_ONLY attachment and you get the 400 above. And the zone affinity has to be there, because the disk is zonal, and without it the scheduler learns about the mismatch from a failed attach rather than from placement.

The pod then needs the capability to mount, and nothing else. A container needs CAP_SYS_ADMIN to call mount(2), and the default seccomp profile allows the syscall when that capability is held. But GKE's default AppArmor profile for containers denies mount regardless of capabilities, and the failure it produces is an unhelpful "cannot mount /dev/disk read-only". We tried every combination to be sure: the default profile with SYS_ADMIN fails, the default profile with seccomp disabled fails, and only an unconfined AppArmor profile lets the mount through. The container is otherwise as locked down as it can be:

securityContext:
  privileged: false
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]
    add: ["SYS_ADMIN", "SETPCAP"]
  seccompProfile: {type: RuntimeDefault}
  appArmorProfile: {type: Unconfined}
volumeDevices:
  - name: data
    devicePath: /dev/data-disk

Step 4: Check the journal, mount read-only from inside the pod, and verify

Before the server opens a single file, it does four things with the device.

First, it checks that the filesystem is clean, because of the preempted filler from Step 1. The server reads the ext4 superblock at byte 1024 of the device and refuses to continue if the needs_recovery feature flag is set or the VALID_FS state bit is clear. The refusal is written to the container's termination message, the controller reads it from the pod, discards that disk, and fills a fresh one. It is a few lines of code, and it is what makes the mount options below safe to use. Without it, a plain ro mount of a dirty filesystem fails with a kernel message the controller cannot act on, and a ro,noload mount does something worse: it succeeds, and the last writes before the crash are simply invisible, with no error anywhere. A version served from that disk would be missing an unknown number of files and nobody would know.

Then the three commands:

blockdev --setro /dev/data-disk
mount -t ext4 -o ro,noload /dev/data-disk /data
grep -E '^\S+ /data ext4 ro,.*\bnorecovery\b' /proc/mounts

blockdev --setro marks the device read-only at the block layer, for the whole node, and it is the mechanism that does the real work: once the block device is read-only, ext4 will not replay a journal, and a filesystem whose journal needs recovery fails to mount rather than mounting with stale metadata. A later remount,rw by some other process is refused by the kernel for the same reason. ro is the mount flag everyone knows. noload goes one step further and keeps ext4 from loading the journal at all, which is what you want on a disk that seven other nodes may be reading at that moment; its price is that it also skips the kernel's own dirty-journal check, which is why the superblock check comes first. Checking /proc/mounts afterwards, for both ro and norecovery (the kernel's name for noload), is what makes a pod fail loudly rather than serve from an empty directory.

Do the mount in the serving container itself. A privileged init container could leave the mount in the node's namespace for the serving container to pick up through mount propagation, but a mount that outlives the pod has to be cleaned up by something, and until it is, kubelet cannot unmap the device or detach the disk. A mount that lives in the serving container's own mount namespace is torn down with the container and needs no cleanup at all.

Step 5: Drop the privilege, then run the server

CAP_SYS_ADMIN is a capability nobody wants a long-running server to hold, even one that is never reachable by a user. Give it up the moment the mount is verified. The clean way to do that from a shell is setpriv --bounding-set=-all --inh-caps=-all --ambient-caps=-all --no-new-privs, and then exec the real binary.

If the server is a Go binary that ships without a shell, Go makes this harder than it should be. Capabilities are per thread on Linux. The Go runtime has already spawned threads by the time main runs, and the mechanism the standard library offers for applying a syscall to all of them, syscall.AllThreadsSyscall, returns ENOTSUP in any binary that links cgo. What works is to do the mount and the capability drop on one locked OS thread and then execve the binary itself, with a marker flag appended. The new process image takes its credentials from the calling thread, the mount belongs to the process's mount namespace and survives the exec, and the second incarnation refuses to serve unless /proc/self/status shows every capability set at zero and NoNewPrivs: 1, and /proc/mounts shows the device mounted ro without recovery. Two readers on two nodes then see identical bytes, every write attempt gets EROFS, and the disk is never modified, not even by a journal replay.

Once serving, relax the disk's performance. GCE permits one performance change per disk every four hours, and the second call in a window fails with "rate limited," so provision high for the fill and make exactly one change, down to the included baseline of 3,000 IOPS and 140 MiB/s, an hour into serving. Provisioned IOPS also cannot exceed 500 per GiB, with throughput at most a quarter of the IOPS; a request outside those bounds is reported by the CSI driver as "infeasible" and never retried, which leaves a claim Pending forever, so compute requests that fit before asking.

Operating it: rollouts, limits and the tests to run

  • Never take a reader down before its replacement is Ready. On a shared disk a replacement pod is just one more attachment, so roll with maxSurge: 1, maxUnavailable: 0 and a PodDisruptionBudget.
  • Eight attachments per disk is the limit. At eight replicas on eight nodes there is nowhere to surge, so roll one pod at a time for that case, and refuse configurations above eight. Past eight, a second copy of the disk is a snapshot or a clone, not a second fill, and a second group of readers attaches to that one; nothing on the copies changes, so nothing has to keep them in sync.
  • Replicas are for CPU, RAM and request concurrency, never for disk bandwidth. A Hyperdisk's provisioned IOPS and throughput are shared by every VM attached to it.
  • Everything is zonal. That is fine for immutable data: serving a version from a second zone is a second fill of the same bytes.
  • Graph write IOPS on every shared disk. That line should read zero. A nonzero number is the alarm.
  • Run the failure cases before trusting any of it. Delete a reader pod and time its replacement (ours mounted and was Ready in about ten seconds). Delete a reader's VM out from under it (the stale attachment was cleaned up, GCE's user list shrank to the survivor, the survivor kept serving, and the replacement pod mounted on a brand-new node of a different family). Hash the raw device end to end from a pod that never mounted it, before the readers start and again after restarts and node loss, and check the hashes match.

Performance and cost

List prices in us-central1 as of October 2026, for a 3.3 TiB disk:

Option Capacity $/month Readers Machine series Read-only enforced by
Hyperdisk Balanced, multi-attach, pod mounts ro $264 (3,000 IOPS and 140 MiB/s included) 8 per disk C3 and newer the mount and the block layer
pd-balanced attached READ_ONLY $330 10 2nd and 3rd generation the hypervisor
pd-ssd attached READ_ONLY $561 100 recommended 2nd and 3rd generation the hypervisor
Hyperdisk ML, READ_ONLY_MANY 312(264 plus a 400 MiB/s floor at $0.12 per MiB/s) 300 at this size (2,500 below 512 GiB) C3 and newer, plus accelerator series the hypervisor
Filestore Zonal $825 (1 TiB minimum) unlimited any NFS export

What we measured is utilization: over a week, no dataset came near the included baseline of 3,000 IOPS and 140 MiB/s at the 99th percentile, because index blocks live in RAM and a lookup is one seek. The fill is faster, because the disk can be provisioned to whatever the copy can drive and then relaxed. And the capacity line is a fifth cheaper than pd-balanced for the same bytes, on hardware you can actually get.

Against pd-balanced attached read-only, which is what the design ran on before:

  • Same. The server binary, the fill-and-switch lifecycle, the never-kill-a-reader rule, and utilization, since no dataset comes near either disk's baseline.
  • Better. A fifth cheaper per gigabyte, a faster fill, and machines you can actually get.
  • Worse. Eight readers per disk instead of ten, and read-only enforced by the kernel instead of the hypervisor. The next section is about that one.

What you give up

The column in that table that should give you pause is the last one. With a Persistent Disk attached READ_ONLY, a pod that somehow gains CAP_SYS_ADMIN on a reader node still cannot write, because the VM's attachment is read-only and the hypervisor is the referee. With Hyperdisk Balanced every attachment is read-write, and the only things standing between a reader and the shared bytes are the block-layer flag, the mount options, a process with no capabilities, and the rule that a disk is never attached writable again once it has been published. That is a real downgrade in enforcement. An attacker in that scenario already has code running inside the cluster, since nothing outside it can reach these servers, but that foothold should not be worth more than it is.

We think it is the right trade for this kind of data. If the data is a derived artifact that can be rebuilt from its source, corruption is an outage, not a loss. If the file format carries a checksum on every block, an accidentally corrupted block fails the read rather than answering wrong. A deliberate writer who recomputed the checksums would produce answers that look right, and nothing about watching the disk authenticates the bytes. What you can do is notice the writer, which is why write IOPS on every shared disk is graphed next to the service it serves.

If you need the storage layer itself to enforce read-only, this scheme is not for you. Use Hyperdisk ML or Filestore, and pay for the guarantee.

How Stairwell uses this

Stairwell collects every executable and script file that lands on a customer's machines, and keeps it. A lightweight process on each endpoint hashes anything that looks like code the moment it touches disk, whether or not it ever runs, and uploads the bytes we have never seen before. Nothing is thrown away. The premise of the product is that the files on your fleet are evidence, and evidence is only useful if you still have it when you learn what to look for.

Storing the files is the easy part. The hard part is the derived data: nearly every interesting question Stairwell answers is served from indexes built over the whole collection, such as how many machines in your environment and in the world have a given file, or years of DNS resolution history, hot and searchable in under a second. Those indexes are rebuilt in batch rather than edited in place, and each rebuild publishes a complete new version that is never modified afterwards. That makes them exactly the kind of data this article is about, and the single-shared-disk design described here is how we serve them.

Moving a dataset onto Hyperdisk Balanced is a one-line configuration change per dataset, and datasets are moving as the new node pools come online. The dirty-journal refusal is exercised against a genuinely dirty ext4 superblock in tests, as is the response to it, since we have not yet seen a real preemption land in that window.

Final thoughts

The lesson we keep relearning is that immutable data is easy to serve. Because a version never changes after it is published, almost every hard problem in serving it goes away: there is no cache invalidation, no replication lag, no partial write to recover from, and a fleet of machines can read the same bytes through the same disk without a word of coordination between them. Hyperdisk took away the hypervisor's read-only flag, which was a real guarantee and one we miss. But the property the design rests on is the immutability, and that one survived. Everything in this article is a way of making the kernel respect that on hardware that does not do it for you.

If problems like this are the kind you enjoy, we would love to chat with you.

Table of Contents

SHARE THIS ARTICLE

RELATED ARTICLES

Read More