98 Container Internals Interview Questions and Answers (2026)

A container is a normal Linux process (or group of processes) whose view of the system is restricted by kernel features so it looks like it has its own machine: it shares the host kernel rather than running its own.

• Isolation model:

  • Containers use kernel primitives: namespaces (isolate what a process sees) and cgroups (limit what it can use).

  • VMs use a hypervisor to virtualize hardware; each guest runs its own full kernel and OS.

• Boundary:

  • Container boundary is the kernel syscall interface: all containers share one host kernel.

  • VM boundary is the virtual hardware/hypervisor: a much stronger, lower-level barrier.

• Trade-offs:

  • Containers: fast startup (milliseconds), small footprint, high density, but weaker isolation and kernel-version coupling.

  • VMs: stronger security/multi-tenancy isolation and different-OS support, but slower boot, more RAM/disk, and heavier overhead.

• Consequence: a kernel exploit escaping a container can reach the host; escaping a VM requires breaking the hypervisor.

Containers win on speed, density, and portability because they share the host kernel; VMs win on isolation strength and OS flexibility because they virtualize full hardware. The choice is really about how strong a security/tenancy boundary you need.

• Container advantages:

  • Near-instant startup and low overhead: no guest OS to boot.

  • High density: dozens per host since they share one kernel.

  • Portable, reproducible images that bundle app plus dependencies.

• Container trade-offs:

  • Weaker isolation: shared kernel means a wider attack surface.

  • Must match the host kernel; can't run a fundamentally different OS.

• VM advantages:

  • Strong isolation suited to untrusted or strict multi-tenant workloads.

  • Can run different OSes/kernels on the same host.

• VM trade-offs: Slower boot, larger images, more RAM/CPU per instance.

• Common pattern: combine them (VMs to segment tenants, containers inside for packaging), or use lightweight VMs like Kata Containers / Firecracker to get both.

An image is an immutable, read-only template (a stack of layers plus metadata); a container is a running (or stopped) instance of that image with a thin writable layer on top. Layers let images share storage and give containers cheap copy-on-write behavior.

• Image = read-only blueprint:

  • A set of ordered filesystem layers (tarballs) plus a config describing how to run it (entrypoint, env, etc.).

  • Immutable and content-addressed, so it can be shared and cached.

• Container = image + writable layer + runtime state:

  • The runtime stacks the image's read-only layers and adds one thin read-write layer on top.

  • It also carries live process/namespace/cgroup state while running.

• Layers form a union filesystem:

  • A driver like overlay2 merges lower (image) layers and the upper (container) layer into one view.

  • Upper-layer files shadow identical paths in lower layers.

• Copy-on-write semantics:

  • Reads come straight from the shared read-only layers, so many containers share the same on-disk bytes.

  • On first write to an existing file, the driver copies it up into the writable layer, then modifies the copy.

  • Deletes are recorded as whiteout files rather than touching lower layers.

  • Result: fast startup, low disk use; writable data vanishes when the container is removed (use volumes for persistence).

Namespaces are the Linux kernel feature that gives each container its own private view of a global system resource, so processes inside think they have their own OS. Each namespace type virtualizes one class of resource.

• PID namespace: Isolates the process ID tree: the container's first process is PID 1 and it can't see or signal host processes.

• Network namespace: Gives its own interfaces, routing table, and ports; typically wired to the host via a veth pair and a bridge.

• Mount namespace: Isolates the filesystem mount table, so the container sees its own rootfs and mounts without affecting the host.

• User namespace: Maps UIDs/GIDs so root (UID 0) inside maps to an unprivileged UID on the host: a major security boundary.

• Others round out the picture: UTS (hostname), IPC (shared memory/semaphores), and cgroup (cgroup root view).

• Created with clone(), unshare(), or joined with setns().

No, they are complementary kernel features that solve different problems: namespaces control what a process can see (isolation of resource visibility), while cgroups control how much a process can use (accounting and limiting of resource consumption).

• Namespaces = isolation (visibility):

  • They partition global kernel resources so each process group sees its own view: PIDs, mounts, network stack, hostname, users, IPC.

  • Types include pid, net, mnt, uts, ipc, user, cgroup, time.

• Cgroups = control (consumption):

  • They meter and cap CPU, memory, I/O, and PIDs, and can throttle or OOM-kill processes that exceed limits.

  • Organized as a hierarchy of controllers (cpu, memory, io, pids).

• Together they make a container: Namespaces give it a private world; cgroups keep it from starving the host. A container is roughly namespaces + cgroups + a root filesystem.

The UTS (UNIX Time-sharing System) namespace isolates two identifiers: the hostname (nodename) and the NIS domain name. This lets a container set its own hostname without affecting the host or other containers.

• What it scopes: The values returned by uname() for nodename and domainname, and set via sethostname() / setdomainname().

• Why a container wants its own:

  • Identity: many apps derive logging tags, cluster membership, or config from the hostname; each container needs a stable, distinct name.

  • Safety: a containerized process calling sethostname() must not rename the host machine.

• Scope note: It only isolates these strings, not DNS resolution or networking, which belong to the network namespace and /etc/resolv.conf.

The runtime builds a root filesystem from the image layers, then combines a mount namespace with a root-changing syscall so the container sees that filesystem as / and cannot reach the host's tree.

• Assemble the rootfs: Union-mount the image's read-only layers plus a writable layer (OverlayFS) into a single directory.

• Isolate the mounts: Create a new mount namespace (CLONE_NEWNS) so the container's mounts are private and don't leak to the host.

• Switch the root:

  • Use pivot_root (or chroot) to make the assembled rootfs the new /, then unmount the old root so it's unreachable.

  • Mount virtual filesystems (/proc, /sys, /dev) inside the new root.

cgroups are a Linux kernel feature that organizes processes into hierarchical groups and meters, limits, and accounts their use of resources like CPU, memory, and I/O. They are one of the two pillars of containers (cgroups for resource control, namespaces for isolation).

• What they do:

  • Limit: cap a group's resource usage (max memory, CPU quota).

  • Prioritize: give groups relative shares under contention.

  • Account: track exactly how much each group consumes.

  • Control: freeze, throttle, or OOM-kill members of a group.

• Controllers (subsystems): Each resource has a controller: cpu, memory, io, pids, etc., exposed as files under a cgroup directory.

• v1 vs v2: v1 has a separate hierarchy per controller; v2 uses a single unified hierarchy (the modern default) with a cleaner, consistent interface.

• How containers use them: The runtime (runc) creates a cgroup per container and writes limits from the OCI spec (e.g. cpu.max, memory.max) so the container can't starve the host or its neighbors.

Docker is just one product built on a stack of lower-level pieces; alternatives differ mainly in whether they need a long-running root daemon and where they sit in the runtime stack.

• containerd:

  • A high-level runtime that pulls images, manages storage/snapshots, and supervises containers; Docker itself uses it under the hood.

  • Used directly by Kubernetes via the CRI, so Docker-the-tool isn't needed in clusters.

• CRI-O: A minimal runtime built specifically to implement Kubernetes' CRI, nothing more.

• Podman:

  • Daemonless and rootless: runs containers as child processes of your shell, not under a root daemon, improving security and auditing.

  • CLI-compatible with Docker (alias docker=podman).

• runc: The low-level OCI runtime that actually creates namespaces/cgroups; most of the above call it.

• Sandboxed runtimes: gVisor (user-space kernel) and Kata Containers (micro-VMs) trade some speed for VM-like isolation.

The story is a steady progression from crude filesystem isolation to full process isolation to standardized, interchangeable tooling: each step added a stronger boundary or better ergonomics, ending in open specs so no single vendor owns the format.

1. chroot (1979/1982): Changes a process's root directory: filesystem-only isolation, easily escaped, no resource or process isolation.

2. Namespaces + cgroups (2000s): Kernel gains per-process isolation of PIDs/network/mounts plus resource limits, the real foundations.

3. LXC (2008): First mainstream toolset combining namespaces and cgroups into usable containers, but hard to configure and not portable.

4. Docker (2013): Added layered images, a Dockerfile build model, and a registry: made containers portable and developer-friendly (initially built on LXC, later libcontainer).

5. OCI (2015): Open Container Initiative standardizes the image, runtime, and distribution specs; Docker donated runc as the reference runtime.

6. containerd (2017): Extracted from Docker as a standalone high-level runtime, donated to the CNCF, now used directly by Kubernetes.

The OCI Image Spec defines what a container image is (its layers, manifest, and config), and the OCI Distribution Spec defines how images move to and from registries (the HTTP API). Together they make images and registries vendor-neutral and interoperable.

• Image Specification:

  • Layers: filesystem changes stored as tar archives, referenced by digest.

  • Manifest: a JSON document listing the config and layer digests plus media types.

  • Config: JSON with runtime settings (entrypoint, env, working dir) and the ordered layer diff IDs.

  • Index: an optional multi-arch manifest pointing to per-platform manifests.

• Distribution Specification:

  • Standardizes the registry HTTP API for push/pull of blobs and manifests (the /v2/ endpoints).

  • Defines how content is addressed by digest and how tags resolve to manifests.

• Why it matters: Any compliant tool (Docker, Podman, containerd, Buildah) can build, store, and run the same images against any compliant registry.

Content-addressable storage means each piece of an image is identified by the hash of its own contents (a SHA256 digest) rather than by name or location, so the address is derived from the data itself. This makes images immutable, verifiable, and efficiently deduplicated.

• How it works:

  • Every layer blob, config, and manifest is hashed; the digest looks like sha256:abc123....

  • The manifest references layers by digest, and a tag ultimately resolves to a manifest digest.

• Benefits:

  • Immutability: changing any byte changes the digest, so a digest always points to exactly the same content.

  • Integrity: the client recomputes the hash on pull to detect corruption or tampering.

  • Deduplication: identical layers share one blob across images, saving storage and bandwidth.

  • Caching: an already-present digest need not be re-pulled.

• Practical note: Pinning by digest (image@sha256:...) is reproducible; tags like latest are mutable pointers and can change.

They are complementary OCI standards: the Image Spec defines what a packaged image looks like on disk and in a registry, while the Runtime Spec defines how an already-unpacked filesystem bundle is configured and executed as a container. One produces the artifact; the other runs it.

• OCI Image Specification:

  • Describes the manifest, config JSON, layers, and image index (the distributable, content-addressed artifact).

  • Concerned with packaging, distribution, and content addressing.

• OCI Runtime Specification:

  • Describes a filesystem bundle: a root filesystem plus a config.json defining process, mounts, namespaces, cgroups, and capabilities.

  • Implemented by runtimes like runc.

• The bridge between them:

  • A higher-level runtime (e.g. containerd) pulls an OCI image, unpacks its layers into a rootfs, and generates the runtime config.json from the image config.

  • That bundle is then handed to an OCI runtime to actually start the container.

The manifest is a small index that lists, by digest, the config blob and the ordered layer blobs that make up one image for one platform. The config JSON is the actual metadata describing how the image was built and how it should run. Both are content-addressed blobs, but they serve different roles.

• Image manifest contains:

  • A config descriptor: media type, digest, and size of the config blob.

  • A layers array: ordered descriptors (digest, size, media type) for each layer blob.

  • Optional annotations and a schemaVersion.

• Image config JSON contains:

  • Runtime settings: Entrypoint, Cmd, Env, WorkingDir, User, exposed ports.

  • Platform info: architecture and os.

  • A rootfs section listing the layer diffIDs (uncompressed digests) in order.

  • A history of the build steps.

• Key distinction: Manifest = "which blobs and where"; config = "what those blobs mean and how to run them."

A manifest list (called an image index in the OCI spec) is a top-level manifest that points to multiple platform-specific manifests, each tagged with its architecture and OS. It lets a single tag serve the right image for whatever platform pulls it.

• Structure:

  • An index blob contains a manifests array of descriptors, each with a digest and a platform object (architecture, os, optional variant).

  • Each referenced manifest is a normal single-platform image manifest.

• How multi-arch resolution works:

  1. Client requests myimage:latest and receives the index.

  2. It matches its own platform (e.g. linux/arm64) against the entries.

  3. It pulls only the matching manifest and its layers.

• Benefits:

  • One tag, many architectures: no per-arch tag juggling.

  • Built with tools like docker buildx or docker manifest.

These two primitives split the job: namespaces decide what a container can see, and cgroups decide how much it can use. Together they let ordinary host processes behave like isolated, resource-bounded containers without a VM.

• Namespaces = isolation (visibility):

  • Each container gets a private view of PIDs, network, mounts, users, hostname, and IPC.

  • Prevents processes from seeing or interfering with resources outside their namespaces.

• cgroups = resource control (limits):

  • Group processes and enforce limits/accounting on CPU, memory, block I/O, and PIDs.

  • Exceeding a memory limit triggers the OOM killer; CPU shares/quotas throttle scheduling.

  • Managed via the cgroup v2 hierarchy under /sys/fs/cgroup.

• How they combine:

  • A runtime like runc starts a process in fresh namespaces and places it in a cgroup as defined by the OCI config.json.

  • Namespaces alone don't cap usage; cgroups alone don't hide the host: you need both for real containment.

  • They are complemented by capabilities, seccomp, and MAC (SELinux/AppArmor) for a full security posture.

A PID namespace gives a container its own isolated process-ID number space: processes inside see a fresh tree starting at PID 1 and cannot see or signal processes outside it. The first process in that namespace becomes PID 1 and inherits special init responsibilities.

• Isolated, nested numbering:

  • The same process has one PID inside the namespace and a different (higher) PID on the host.

  • Processes in a child PID namespace are invisible to it but visible to its parent.

• Why PID 1 is special:

  • It must reap orphaned/zombie children; if it doesn't, zombies accumulate.

  • Signals behave differently: the kernel won't deliver signals lacking a handler to PID 1, so a naive app may ignore SIGTERM and not shut down gracefully.

  • If PID 1 exits, the whole namespace is torn down and all its processes are killed.

• Practical consequence: Use a proper init (tini, docker run --init) to handle reaping and signal forwarding for the app.

The mount namespace isolates the set of filesystem mount points a process sees, so a container can have its own root filesystem and mount table independent of the host. Combined with pivot_root (or chroot), it gives the container a completely separate view of the filesystem.

• Per-namespace mount table: Each namespace has its own list of mounts; a mount or unmount inside doesn't affect the host by default.

• Building the container root:

  • The runtime prepares a rootfs (often an overlayfs of image layers) and switches to it with pivot_root, making it the container's /.

  • It then mounts /proc, /sys, tmpfs, and bind-mounts volumes.

• Mount propagation: Propagation types (private, shared, slave) control whether mount events cross the namespace boundary, which is how bind-mounts and volumes are shared or kept isolated.

A network namespace gives a container its own isolated network stack: its own interfaces, routing table, iptables rules, and port space. A container starts with only a loopback device and gets connectivity via a virtual link (typically a veth pair) into the host.

• Isolated stack:

  • Each namespace has independent interfaces, IP addresses, routes, ARP tables, and firewall rules.

  • Two containers can both bind port 80 because port spaces are separate.

• Connecting to the outside:

  • A veth pair acts like a virtual cable: one end sits in the container, the other on the host (often plugged into a bridge).

  • NAT/masquerading via iptables gives outbound access; port publishing adds DNAT rules.

• Management: You can inspect one with ip netns exec or by entering /proc/<pid>/ns/net.

Containers in the same Pod share the network and IPC namespaces (and optionally PID), plus any shared volumes, while each keeps its own mount and (usually) PID namespace and its own filesystem root. This is implemented via a "pause" container that holds the shared namespaces.

• Shared by default:

  • Network namespace: all containers share one IP and port space, so they reach each other over localhost.

  • IPC namespace: they can use SysV IPC / POSIX shared memory together.

  • Shared volumes: mounted into multiple containers to exchange files.

• Not shared by default:

  • Mount namespace and root filesystem: each container has its own image/rootfs.

  • PID namespace: separate unless shareProcessNamespace: true is set.

• The pause container: It's created first and owns the shared namespaces so they persist even as app containers restart.

Container networking is built from network namespaces plus virtual devices: each container gets an isolated stack, and veth pairs connect those namespaces to the host, where a software switch (Linux bridge) and NAT rules tie everything together. The bridge model is the classic single-host design.

• Primitives involved:

  • Network namespace for isolation; veth pair as the virtual cable; a Linux bridge (e.g. docker0) as an L2 switch.

  • iptables for NAT (outbound masquerade) and port forwarding (DNAT).

• Linux bridge model:

  • Each container's host-side veth end is attached to the bridge; the bridge forwards frames between them like a switch.

  • Containers on the same bridge talk directly at L2; traffic leaving the host is NATed through the host IP.

• Other conceptual models:

  • Overlay networks (VXLAN) for cross-host connectivity.

  • Macvlan/ipvlan to give containers IPs directly on the physical network.

  • host mode: the container shares the host's network namespace with no isolation.

The IPC namespace isolates System V IPC objects and POSIX message queues, so processes in one container cannot see or attach to the IPC resources of another container or the host.

• Resources it scopes:

  • System V shared memory segments (shmget), semaphores (semget), and message queues (msgget).

  • POSIX message queues, visible under /dev/mqueue.

• Why it matters:

  • These objects have global identifiers/keys; without isolation, one container could read or clobber another's shared memory.

  • Each namespace gets its own limits (/proc/sys/kernel/shmmax etc.).

• Consequence: Two containers must share an IPC namespace to communicate via shared memory (e.g. Docker --ipc=container:<id>); otherwise the segments are invisible to each other.

nsenter enters one or more existing namespaces of a target process and then runs a command inside them. It works by opening the magic symlink files under /proc/<pid>/ns/ and calling setns() on each, which reassociates the caller with those namespaces.

• The /proc/<pid>/ns/* files:

  • One entry per namespace type: net, pid, mnt, uts, ipc, user, cgroup, time.

  • Each is a handle to a namespace; holding an open fd keeps that namespace alive even if the original process exits.

• How joining works:

  • Open the ns file, call setns(fd, type); the calling thread now belongs to that namespace.

  • For the PID namespace, setns() only affects children: nsenter then fork()s so the new child is actually inside the target PID namespace.

• Typical use: Debugging a container's network from the host without a shell in the container.

A fresh network namespace starts empty: it has only a loopback interface (down), no physical NICs, no routes, and no way out. Connectivity must be wired up explicitly by connecting it to the host's networking.

• What the namespace gives you:

  • Its own interfaces, routing table, ARP table, iptables/nftables rules, and sockets, fully isolated from the host.

  • Only a lo interface exists, and it's down until you bring it up.

• Steps to give it network access (typical bridge setup):

  1. Create a veth pair: two connected virtual interfaces.

  2. Move one end into the container's netns; leave the other on the host.

  3. Attach the host end to a bridge (e.g. docker0) and bring both ends up.

  4. Assign an IP to the container end and bring up lo.

  5. Add a default route in the container pointing at the bridge/gateway IP.

  6. Enable IP forwarding and add a NAT/masquerade rule on the host so container traffic can reach external networks.

• Result: Packets leave the container through the veth, hit the bridge, get forwarded and masqueraded by the host to the outside world.

OverlayFS is a union filesystem that presents a single merged view by stacking one or more read-only lower layers beneath a single writable upper layer, resolving files top-down.

• The four components:

  • lowerdir: one or more read-only layers (the image layers); listed top-to-bottom, higher wins.

  • upperdir: the single writable layer where all changes land.

  • workdir: an empty scratch dir on the same filesystem as upper, used internally for atomic operations like copy-up.

  • merged: the unified mountpoint the container sees as its root.

• How lookups resolve:

  • A file is served from the highest layer that has it: upper shadows lower.

  • Reads of unmodified files come straight from the lower (read-only) layers.

• Writes and deletions:

  • Modifying a lower file triggers copy-up: the file is copied into upper, then edited there.

  • Deleting a lower file creates a whiteout (a special char device 0/0) in upper that hides it in merged.

Copy-on-Write means layers are shared read-only until a container tries to modify a file: only then is that file copied into the container's writable layer and changed there, leaving the original untouched.

• How it works:

  • Reads and unchanged files reference the shared lower layers directly (no copy).

  • First write to a file copies it up to the writable layer (copy-up), then applies the change; subsequent writes hit the copy.

• Benefits:

  • Fast startup: no need to duplicate the whole image, the container just gets a thin writable layer.

  • Storage efficiency: many containers from one image share the same read-only layers on disk.

  • Memory efficiency: shared pages of shared files can be cached once across containers.

• Trade-off: First modification of a large file pays a copy-up latency cost; write-heavy workloads on image files are better served by a volume.

An image is an ordered stack of read-only layers, each a filesystem diff; a union filesystem stacks them (plus a thin writable layer on top) into one root filesystem for the running container.

• Stacking:

  • Each layer records the changes (added/modified/deleted files) relative to the layers below it.

  • A union mount (e.g. OverlayFS) merges them so the container sees a single coherent tree, with upper layers shadowing lower ones.

• Why read-only: Immutability makes layers safely shareable and content-addressable, and guarantees the image is reproducible; runtime changes go to the separate writable layer.

• Why shared and deduplicated:

  • Each layer is identified by a content digest (SHA-256), so identical layers are stored once and reused across images and containers.

  • Pulling an image only downloads layers not already present locally, saving bandwidth and disk.

Data written to a container's writable layer is ephemeral: it lives only as long as the container, so removing the container deletes that layer and everything written to it. Volumes and bind mounts persist data by storing it outside that layer.

• Writable layer lifecycle: It's the thin CoW layer on top of the image; docker rm destroys it along with all runtime changes.

• Volumes:

  • Managed by the runtime, stored outside the union filesystem (e.g. under /var/lib/docker/volumes) and mounted into the container.

  • Survive container removal, can be shared between containers, and bypass CoW so writes are faster.

• Bind mounts:

  • Mount an arbitrary host path directly into the container; data lives on the host filesystem.

  • Good for sharing config/source with the host, but tie the container to the host's directory layout.

• Key point: Both mount points sit outside the writable layer, so their contents are unaffected by container deletion.

A container image is a stack of read-only layers; immutability comes from the fact that no running container ever writes to those layers. Instead a thin writable layer is added on top, and the union filesystem presents the merged view.

• Layers are read-only and content-addressed:

  • Each layer is identified by the digest of its contents (e.g. sha256:...), so any change produces a new layer with a new ID rather than mutating an old one.

  • Identical layers are shared across images, saving space and enabling caching.

• Copy-on-write isolates changes: When a container modifies a file, the union FS copies it up into the writable layer; the underlying read-only layer is untouched.

• The writable layer is ephemeral: It lives only for the container's lifetime and is discarded on removal, so the image itself stays pristine and reproducible.

• Result: verifiability and reuse: Because layers can't change, digests can be verified and many containers can safely share the same base layers.

The writable layer is a copy-on-write overlay mount tied to the container's lifecycle and stacked on the image; bind mounts and volumes are separate mounts of host (or driver-managed) storage grafted into the container's mount namespace, bypassing the overlay entirely.

• Ephemeral writable layer:

  • An overlay upperdir merged with the image's read-only lowerdirs, so writes trigger copy-up and disappear when the container is removed.

  • Goes through the storage driver, so heavy I/O pays copy-on-write overhead.

• Bind mounts:

  • A mount --bind of an existing host path into the container namespace: same inodes as the host, no overlay, no copy-on-write.

  • Data lives on the host filesystem and persists independently of the container.

• Volumes:

  • Runtime-managed storage (a directory the engine controls, or a volume-driver backend) mounted into the container.

  • Also bypasses the overlay and outlives the container; portable and managed by the engine rather than a raw host path.

• Key kernel-level distinction: Overlay writes are layered COW that vanish on removal; bind mounts and volumes are direct mounts to durable storage that shadow whatever the image had at that path.

cgroups enforce limits through the kernel scheduler and memory controller: CPU is throttled by capping how much runtime a group gets per scheduling period, while memory is bounded by a hard ceiling that triggers reclaim and ultimately the OOM killer when exceeded.

• CPU (CFS bandwidth control):

  • cpu.cfs_period_us defines a period (default 100ms) and cpu.cfs_quota_us the runtime allowed within it; quota/period = number of cores. E.g. 50000/100000 = 0.5 CPU.

  • When a group uses its quota, the scheduler throttles it until the next period, so limits are enforced by withholding CPU time, not by killing.

  • cpu.shares (v1) / cpu.weight (v2) instead set relative proportions under contention, not hard caps.

  • In cgroup v2 both are unified as cpu.max ("quota period").

• Memory:

  • memory.max (v2) / memory.limit_in_bytes (v1) sets the hard limit on the group's usage.

  • As usage approaches the limit the kernel reclaims pages (drops cache, swaps); if it can't stay under, the cgroup OOM killer terminates a process in the group.

  • memory.high is a soft throttle that applies reclaim pressure before the hard cap, slowing the group instead of killing it.

• Key contrast: CPU overuse is throttled (elastic); memory overuse is fatal (OOM kill), because you can defer CPU but not fabricate RAM.

When a container's memory usage hits its cgroup memory limit and cannot reclaim enough, the kernel's OOM-killer picks and kills a process inside that cgroup, so the failure is scoped to the container rather than the whole host.

• cgroup-scoped, not host-wide: Hitting the memory cgroup limit triggers a cgroup OOM event; the killer targets tasks within that cgroup, protecting other containers and the host.

• Reclaim first, then kill: At the limit the kernel tries to reclaim (drop page cache, swap if allowed); only when it can't free enough does it invoke the OOM-killer.

• PID 1 matters in containers: If the main process (PID 1 in the container) is killed, the whole container dies, typically showing exit code 137 (128 + SIGKILL).

• Observability: Look for OOMKilled status and kernel log lines; memory.oom_control (v1) / memory.events (v2) expose OOM counters.

The freezer controller suspends and resumes all tasks in a cgroup atomically, which is how a container is paused: every process is stopped in place without being killed, then can be thawed to continue exactly where it left off.

• What it does:

  • Moves all tasks in the cgroup into an uninterruptible frozen state so they receive no CPU time.

  • Unlike SIGSTOP, processes can't observe or interfere with the freeze, and it applies to the whole group atomically.

• Interface:

  • v1: write FROZEN or THAWED to freezer.state.

  • v2: write 1 or 0 to cgroup.freeze.

• How containers use it:

  • docker pause freezes the container's cgroup; processes and memory stay resident, so resume is instant.

  • Also used by checkpoint/restore (CRIU) to get a stable, consistent snapshot of processes.

The pids controller caps the number of processes/threads (tasks) a cgroup may create, which prevents a container from exhausting the host's finite PID space via runaway forking or a fork bomb.

• What it limits:

  • A maximum task count set in pids.max; current usage is visible in pids.current.

  • When the limit is reached, fork()/clone() fails with EAGAIN.

• Why it matters for containers:

  • PIDs are a global, finite host resource; one container spawning endlessly can starve every other container and the host itself.

  • Protects against fork bombs and buggy apps leaking threads/zombies.

• Practical note: Runtimes expose this (e.g. docker run --pids-limit, Kubernetes pod/node PID limits) as a stability guardrail.

The io controller (v2) / blkio controller (v1) throttles and weights a container's access to block devices, so one container's disk activity can't monopolize storage bandwidth or IOPS.

• Two enforcement styles:

  • Throttling (hard caps): absolute limits on bytes/sec and ops/sec per device (read/write).

  • Weighting (proportional): shares of available I/O when the device is contended, similar to CPU shares.

• Interface:

  • v2: a single io.max (limits per major:minor device: rbps, wbps, riops, wiops) and io.weight.

  • v1: split across files like blkio.throttle.read_bps_device and blkio.weight.

• Key caveat:

  • v1 weighting only worked well for direct I/O with the CFQ scheduler; buffered writes weren't accounted properly.

  • v2 fixes this because the unified hierarchy lets memory writeback be attributed to the right cgroup, so buffered write throttling actually works.

cpuset pins a container to specific CPU cores (and memory NUMA nodes), controlling where it runs, whereas quota-based limits (CFS) control how much CPU time it gets across whatever cores it lands on.

• cpuset = affinity/placement:

  • Set via cpuset.cpus (which cores) and cpuset.mems (which NUMA memory nodes).

  • The process can only be scheduled on the listed cores, giving hard isolation and cache/NUMA locality.

• Quota (CFS) = amount of time: Set via cpu.cfs_quota_us/cpu.cfs_period_us (or cpu.max in v2): caps CPU-seconds per period regardless of which cores.

• Key differences:

  • cpuset gives dedicated cores (no noisy-neighbor jitter, better cache locality); quota only throttles total time.

  • cpuset is coarse (whole cores); quota is fine-grained (fractional CPU like 1.5).

• They compose: pin latency-sensitive workloads with cpuset, and use quota for elastic throttling. Many use both.

Accounting means cgroups measure resource usage (counters/statistics), while limiting means they enforce a ceiling. Accounting is observation; limiting is control, and one can exist without the other.

• Accounting:

  • Tracks consumption: memory.current, cpu.stat, io.stat, pids.current.

  • Always on when a controller is enabled; drives metrics, billing, and monitoring.

• Limiting:

  • Enforces thresholds: memory.max, cpu.max, pids.max.

  • Hitting a limit triggers action: throttling (CPU/IO), reclaim or OOM kill (memory), or fork failure (pids).

• Why the distinction matters:

  • You can account without limiting (measure usage, no cap) to size limits before enforcing them.

  • Limiting builds on accounting: the kernel must count usage to know when to enforce.

The main modes differ in how much network isolation the container gets: none (fully isolated), host (shares the host stack), bridge (private network with NAT, the default), and container (shares another container's namespace).

• none:

  • Only a loopback interface; no external connectivity.

  • Use for maximum isolation or batch/compute jobs that need no network.

• host:

  • Shares the host network namespace: no isolation, no NAT, container ports are host ports.

  • Use for max performance or apps needing many/dynamic ports; costs isolation and risks port conflicts.

• bridge:

  • Container gets a private IP on a virtual bridge; outbound via NAT, inbound via published ports.

  • The default and the general-purpose choice for most single-host containers.

• container:

  • Joins an existing container's netns, sharing its IP and localhost.

  • Use for sidecar patterns (this is how a Kubernetes pod's containers share networking).

CNI is a specification (and set of plugins) that standardizes how container runtimes configure network interfaces for containers, decoupling the runtime from the networking implementation so any orchestrator can work with any network provider.

• The problem it solves:

  • Without a standard, every runtime would need custom integration with every network solution (bridge, overlay, cloud VPC, etc.).

  • CNI gives one contract so networking becomes pluggable.

• How the contract works:

  • The runtime calls a plugin binary with ADD/DEL/CHECK commands, passing the container's netns and JSON config on stdin.

  • The plugin sets up the interface, IP, and routes, then returns the result as JSON.

• Ecosystem:

  • Plugins can chain (e.g. an IPAM plugin allocates IPs while a main plugin wires the interface).

  • Used by Kubernetes and most runtimes; implementations include Calico, Cilium, Flannel, and the reference bridge plugin.

In the default bridge model, outbound traffic is source-NAT'd (masqueraded) to the host's IP, and inbound traffic to a published port is destination-NAT'd from the host port to the container's private IP, all implemented with iptables (or nftables) rules.

• Outbound (egress) via SNAT/masquerade:

  • The container has a private IP (e.g. 172.17.0.x) that isn't routable outside the host.

  • A MASQUERADE rule in the nat table's POSTROUTING chain rewrites the source to the host IP so replies can return.

• Inbound (ingress) via DNAT / port mapping:

  • Publishing a port (e.g. -p 8080:80) adds a DNAT rule in PREROUTING that rewrites host:8080 to containerIP:80.

  • A userland proxy (docker-proxy) or the rules also cover host-local traffic to the mapped port.

• Why NAT is needed here:

  • Container IPs live on a private, host-internal subnet that the outside world can't route to directly.

  • conntrack tracks flows so return packets are un-NAT'd back to the right container automatically.

By default container root is the same UID 0 as host root (no user namespace remapping), so if the process breaks out of its isolation it retains full root power on the host. Even without escaping, root inside the container weakens every other defense layer.

• Same UID as the host: Without a user namespace, UID 0 in the container is UID 0 on the host. A kernel bug or a misconfigured mount then gives real host root.

• Amplifies escape impact: Container root holds capabilities like CAP_DAC_OVERRIDE and can access mounted host paths, sockets (/var/run/docker.sock), or devices with full privilege.

• Weakens defenses: Root can attempt to modify files it shouldn't, exploit setuid binaries, and generally has more capabilities that must be individually dropped.

• Mitigations: Run as a non-root user (USER in the Dockerfile, runAsNonRoot in Kubernetes), enable user namespace remapping, and drop all capabilities you don't need.

Linux capabilities split the monolithic power of root into ~40 distinct privileges, so a process can be granted only the specific abilities it needs instead of all-or-nothing root. Containers use this to run as root-ish but with a minimal capability set.

• The idea: Instead of checking UID 0, the kernel checks for a specific capability like CAP_NET_BIND_SERVICE (bind ports < 1024), CAP_NET_ADMIN (network config), CAP_SYS_ADMIN (the broad, dangerous one).

• How containers use them:

  • Runtimes start with a reduced default set and let you add/drop precisely: --cap-drop=ALL then --cap-add=NET_BIND_SERVICE.

  • In Kubernetes via securityContext.capabilities.

• Best practice: Drop all, add back only what's required (least privilege). Avoid CAP_SYS_ADMIN, which is nearly equivalent to full root.

A read-only root filesystem mounts the container's image layers as immutable, so processes cannot write anywhere except explicitly provided writable volumes or tmpfs mounts.

• How it works: The root mount is set read-only (e.g. Docker --read-only or Kubernetes readOnlyRootFilesystem: true); any needed writable paths are added as explicit volumes.

• Security benefits:

  • Blocks an attacker from dropping malware, web shells, or modified binaries onto disk.

  • Prevents tampering with app config or system files to establish persistence.

  • Enforces immutability, so runtime state matches the vetted image (better auditing and reproducibility).

• Practical notes:

  • Apps that need scratch space get a tmpfs or emptyDir for /tmp, caches, or PID files.

  • It's a layer, not a full solution: it doesn't stop in-memory attacks or kernel exploits.

A privileged container (--privileged) runs with essentially all Linux capabilities, unrestricted device access, and most security filters disabled, so its root is effectively root on the host: it removes almost every isolation layer that normally contains a container.

• What it grants: All capabilities (including CAP_SYS_ADMIN), access to all host devices under /dev, and a lifted default seccomp/AppArmor profile.

• Why isolation collapses:

  • With device access it can mount the host disk and read/write host files directly.

  • It can load kernel modules, manipulate cgroups, and access host resources to break out.

  • Namespaces still exist, but the capabilities to escape them are handed over.

• When it's (rarely) needed: Docker-in-Docker or low-level system tooling; prefer granting specific capabilities/devices instead of full privilege.

By default a runtime like Docker keeps a small allowlist of about 14 capabilities and drops everything else (including all the dangerous ones), following the principle of least privilege so a container's root can do ordinary app work but not administer the host.

• Kept by default (examples): CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_SETUID/CAP_SETGID, CAP_NET_BIND_SERVICE, CAP_KILL, CAP_CHROOT: enough to run typical services.

• Dropped by default (examples): CAP_SYS_ADMIN, CAP_SYS_MODULE, CAP_SYS_PTRACE, CAP_NET_ADMIN, CAP_SYS_TIME: host-level admin powers.

• The reasoning:

  • Least privilege: keep only what common workloads need so a compromise has minimal reach.

  • Full root is really a bundle of capabilities; dropping most of them lets "root in a container" stay far weaker than host root.

• Best practice: Harden further with --cap-drop=ALL then add back only the specific capabilities the app requires.

The OCI Runtime Specification is an industry standard that defines how to run a container from an on-disk bundle: it standardizes the config.json configuration and the container lifecycle so any compliant runtime (like runc) behaves interchangeably.

• The filesystem bundle: A config.json plus a root filesystem directory the runtime is handed to launch.

• The runtime configuration: What config.json contains: the process to run, env, mounts, namespaces, cgroup limits, capabilities, seccomp, and other Linux-specific settings.

• The lifecycle and operations: State transitions (creating, created, running, stopped) and standard operations: create, start, kill, delete.

• What it does NOT define: Image format (that's the OCI Image Spec) or how images are pulled/stored: those belong to higher-level runtimes.

The CRI is a gRPC API that the kubelet uses to talk to a container runtime, so Kubernetes can manage pods and containers without knowing the runtime's internals: it's the pluggable boundary between the orchestrator and the runtime.

• A stable contract: Defines two services: RuntimeService (pod sandboxes, containers, exec/attach) and ImageService (pull, list, remove images).

• Decouples Kubernetes from any one runtime: Any runtime implementing CRI (containerd via its CRI plugin, CRI-O) can be plugged in; this is what allowed the removal of the built-in dockershim.

• Pod-centric model: The kubelet first asks the runtime to create a pod sandbox (shared namespaces, the pause container), then to run containers inside it.

• Where it sits: kubelet → CRI → high-level runtime → OCI runtime (runc). CRI is the top boundary; OCI is the bottom one.

High-level runtimes (containerd, CRI-O) manage the full container lifecycle above the OS: images, storage, networking, and orchestrator APIs; low-level runtimes (runc, crun) do the narrow, privileged job of actually spawning the container process from a bundle. The high-level runtime ultimately calls the low-level one.

• High-level runtimes:

  • Pull and unpack OCI images, manage snapshots/layers, set up CNI networking, expose CRI/gRPC to Kubernetes.

  • They produce an OCI bundle and delegate the actual run to an OCI runtime.

• Low-level (OCI) runtimes:

  • Read config.json and set up namespaces, cgroups, capabilities, seccomp, then exec the container process. No image or network knowledge.

  • runc is the Go reference implementation; crun is a smaller/faster C implementation of the same OCI spec.

• The relationship: Because both honor the OCI spec, low-level runtimes are swappable (e.g. point containerd at crun or kata-runtime) without changing the high-level runtime.

An OCI bundle is the on-disk, ready-to-run representation of a container: a directory containing everything an OCI runtime needs to launch it. It is the input contract to a runtime like runc.

• The two components:

  1. config.json: the runtime configuration (process to run, env, mounts, namespaces, cgroup limits, capabilities, seccomp).

  2. A root filesystem directory (typically rootfs/): the unpacked image layers that become the container's /.

• Its role: A high-level runtime (containerd) unpacks an image into this bundle, then hands the directory to runc, which reads config.json and pivots into the rootfs.

• Clean separation: The bundle is what makes image handling and container execution independent: the runtime never touches image registries, only bundles.

runc is the low-level reference runtime that actually creates and runs a container: it takes an OCI bundle (a root filesystem plus a config.json) and asks the kernel to set up namespaces, cgroups, capabilities, and mounts, then executes the process.

• It is the reference implementation of the OCI Runtime Spec:

  • The spec defines the on-disk bundle format and the lifecycle operations (create, start, kill, delete); runc implements them.

  • Because the interface is standardized, runc is swappable with any other OCI-compliant runtime.

• It sits at the bottom of the stack: High-level managers (containerd, CRI-O) do image pulls, networking, and lifecycle bookkeeping, then invoke runc per container.

• It is short-lived, not a daemon: runc execs the container process and exits; the container keeps running under a shim, not under runc.

• Written in Go and uses libcontainer to do the actual namespace/cgroup wiring.

crun is a fast, lightweight OCI runtime written in C by Red Hat that is functionally a drop-in replacement for runc: it implements the same OCI Runtime Spec but with a smaller footprint and quicker startup.

• Language and size:

  • Written in C with no garbage-collected runtime, so its binary and memory usage are much smaller than Go-based runc.

  • Lower per-container overhead matters at high container density.

• Performance: Faster create/start times and lower memory per container, which helps short-lived and high-churn workloads.

• Feature support: Early and strong cgroup v2 support, and it can be embedded as a library (libcrun).

• Compatibility: Same OCI interface, so containerd or CRI-O can call it wherever they call runc; it is the default runtime on some distros (e.g. Fedora/RHEL with CRI-O).

youki is an OCI-compliant container runtime written in Rust: like runc and crun it implements the OCI Runtime Spec, aiming for memory safety and performance without a garbage-collected runtime.

• Language and goals: Written in Rust, giving memory safety without a GC, plus low startup latency and small footprint comparable to crun.

• Compared to runc: runc is Go with GC overhead; youki targets faster create/start and lower memory, similar to crun's advantages over runc.

• Compared to crun: Same niche (small, fast, native), but Rust vs C; a design and safety choice more than a big behavioral difference.

• Interchangeability: Being OCI-compliant, it plugs into containerd or CRI-O in place of runc; it is younger and less battle-tested in production.

A zombie is a terminated child whose exit status hasn't been reaped by its parent; in containers this piles up when PID 1 fails to reap orphaned processes, which is a direct consequence of the PID 1 problem.

• What a zombie is: When a process exits, it stays as a defunct entry in the process table until its parent calls wait() to collect the exit code.

• How PID 1 gets involved:

  • When a process's parent dies, its children are reparented to PID 1, which is then responsible for reaping them.

  • If the container's PID 1 (your app) never reaps, those orphans accumulate as zombies.

• Why it matters: Zombies consume PID table slots; enough of them can exhaust PIDs and prevent new processes from starting.

• The fix (same as PID 1 problem): Run a proper init as PID 1 (tini, --init) that reaps children by design; this is especially important for apps that spawn subprocesses.

They act as a lightweight init process at PID 1 that forwards signals to your real application and reaps zombie processes, giving you correct signal handling and clean shutdown without your app having to implement init duties.

• The PID 1 problem they solve:

  • Linux gives PID 1 special treatment: signals without a registered handler are ignored, so a naive app as PID 1 may never receive SIGTERM.

  • PID 1 is also responsible for reaping orphaned children; if it doesn't, defunct (zombie) processes accumulate.

• Signal forwarding:

  • tini/dumb-init run as PID 1 and re-send received signals (SIGTERM, SIGINT) to the child app, so it can shut down gracefully.

  • They exit with the child's exit status, preserving correct container exit codes.

• Zombie reaping: They call wait() on reparented children, preventing zombie buildup in long-running containers.

• How you enable them: Set the init as ENTRYPOINT so it wraps your app, or use Docker's built-in --init flag (which injects tini automatically).

Because the container's lifecycle is driven by signals: orchestrators send SIGTERM to PID 1 to ask for a clean stop, and if that signal is ignored or not passed to the real app, the app is killed abruptly, dropping requests and leaking resources.

• How the shutdown sequence works:

  • On docker stop or a Kubernetes pod termination, the runtime sends SIGTERM to PID 1, then SIGKILL after a grace period (default 10s in Docker, terminationGracePeriodSeconds in k8s).

  • The app should use that window to finish in-flight requests, flush buffers, and close connections.

• Why PID 1 makes this fragile:

  • PID 1 ignores signals lacking an explicit handler, so an app without a SIGTERM handler simply doesn't react and is force-killed.

  • A shell-form CMD wraps the app in /bin/sh -c, which becomes PID 1 and often doesn't forward the signal to the child.

• Consequences of getting it wrong:

  • Dropped or errored client requests, corrupted writes, unreleased locks or DB connections.

  • Slow stops: the container hangs until SIGKILL, wasting the whole grace period on every deploy or scale-down.

• How to do it right: Use exec-form CMD/ENTRYPOINT so the app is PID 1 and register a SIGTERM handler, or front it with an init like tini/dumb-init for correct forwarding.

Namespaces were added to the Linux kernel to give a process a private, isolated view of a global system resource, so multiple processes could each believe they own resources like the process tree, network stack, or mounts. They are the core building block that made lightweight containers possible.

• Origin:

  • Inspired by Plan 9's per-process namespaces; the mount namespace landed first in Linux 2.4.19 (2002).

  • Most others (pid, net, ipc, uts) arrived through the 2.6 series, with user namespaces largely complete by 3.8 (2013).

• Motivation:

  • Provide lightweight isolation without a full VM: resource partitioning inside one kernel.

  • Enable use cases like containers, checkpoint/restore, and running services with their own network/hostname.

• Key idea: Each namespace virtualizes one resource type; a process can be in different namespaces per type, created via clone(), unshare(), and setns().

Windows containers apply the same idea as Linux (isolate a process's view of the OS) but through Windows kernel mechanisms, and they offer two isolation modes: process isolation shares the host kernel like Linux containers, while Hyper-V isolation wraps each container in a lightweight VM for a stronger boundary.

• How they work:

  • Windows provides its own namespace/job-object style isolation plus a compute service (HCS) that runtimes drive.

  • Images must match the host's Windows build closely because of tight kernel coupling.

• Process isolation:

  • Containers share the host kernel: fast and dense, like Linux containers.

  • Requires host and container OS versions to be compatible.

• Hyper-V isolation:

  • Each container runs in a minimal, optimized VM with its own kernel: stronger security and version flexibility.

  • Higher overhead and slower startup than process isolation.

• Same image can typically run in either mode; you pick isolation at runtime.

They are two different digests of the same layer computed over different forms of the data: the layer digest is the hash of the compressed blob as stored/transferred, while the diffID is the hash of the uncompressed tar. One addresses distribution; the other identifies filesystem content for stacking.

• Layer digest (blob digest):

  • SHA-256 of the layer as stored: usually the gzip-compressed tarball.

  • Appears in the manifest's layers array and is what the registry uses to store and serve the blob.

• diffID:

  • SHA-256 of the uncompressed tar (the actual filesystem changeset).

  • Listed in the config's rootfs.diff_ids and used to compute the chainID for local layer caching.

• Why two exist:

  • Compression is nondeterministic, so the compressed hash can differ even when content is identical; the diffID gives a stable identity for the filesystem content.

  • The layer digest optimizes transfer/dedup on the wire; the diffID optimizes stacking and dedup on disk.

The OCI distribution spec models a registry as content-addressed blob storage plus a tag/name index, exposed over a small set of HTTP endpoints under /v2/. Blobs (layers and configs) are stored and fetched by digest; manifests are stored by digest and also reachable by tag.

• Two kinds of content:

  • Blobs: layer tarballs and the config JSON, addressed by their SHA-256 digest.

  • Manifests: the manifest/index, addressed by digest or by a human tag.

• Key endpoints:

  • GET /v2/<name>/blobs/<digest> fetches a blob by content address.

  • GET /v2/<name>/manifests/<reference> fetches a manifest by tag or digest.

  • Uploads use POST/PATCH/PUT to /v2/<name>/blobs/uploads/ (chunked or monolithic).

• How a pull works:

  1. Client GETs the manifest; the registry returns config and layer digests.

  2. Client GETs each blob by digest, skipping any it already has cached.

• Why content addressing matters: Digests give automatic dedup (shared layers stored once) and integrity verification (fetched bytes must hash to the requested digest).

A user namespace maps a range of UIDs/GIDs inside the namespace to a different range on the host, so a process can be root (UID 0) inside while being an unprivileged user outside. This decoupling is what lets an ordinary user create and own containers without real host root.

• UID/GID mapping:

  • Mappings are written to /proc/<pid>/uid_map and gid_map, e.g. inside UID 0 maps to host UID 100000.

  • A file owned by host UID 100000 appears owned by root inside the namespace.

• Capabilities are namespace-scoped: The process gets a full capability set inside its user namespace, but those capabilities are powerless against resources owned by the host root.

• Enables rootless containers:

  • An unprivileged user can unshare() a user namespace and then create other namespaces (mount, net, PID) that normally require CAP_SYS_ADMIN.

  • Tools like Podman and newuidmap/newgidmap (with /etc/subuid) set up the subordinate ID ranges.

• Security benefit: A container breakout to "root" lands only on an unprivileged host user, shrinking the blast radius.

These three syscalls are the kernel API for namespaces: clone() creates a new process in new namespaces, unshare() moves the calling process into new namespaces, and setns() attaches an existing process to an already-existing namespace.

• clone():

  • Like fork() but takes CLONE_NEW* flags (e.g. CLONE_NEWPID, CLONE_NEWNET) to create the child in fresh namespaces.

  • Used by runtimes to launch the container's first process directly into its isolation.

• unshare():

  • Puts the current process into new namespaces without creating a child (the unshare CLI uses it).

  • Note: CLONE_NEWPID affects children only, since your own PID can't change.

• setns():

  • Joins an existing namespace referenced by an fd from /proc/<pid>/ns/*.

  • This is how docker exec and nsenter enter a running container.

The container engine launches the container's first process inside a new PID namespace, so it appears as PID 1 to the container while the kernel simultaneously assigns it a normal, larger PID on the host. Every container process therefore has two PIDs: one in its namespace and one on the host.

• Creation: The runtime uses clone() with CLONE_NEWPID so the entrypoint becomes PID 1 in the new namespace.

• Dual-PID relationship:

  • Container PID 1234 might be host PID 48211; the host sees the real number, the container sees the namespaced one.

  • The host (parent namespace) can see and signal container processes, but not vice versa.

• Why it matters:

  • Killing a container really means signaling its PID 1 by host PID; when PID 1 exits the namespace collapses.

  • Tools map between views: docker top or /proc show host PIDs, while ps inside the container shows namespaced ones.

The cgroup namespace virtualizes a process's view of the cgroup hierarchy, so that its own cgroup appears as the root / in /proc/self/cgroup and under /sys/fs/cgroup. It was added because cgroups alone limit resources but still leak the full host cgroup paths into the container.

• The problem it solves:

  • Without it, a container reading /proc/self/cgroup saw its absolute path on the host (e.g. /docker/<long-id>/...), leaking host layout and container identity.

  • That also made it awkward to bind-mount and use cgroupfs inside the container safely.

• What it does:

  • It sets a cgroup root for the namespace; paths are shown relative to that root, hiding ancestors above it.

  • Improves isolation and makes running systemd or nested cgroup managers inside a container cleaner.

• Key distinction: It virtualizes the view of the hierarchy, not the limits themselves: the actual resource caps still come from the cgroup the process is placed in.

The time namespace lets a container present different values for certain system clocks, primarily CLOCK_MONOTONIC and CLOCK_BOOTTIME, by applying per-namespace offsets to the host clocks.

• What it virtualizes:

  • Offsets for CLOCK_MONOTONIC and CLOCK_BOOTTIME, configured via /proc/<pid>/timens_offsets.

  • This makes a container appear to have its own uptime/boot time.

• What it does NOT virtualize: Wall-clock time (CLOCK_REALTIME) is not offset; you cannot give a container a different calendar date this way.

• Why it exists: Chiefly for checkpoint/restore (CRIU): a restored process must see monotonic time that never goes backwards even on a new host.

Mount propagation controls whether a mount or unmount event under a directory is shared with (propagated to) other mount namespaces or peer mounts. It matters for containers because it decides whether mounts done on the host become visible inside a container and vice versa.

• shared: Mount/unmount events propagate both ways between peers in a peer group: mounting on the host shows up in the container and the reverse.

• private: No propagation in either direction: events stay local to that mount. This is the safe default for isolating a container.

• slave: One-way: the slave receives events from its master but does not send its own back up. Used so a container sees host mounts (like a device appearing) without polluting the host.

• unbindable: Private and additionally cannot be used as the source of a bind mount, preventing recursive/duplicated bind explosions.

• Container relevance:

  • Runtimes often make the container rootfs private or slave; rshared bind mounts are how tools like a CSI driver push new mounts into a running container.

  • Set with mount --make-shared, --make-private, --make-slave, --make-unbindable (and recursive r* variants).

PID namespaces form a hierarchy: creating a new PID namespace nests it under the current one. A process gets a PID in its own namespace and in every ancestor namespace, so the parent can see and signal it, but the process itself only sees the innermost view.

• Nesting rules:

  • A clone()/unshare() with CLONE_NEWPID makes children live in a new child namespace; can nest up to a fixed depth (32).

  • The first process in the new namespace becomes PID 1 there.

• Multiple PIDs per process:

  • A process has one PID in its own namespace and a different (usually higher) PID in each ancestor, up to the host.

  • Visibility is one-way: ancestors see descendants; a process cannot see or signal PIDs in child or sibling namespaces.

• PID 1 responsibilities:

  • It reaps orphaned children and receives default signal handling; if PID 1 dies, the whole namespace's processes are killed.

  • This is why containers need a proper init (tini, dumb-init) to avoid zombie processes.

A user namespace gives a process its own uid/gid mapping table: a uid inside the namespace is translated to a different uid outside it, so uid 0 (root) inside can map to an unprivileged uid on the host.

• The mapping is defined per-namespace:

  • Written to /proc/<pid>/uid_map and /proc/<pid>/gid_map as triples: inside-id, outside-id, length.

  • Example: 0 100000 65536 maps container uid 0 to host uid 100000, uid 1 to 100001, and so on.

• subuid/subgid delegate a safe range:

  • /etc/subuid and /etc/subgid grant an unprivileged user a block of host ids (e.g. alice:100000:65536) they are allowed to remap into.

  • Helpers newuidmap/newgidmap (setuid) write the map using only ranges the user owns, enabling rootless containers.

• Why container-root is not host-root:

  • The kernel evaluates permissions and capabilities against the host uid the process actually maps to, not the in-namespace uid.

  • So root in the container has full capabilities only over resources inside its namespace; on the host it is just unprivileged uid 100000 with no rights over other files or processes.

pivot_root swaps the process's root mount for a new one and lets you detach the old root entirely, whereas chroot only changes a pointer to / without removing access to the original filesystem, making it weaker for isolation.

• What pivot_root does:

  • Moves the current root mount to a specified old-root directory and makes a new directory the root of the mount namespace.

  • After it, you umount the old root so the host filesystem is genuinely gone from the container's view.

• Why chroot is insufficient:

  • It only changes the process's root directory; the old root mount still exists and is reachable.

  • A process with CAP_SYS_CHROOT (or via open fds, .. tricks) can escape a chroot jail.

• Why containers prefer pivot_root: Combined with a mount namespace, detaching the old root removes any mount the container could climb back through, giving true filesystem isolation.

FUSE (Filesystem in Userspace) lets you implement image storage logic in a userspace process rather than the kernel, which is powerful for features like lazy pulling and on-demand fetching, but it costs performance because every I/O crosses the user/kernel boundary.

• Why FUSE can be the right choice:

  • Lazy / on-demand loading: filesystems like stargz or nydus can start a container before all layers download, fetching blocks as they're read.

  • No kernel privileges or modules needed, so it works in restricted or rootless environments.

  • Flexibility: custom logic (decompression, remote fetch, dedup) is easy to implement and iterate in userspace.

• Tradeoffs versus OverlayFS:

  • Performance: FUSE adds context switches and data copies between kernel and the userspace daemon; OverlayFS runs entirely in-kernel and is much faster.

  • Reliability: if the FUSE daemon crashes, the mount breaks; OverlayFS has no such daemon dependency.

  • Maturity: OverlayFS is the default snapshotter, well-tested and simple, but it needs full layers present locally and lacks native lazy-pull.

• Rule of thumb: use OverlayFS for speed and simplicity; reach for FUSE when you need lazy pulling, custom formats, or unprivileged operation.

A whiteout is a special marker in an upper layer that says "this file, which exists in a lower layer, is deleted." Because lower layers are read-only, you can't actually remove the file; you record its absence with a whiteout so the merged view hides it.

• How OverlayFS represents them:

  • A whiteout is a character device with device number 0/0 placed in the upper layer at the deleted file's path.

  • When overlay merges layers, that name is masked and the file disappears from the unified view.

• How OCI image layers represent them:

  • In a tar layer, a deletion is a zero-length file named .wh.<filename> (the AUFS-style convention) in the same directory.

  • When the layer is unpacked/applied, the runtime translates .wh. entries into the storage driver's native whiteout (e.g. the 0/0 char device for overlay).

• Purpose: Preserves immutability of lower layers while still expressing deletions as forward-only diffs.

An opaque directory is a marker saying "this directory in the upper layer completely replaces any directory of the same name in lower layers": nothing from below shows through. It's a directory-level mask, whereas a whiteout hides a single named entry.

• When it's used: Typically when a directory is deleted and recreated: the new directory must not inherit the old contents merged from lower layers.

• How OverlayFS marks it: An extended attribute trusted.overlay.opaque="y" is set on the upper directory, so overlay ignores the lower versions entirely.

• How OCI layers mark it: A special entry .wh..wh..opq inside the directory in the tar layer signals opacity.

• Difference from a whiteout:

  • Whiteout: hides one specific path (.wh.<name>).

  • Opaque: hides all lower-layer contents of an entire directory at once, without needing a whiteout per file.

Snapshotters (containerd) and storage/graph drivers (the older Docker term) are the components that turn image layers into a usable root filesystem for a container. They manage how layers are stored, stacked, and given a writable top, using a backend like OverlayFS.

• Graph drivers (Docker's model): Pluggable backends (overlay2, btrfs, devicemapper) that know how to assemble read-only layers plus a copy-on-write layer.

• Snapshotters (containerd's model):

  • A cleaner abstraction based on immutable "committed" snapshots and mutable "active" snapshots, each layered on a parent.

  • containerd asks the snapshotter to produce a set of mounts; it doesn't manage the filesystem itself.

  • Pluggable implementations: overlayfs (default), plus lazy-pull ones like stargz and nydus.

• Their role in the runtime pipeline:

  • containerd unpacks image layers into the snapshotter, requests a writable snapshot, and hands the resulting mounts to the OCI runtime (runc) as the container's rootfs.

  • They decouple image storage strategy from image distribution and execution, so you can swap backends without changing the rest of the stack.