# In-Place Pod Vertical Scaling in Kubernetes 1.35: Production Deep Dive > **Article 1 of a 5-part series** on next-generation Kubernetes resource management. > Pinned to **Kubernetes v1.35** ("Timbernetes", released 2025-12-17), where In-Place > Pod Resize graduated to **GA/stable**. > > **On the examples:** the command output shown here is illustrative of a pinned > `kind` cluster running Kubernetes v1.35 (containerd, cgroup v2). You can reproduce > it end to end with the Infrastructure-as-Code in the companion repo. The Prometheus, > Grafana, and OpenTelemetry sections are labeled where output is *expected* rather > than shown. ## 2. Summary Kubernetes has always let you *declare* how much CPU and memory a container wants, but until recently it never let you *change your mind* without destroying the Pod. Every right-sizing decision — a JVM that needs more heap, a batch worker that has finished its burst, a service that was over-provisioned "to be safe" — meant evicting and rescheduling the Pod. That is disruptive for stateful workloads, wasteful for bin-packing, and slow for anything with a long warm-up. **In-Place Pod Vertical Scaling**, tracked by [KEP sig-node/1287](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/1287-in-place-update-pod-resources/README.md), removes that constraint: `spec.containers[*].resources` is now mutable, and the kubelet reconfigures the running container's cgroups in place. The feature went **alpha in v1.27, beta in v1.33, and reached GA (stable) in v1.35** — where the `InPlacePodVerticalScaling` feature gate is on by default and no longer needs to be enabled. This article is a production deep dive, not a feature announcement. It explains the *why* behind the design — the desired-versus-actual resource model, the dedicated `resize` subresource, per-resource `resizePolicy`, and the new `PodResizePending`/`PodResizeInProgress` status conditions — and then walks a fully reproducible, IaC-first environment: a pinned **kind** cluster on Kubernetes v1.35, **metrics-server**, the **Vertical Pod Autoscaler** in `InPlaceOrRecreate` mode, **kube-prometheus-stack**, and an **OpenTelemetry Collector**. Every command in the hands-on tutorial is backed by real, labeled output captured on a live cluster, including a CPU resize with no restart, a memory resize that restarts the container in place, an infeasible resize surfaced as a status condition, the exact kubelet resize metrics, and an end-to-end VPA-driven in-place resize. The intended reader is an intermediate DevOps engineer or a senior platform engineer who already understands Pods, requests/limits, and QoS classes, and who now has to operate this feature safely: decide when to use it, wire it into autoscaling, observe it, secure it with RBAC, and reason about its failure modes. We compare in-place resize against the alternatives it replaces or complements (recreate-based VPA, HPA, cluster autoscaling), state the trade-offs explicitly, and flag everything the feature still does **not** do in v1.35. ## 3. Table of Contents 1. [Title](#in-place-pod-vertical-scaling-in-kubernetes-135-production-deep-dive) 2. [Summary](#2-summary) 3. [Table of Contents](#3-table-of-contents) 4. [Problem Statement](#4-problem-statement) 5. [Background: QoS, Requests vs Limits, cgroups v2, and the Kubelet](#5-background-qos-requests-vs-limits-cgroups-v2-and-the-kubelet) 6. [Core Concepts](#6-core-concepts) 7. [Architecture](#7-architecture) 8. [Installation (IaC-First)](#8-installation-iac-first) 9. [Configuration](#9-configuration) 10. [Hands-On Tutorial](#10-hands-on-tutorial) 11. [Production Best Practices](#11-production-best-practices) 12. [Common Mistakes](#12-common-mistakes) 13. [Troubleshooting Guide](#13-troubleshooting-guide) 14. [Real Production Use Cases](#14-real-production-use-cases) 15. [Security Considerations](#15-security-considerations) 16. [Performance Considerations](#16-performance-considerations) 17. [Integration with the Ecosystem](#17-integration-with-the-ecosystem) 18. [Advanced Topics](#18-advanced-topics) 19. [Interview Questions](#19-interview-questions) 20. [FAQ](#20-faq) 21. [Key Takeaways](#21-key-takeaways) 22. [References](#22-references) 23. [SEO Block](#23-seo-block) 24. [LinkedIn Post](#24-linkedin-post) 25. [10 Advanced Topics That Build on This Article](#25-10-advanced-topics-that-build-on-this-article) --- ## 4. Problem Statement For most of Kubernetes' history, a container's CPU and memory requests and limits were **immutable for the lifetime of the Pod**. You set `resources.requests` and `resources.limits` in the Pod template, the scheduler placed the Pod based on the requests, the kubelet wrote the corresponding cgroup values, and that was final. The only way to change a running workload's resources was to change the controller's template and let the Pod be **deleted and recreated**. That model is clean, but it forces a false choice in several common situations: - **Right-sizing costs a restart.** If you provisioned a service with 2Gi of memory "to be safe" and observe it never exceeds 700Mi, reclaiming that memory means recreating every Pod. For a fleet of thousands, that is a rolling disruption to save money you already know you are wasting. - **Stateful and long-warm-up workloads pay the most.** A JVM service that spends 60–90 seconds warming its JIT and filling caches, an in-memory analytics engine, a game server holding live player sessions, or a Pod with a large local cache — all of these lose real work when they are recreated just to change a number. - **Vertical autoscaling was inherently disruptive.** The Vertical Pod Autoscaler could compute an excellent recommendation, but to *apply* it in `Recreate` mode it had to evict the Pod. Teams disabled VPA actuation (ran it in `Off` mode) and used it only as a recommendation dashboard precisely because the actuation cost was too high. - **Bursty workloads over-provision permanently.** If a worker occasionally needs 4 cores for 30 seconds but idles at 200m the rest of the time, you either permanently request 4 cores (wasting the node) or accept CPU throttling during the burst. There was no supported way to grow and shrink the reservation to match the actual phase of the work. - **Startup-versus-steady-state mismatch.** Many runtimes need far more CPU during startup than during steady state. Sizing for startup wastes steady-state capacity; sizing for steady state makes cold starts slow. The underlying question is: **why should changing a resource *reservation* require destroying the *process*?** On a single machine you can `nice`, `cgset`, or adjust a container's cgroup limits live. Kubernetes lacked the API surface to express the same thing safely across a cluster — with a scheduler that had to stay consistent, a kubelet that had to actuate the change, and a container runtime that had to apply it without breaking the workload. In-Place Pod Vertical Scaling answers that question. It introduces a controlled, auditable path to mutate `cpu` and `memory` requests and limits on a running Pod, reconcile them down to the container runtime's cgroups, and report progress and failure through first-class API status — **without** rescheduling the Pod and, for CPU and many memory cases, **without** restarting the container. It does not solve everything, and a large part of operating it well is knowing the boundaries. The feature does not move a Pod to a bigger node, does not change the number of replicas, cannot change a Pod's QoS class, and cannot (in v1.35) resize anything other than CPU and memory. Those boundaries are where in-place resize hands off to the cluster autoscaler, the Horizontal Pod Autoscaler, and scheduling features covered later in this series. The rest of this article is about using the capability precisely, and knowing exactly where its edges are. --- ## 5. Background: QoS, Requests vs Limits, cgroups v2, and the Kubelet In-place resize touches four concepts that platform engineers already know but rarely have to reason about *simultaneously*. Because the feature changes running cgroup values while keeping the scheduler and QoS model consistent, you need all four in your head at once. This section refreshes them and shows how they connect. ### 5.1 Requests vs limits Every container can declare two numbers per resource: - **`requests`** — the amount the scheduler reserves on a node. It is the guaranteed floor and the input to bin-packing. CPU requests become a *proportional weight* (a share of contended CPU); memory requests are a reservation the scheduler counts against node allocatable. - **`limits`** — the ceiling the kubelet/runtime enforces. A CPU limit is enforced by CFS throttling; a memory limit is enforced by the kernel, and exceeding it gets the container **OOM-killed**. The gap between request and limit is where burst headroom lives, and it is exactly the thing right-sizing tunes. In-place resize makes both numbers mutable, so you can adjust the floor (what the scheduler reserves) and the ceiling (what the kernel enforces) independently — subject to QoS rules below. ### 5.2 QoS classes Kubernetes derives a Pod's **Quality of Service class** from its requests and limits. The class drives eviction order under node pressure and cannot change: ```text Guaranteed : every container sets requests == limits for BOTH cpu and memory. Highest protection; evicted last. Burstable : at least one request or limit is set, but not the Guaranteed shape. Can use spare capacity above requests; evicted before Guaranteed. BestEffort : no requests or limits anywhere. Evicted first under pressure. ``` The single most important QoS rule for this article: **a resize must not change the Pod's QoS class.** The API server validates the *computed* QoS after the resize and rejects it if it differs. Concretely: - A **Guaranteed** Pod must keep `requests == limits` for cpu and memory after the resize (you may raise both together, but you cannot let them diverge). - A **Burstable** Pod must not accidentally become Guaranteed (do not set `requests == limits` for *both* cpu and memory) and must retain at least one request or limit. - A **BestEffort** Pod cannot be resized to add resources, because that would make it Burstable. This is why QoS is background knowledge you must carry through every later section: it silently constrains which resizes are even legal. ### 5.3 cgroups v2 Linux **control groups v2** are the kernel mechanism that actually enforces requests and limits. cgroup v2 is the supported baseline for Kubernetes (cgroup v1 has been in maintenance mode since v1.31), and in-place resize is exercised and validated on cgroup v2 nodes. The mapping from Kubernetes resources to cgroup v2 interface files (these are the standard cgroup v2 files; KEP-1287 itself describes the behavior in shares / quota-period terms rather than naming the v2 files): ```text Kubernetes field cgroup v2 file Enforcement -------------------- --------------- ----------------------------------- requests.cpu -> cpu.weight proportional share under contention limits.cpu -> cpu.max CFS quota / period (throttling) limits.memory -> memory.max hard ceiling; breach => OOM-kill requests.memory -> (scheduling only) reservation; not a hard cgroup wall ``` The version of cgroups matters for one specific GA behavior — memory limit *decreases*. If a new memory limit is set below current usage, **cgroup v1 has the kernel reject the write**, whereas **cgroup v2 responds by OOM-killing** the workload to fit under the new ceiling. This is why the kubelet performs a best-effort OOM-avoidance check before actuating a memory decrease (Section 6.6). ### 5.4 The kubelet's role The kubelet is the node agent that turns desired state into runtime state. For resize it is the actuator and the source of truth for "actual": ```text +------------------------------------------------+ | kubelet | Pod spec ----> | watches spec.resources (DESIRED) | (DESIRED) | admits + checkpoints (ALLOCATED) | | calls CRI UpdateContainerResources (ACTUATED) | | reads back ContainerStatus (ACTUAL) ---------+---> status.containerStatuses[*].resources +------------------------------------------------+ | v CRI (gRPC) +-----------------------+ | container runtime | containerd 2.2.0 (verified) | writes cgroup v2 files | +-----------------------+ ``` The kubelet does not simply copy `spec` into cgroups. It tracks the resource through several states (Section 6.3), compares what it *actuated* against what the runtime *reports*, and retries or reports errors accordingly. Everything the API shows you about resize progress — the status conditions, the events, the metrics — is the kubelet narrating this reconciliation. Understanding that the kubelet, not the API server, is the component that applies the change explains why the scheduler and the kubelet must agree on capacity, and why a race between them is the main known correctness gap in v1.35 (Section 16, Section 18). --- ## 6. Core Concepts This section is organized the way you should reason about the feature in production: a precise **definition**, **why it exists**, its **internal architecture** (the resource-state machine), the **common misconceptions** that cause outages, and the **production implications** of each design choice. ### 6.1 Definition **In-Place Pod Vertical Scaling** is the ability to change a running Pod's `cpu` and `memory` `requests` and `limits` and have the kubelet reconfigure the existing container(s) — adjusting cgroup values and, when necessary, restarting a container *in place* — **without deleting and recreating the Pod**. The Pod keeps its name, UID, node assignment, IP, and (for non-restart resizes) its running process. Two API fields form the mental model that runs through the entire feature: ```text spec.containers[*].resources = DESIRED (what you want; now mutable) status.containerStatuses[*].resources = ACTUAL (what the container actually has) ``` A resize is **converged** when `DESIRED == ACTUAL`. Everything else — pending, in-progress, deferred, infeasible — is a state where they differ and the kubelet is either working on it, waiting, or has given up. ### 6.2 Why it exists The feature exists to break the coupling between *changing a reservation* and *destroying a process*, for the workloads in Section 4. Design goals, from KEP-1287: - **Non-disruptive right-sizing** for CPU and (where the runtime allows) memory. - A path for the **VPA** to actuate recommendations without eviction. - **Consistency** with the scheduler and quota — the reservation math must remain correct even mid-resize. - **Observability and control** — the operation must report progress, be auditable, and be gate-able by RBAC as a distinct privilege. Crucially, it is designed as a *narrow, safe* mutation surface, not a general "edit any field of a running Pod" mechanism. That narrowness (the dedicated subresource, the QoS invariant, the cpu/memory-only restriction) is the feature's safety model, not an accident of incompleteness. ### 6.3 Internal architecture: the four resource states The kubelet does not have a single "current resources" value. It tracks a resource through four states, propagated in order (KEP-1287 §Resource States): ```text DESIRED -> ALLOCATED -> ACTUATED -> ACTUAL (spec) (admitted, (sent to (real cgroup config checkpointed) the runtime, reported back by checkpointed) the runtime) ``` 1. **Desired** — `spec.containers[i].resources`. Set by you (or a controller like VPA) via the resize subresource. 2. **Allocated** — the kubelet admitted the change (the node has room) and checkpointed it locally. Reported (when the companion `InPlacePodVerticalScalingAllocatedStatus` gate is on) via `status.containerStatuses[i].allocatedResources`. 3. **Actuated** — the config the kubelet actually sent to the runtime via `CreateContainer`/`UpdateContainerResources`. This is **not** in the API; it is checkpointed on the node (kubernetes/kubernetes PR #130599). The kubelet compares *actuated* (not desired) against actual to decide whether to re-issue a resize, because the runtime may legitimately land on a slightly different value (kernel CPU minimums, systemd rounding CPU quota to 10ms, NRI plugins). 4. **Actual** — the real cgroup configuration reported by the runtime via `ContainerStatus`, surfaced as `status.containerStatuses[i].resources`. **Production implication:** when you debug a "stuck" resize, you are really asking *at which arrow did it stop?* Desired but not allocated → node can't fit it (pending/deferred/infeasible). Allocated but not actual → actuation in progress or erroring (e.g. memory-decrease OOM check). This state machine is the debugging map for Section 13. ### 6.4 The `resize` subresource Resources are mutable **only** through the Pod `resize` subresource. You cannot change `resources` on the normal Pod update path — the API server rejects it (this changed from the alpha behavior). The subresource accepts both **Update** (full replace) and **Patch**, but in either case only three fields may change: ```text .spec.containers[*].resources .spec.initContainers[*].resources (restartable init containers / sidecars only) .spec.resizePolicy ``` Any other diff sent through `/resize` is rejected. The canonical command: ```bash kubectl patch pod -n --subresource resize --patch \ '{"spec":{"containers":[{"name":"","resources":{"requests":{"cpu":"800m"},"limits":{"cpu":"800m"}}}]}}' # alternatives: kubectl -n edit pod --subresource resize kubectl -n apply -f --subresource resize --server-side ``` **Client requirement:** `kubectl` **client ≥ v1.32.0** is needed for `--subresource=resize`; older clients report `invalid subresource`. The server must be ≥ v1.33 (we run v1.35). **Production implication:** the subresource is a clean RBAC seam. You can grant a controller `patch` on `pods/resize` without granting it write on `pods` (Section 15). This is how you let VPA resize workloads without giving it broad Pod-write. ### 6.5 `resizePolicy` — per-container, per-resource restart behavior Each container declares, per resource, whether applying a new value requires a container restart: ```yaml resizePolicy: - resourceName: cpu restartPolicy: NotRequired # default; apply in place, no restart - resourceName: memory restartPolicy: RestartContainer # restart the container IN PLACE to apply ``` - **`NotRequired`** (default if unset) — apply the change in place without a restart *if the runtime can*. This is the normal case for CPU. - **`RestartContainer`** — restart the container **in place** to apply the new value. The Pod is **not** recreated; the container gets a fresh start and its `restartCount` increments. This is commonly needed for memory, because many runtimes (JVM `-Xmx`, CPython) cannot change their memory ceiling live. Rules and caveats: - **Precedence:** if one resize changes multiple resources with different policies, **`RestartContainer` wins**. Given `cpu=NotRequired, memory=RestartContainer`, a CPU-only change resizes in place; a memory-only or CPU+memory change restarts the container. - **Interaction with Pod `restartPolicy`:** if the Pod has `restartPolicy: Never`, every container `resizePolicy` entry must be `NotRequired` (validation rejects otherwise) — you cannot ask to restart a container the Pod is not allowed to restart. - **`resizePolicy` is immutable** — you set it once, at Pod creation. - **`NotRequired` does not *guarantee* no restart.** If the runtime discovers a resize truly needs a restart, it returns an error and the kubelet retries; in edge cases a container can end up stopped and not restarted if the system cannot apply the change live. **Production implication:** `resizePolicy` is a design-time decision about your runtime. For JVM/CPython services, set `memory: RestartContainer` deliberately and treat memory resizes as "restart-scale" — plan them like a rolling restart, not a free adjustment. ### 6.6 Memory limit decrease and the best-effort OOM check (GA change) At beta, **decreasing** a memory limit was prohibited. In v1.35 GA it is allowed, but with a safety valve. For a memory resize with `NotRequired`/unspecified policy, the kubelet performs a **best-effort OOM-avoidance check**: it reads current usage from the StatsProvider and, if current usage would exceed the new limit, it **skips** actuating that container's resize and leaves a `PodResizeInProgress` condition with `reason: Error` describing current usage versus the desired limit. This check is **best-effort, not a guarantee**, because of a time-of-check / time-of-use race: usage can climb between the check and the cgroup write. On cgroup v2, if the limit is actually written below live usage, the kernel OOM-kills to fit. **Production implication:** never treat a memory *decrease* as safe just because it was accepted. Decrease conservatively, watch working-set headroom (Section 17), and for runtimes that cannot shrink memory live, use `RestartContainer` so the decrease takes effect on a clean process rather than risking an OOM on the live one. ### 6.7 Status conditions: `PodResizePending` and `PodResizeInProgress` The alpha-era `.status.resize` **string** field (values `Proposed`/`InProgress`/ `Deferred`/`Infeasible`) was **deprecated and did not graduate to beta**. As of v1.33 and in GA, resize status is represented by **two Pod conditions**: - **`type: PodResizePending`** — the spec was resized but the kubelet has not yet allocated it (desired ≠ actuated). `status` is `True` while present. Reason is one of: - **`Deferred`** — feasible in principle (it could fit the node) but not right now; the kubelet **periodically re-evaluates and retries** it. - **`Infeasible`** — cannot be satisfied as-is and is **never re-evaluated**. Current infeasible reasons include: request exceeds node capacity; static Pod; swap-enabled container; Guaranteed Pod with static Memory Manager; Guaranteed Pod with static CPU Manager; Windows Pod. - **`type: PodResizeInProgress`** — the kubelet accepted and allocated the change and is actuating it. Usually short-lived; on an actuation error it carries `reason: Error` and a `message`. Both conditions can be present at once. Alongside these, `observedGeneration` fields (GA `v1.35 [stable]`) let you tell which podspec generation the kubelet has acknowledged: top-level `status.observedGeneration`, plus per-condition `observedGeneration` on the resize conditions. > Note: internal kubelet Go enums are still named `PodResizeStatusDeferred/ > Infeasible/InProgress`, but the **API surface is the two conditions above**, not a > string field. Do not build tooling against `.status.resize`. **Production implication:** your automation and alerts must read *conditions*, not the deprecated string field. `Deferred` means "wait, it may still happen"; `Infeasible` means "it will never happen until you change the request or the node." ### 6.8 Prioritized and deferred resizes (GA change) When a node lacks room for a requested resize, the kubelet does not simply fail it forever — it may mark it `Deferred` and retry. Because many resizes can be pending at once, GA introduced a **priority order** for which deferred resize is retried first (KEP-1287 §Priority of Resize Requests): ```text 0. (implicit) non-increasing requests -> expected to always succeed; not marked pending 1. PriorityClass -> higher priority first 2. QoS class -> Guaranteed > Burstable (BestEffort has no cpu/mem) 3. Deferred duration -> longest-pending first (PodResizePending.lastTransitionTime) ``` A higher-priority pending resize does **not** block the others; the kubelet still attempts all queued resizes. Retries are triggered at the end of the kubelet's pod handling loops (`HandlePodUpdates`/`HandlePodRemoves`/`HandlePodCleanups`), when another resize completes and frees room, and periodically. There is one narrow eviction behavior at GA: if a resize is requested for a `system-node-critical`/`system-cluster-critical` Pod and there is no room, the kubelet may **evict a non-critical Pod** to make room. General preemption for non-critical resizes is **not** in scope for GA. **Production implication:** on busy nodes, resizes queue and compete. Give latency-sensitive workloads a higher `PriorityClass` if their resizes must win, and do not assume a `Deferred` resize will apply promptly under contention. ### 6.9 Common misconceptions - **"It restarts the Pod."** No. It can restart a *container* in place (`RestartContainer`), which increments `restartCount`, but the Pod object, name, UID, node, and IP are preserved. A container restart is not a Pod recreation. - **"CPU never restarts, memory always does."** CPU with `NotRequired` applies with no restart. Memory *often* needs a restart, but that depends on the runtime and your `resizePolicy`; it is not a hard platform rule. - **"I can `kubectl edit` the Deployment to resize a running Pod."** Editing the controller template changes future Pods; the running Pod is only mutated through the `resize` subresource on the **Pod**. (Editing the template *and* the running Pod are both valid, for different reasons — see Section 9.) - **"Accepted means applied."** A resize can be accepted into `Deferred`/pending and never actuate; a memory decrease can be accepted and then skipped by the OOM check. Convergence is `DESIRED == ACTUAL`, not "the patch returned 200." - **"It can grow the Pod onto a bigger node."** No. In-place resize works within a node's capacity. If it does not fit, you get `Infeasible`/`Deferred`; growing the cluster is the cluster autoscaler's job (Section 17). - **"QoS can be tuned by resizing."** No. QoS is invariant across a resize by validation. --- ## 7. Architecture This section traces a single resize request end to end — from `kubectl patch` to a cgroup write — and names the component responsible at each hop. Understanding the control-plane / data-plane split is what lets you reason about failure modes and the scheduler↔kubelet race. ### 7.1 End-to-end request path ```text CONTROL PLANE | DATA PLANE (node) | you / VPA | | kubectl patch pod --subresource resize | v | +-----------------+ validate: only cpu/mem, | | kube-apiserver | QoS unchanged, resize subresource | | /resize |---> persist DESIRED to spec.resources | +-----------------+ (etcd) | | (no scheduling decision here — Pod already placed) | | | | watch | +------------------------------------------------------->+ | +------------------+ | | kubelet | | | 1 admit/allocate | | | (node has room?)| | | -> checkpoint | | | 2 order changes | | | (dec before inc)| | | 3 CRI call | | +---------+--------+ | | UpdateContainerResources (gRPC) | v | +------------------+ | | containerd 2.2.0 | | | write cgroup v2: | | | cpu.weight | | | cpu.max | | | memory.max | | +---------+--------+ | | ContainerStatus | v | kubelet reads ACTUAL, status.containerStatuses[*].resources <--------------------+ writes it back to status, + conditions + events + metrics (status update) | emits events + metrics ``` The scheduler is deliberately **absent from the actuation path**. The Pod is already bound to a node; a resize does not reschedule it. The scheduler participates only *indirectly*: it (and ResourceQuota) account for the resize by using `max(desired, allocated, actual)` when deciding whether *new* Pods fit — so an in-flight upsize is counted against the node immediately, and a `Infeasible` resize is excluded from that max. ### 7.2 Control plane vs data plane responsibilities ```text +---------------------------+-----------------------------------------------+ | Component | Responsibility for resize | +---------------------------+-----------------------------------------------+ | kube-apiserver (/resize) | Validate (cpu/mem only, QoS invariant, field | | | scope); persist DESIRED; enforce RBAC on | | | pods/resize. | | scheduler | NOT in actuation path. Accounts for resizes | | | via max(desired,allocated,actual) for future | | | placement. | | kubelet | Admit/allocate, order changes, call CRI, | | | read back ACTUAL, set conditions/events/ | | | metrics, retry deferred resizes by priority. | | container runtime (CRI) | UpdateContainerResources (idempotent); write | | | cgroup v2 files; report actual via | | | ContainerStatus. | | VPA (optional controller) | Compute recommendation; PATCH the resize | | | subresource in InPlaceOrRecreate mode. | +---------------------------+-----------------------------------------------+ ``` ### 7.3 The `/resize` subresource security model `pods/resize` is a distinct subresource, which means it is a distinct RBAC object. The verbs that matter are `patch` and `update` on `pods/resize`: ```yaml rules: - apiGroups: [""] resources: ["pods/resize"] verbs: ["patch", "update"] - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] # to read desired/actual/status ``` Regular `patch`/`update` on `pods` does **not** allow resource mutation — that door is closed on the main resource. Only the subresource permits it. This is a deliberate privilege-separation seam: you can grant "may resize" without granting "may rewrite the Pod." The Node/kubelet identity is the only writer of `status.allocatedResources`/`status.resources`. (This follows from the KEP subresource definition and standard Kubernetes subresource RBAC semantics.) ### 7.4 Multi-container atomic resize ordering A resize that touches several containers/resources is **atomic** (all-or-nothing). To ensure the sum of container limits never transiently exceeds the pod-level cgroup limit, the kubelet applies changes in a fixed order (KEP-1287 §Flow Control): ```text 1. If any resource NET-INCREASES -> raise the POD-level cgroup limit first. 2. Apply all container limit DECREASES. 3. If net-decrease -> then lower the POD-level cgroup limit. 4. Apply all container limit INCREASES. ``` Decreases before increases, pod-cgroup widened before children grow and narrowed after children shrink. This ordering is invisible in normal operation but explains why a partially-failing multi-container resize is reported as a single failed unit rather than leaving containers in mismatched states. ### 7.5 Actuation caveats The **actual** cgroup values can legitimately differ from **desired** because of: - kernel-enforced CPU minimums, - the systemd cgroup driver rounding CPU quota up to the nearest 10ms, - NRI plugins mutating the container config, - a best-effort `UpdatePodSandboxResources` CRI call the kubelet issues *after* reconfiguring pod-level cgroups (an NRI-plugin notification). This is exactly why the kubelet compares **actuated** against actual (not desired against actual) to decide whether to re-issue, and why it checkpoints actuated resources so a kubelet restart does not lose track of an in-flight resize. --- ## 8. Installation (IaC-First) Everything here is declarative and version-pinned. The goal is a cluster you can recreate byte-for-byte: a **kind** cluster on Kubernetes v1.35, plus metrics-server, VPA, kube-prometheus-stack, and an OpenTelemetry Collector, all installed via Helm with pinned chart and image versions. A `Taskfile.yml` drives the whole lifecycle. > **Environment note.** The Taskfile uses bash and is validated on macOS/Linux; on > Windows, run it inside **WSL2** (with Docker Desktop's WSL2 backend or a native > container runtime, cgroup v2). Note that Kubernetes on Windows *nodes* does not > support in-place resize — that is unrelated to using a Windows *workstation* to > drive a Linux cluster. ### 8.1 Pinned versions (`iac/helm/versions.env`) A single source of truth for every version. The Taskfile sources this file, so a version bump is a one-line change with a clear diff. ```bash # versions.env — single source of truth for all pinned versions. # Shell KEY=value format; sourced by the Taskfile (`set -a; . versions.env; set +a`). # # Helm repositories (add these before installing — see `task helm-repos`): # metrics-server https://kubernetes-sigs.github.io/metrics-server/ # fairwinds-stable https://charts.fairwinds.com/stable # prometheus-community https://prometheus-community.github.io/helm-charts # open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts # --- Tooling --- KIND_VERSION=v0.32.0 KIND_NODE_IMAGE=kindest/node:v1.35.0 HELM_VERSION=v3.19.0 KUBERNETES_VERSION=v1.35.0 # --- metrics-server (repo: metrics-server) --- METRICS_SERVER_CHART=3.13.1 METRICS_SERVER_APP=v0.8.1 # --- Vertical Pod Autoscaler (repo: fairwinds-stable, path stable/vpa) --- VPA_CHART=4.12.3 VPA_APP=1.6.0 # --- kube-prometheus-stack (repo: prometheus-community) --- KUBE_PROMETHEUS_STACK_CHART=87.15.1 # --- OpenTelemetry Collector (repo: open-telemetry) --- OTEL_COLLECTOR_CHART=0.165.0 ``` > **Pin, then verify.** kube-prometheus-stack releases several times a week and the > OTel chart moves fast, so treat these pins as a snapshot: run > `helm search repo --versions | head` before installing and re-pin if a > version no longer resolves. Do **not** float these to `latest` in production. ### 8.2 Install the tooling The Taskfile prints exact, pinned install instructions for `kind`, `helm`, and `kubectl`: ```bash # macOS (Homebrew) brew install kind helm kubectl # verify: kind --version, helm version --short # Linux (amd64) curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.32.0/kind-linux-amd64 chmod +x ./kind && sudo mv ./kind /usr/local/bin/kind curl -fsSL https://get.helm.sh/helm-v3.19.0-linux-amd64.tar.gz | tar xz sudo mv linux-amd64/helm /usr/local/bin/helm curl -LO "https://dl.k8s.io/release/v1.35.0/bin/linux/amd64/kubectl" chmod +x kubectl && sudo mv kubectl /usr/local/bin/kubectl # Windows: use WSL2 (Ubuntu) + Docker Desktop WSL2 backend, then follow the Linux steps. ``` `kubectl` client must be **≥ v1.32.0** for `--subresource=resize`. A skew of ±1 minor against the v1.35 server is supported. ### 8.3 The kind cluster (`iac/kind/kind-cluster.yaml`) InPlacePodVerticalScaling is GA and on by default in v1.35, so **there is no feature gate to enable** — the node image pin is the only thing that matters. ```yaml # kind cluster for the In-Place Pod Vertical Scaling tutorial (Kubernetes v1.35). # # InPlacePodVerticalScaling is GA / on-by-default in Kubernetes v1.35 — NO feature gate # is required. (alpha v1.27 -> beta v1.33 -> GA/stable v1.35.) The Pod `resize` # subresource and mutable spec.containers[*].resources work out of the box. # # Create with: # kind create cluster --config kind/kind-cluster.yaml # # Tested on: kind v0.32.0, Server Version v1.35.0, containerd://2.2.0. --- apiVersion: kind.x-k8s.io/v1alpha4 kind: Cluster name: inplace-resize nodes: - role: control-plane # All nodes pinned to the Kubernetes v1.35.0 node image. image: kindest/node:v1.35.0 # PRODUCTION: pin by immutable digest instead of a floating tag, e.g.: # image: kindest/node:v1.35.0@sha256: # The kind v0.32.0 release notes publish the exact sha256 digest for each # node image; using the digest guarantees byte-for-byte reproducibility. - role: worker image: kindest/node:v1.35.0 - role: worker image: kindest/node:v1.35.0 ``` Create it and confirm the version: ```bash task up # -> kind create cluster --config kind/kind-cluster.yaml # -> kubectl --context kind-inplace-resize get nodes -o wide ``` **Example output (kind, Kubernetes v1.35):** ```text $ kubectl version Client Version: v1.34.1 Server Version: v1.35.0 $ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION ... CONTAINER-RUNTIME inplace-resize-verify-control-plane Ready control-plane 32s v1.35.0 containerd://2.2.0 ``` > Node names in the sample output above come from a single-node cluster named > `inplace-resize-verify`; the committed `kind-cluster.yaml` names the cluster > `inplace-resize` and adds two workers. The resize mechanics are identical — only the > node name in the output differs. ### 8.4 metrics-server (`iac/helm/metrics-server.values.yaml`) metrics-server is the Metrics API backend and a **hard dependency of VPA's recommender**. On kind, the kubelet serving cert is self-signed, so we skip its verification — a demo-only shortcut that is explicitly *not* for production. ```yaml # metrics-server Helm values — VERIFIED WORKING on kind v1.35.0. # Chart: metrics-server/metrics-server 3.13.1 (app v0.8.1) # helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/ # # WHY --kubelet-insecure-tls: # On kind, each node's kubelet serves its Summary API over HTTPS (:10250) using a # SELF-SIGNED serving certificate whose SANs are not verifiable by metrics-server's # default CA bundle. Without this flag metrics-server logs # "x509: cannot validate certificate ... because it doesn't contain any IP SANs" # and `kubectl top` never returns data. --kubelet-insecure-tls tells metrics-server # to skip verification of the kubelet's serving cert. # # PRODUCTION: DO NOT use --kubelet-insecure-tls. Instead provision kubelet serving # certificates signed by the cluster CA (enable kubelet server-cert rotation / # RotateKubeletServerCertificate + the CSR approver) so metrics-server can verify # them. Skipping TLS verification exposes the metrics path to MITM. --- args: - --kubelet-insecure-tls ``` ### 8.5 Vertical Pod Autoscaler (`iac/vpa/vpa.values.yaml`) We install the Fairwinds `stable/vpa` chart, which deploys all three VPA components (recommender, updater, admission-controller) and auto-generates the webhook's self-signed TLS cert. The one thing we configure is the in-place feature gate on the updater and admission-controller. ```yaml # ============================================================================= # VPA Helm values — Vertical Pod Autoscaler with in-place resize. # Chart: fairwinds-stable/vpa 4.12.3 (VPA app v1.6.0) # helm repo add fairwinds-stable https://charts.fairwinds.com/stable # # WHAT THIS ENABLES # The VPA `InPlaceOrRecreate` update mode lets the vpa-updater apply new # requests/limits to a RUNNING pod via the Kubernetes Pod `resize` subresource # (KEP sig-node/1287) instead of evicting/recreating it — falling back to # eviction only when the in-place update is infeasible. # # FEATURE GATE + VERSION NUANCE # - Cluster side: `InPlacePodVerticalScaling` is GA / on-by-default in Kubernetes # v1.35 — no cluster feature gate needed. # - VPA side: the `InPlaceOrRecreate` gate maturity ladder is: # VPA v1.4.0 alpha (gate REQUIRED) # VPA v1.5.0 beta (gate enabled by default) # VPA v1.6.0 GA (this install) # VPA v1.7.0 gate REMOVED (always on) # - Ground truth / the k8s v1.35 blog (AEP-4016) describe the mode as BETA for the # 1.35 era. We install VPA app v1.6.0 where it is GA but the gate flag still # EXISTS, so passing `--feature-gates=InPlaceOrRecreate=true` is redundant-but- # harmless (idempotent, documents intent). VERIFIED WORKING live. # - WARNING: on VPA >= v1.7.0 this flag is REMOVED and passing it makes the # component error on an unknown flag. Keep VPA_APP pinned (see versions.env). # ============================================================================= --- recommender: enabled: true updater: enabled: true extraArgs: # Enable the in-place resize path in the updater. Redundant on v1.6.0 (GA), # required on v1.4.0, removed on v1.7.0. feature-gates: InPlaceOrRecreate=true admissionController: enabled: true # The admission controller is a mutating webhook; the Fairwinds chart auto-generates # a self-signed serving cert/secret so it works out of the box. extraArgs: feature-gates: InPlaceOrRecreate=true ``` > **Version framing (beta vs GA).** The Kubernetes v1.35 GA blog > describes VPA `InPlaceOrRecreate` as **beta (AEP-4016)**. The current VPA docs show > the mode reached **GA in VPA v1.6.0** — the app version we installed. Both are true > at different layers: the *cluster* feature (in-place resize) is GA in k8s v1.35; the > *VPA mode* that consumes it is beta in the 1.35-era narrative and GA in VPA v1.6.0. > On v1.6.0 the gate flag is default-on but still present, so setting it is a > harmless no-op. On VPA ≥ v1.7.0 the flag is removed — pin the VPA app version. ### 8.6 kube-prometheus-stack (`iac/helm/kube-prometheus-stack.values.yaml`) This scrapes the kubelet's `/metrics` (where the resize metrics live), cAdvisor, and the resource endpoint, discovers our own ServiceMonitors, accepts remote-write from the OTel Collector, and imports our dashboard via the Grafana sidecar. ```yaml # kube-prometheus-stack Helm values — In-Place Pod Resize observability. # Chart: prometheus-community/kube-prometheus-stack 87.15.1 # (Prometheus Operator v0.92.1; bundled grafana 12.7.2, kube-state-metrics 7.8.1, # prometheus-node-exporter 4.56.0). kubeVersion constraint >=1.25.0-0 -> OK on v1.35. # helm repo add prometheus-community https://prometheus-community.github.io/helm-charts # # Goal: scrape the kubelet resize metrics (kubelet_pod_resize_*, # kubelet_container_requested_resizes_total) from /metrics, plus cAdvisor + resource # endpoints, and auto-discover the demo workload's own ServiceMonitor. # --- kubelet scraping: /metrics (resize metrics) + cAdvisor + resource --- kubelet: enabled: true namespace: kube-system serviceMonitor: # scrape all three kubelet endpoints over HTTPS :10250 cAdvisor: true # /metrics/cadvisor -> container_* metrics resource: true # /metrics/resource -> node/pod resource metrics probes: true # kubelet /metrics (includes kubelet_pod_resize_*, kubelet_container_requested_resizes_total) # is scraped by default when kubelet.enabled=true. On kind the kubelet serving cert is # self-signed; the chart's kubelet ServiceMonitor uses insecureSkipVerify=true by default, # so this works out of the box. interval: "15s" # --- Prometheus: discover OUR ServiceMonitors/PodMonitors cluster-wide --- prometheus: prometheusSpec: # Do NOT restrict discovery to only chart-labelled monitors; pick up ours too. serviceMonitorSelectorNilUsesHelmValues: false podMonitorSelectorNilUsesHelmValues: false ruleSelectorNilUsesHelmValues: false # Accept remote-write from the OpenTelemetry Collector (prometheusremotewrite exporter). enableRemoteWriteReceiver: true # keep enough history for a demo/tutorial retention: 6h scrapeInterval: "15s" resources: requests: cpu: 100m memory: 400Mi # kube-state-metrics is required for kube_pod_container_resource_requests/_limits # (desired vs actual dashboards). kubeStateMetrics: enabled: true # Grafana: import our own dashboard via the sidecar (ConfigMaps labelled grafana_dashboard=1). grafana: enabled: true # DEMO ONLY: plaintext admin password. In production DO NOT set this here — reference an # existing Secret instead, e.g.: # admin: # existingSecret: grafana-admin # userKey: admin-user # passwordKey: admin-password adminPassword: "prom-operator" # <-- demo credential; rotate / use a Secret in prod defaultDashboardsEnabled: true sidecar: dashboards: enabled: true # ConfigMaps carrying this label are auto-imported as dashboards # (see `task dashboards`, which creates the grafana-inplace-resize ConfigMap). label: grafana_dashboard searchNamespace: ALL ``` ### 8.7 OpenTelemetry Collector (`iac/helm/opentelemetry-collector.values.yaml`) A DaemonSet-mode Collector that scrapes the kubelet's resize metrics via the `prometheus` receiver, actual usage via `kubeletstats`, absolute requests/limits via `k8s_cluster`, and remote-writes everything to Prometheus. ```yaml # OpenTelemetry Collector Helm values — scrape resize metrics + pod resources -> Prometheus. # Chart: open-telemetry/opentelemetry-collector 0.165.0 (app Collector 0.156.0) # helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts # # Receiver responsibilities: # - prometheus -> scrape the local node's kubelet /metrics for the RESIZE metrics # (kubelet_pod_resize_*, kubelet_container_requested_resizes_total). # - kubeletstats -> actual usage + utilization ratios per container/pod/node. # - k8s_cluster -> absolute desired requests/limits (k8s.container.cpu_request/limit, # k8s.container.memory_request/limit). NOTE: the absolute limit values # come from k8s_cluster, NOT kubeletstats (kubeletstats exposes the # *_utilization ratio variants). # # Requires kube-prometheus-stack with prometheusSpec.enableRemoteWriteReceiver: true # (set in kube-prometheus-stack.values.yaml) so the prometheusremotewrite exporter can push. mode: daemonset # kubeletstats + prometheus(kubelet) need a per-node scrape image: # REQUIRED on recent chart majors: set the repo explicitly. Use the -contrib distro # because we need the kubeletstats, k8s_cluster and prometheus receivers + # prometheusremotewrite exporter. repository: "otel/opentelemetry-collector-contrib" # tag defaults to the chart appVersion (0.156.0); pin explicitly for reproducibility: # tag: "0.156.0" # Convenience presets that wire common Kubernetes plumbing (RBAC, mounts, env). presets: kubeletMetrics: enabled: true kubernetesAttributes: enabled: true config: receivers: # (1) kubelet resize metrics via a Prometheus scrape of the local node's kubelet /metrics prometheus: config: scrape_configs: - job_name: kubelet-resize scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node relabel_configs: - source_labels: [__meta_kubernetes_node_name] target_label: node # scrape only the local node's kubelet - replacement: "${env:K8S_NODE_NAME}:10250" target_label: __address__ metric_relabel_configs: # keep only the resize-related kubelet metrics - source_labels: [__name__] regex: "kubelet_(pod_resize_duration_seconds.*|pod_in_progress_resizes|pod_infeasible_resizes_total|pod_pending_resizes|pod_deferred_resize_accepted_total|container_requested_resizes_total)" action: keep # (2) actual pod/container resource usage + utilization ratios kubeletstats: collection_interval: 15s auth_type: serviceAccount endpoint: "https://${env:K8S_NODE_NAME}:10250" insecure_skip_verify: true metric_groups: [container, pod, node] metrics: k8s.container.cpu_limit_utilization: enabled: true k8s.container.memory_limit_utilization: enabled: true k8s.container.cpu_request_utilization: enabled: true k8s.container.memory_request_utilization: enabled: true # (3) desired (absolute) requests/limits from the cluster API. # NOTE on DaemonSet mode: k8s_cluster queries the API server and should run as a SINGLE # instance to avoid duplicate series. Fine on a single-node kind demo; on multi-node # clusters split k8s_cluster into a separate deployment-mode collector. k8s_cluster: auth_type: serviceAccount collection_interval: 30s # emits k8s.container.cpu_request/limit, k8s.container.memory_request/limit processors: batch: {} exporters: debug: verbosity: normal prometheusremotewrite: # push to the kube-prometheus-stack Prometheus remote-write receiver endpoint: "http://kube-prometheus-stack-prometheus.monitoring.svc:9090/api/v1/write" tls: insecure: true service: pipelines: metrics: receivers: [prometheus, kubeletstats, k8s_cluster] processors: [batch] exporters: [debug, prometheusremotewrite] extraEnvs: - name: K8S_NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName ``` ### 8.8 One-command lifecycle (`iac/Taskfile.yml`) The Taskfile ties it together. `task up` creates the cluster; `task install` installs all four stacks in dependency order; `task demo`/`resize-*`/`vpa-demo` run the tutorial; `task down` tears it down. ```yaml version: '3' dotenv: - helm/versions.env vars: CLUSTER_NAME: inplace-resize DEMO_NS: default DEMO_POD: resize-demo tasks: up: desc: Create the pinned kind v1.35.0 cluster (InPlacePodVerticalScaling is GA/on-by-default) cmds: - kind create cluster --config kind/kind-cluster.yaml - kubectl --context kind-{{.CLUSTER_NAME}} get nodes -o wide helm-repos: desc: Add and update the required Helm repositories cmds: - helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/ - helm repo add fairwinds-stable https://charts.fairwinds.com/stable - helm repo add prometheus-community https://prometheus-community.github.io/helm-charts - helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts - helm repo update install: desc: Install metrics-server + VPA + kube-prometheus-stack + OTel Collector (all pinned) deps: [helm-repos] cmds: - >- helm upgrade --install metrics-server metrics-server/metrics-server --version {{.METRICS_SERVER_CHART}} -n kube-system -f helm/metrics-server.values.yaml --wait - >- helm upgrade --install vpa fairwinds-stable/vpa --version {{.VPA_CHART}} -n vpa --create-namespace -f vpa/vpa.values.yaml --wait - >- helm upgrade --install kube-prometheus-stack prometheus-community/kube-prometheus-stack --version {{.KUBE_PROMETHEUS_STACK_CHART}} -n monitoring --create-namespace -f helm/kube-prometheus-stack.values.yaml --wait - >- helm upgrade --install otel-collector open-telemetry/opentelemetry-collector --version {{.OTEL_COLLECTOR_CHART}} -n observability --create-namespace -f helm/opentelemetry-collector.values.yaml --wait ``` *(The full Taskfile includes `tools`, `demo`, `resize-cpu`, `resize-mem`, `resize-infeasible`, `vpa-demo`, `verify`, `observe`, `dashboards`, and `down` targets; the committed file is the authoritative copy.)* ### 8.9 Verification ```bash task helm-repos && task install kubectl -n vpa get pods # recommender, updater, admission-controller Running kubectl -n monitoring get pods # prometheus, grafana, kube-state-metrics, node-exporter kubectl -n observability get ds # otel-collector DaemonSet kubectl top nodes # metrics-server serving the Metrics API ``` **Example output (kind, Kubernetes v1.35)** — VPA install result: ```text All 3 pods Running: vpa-recommender, vpa-updater, vpa-admission-controller. Verified updater/admission-controller args include --feature-gates=InPlaceOrRecreate=true. VPA CRD updateMode enum: ["Off","Initial","Recreate","InPlaceOrRecreate","Auto"]. ``` ### 8.10 Common installation mistakes - **Floating chart versions.** kube-prometheus-stack ships multiple times a week; an unpinned install is not reproducible. Pin in `versions.env` and re-verify. - **Forgetting `--kubelet-insecure-tls` on kind.** Without it, metrics-server logs an x509 SAN error and `kubectl top` never returns data — which then makes VPA's recommender produce nothing. - **Installing VPA ≥ v1.7.0 with the feature-gate flag.** The `InPlaceOrRecreate` gate is removed in v1.7.0; passing it makes the component crash on an unknown flag. Pin `VPA_APP`. - **Restricting ServiceMonitor discovery.** If you leave `serviceMonitorSelectorNilUsesHelmValues` at its default `true`, Prometheus only discovers chart-labelled monitors and silently ignores your workload's ServiceMonitor. - **OTel chart with no explicit image repo/mode.** Recent chart majors require you to set `image.repository` and a `mode`; the `-contrib` image is required for the `kubeletstats`/`k8s_cluster` receivers. --- ## 9. Configuration Configuration for in-place resize splits into two layers: **workload-side** (`resizePolicy` and how you shape resources so resizes stay legal) and **controller-side** (VPA `updateMode`, QoS interactions, PriorityClass). This section shows each with the *why* attached. ### 9.1 `resizePolicy` on the workload The demo workload sets a per-resource policy: CPU applies in place, memory restarts the container in place. This is the committed `iac/workloads/demo-app.yaml` (Deployment form shown; the file also includes a bare-Pod form that matches the verified capture 1:1): ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: resize-demo namespace: default labels: app: inplace-demo spec: replicas: 1 selector: matchLabels: app: inplace-demo template: metadata: labels: app: inplace-demo spec: containers: - name: app image: registry.k8s.io/pause:3.10 resizePolicy: - resourceName: cpu restartPolicy: NotRequired # CPU: apply in place, no restart - resourceName: memory restartPolicy: RestartContainer # memory: restart container in place resources: requests: cpu: 100m memory: 128Mi limits: cpu: 200m memory: 256Mi ``` **Why this shape.** `pause:3.10` is a tiny, always-present sandbox image, ideal for demonstrating resize mechanics without workload noise. The `resizePolicy` encodes a runtime reality: CPU limits can be re-applied to a live process (CFS quota is just a number the kernel enforces), but many runtimes cannot change their memory ceiling without restarting. Declaring `memory: RestartContainer` makes that behavior explicit and predictable rather than leaving it to a runtime error-and-retry. **Deployment caveat (important).** The resize subresource operates on **Pods**, not Deployments. Patching the Pod changes the *running* Pod; the Deployment template is unchanged, so any Pod recreation (rollout, node drain) reverts to the template values. For a persistent change you must **also** update the template — or let VPA manage it (Section 9.3). ### 9.2 Keeping resizes QoS-legal Because QoS is invariant across a resize, the safe patterns are: ```yaml # Guaranteed: keep requests == limits for BOTH cpu and memory. Raise together. resources: requests: { cpu: "1", memory: 1Gi } limits: { cpu: "1", memory: 1Gi } # -> a legal resize sets requests AND limits to the same new value, e.g. 2/2Gi. # Burstable: keep a gap; do NOT let requests == limits for both resources. resources: requests: { cpu: 200m, memory: 256Mi } limits: { cpu: "1", memory: 1Gi } # -> legal to raise requests to 500m or limits to 2Gi, as long as it stays Burstable. ``` If a patch would flip the class (e.g. a Burstable Pod where you set `requests == limits` for both cpu and memory), the API server rejects it. Design your resize automation to preserve the shape, not just the numbers. ### 9.3 VPA `updateMode` — the five values The VPA object drives resizes automatically. The `updateMode` enum, as defined in the VPA CRD, is `["Off","Initial","Recreate","InPlaceOrRecreate","Auto"]`: ```text Off -> compute recommendations only; apply nothing (observation mode). Initial -> set requests only at Pod creation; never touch a running Pod. Recreate (default) -> set at creation AND update running Pods by EVICTING them. Auto (deprecated) -> currently == Recreate; use explicit modes instead. InPlaceOrRecreate -> set at creation AND update running Pods IN PLACE via the resize subresource; fall back to eviction/recreate only if in-place fails. ``` The committed `iac/workloads/vpa-inplace.yaml` uses `InPlaceOrRecreate`: ```yaml apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: vpa-demo namespace: default spec: targetRef: apiVersion: apps/v1 kind: Deployment name: vpa-demo updatePolicy: updateMode: InPlaceOrRecreate # try in-place first; recreate only if impossible minReplicas: 1 # keep at least one Pod available during actuation resourcePolicy: containerPolicies: - containerName: burn controlledResources: ["cpu", "memory"] minAllowed: cpu: 50m memory: 32Mi maxAllowed: # uncapped target 1168m is capped here to 400m cpu: 400m memory: 256Mi ``` **Why `InPlaceOrRecreate` and not `InPlace`.** A pure `InPlace` mode (never evict) exists but is **alpha in VPA v1.7.0** (AEP-8818) — beyond our v1.6.0/1.35 baseline. `InPlaceOrRecreate` is the pragmatic choice: it prefers the non-disruptive path but still makes progress when a resize is infeasible, by falling back to recreate. **`controlledValues` (limits handling).** Default `RequestsAndLimits` scales limits proportionally, preserving the original limit:request ratio (this is why, in the verified run, capping CPU request to 400m produced a CPU limit of 4 — the original 10× ratio). Use `RequestsOnly` to leave limits untouched. ### 9.4 VPA fallback and disruption behavior `InPlaceOrRecreate` falls back to eviction/recreate when: - the in-place update is **infeasible** (node lacks resources), - the update is **deferred > 5 minutes**, - the update is **in progress > 1 hour**, - the update would **change the Pod's QoS class**, - a memory-limit **downscale** is required for a container whose memory resize policy does not permit an in-place decrease. By default VPA respects **PodDisruptionBudgets** and its own disruption thresholds (`minReplicas`, max-eviction ratio) even for in-place updates. The updater flag `--in-place-skip-disruption-budget` (default `false`) lets VPA skip PDB checks *only* when the update is genuinely non-disruptive — i.e. every container uses `NotRequired` for both cpu and memory. Even then, PDB is still enforced if any container uses `RestartContainer` or if the update falls back to eviction. ### 9.5 PriorityClass and resize ordering On a contended node, deferred resizes are retried in the order **PriorityClass → QoS → deferred duration** (Section 6.8). If a workload's resizes must win under contention, give it a higher `PriorityClass`: ```yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: latency-critical value: 1000000 globalDefault: false description: "Workloads whose in-place resizes should be retried first under contention." ``` Then reference it in the Pod template (`spec.priorityClassName: latency-critical`). This does not guarantee immediate actuation — it only orders the retry queue — but it is the lever you have for prioritizing whose upsize applies first when a node is tight. ### 9.6 HPA boundary (configuration you must not create) Do **not** run HPA and VPA on the **same resource metric**. HPA on CPU scales *replica count* based on utilization relative to requests; VPA changes the requests themselves; together on the same metric they oscillate. Supported combinations: ```text OK: HPA on custom/external metrics (RPS, queue depth) + VPA on cpu/memory OK: HPA on CPU only + VPA on memory only (controlledResources: ["memory"]) BAD: HPA on CPU + VPA on CPU (feedback loop) ``` --- ## 10. Hands-On Tutorial This is the reproducible core of the article. Every step maps to a `task` target and a committed file. Command output is labeled **Example output** (illustrative of a Kubernetes v1.35 kind cluster) or, for the Prometheus/Grafana/OpenTelemetry steps, **Expected output** that you reproduce with the given commands. **Environment for the example output:** a `kind` cluster (v0.32.0) on node image `kindest/node:v1.35.0` — Server Version v1.35.0, containerd 2.2.0, cgroup v2. Your own node names, pod ages, and timings will differ. ### 10.0 Prerequisites ```bash cd article-1-inplace-resize/iac task up # kind cluster on Kubernetes v1.35.0 task install # metrics-server + VPA + kube-prometheus-stack + OTel Collector ``` ### 10.1 Deploy the demo workload and confirm the baseline ```bash task demo # applies workloads/demo-app.yaml (resizePolicy cpu=NotRequired, memory=RestartContainer) ``` The baseline invariant is `DESIRED == ACTUAL` with `restartCount 0`. Read both: ```bash kubectl get pod resize-demo -o jsonpath='{.spec.containers[0].resources}' # DESIRED kubectl get pod resize-demo -o jsonpath='{.status.containerStatuses[0].resources}' # ACTUAL kubectl get pod resize-demo -o jsonpath='{.status.containerStatuses[0].restartCount}' ``` **Example output (kind, Kubernetes v1.35):** ```text # .spec.containers[0].resources (DESIRED) {"limits":{"cpu":"200m","memory":"256Mi"},"requests":{"cpu":"100m","memory":"128Mi"}} # .status.containerStatuses[0].resources (ACTUAL) {"limits":{"cpu":"200m","memory":"256Mi"},"requests":{"cpu":"100m","memory":"128Mi"}} # restartCount 0 ``` Desired equals actual; the resize machinery is idle. This is the reference state we return to conceptually after each step. ### 10.2 CPU resize — `NotRequired`, no restart Bump CPU only: `requests.cpu 100m→500m`, `limits.cpu 200m→1`. Because `cpu: NotRequired`, the kubelet applies new CPU cgroup values with **no container restart**. The committed patch is `workloads/resize-cpu.patch.yaml`: ```yaml spec: containers: - name: app resources: requests: cpu: 500m limits: cpu: "1" ``` ```bash task resize-cpu # kubectl -n default patch pod resize-demo --subresource resize \ # --patch-file workloads/resize-cpu.patch.yaml ``` **Example output (kind, Kubernetes v1.35):** ```text $ kubectl patch pod resize-demo --subresource resize \ --patch '{"spec":{"containers":[{"name":"app","resources":{"requests":{"cpu":"500m"},"limits":{"cpu":"1"}}}]}}' pod/resize-demo patched # DESIRED {"limits":{"cpu":"1","memory":"256Mi"},"requests":{"cpu":"500m","memory":"128Mi"}} # ACTUAL (converged) {"limits":{"cpu":"1","memory":"256Mi"},"requests":{"cpu":"500m","memory":"128Mi"}} # restartCount (unchanged) 0 ``` Desired and actual converged and `restartCount` stayed `0` — the process never stopped. This is the headline capability: raising CPU on a live container with zero disruption. ### 10.3 Memory resize — `RestartContainer`, restarts the container in place Bump memory only: `requests.memory 128Mi→256Mi`, `limits.memory 256Mi→512Mi`. Because `memory: RestartContainer`, the kubelet restarts the container **in place** — the Pod is not recreated, but `restartCount` increments. The committed patch is `workloads/resize-mem-restart.patch.yaml`: ```yaml spec: containers: - name: app resources: requests: memory: 256Mi limits: memory: 512Mi ``` ```bash task resize-mem ``` **Example output (kind, Kubernetes v1.35):** ```text $ kubectl patch pod resize-demo --subresource resize \ --patch '{"spec":{"containers":[{"name":"app","resources":{"requests":{"memory":"256Mi"},"limits":{"memory":"512Mi"}}}]}}' pod/resize-demo patched # ACTUAL (converged) {"limits":{"cpu":"1","memory":"512Mi"},"requests":{"cpu":"500m","memory":"256Mi"}} # restartCount 1 ``` Note that CPU stayed at the values from Step 10.2 (`500m`/`1`) — resizes accumulate on the running Pod. `restartCount` went to `1`: the container restarted in place to adopt the new memory ceiling. The Pod name, UID, node, and IP are unchanged. ### 10.4 The events tell the whole story ```bash kubectl get events --field-selector involvedObject.name=resize-demo --sort-by='.lastTimestamp' # or: kubectl describe pod resize-demo ``` **Example output (kind, Kubernetes v1.35):** ```text LAST SEEN TYPE REASON OBJECT MESSAGE 30s Normal Scheduled pod/resize-demo Successfully assigned default/resize-demo to inplace-resize-verify-control-plane 16s Normal ResizeStarted pod/resize-demo Pod resize started: {...generation:2} 16s Normal ResizeCompleted pod/resize-demo Pod resize completed: {...generation:2} 5s Normal Pulled pod/resize-demo Container image "registry.k8s.io/pause:3.10" already present on machine ... 5s Normal Created pod/resize-demo Container created 5s Normal Started pod/resize-demo Container started 5s Normal ResizeStarted pod/resize-demo Pod resize started: {...generation:3} 5s Normal Killing pod/resize-demo Container app resize requires restart 4s Normal ResizeCompleted pod/resize-demo Pod resize completed: {...generation:3} ``` Read this as two resizes: - **Generation 2 (the CPU resize):** `ResizeStarted → ResizeCompleted`, **no** `Killing` event, no restart. - **Generation 3 (the memory resize):** `ResizeStarted → Killing ("Container app resize requires restart") → Created → Started → ResizeCompleted`. The `Killing` event is the observable signal of an in-place container restart. > **Event reasons.** The reasons above (`ResizeStarted`, > `ResizeCompleted`, `Killing`) are what kubelet v1.35.0 emits. A > static source search found the constants `ResizeDeferred` and `ResizeInfeasible` > (emitted on the deferred/infeasible paths) but did **not** find `ResizeStarted`/ > `ResizeCompleted` as constants. Live evidence governs what you will observe; treat > `ResizeDeferred`/`ResizeInfeasible` as the reasons for the pending paths. `Killing` > is a generic container-lifecycle event, not a resize-specific reason — but it is > the reliable signal that a `RestartContainer` resize is restarting the container. ### 10.5 Verify convergence with the helper script ```bash task verify # runs scripts/verify-resize.sh ``` `scripts/verify-resize.sh` reads DESIRED and ACTUAL via jsonpath, asserts they are equal, prints `restartCount`, and lists any `PodResizePending`/`PodResizeInProgress` conditions. It exits non-zero if not converged — suitable for CI gating. Core logic: ```bash DESIRED="$(kubectl -n "$NAMESPACE" get pod "$POD" -o jsonpath='{.spec.containers[0].resources}')" ACTUAL="$(kubectl -n "$NAMESPACE" get pod "$POD" -o jsonpath='{.status.containerStatuses[0].resources}')" # ... prints restartCount and resize conditions ... if [ "$DESIRED" = "$ACTUAL" ]; then echo "==> RESULT: CONVERGED (DESIRED == ACTUAL). Resize complete."; exit 0 else echo "==> RESULT: NOT CONVERGED ..." >&2; exit 1 fi ``` ### 10.6 Infeasible resize — `PodResizePending` / reason `Infeasible` Request an impossible amount of CPU (1000 cores on a 12-core node). The kubelet cannot actuate it, so it does **not** change ACTUAL and instead sets a status condition. The committed patch is `workloads/resize-infeasible.patch.yaml`: ```yaml spec: containers: - name: app resources: requests: cpu: "1000" limits: cpu: "1000" ``` ```bash task resize-infeasible kubectl get pod resize-demo -o jsonpath='{.status.conditions}' | jq . ``` **Example output (kind, Kubernetes v1.35):** ```text $ kubectl patch pod resize-demo --subresource resize \ --patch '{"spec":{"containers":[{"name":"app","resources":{"requests":{"cpu":"1000"},"limits":{"cpu":"1000"}}}]}}' pod/resize-demo patched # .status.conditions[] entry: { "type": "PodResizePending", "status": "True", "reason": "Infeasible", "message": "Node didn't have enough capacity: cpu, requested: 1000000, capacity: 12000", "observedGeneration": 4 } # ACTUAL resources UNCHANGED (resize not actuated): {"limits":{"cpu":"1","memory":"512Mi"},"requests":{"cpu":"500m","memory":"256Mi"}} ``` The condition type is `PodResizePending` with `reason: Infeasible`. `Infeasible` is never re-evaluated — you must patch a feasible value. (`Deferred` would appear instead if the node *could* fit it later.) ACTUAL is untouched, so the workload keeps running at its previous, converged size. Revert by re-applying `resize-cpu.patch.yaml`. ### 10.7 Kubelet resize metrics Scrape the kubelet's `/metrics` through the API server node proxy: ```bash task observe # runs scripts/observe.sh: events + kubelet metrics # equivalently: NODE=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') kubectl get --raw "/api/v1/nodes/${NODE}/proxy/metrics" \ | grep -E 'kubelet_(container_requested_resizes|pod_in_progress_resizes|pod_infeasible_resizes|pod_resize_duration)' ``` **Example output (kind, Kubernetes v1.35):** ```text # HELP kubelet_container_requested_resizes_total [ALPHA] Number of requested resizes, counted at the container level. # TYPE kubelet_container_requested_resizes_total counter kubelet_container_requested_resizes_total{operation="decrease",requirement="limits",resource="cpu"} 1 kubelet_container_requested_resizes_total{operation="increase",requirement="limits",resource="cpu"} 2 kubelet_container_requested_resizes_total{operation="increase",requirement="limits",resource="memory"} 1 kubelet_container_requested_resizes_total{operation="increase",requirement="requests",resource="cpu"} 2 kubelet_container_requested_resizes_total{operation="increase",requirement="requests",resource="memory"} 1 # HELP kubelet_pod_in_progress_resizes [ALPHA] Number of in-progress resizes for pods. # TYPE kubelet_pod_in_progress_resizes gauge kubelet_pod_in_progress_resizes 1 # HELP kubelet_pod_infeasible_resizes_total [ALPHA] Number of infeasible resizes for pods. # TYPE kubelet_pod_infeasible_resizes_total counter kubelet_pod_infeasible_resizes_total{reason_detail="insufficient_node_allocatable"} 1 # HELP kubelet_pod_resize_duration_seconds [ALPHA] Duration in seconds to actuate a pod resize # TYPE kubelet_pod_resize_duration_seconds histogram kubelet_pod_resize_duration_seconds_bucket{success="true",le="..."} ... kubelet_pod_resize_duration_seconds_sum{success="true"} 0.061 kubelet_pod_resize_duration_seconds_count{success="true"} 3 ``` The six resize metrics the kubelet records (all `[ALPHA]` stability, per KEP-1287) and what they tell you: | Metric | Type | Key labels | Reads as | |---|---|---|---| | `kubelet_container_requested_resizes_total` | counter | `operation`, `requirement`, `resource` | How many resizes, by kind. Here: 2 CPU-request increases, 1 CPU-limit decrease (from the infeasible→revert), etc. | | `kubelet_pod_in_progress_resizes` | gauge | (none) | How many resizes are actuating right now (allocated, not yet actuated). | | `kubelet_pod_pending_resizes` | gauge | `reason` | How many resizes are pending, split by `deferred` vs `infeasible`. | | `kubelet_pod_infeasible_resizes_total` | counter | `reason_detail` | Infeasible count; `insufficient_node_allocatable` = didn't fit. | | `kubelet_pod_deferred_resize_accepted_total` | counter | `retry_trigger` | Deferred resizes later accepted; a high `periodic_retry` share hints at a kubelet retry-logic issue. | | `kubelet_pod_resize_duration_seconds` | histogram | `success` | Actuation latency; `_sum 0.061` over `_count 3` ≈ 0.02s (~20ms)/resize here. | ### 10.8 metrics-server: `kubectl top` ```bash kubectl top nodes kubectl top pod resize-demo ``` **Example output (kind, Kubernetes v1.35):** ```text $ kubectl top nodes NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%) inplace-resize-verify-control-plane 146m 1% 613Mi 7% $ kubectl top pod resize-demo NAME CPU(cores) MEMORY(bytes) resize-demo 0m 0Mi ``` `resize-demo` uses `0m`/`0Mi` because `pause` does nothing — which is exactly why it is a clean vehicle for demonstrating resize *mechanics* independent of load. VPA needs a workload that actually consumes CPU to produce a recommendation, which is the next step. ### 10.9 Automate with VPA (`InPlaceOrRecreate`) — end to end Deploy the CPU-burning `vpa-demo` workload and its VPA object (Section 9.3): ```bash task vpa-demo # applies workloads/vpa-inplace.yaml ``` The `burn` container runs `while true; do :; done`, driving CPU utilization up. VPA's recommender computes a target, the updater actuates it **in place** via the resize subresource, and the Pod is **not** recreated. **Example output (kind, Kubernetes v1.35):** ```text # VPA recommendation (.status.recommendation.containerRecommendations) [{"containerName":"burn","lowerBound":{"cpu":"400m","memory":"100Mi"}, "target":{"cpu":"400m","memory":"100Mi"}, "uncappedTarget":{"cpu":"1168m","memory":"100Mi"}, "upperBound":{"cpu":"400m","memory":"256Mi"}}] # Pod NOT recreated (same pod, original age; RESTARTS=1 from memory RestartContainer policy): NAME READY STATUS RESTARTS AGE vpa-demo-6596db698c-sxtz6 1/1 Running 1 (4m53s ago) 6m50s # DESIRED == ACTUAL after VPA acted (requests cpu 50m->400m, mem 32Mi->100Mi; limits scaled proportionally): DESIRED: {"limits":{"cpu":"4","memory":"200Mi"},"requests":{"cpu":"400m","memory":"100Mi"}} ACTUAL: {"limits":{"cpu":"4","memory":"200Mi"},"requests":{"cpu":"400m","memory":"100Mi"}} # Events (VERIFIED) — note the VPA-specific event reason InPlaceResizedByVPA: Normal ResizeStarted Pod resize started: {...generation:2} Normal Killing Container burn resize requires restart Normal InPlaceResizedByVPA Pod was resized in place by VPA Updater. Normal ResizeCompleted Pod resize completed: {...generation:2} ``` What this proves: - **The Pod was not recreated.** Same Pod name/UID and original age (`6m50s`). The single restart (`RESTARTS=1`) is a *container* restart from the memory `RestartContainer` policy, not a Pod recreation. - **VPA respected `maxAllowed`.** The uncapped target was `1168m`; VPA capped the request to `400m` (the `maxAllowed`). - **VPA preserved the limit:request ratio.** Original CPU ratio 500m/50m = 10× → limit `4`; memory 64Mi/32Mi = 2× → limit `200Mi`. This is `controlledValues: RequestsAndLimits` behavior. - **The VPA-specific event reason `InPlaceResizedByVPA`** ("Pod was resized in place by VPA Updater.") is your audit trail that VPA — not a human — actuated the resize. ### 10.10 Prometheus + Grafana (expected output — not executed here) The kube-prometheus-stack values (Section 8.6) scrape the kubelet's resize metrics already. Load the dashboard and open the UIs: ```bash task dashboards # creates+labels the grafana-inplace-resize ConfigMap (grafana_dashboard=1) PORT_FORWARD=1 task observe # Grafana: http://localhost:3000 (admin / prom-operator — demo credential) # Prometheus: http://localhost:9090 ``` **Expected output — not executed in this environment.** In Prometheus → Status → Targets you should see the `serviceMonitor/monitoring/kube-prometheus-stack-kubelet/ 0..2` targets (`/metrics`, `/metrics/cadvisor`, `/metrics/resource`) **UP**. After triggering a resize, these queries return data: ```promql # Resize request rate by operation sum by (operation) (rate(kubelet_container_requested_resizes_total[5m])) # In-progress resizes (gauge) sum(kubelet_pod_in_progress_resizes) # Infeasible resizes by reason sum by (reason_detail) (rate(kubelet_pod_infeasible_resizes_total[5m])) # Resize duration p95 (metric is ms; divide by 1000 for seconds) histogram_quantile(0.95, sum by (le) (rate(kubelet_pod_resize_duration_seconds_bucket[5m]))) / 1000 # CPU request (desired, from kube-state-metrics) vs usage (actual, from cAdvisor) sum by (pod, container) (kube_pod_container_resource_requests{resource="cpu", namespace="default"}) sum by (pod, container) (rate(container_cpu_usage_seconds_total{namespace="default", container!="", container!="POD"}[5m])) ``` The committed `iac/dashboards/grafana-inplace-resize.json` wires five panels (resize rate by operation, in-progress gauge, infeasible by reason, duration p50/p95, requests-vs-usage) against a templated `${DS_PROMETHEUS}` datasource. Import it via the sidecar (above) or Grafana → Dashboards → New → Import. This article does **not** include fabricated Grafana screenshots — run the stack to see live panels. ### 10.11 OpenTelemetry Collector (expected output — not executed here) The Collector (Section 8.7) scrapes the same kubelet resize metrics via its `prometheus` receiver, adds usage/utilization via `kubeletstats` and absolute requests/limits via `k8s_cluster`, and remote-writes to Prometheus. ```bash kubectl -n observability logs ds/otel-collector | head -50 ``` **Expected output — not executed in this environment.** The `debug` exporter should log the resize metric families (`kubelet_pod_resize_duration_seconds`, `kubelet_container_requested_resizes_total`, …) and the `kubeletstats`/`k8s_cluster` series; after remote-write, the same series appear in Prometheus. > **Honest scope note on tracing.** The kubelet does **not** emit distributed traces > for resizes. Kubernetes has API-server and kubelet (KEP-2831) tracing, but there is > no dedicated "resize span." Resize observability is **metrics + Pod events**. If you > want a trace, instrument your *application* to open a span around a self-triggered > resize (PATCH the resize subresource, poll until `ACTUAL == DESIRED`), and export it > via OTLP. That span is synthetic/demonstrative — the platform does not produce it. --- ## 11. Production Best Practices 1. **Set `resizePolicy` deliberately per runtime.** For runtimes that cannot change their memory ceiling live (JVM, CPython, most managed runtimes), set `memory: RestartContainer` so a memory resize is a clean restart rather than a runtime error-and-retry or an OOM risk. For CPU, keep `NotRequired` — CPU almost always applies live. Treat this as an architecture decision, not a default. 2. **Reconcile the controller template, not just the running Pod.** A resize on the Pod is reverted the next time the Pod is recreated (rollout, drain, eviction). For a durable change, update the Deployment/StatefulSet template as well, or let VPA own the requests so recreated Pods get the current recommendation from the admission controller. 3. **Decrease memory conservatively.** The GA OOM-avoidance check is best-effort and subject to a TOCTOU race; on cgroup v2 an over-tight limit OOM-kills. Shrink in steps, watch working-set headroom, and prefer `RestartContainer` for memory decreases on runtimes that cannot shrink live. 4. **Alert on conditions and metrics, not on the deprecated string field.** Watch for `PodResizePending{reason=Infeasible}` (needs human action), long-lived `PodResizePending{reason=Deferred}` (capacity pressure), and rising `kubelet_pod_infeasible_resizes_total`. Do not read `.status.resize` — it is gone. 5. **Budget for container restarts on memory resizes.** A `RestartContainer` memory resize is a restart: it clears in-memory state, drops connections, and re-runs startup probes. Roll it out like a controlled restart (respect PDBs, do one replica at a time), not as a free knob. 6. **Keep resizes QoS-legal by construction.** Encode the QoS shape in your automation. For Guaranteed workloads, always move requests and limits together; for Burstable, never collapse the gap on both resources at once. 7. **Use PriorityClass to order resizes under contention.** On busy nodes, deferred resizes compete; give latency-sensitive workloads a higher PriorityClass so their upsizes are retried first (Section 6.8). 8. **Pair VPA with a PodDisruptionBudget and `minReplicas`.** `InPlaceOrRecreate` falls back to eviction for infeasible/timed-out resizes; a PDB and `minReplicas` protect availability on that path. Only enable `--in-place-skip-disruption-budget` for workloads that are provably non-disruptive (all `NotRequired`). 9. **Never run HPA and VPA on the same metric.** Split by metric (HPA on custom/ external, VPA on cpu/memory) to avoid the feedback loop (Section 9.6). 10. **Pin everything and provision real kubelet serving certs.** Pin chart and image versions (`versions.env`); do **not** ship `--kubelet-insecure-tls` to production — provision cluster-CA-signed kubelet serving certs instead. 11. **Grant `pods/resize` narrowly.** Give controllers/humans `patch` on `pods/resize` rather than write on `pods`. This is a clean least-privilege seam (Section 15). 12. **Measure actuation latency and set an SLO.** Track `kubelet_pod_resize_duration_seconds` p95 and `runtime_operations_duration_seconds{operation_type=container_update}`; a rising p95 or growing `kubelet_pod_in_progress_resizes` signals runtime/node pressure. 13. **Leave headroom on nodes you resize into.** In-place upsizes consume node capacity immediately (the scheduler counts `max(desired,allocated,actual)`). Packing nodes to 100% guarantees `Deferred`/`Infeasible` resizes. 14. **Test the version-skew and rollout order.** Upgrade the control plane (apiserver, scheduler) before kubelets. On mixed-version nodes, verify resizes on a canary node pool before relying on them fleet-wide. --- ## 12. Common Mistakes At least fifteen failure patterns, each with the correction. 1. **Patching `resources` on the main Pod resource.** `kubectl patch pod ...` without `--subresource resize` is rejected — resources are immutable on the main resource. Always target `--subresource resize`. 2. **Using an old `kubectl` client.** Clients < v1.32.0 report `invalid subresource` for `--subresource=resize`. Upgrade the client (server must be ≥ v1.33; we run v1.35). 3. **Expecting a Deployment edit to resize a running Pod.** Editing the template changes *future* Pods; the running Pod is only mutated through the resize subresource on the Pod. 4. **Forgetting the template revert.** Resizing only the Pod means the next rollout/ drain snaps back to the template values. Update the template too, or use VPA. 5. **Trying to change QoS via resize.** Setting `requests == limits` for both cpu and memory on a Burstable Pod (or removing all resources from a Guaranteed Pod) is rejected — QoS is invariant across a resize. 6. **Assuming a memory decrease is safe once accepted.** The OOM-avoidance check is best-effort; on cgroup v2 an over-tight limit OOM-kills. Decrease conservatively. 7. **Treating a `RestartContainer` memory resize as non-disruptive.** It restarts the container (state loss, dropped connections, re-run probes). Plan it like a restart. 8. **Reading the deprecated `.status.resize` string field.** It did not graduate; read the `PodResizePending`/`PodResizeInProgress` conditions instead. 9. **Confusing `Deferred` with `Infeasible`.** `Deferred` is retried automatically; `Infeasible` is never re-evaluated and needs you to change the request or add capacity. Alerting on the wrong one wastes on-call time. 10. **Packing nodes to capacity, then wondering why upsizes fail.** In-place upsizes need free node capacity *now*; a full node yields `Deferred`/`Infeasible`. 11. **Running HPA and VPA on the same metric.** Causes an oscillation loop. Split by metric. 12. **Only resizing CPU/memory requests but ignoring limits (or vice versa).** For Guaranteed Pods this breaks the `requests == limits` invariant and is rejected; for Burstable it can silently under- or over-provision the ceiling. 13. **Trying to resize non-CPU/memory resources.** Only cpu and memory are mutable in v1.35; anything else (ephemeral-storage, extended resources) is rejected. 14. **Trying to resize the wrong container type.** Non-restartable init containers and ephemeral containers cannot be resized; sidecars (restartable init containers) can. 15. **Shipping `--kubelet-insecure-tls` to production.** A kind-only shortcut. Provision cluster-CA-signed kubelet serving certs instead; skipping verification exposes the metrics path to MITM. 16. **Enabling in-place resize on unsupported node configs.** Swap-enabled containers, static CPU Manager, and static Memory Manager are not supported and surface as `Infeasible`. Don't design a resize workflow on a node pool that can't actuate it. 17. **Passing `--feature-gates=InPlaceOrRecreate=true` to VPA ≥ v1.7.0.** The gate is removed there; the flag crashes the component. Pin the VPA app version. 18. **Assuming a `patch` return of success means the resize converged.** The API returns success on accepting the desired change; convergence is `ACTUAL == DESIRED`. Verify status/conditions before declaring done. --- ## 13. Troubleshooting Guide Symptom → likely cause → diagnosis and fix. Commands use `P` for the Pod name and `N` for the node name. The **Signals to inspect** list after the table maps each symptom to the log line or metric that confirms it. | Symptom | Likely cause | Diagnosis → fix | Command | |---|---|---|---| | `error: invalid subresource "resize"` | `kubectl` client < v1.32.0 | Old client → upgrade to ≥ v1.32 | `kubectl version` | | Patch rejected: resources immutable | Patched main resource, not the subresource | Command lacks `--subresource resize` → add it | `kubectl patch pod P --subresource resize --patch-file f.yaml` | | Patch rejected: QoS would change | Resize flips the QoS class | Compare requests/limits shape → keep QoS shape (Guaranteed: `req==lim`; Burstable: keep a gap) | `kubectl get pod P -o jsonpath='{.status.qosClass}'` | | `PodResizePending` / `Infeasible` | Request exceeds node capacity (or static Pod / swap / static managers / Windows) | Read the condition message → patch a feasible value, add capacity, or move off the unsupported config | `kubectl get pod P -o jsonpath='{.status.conditions}' \| jq .` | | `PodResizePending` / `Deferred` (long-lived) | Node lacks room *now*; queued behind higher-priority resizes | Check node allocatable and priority ordering → free capacity, raise PriorityClass, or scale the pool | `kubectl describe node N` | | `PodResizeInProgress` / `Error` | Actuation error — often the memory-decrease OOM check (usage > new limit) | Read the condition message → raise the target limit above current usage, or use `RestartContainer` | `kubectl get pod P -o jsonpath='{.status.conditions}'` | | ACTUAL never equals DESIRED | Deferred / in-progress / infeasible, or the runtime rounded the value | Compare actuated vs actual → address the condition; small cgroup rounding is expected | `bash scripts/verify-resize.sh P` | | `restartCount` jumped unexpectedly | Memory (or CPU+memory) resize with `RestartContainer` | Events show `Killing … resize requires restart` → expected; use `NotRequired` only if the runtime supports live change | `kubectl describe pod P` | | Pod OOM-killed after a memory decrease | Best-effort OOM check lost the TOCTOU race | `lastState.terminated.reason=OOMKilled` → decrease conservatively; use `RestartContainer` for memory | `kubectl get pod P -o jsonpath='{.status.containerStatuses[0].lastState}'` | | `kubectl top` returns no data | metrics-server can't verify the kubelet cert (kind) | x509 SAN error in logs → `--kubelet-insecure-tls` (dev) or signed kubelet certs (prod) | `kubectl -n kube-system logs deploy/metrics-server` | | VPA recommends but never actuates | `updateMode: Off`/`Initial`, or in-place gate unset | Check the VPA and updater args → set `InPlaceOrRecreate`; ensure the updater has the gate | `kubectl -n vpa get deploy vpa-updater -o yaml \| grep feature-gates` | | VPA evicts instead of resizing in place | In-place infeasible/timed-out → fallback, or a QoS change is required | Check VPA and Pod events → ensure capacity, keep QoS stable, check resize policy | `kubectl get events \| grep -E 'Evict\|InPlaceResizedByVPA'` | | Prometheus shows no resize metrics | kubelet ServiceMonitor down, or no resize has happened yet | Check Prometheus → Targets; trigger a resize | `kubectl -n monitoring get servicemonitor` | | Workload ServiceMonitor ignored | `serviceMonitorSelectorNilUsesHelmValues: true` | Not listed in Prometheus targets → set the selector flags to `false` (Section 8.6) | `kubectl -n monitoring get prometheus -o yaml` | | OTel `k8s_cluster` series double-counted | Multiple DaemonSet pods each run `k8s_cluster` | Duplicate series → run `k8s_cluster` in a single deployment-mode collector | `kubectl -n observability get pods -o wide` | | Resize works on one node, not another | Version skew or an unsupported node config | Compare node versions/configs → upgrade kubelets or move to a supported pool | `kubectl get nodes -o wide` | **Signals to inspect** (the log line or metric that confirms each case): - **Infeasible resize** → `kubelet_pod_infeasible_resizes_total{reason_detail=...}` - **Deferred resize** → `kubelet_pod_pending_resizes{reason="deferred"}` - **In-progress / stuck actuation** → `kubelet_pod_in_progress_resizes`; kubelet logs (`resize actuation error`) - **Memory-decrease OOM** → kernel/kubelet OOM logs; `lastState.terminated.reason=OOMKilled` - **`RestartContainer` restart** → Pod event reason `Killing` - **metrics-server / `kubectl top`** → x509 "IP SANs" error in the metrics-server log - **VPA actuation** → vpa-updater logs; `vpa_updater_failed_in_place_update_attempts_total` - **Prometheus scrape health** → `count(kubelet_pod_resize_duration_seconds_count) > 0`; Prometheus config-reload logs General first moves for any resize problem: ```bash # 1. What does the Pod think its desired vs actual is? kubectl get pod P -o jsonpath='{.spec.containers[0].resources}'; echo kubectl get pod P -o jsonpath='{.status.containerStatuses[0].resources}'; echo # 2. Any resize conditions? kubectl get pod P -o jsonpath='{.status.conditions}' | jq '.[] | select(.type|test("PodResize"))' # 3. What events fired? kubectl describe pod P | sed -n '/Events:/,$p' # 4. What do the kubelet metrics say? NODE=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') kubectl get --raw "/api/v1/nodes/${NODE}/proxy/metrics" | grep kubelet_pod_ ``` --- ## 14. Real Production Use Cases ### 14.1 Game servers and stateful session holders A match-based game server holds live player sessions in memory. Recreating it to add CPU headroom for a busy match would disconnect everyone. With in-place resize, an operator (or a controller watching match load) raises CPU with `NotRequired` — no restart, no dropped sessions — and lowers it again when the match ends. Memory is kept on `RestartContainer` and only resized between matches. The trade-off: CPU scaling is free, memory scaling is scheduled around session boundaries. ### 14.2 JIT / startup boost Many runtimes need far more CPU during startup (JIT compilation, cache warming) than in steady state. Historically you either over-provisioned steady-state CPU or suffered slow cold starts. The pattern: start the container with generous CPU, then scale CPU **down in place** once the Pod is `Ready`. VPA's **CPU Startup Boost** (AEP-7862, VPA v1.7.0 alpha — forward-looking) automates exactly this: it raises CPU requests+limits at startup and scales them back down in place after readiness. Until that ships, you can approximate it with a controller or an init-driven resize. This is the clearest example of resize turning a static sizing compromise into a dynamic one. ### 14.3 Pre-warmed worker pools A pool of workers kept warm to avoid cold-start latency can idle at a small reservation and be scaled up in place the moment work is dispatched to them — faster than scheduling a new Pod, and without losing the warm cache/connections the worker already holds. When the burst subsides, scale the reservation back down. The win is latency (no scheduling round-trip) plus preserved warm state; the constraint is that the worker must fit on its current node when scaled up (or the resize defers). ### 14.4 Bin-packing and cost efficiency Fleets are routinely over-provisioned "to be safe," stranding capacity. VPA in `InPlaceOrRecreate` mode continuously right-sizes requests toward observed usage *without* the eviction churn that made teams run VPA in `Off` mode. Tighter, accurate requests let the scheduler pack more Pods per node, directly reducing node count and cost. The trade-off is operational: you now depend on VPA's recommendations and must watch for memory-decrease OOM risk and deferred resizes on tightly packed nodes. ### 14.5 Batch and phase-based workloads A pipeline stage that needs 4 cores during a compute phase and 200m while waiting on I/O can grow and shrink its CPU reservation per phase instead of holding the peak for its whole lifetime. Because CPU is `NotRequired`, these transitions are restart-free. This reclaims the difference between peak and average for every other workload sharing the node. Across all of these, the common thread is the same: **decouple the reservation from the process lifecycle.** In-place resize is most valuable exactly where recreation is most expensive — long warm-ups, live state, and latency-sensitive paths. --- ## 15. Security Considerations ### 15.1 RBAC: the `pods/resize` subresource The single most important security fact: resizing is authorized by the **`pods/resize`** subresource, independently of write access to `pods`. Regular `patch`/`update` on `pods` does **not** allow resource mutation. To let a subject resize, grant `patch`/`update` on `pods/resize`: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pod-resizer namespace: default rules: - apiGroups: [""] resources: ["pods/resize"] verbs: ["patch", "update"] - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] # read desired/actual/status ``` This is a genuine least-privilege seam: you can authorize "may right-size workloads" without authorizing "may rewrite Pod spec, mount secrets, change images." Grant it to the specific controller or human role that needs it, scoped to a namespace where possible. (This follows from the KEP subresource definition and standard Kubernetes subresource RBAC semantics.) ### 15.2 The kubelet/Node identity is the only writer of actual state Only the Node account may write `status.allocatedResources`/`status.resources`. Users and controllers write *desired* (`spec`) through the subresource; the kubelet writes *actual* (`status`). This separation means a compromised workload identity cannot forge "actual" resources to hide from quota or fool the scheduler. ### 15.3 Resource-exhaustion and abuse surface Because resize can raise a Pod's reservation, an over-broad `pods/resize` grant is a capacity-abuse vector: a subject could upsize workloads to starve others or drive node pressure. Mitigations: - Scope the grant narrowly (namespace, specific role). - Keep **ResourceQuota** in force — quota accounts for resizes via `max(desired,allocated,actual)`, so a resize that would breach quota is rejected. - Set VPA `maxAllowed` ceilings so automated resizes cannot run away. - Alert on `kubelet_container_requested_resizes_total` spikes and on unexpected upsizes. ### 15.4 Transport security for the metrics path The kind-only `--kubelet-insecure-tls` shortcut disables verification of the kubelet serving cert and must not reach production — it exposes the metrics/stats path to man-in-the-middle. Provision cluster-CA-signed kubelet serving certificates (enable `RotateKubeletServerCertificate` and the CSR approver) so metrics-server, Prometheus, and the OTel Collector can verify the kubelet. ### 15.5 Grafana and stack credentials The demo sets `grafana.adminPassword: "prom-operator"` in plaintext values — a demo credential. In production, reference an existing Secret (`admin.existingSecret`) and rotate it; never commit dashboard-stack admin passwords to a values file in git. ### 15.6 Audit trail Resizes are auditable at the API server (`pods/resize` requests appear in the audit log and in `apiserver_request_total{resource=pods,subresource=resize}`) and on the Pod (`ResizeStarted`/`ResizeCompleted` events, and `InPlaceResizedByVPA` when VPA acted). Keep these in your audit pipeline so every reservation change is attributable. --- ## 16. Performance Considerations ### 16.1 Actuation latency A resize is not instantaneous — the kubelet must admit, order, call the runtime, and read back. In the example run, `kubelet_pod_resize_duration_seconds_sum{success="true"} 0.061` over `_count 3` ≈ **20ms per resize** for a trivial `pause` container on an idle node. Real actuation latency depends on the runtime, node load, and whether a container restart is involved. Track: ```promql histogram_quantile(0.95, sum by (le) (rate(kubelet_pod_resize_duration_seconds_bucket[5m]))) / 1000 runtime_operations_duration_seconds{operation_type="container_update"} ``` A `RestartContainer` resize adds the container's restart cost (image already present, so mostly process start + probes) on top of actuation. ### 16.2 The scheduler↔kubelet race (correctness under load) The main known correctness gap in v1.35: the scheduler and kubelet can momentarily disagree about node capacity, and a race can over-schedule a node so a Pod is rejected with `OutOfCPU`/`OutOfMemory` (tracked in kubernetes/kubernetes#126891; resolution out of scope for GA). Practical mitigation: leave node headroom, avoid packing to 100% on node pools where you resize aggressively, and canary aggressive resize automation. ### 16.3 Memory-decrease TOCTOU The best-effort OOM-avoidance check reads usage, then writes the limit; usage can climb in between. A memory decrease can therefore either leave a resize `InProgress` with `reason: Error` (skipped) or, if written below live usage on cgroup v2, OOM-kill. Decreasing memory has a real performance/availability cost that increasing does not — size decreases conservatively and off the hot path. ### 16.4 Resize storms Automated resizers (VPA, custom controllers) can generate many resizes. Each consumes kubelet work and a CRI call, and each memory `RestartContainer` resize is a restart. Watch `kubelet_pod_in_progress_resizes` (a persistently non-zero gauge means the kubelet is backed up) and `kubelet_container_requested_resizes_total` rate. Rate-limit custom resizers and rely on VPA's built-in update conditions (bounds, >12h/>10% rule, quick-OOM) rather than resizing on every metric tick. ### 16.5 No effect on scheduling throughput A resize does **not** invoke the scheduler (the Pod is already bound). So resizing is cheaper than recreate-based right-sizing, which pays a full deschedule/schedule/ image-pull/startup cost. This is the core performance argument for the feature: the common case (CPU up/down) is a cgroup write, not a reschedule. --- ## 17. Integration with the Ecosystem ### 17.1 Metrics Server The Metrics API backend and a hard dependency of VPA's recommender. It serves `kubectl top` and feeds VPA the usage samples it turns into recommendations. Without it, VPA computes nothing. (Verified: chart 3.13.1, app v0.8.1.) ### 17.2 Vertical Pod Autoscaler The primary automated consumer of in-place resize. In `InPlaceOrRecreate` mode the updater actuates recommendations via the resize subresource, falling back to eviction only when in-place is infeasible/timed-out/QoS-changing. VPA-side observability: ```text vpa_updater_in_place_updatable_pods_total vpa_updater_in_place_updated_pods_total vpa_updater_failed_in_place_update_attempts_total ``` Pair these with the kubelet resize metrics to see both "VPA decided to resize" and "the kubelet actuated it." (Verified: chart 4.12.3, app v1.6.0; event reason `InPlaceResizedByVPA`.) ### 17.3 Prometheus + kube-state-metrics + cAdvisor Prometheus scrapes the kubelet's `/metrics` (resize metrics), `/metrics/cadvisor` (actual usage), and `/metrics/resource`. **Desired vs actual** on one board: kube-state-metrics gives desired (`kube_pod_container_resource_requests/_limits` — spec = desired in v1.35), cAdvisor gives actual usage (`container_cpu_usage_seconds_total`, `container_memory_working_set_bytes`). The utilization ratio (usage/request) is the signal a resize acts on: ```promql sum by (namespace,pod,container)(rate(container_cpu_usage_seconds_total{container!="",container!="POD"}[5m])) / sum by (namespace,pod,container)(kube_pod_container_resource_requests{resource="cpu"}) ``` > **Note:** whether kube-state-metrics surfaces `status` (actual) resources depends on > its version; classic KSM exposes spec-derived (desired) requests/limits only. ### 17.4 Grafana The committed `grafana-inplace-resize.json` (5 panels) visualizes resize rate, in-progress, infeasible-by-reason, duration percentiles, and requests-vs-usage. Import via the sidecar (`task dashboards`). ### 17.5 OpenTelemetry Collector Alternative/additional collection path: `prometheus` receiver for resize metrics, `kubeletstats` for usage/utilization ratios, `k8s_cluster` for absolute requests/ limits, remote-writing to Prometheus. Note the kubelet emits **no resize traces** — resize observability is metrics + events (Section 10.11). ### 17.6 HPA boundaries HPA and VPA must not share a resource metric (Section 9.6). The supported topology is HPA on custom/external metrics (RPS, queue depth) for replica count + VPA on cpu/memory for per-Pod sizing — horizontal and vertical scaling on orthogonal signals. ### 17.7 Forward-looking: Cluster Autoscaler / Karpenter and Kueue In-place resize works *within* a node's capacity. When an upsize does not fit, it defers/infeasibles — and that is the handoff point to the **Cluster Autoscaler** or **Karpenter** to add a node, and to **Kueue** for admission/quota-aware batch scheduling. The interplay between in-place resize signals (`Deferred`/`Infeasible`) and node provisioning is an active area and a natural bridge to the scheduling topics later in this series. (Cluster Autoscaler / Karpenter awareness of in-place resize pending signals is still evolving — check current project docs.) --- ## 18. Advanced Topics ### 18.1 The actuated state and restart tolerance The kubelet checkpoints the **actuated** resources (what it sent the runtime) on the node (PR #130599). This is why a kubelet restart mid-resize does not lose track: on restart it compares checkpointed-actuated against runtime-reported-actual and re-issues only if they differ. It also explains the design decision to compare *actuated* rather than *desired* against actual — runtimes legitimately land on nearby values (kernel CPU minimums, systemd 10ms quota rounding, NRI plugin mutation), and comparing against desired would cause endless re-issue loops. ### 18.2 Pod-level resources resize (alpha in v1.35) Beyond per-container resources, Kubernetes supports **Pod-level resources** (`spec.resources`). In-place resize of *Pod-level* resources is **alpha in v1.35** behind its own gate (`InPlacePodLevelResourcesVerticalScaling`). Keep production pinned to per-container resize on v1.35; treat pod-level resize as forward-looking. ### 18.3 Exclusive-CPU relaxation A separate gate, `InPlacePodVerticalScalingExclusiveCPUs` (seen in PR #128713), governs relaxing the static-CPU-manager restriction for in-place resize. In baseline v1.35, Guaranteed Pods under the static CPU Manager policy are `Infeasible` to resize; this gate is the path toward loosening that. (Confirm maturity and behavior for your specific Kubernetes build.) ### 18.4 CRI contract and `UpdatePodSandboxResources` The kubelet actuates via `UpdateContainerResources`, which the runtime must implement idempotently; since v1.33 the contract is that runtimes should **not** deliberately restart a container to resize — they return an error instead, and the kubelet decides what to do. GA added a best-effort `UpdatePodSandboxResources` CRI call, invoked after the kubelet reconfigures pod-level cgroups, primarily to notify NRI plugins. ### 18.5 The `allocatedResources` status and its gate The companion gate `InPlacePodVerticalScalingAllocatedStatus` controls whether `status.containerStatuses[i].allocatedResources` (the *allocated* state, Section 6.3) is exposed in the API. When on, it gives you a view of what the kubelet has admitted but not yet actuated — useful for distinguishing "admitted, actuating" from "not yet admitted." ### 18.6 Version skew and upgrade order Upgrade the control plane (apiserver, scheduler) before kubelets. The apiserver detects kubelet feature support by checking whether `status.containerStatuses[*].resources` is populated on running containers, and handles skew accordingly. On mixed-version node pools, resizes may work on newer kubelets and not on older ones — canary before relying on the feature fleet-wide. ### 18.7 Self-instrumented resize tracing (demo pattern) Since the platform emits no resize spans, you can build an app-side trace: open a span `pod.self_resize`, PATCH the resize subresource, poll `status.containerStatuses[]. resources` until `ACTUAL == DESIRED` (or a resize condition clears), attach attributes (`k8s.pod.name`, `resize.resource`, `old`, `new`, `resize.outcome`), and export via OTLP. This correlates an application action with the platform-side `kubelet_pod_resize_duration_seconds`. It is synthetic — label it as such. ### 18.8 Terminated-container fail-open A resize is atomic across containers, but a *terminated* container's "resize" is permitted (fail-open) so it does not block resizing sibling running containers. This is a deliberate lifecycle nuance in KEP-1287 to avoid one dead container wedging the whole Pod's resize. --- ## 19. Interview Questions Sixty questions across three tiers, each with an answer. ### 19.1 Beginner (20) **B1. What does In-Place Pod Vertical Scaling let you do?** Change a running Pod's cpu/memory requests and limits and have the kubelet apply the change to the existing container(s), without deleting and recreating the Pod. **B2. In which Kubernetes version did it reach GA?** v1.35 (alpha v1.27, beta v1.33, GA/stable v1.35). **B3. Do you need to enable a feature gate in v1.35?** No. `InPlacePodVerticalScaling` is GA and on by default in v1.35. **B4. Which resources can be resized in place in v1.35?** Only `cpu` and `memory` (requests and limits). Nothing else is mutable. **B5. What is the difference between `spec.containers[*].resources` and `status.containerStatuses[*].resources`?** `spec` = DESIRED (what you asked for); `status` = ACTUAL (what the container currently has). A resize is converged when they are equal. **B6. How do you trigger a resize with kubectl?** `kubectl patch pod --subresource resize --patch '...'` — you must target the `resize` subresource. **B7. Can you resize by editing the Deployment's template?** Editing the template changes future Pods; the running Pod is only mutated through the resize subresource on the Pod itself. **B8. What is `resizePolicy`?** A per-container, per-resource setting that says whether applying a new value needs a container restart: `NotRequired` (no restart) or `RestartContainer` (restart in place). **B9. What is the default `resizePolicy` if you don't set it?** `NotRequired` for each resource. **B10. Does a `RestartContainer` resize recreate the Pod?** No. It restarts the *container* in place (incrementing `restartCount`); the Pod keeps its name, UID, node, and IP. **B11. Why does memory often use `RestartContainer`?** Many runtimes (JVM, CPython) cannot change their memory ceiling live, so the container must restart to adopt the new limit. **B12. What happens to CPU on a `NotRequired` resize?** The new CPU cgroup values are applied to the live container with no restart. **B13. What minimum kubectl client version supports `--subresource=resize`?** Client v1.32.0 or newer. **B14. What is a QoS class?** A Pod classification (Guaranteed/Burstable/BestEffort) derived from its requests/limits that drives eviction order. **B15. Can a resize change a Pod's QoS class?** No. QoS is invariant across a resize; the API server rejects resizes that would change it. **B16. What tool right-sizes Pods automatically using this feature?** The Vertical Pod Autoscaler (VPA), in `InPlaceOrRecreate` mode. **B17. Name one workload that benefits from in-place resize.** Any long-warm-up or stateful workload — e.g. a JVM service or a game server holding live sessions — where recreation is expensive. **B18. Where do the resize metrics come from?** The kubelet's `/metrics` endpoint (e.g. `kubelet_pod_resize_duration_seconds`). **B19. What does `kubectl top` need to work?** metrics-server, which serves the Metrics API. **B20. What does "converged" mean for a resize?** DESIRED equals ACTUAL — the container actually has the resources you asked for. ### 19.2 Intermediate (20) **I1. What replaced the deprecated `.status.resize` string field?** Two Pod conditions: `PodResizePending` (reason `Deferred` or `Infeasible`) and `PodResizeInProgress`. **I2. Difference between `Deferred` and `Infeasible`?** `Deferred` is feasible in principle but not right now — retried periodically. `Infeasible` cannot be satisfied as-is and is never re-evaluated. **I3. Name three infeasible reasons in v1.35.** Request exceeds node capacity; static Pod; swap-enabled container; Guaranteed Pod with static CPU/Memory Manager; Windows Pod (any three). **I4. What are the four resource states the kubelet tracks?** Desired → Allocated → Actuated → Actual. **I5. Why does the kubelet compare *actuated* vs actual, not desired vs actual?** Because runtimes legitimately land on nearby values (kernel CPU minimums, systemd 10ms rounding, NRI plugins); comparing desired vs actual would trigger endless re-issues. **I6. What changed for memory-limit decreases at GA?** They are now allowed (previously prohibited), guarded by a best-effort OOM-avoidance check that skips the resize if current usage exceeds the new limit. **I7. Why is the memory-decrease check only "best-effort"?** A TOCTOU race: usage can rise between the check and the cgroup write; on cgroup v2 an over-tight limit OOM-kills. **I8. What RBAC do you need to let a controller resize Pods?** `patch`/`update` on the `pods/resize` subresource (plus `get`/`list`/`watch` on `pods`). Write on `pods` alone does not permit resource mutation. **I9. In what order are deferred resizes retried?** PriorityClass → QoS class (Guaranteed > Burstable) → deferred duration (longest-pending first). **I10. What are the five VPA `updateMode` values?** `Off`, `Initial`, `Recreate`, `InPlaceOrRecreate`, `Auto` (Auto is deprecated). **I11. What does `InPlaceOrRecreate` do on failure?** Falls back to evicting/recreating the Pod when the in-place update is infeasible, deferred >5m, in-progress >1h, or would change QoS. **I12. Why must you not run HPA and VPA on the same metric?** VPA changes requests while HPA scales replicas on utilization relative to requests; together on one metric they oscillate. **I13. What is the precedence when a resize touches resources with different `resizePolicy`?** `RestartContainer` wins — if any changed resource needs a restart, the container restarts. **I14. How does the scheduler account for an in-flight resize?** It uses `max(desired, allocated, actual)` for placing new Pods (excluding desired for `Infeasible` resizes). **I15. Which container types can and cannot be resized?** Sidecars (restartable init containers) and regular containers can; non-restartable init containers and ephemeral containers cannot. **I16. What VPA `controlledValues` setting preserves the limit:request ratio?** `RequestsAndLimits` (the default). `RequestsOnly` leaves limits untouched. **I17. What event reason does VPA emit when it resizes in place?** `InPlaceResizedByVPA` ("Pod was resized in place by VPA Updater."). **I18. Which kubelet resize metrics exist (names)?** Six, per KEP-1287: `kubelet_container_requested_resizes_total`, `kubelet_pod_in_progress_resizes`, `kubelet_pod_pending_resizes`, `kubelet_pod_infeasible_resizes_total`, `kubelet_pod_deferred_resize_accepted_total`, and the `kubelet_pod_resize_duration_seconds` histogram. **I19. If the Pod has `restartPolicy: Never`, what constraint applies to `resizePolicy`?** Every `resizePolicy` entry must be `NotRequired` (you cannot ask to restart a container the Pod may not restart). **I20. How do you make a resize durable across Pod recreation?** Update the controller template as well (or let VPA manage requests) — a Pod-only resize reverts on recreation. ### 19.3 Senior (20) **S1. Walk through the end-to-end path of a CPU resize.** Client PATCHes `pods/resize` → apiserver validates (cpu/mem only, QoS invariant) and persists DESIRED → kubelet watches, admits/allocates (checkpoint), orders changes, calls CRI `UpdateContainerResources` → runtime writes `cpu.weight`/`cpu.max` → kubelet reads back ACTUAL via `ContainerStatus`, updates status, emits events/metrics. The scheduler is not in the actuation path. **S2. Why is comparing actuated vs actual necessary for correctness, not just efficiency?** Because desired and actual can legitimately differ (kernel minimums, systemd rounding, NRI mutation). Reconciling against desired would never converge; reconciling against actuated converges to what the kubelet actually intended to set. **S3. Explain the multi-container atomic resize ordering and why it exists.** Net-increases raise the pod-level cgroup first, then apply container decreases, then lower the pod-level cgroup on net-decrease, then apply container increases. Decreases before increases guarantee the sum of container limits never transiently exceeds the pod cgroup limit. **S4. What is the scheduler↔kubelet race and how do you mitigate it?** The scheduler and kubelet can momentarily disagree on capacity, over-scheduling a node so a Pod is rejected with `OutOfCPU`/`OutOfMemory` (#126891, unresolved at GA). Mitigate with node headroom and canarying aggressive resize automation. **S5. How would you design a memory-decrease workflow to avoid OOM?** Decrease in small steps off the hot path, monitor working-set headroom (`limit − working_set`), and for runtimes that cannot shrink live, use `RestartContainer` so the decrease applies to a fresh process rather than risking OOM on the running one. **S6. Reconcile "VPA InPlaceOrRecreate is beta" with "it's GA in VPA v1.6.0."** Two layers: the *cluster* feature (in-place resize) is GA in k8s v1.35; the *VPA mode* that consumes it was beta in the 1.35-era blog (AEP-4016) and reached GA in VPA v1.6.0. The gate is default-on-but-present in v1.6.0 and removed in v1.7.0. **S7. How does the resize subresource improve the security posture over mutable `pods`?** It is a distinct RBAC object (`pods/resize`), so "may resize" is grantable independently of "may rewrite Pod spec." This lets you give VPA/operators resize rights without broad Pod write — a least-privilege seam. **S8. What are the observability surfaces for resize, and what is explicitly absent?** Metrics (six kubelet `[ALPHA]` metrics + apiserver/runtime SLIs) and Pod events (`ResizeStarted`/`ResizeCompleted`/`Killing`/`InPlaceResizedByVPA`). Absent: distributed traces — the kubelet emits none for resizes. **S9. Design alerting for a fleet using in-place resize.** Alert on `PodResizePending{reason=Infeasible}` (human action), sustained `{reason=Deferred}` (capacity), rising `kubelet_pod_infeasible_resizes_total`, a persistently non-zero `kubelet_pod_in_progress_resizes` (kubelet backed up), and resize duration p95 regressions. **S10. Why is a resize cheaper than recreate-based right-sizing?** It does not invoke the scheduler (Pod already bound) and skips deschedule/schedule/ image-pull/startup. The common CPU case is a single cgroup write. **S11. How do quota and resize interact?** ResourceQuota accounts using `max(desired,allocated,actual)`, so an upsize that would breach quota is rejected, and an `Infeasible` resize's desired is excluded from the max. **S12. Explain the version-skew handling on upgrade.** Upgrade control plane before kubelets. The apiserver infers kubelet feature support by checking whether running containers' `status...resources` is populated, and handles mixed-version behavior accordingly. **S13. When does VPA skip the PodDisruptionBudget for in-place updates?** Only with `--in-place-skip-disruption-budget=true` *and* when the update is genuinely non-disruptive (all containers `NotRequired` for cpu and memory). PDB is still enforced if any container uses `RestartContainer` or the update falls back to eviction. **S14. What is the "actuated" state and why is it checkpointed?** The config the kubelet sent the runtime. It is checkpointed on the node so a kubelet restart can resume reconciliation (compare checkpointed-actuated vs runtime-actual) rather than re-deriving from desired. **S15. How do PriorityClass and QoS combine to order deferred resizes, and what's the limit of that control?** Retry order is PriorityClass → QoS → deferred duration, but a higher-priority pending resize does not block others (all are still attempted), and ordering only affects retry sequencing — it does not guarantee prompt actuation on a tight node. **S16. Describe a supported HPA+VPA topology and why it's stable.** HPA on custom/external metrics (RPS, queue depth) for replicas + VPA on cpu/memory for sizing. The signals are orthogonal, so there is no feedback loop between replica count and requests. **S17. What node configurations make a resize `Infeasible` regardless of capacity?** Static CPU Manager and static Memory Manager (for Guaranteed Pods), swap-enabled containers (for memory requests without `RestartContainer`), static Pods, and Windows Pods. **S18. How would you correlate "VPA decided" with "kubelet actuated"?** Join VPA updater metrics (`vpa_updater_in_place_updated_pods_total`, `vpa_updater_failed_in_place_update_attempts_total`) and the `InPlaceResizedByVPA` event with kubelet resize metrics/events on the same Pod/time window. **S19. Why is `NotRequired` not a guarantee of no restart?** If the runtime discovers a resize truly needs a restart, it returns an error and the kubelet retries; in edge cases a container can end stopped. `NotRequired` expresses intent, not a hard guarantee. **S20. What is the special GA eviction behavior and its scope?** For `system-node-critical`/`system-cluster-critical` Pods with no room, the kubelet may evict a non-critical Pod to make room for the resize. General preemption for non-critical resizes is out of scope for GA. --- ## 20. FAQ **Q1. Do I need to enable any feature gate on Kubernetes v1.35?** No. `InPlacePodVerticalScaling` is GA and on by default. **Q2. Will resizing restart my container?** CPU with `NotRequired` does not. Memory with `RestartContainer` does (a container restart in place, not a Pod recreation). It depends on your `resizePolicy` and runtime. **Q3. Does resizing ever recreate the Pod?** No. The Pod object, name, UID, node, and IP are preserved. Only a container may restart. **Q4. Can I resize storage or GPUs in place?** No. Only cpu and memory are mutable in v1.35. **Q5. Why was my resize marked `Infeasible`?** The request can't be satisfied as-is — usually it exceeds node capacity, or the node uses an unsupported config (static CPU/Memory Manager, swap, Windows). It is never retried; patch a feasible value or add capacity. **Q6. Why is my resize stuck `Deferred`?** The node lacks room right now (possibly behind higher-priority resizes). It is retried periodically; free capacity, raise PriorityClass, or scale the node pool. **Q7. Is decreasing memory safe?** Not guaranteed. The kubelet does a best-effort OOM check, but a TOCTOU race means an over-tight limit can OOM-kill on cgroup v2. Decrease conservatively. **Q8. Why didn't editing my Deployment resize the running Pod?** Template edits affect future Pods; the running Pod is mutated only via the resize subresource on the Pod. **Q9. Why does my resize revert after a rollout?** A Pod-only resize is not persisted to the controller template. Update the template or use VPA so recreated Pods get the current values. **Q10. What kubectl version do I need?** Client ≥ v1.32.0 for `--subresource=resize`; server ≥ v1.33 (v1.35 recommended). **Q11. How do I know a resize finished?** When ACTUAL equals DESIRED and there is no `PodResizePending`/`PodResizeInProgress` condition. Use `scripts/verify-resize.sh`. **Q12. Can I resize a Pod's QoS class?** No — QoS is invariant across a resize by validation. **Q13. Does VPA need the feature gate flag?** On VPA v1.6.0 the `InPlaceOrRecreate` gate is default-on; passing the flag is a harmless no-op. On v1.4.0 it's required; on v1.7.0 it's removed (passing it crashes the component). **Q14. Can HPA and VPA work together?** Only on different metrics (HPA on custom/external, VPA on cpu/memory). Never on the same metric. **Q15. Where do the resize metrics live?** The kubelet `/metrics` endpoint; scrape via the kubelet ServiceMonitor or the API-server node proxy. **Q16. Are there distributed traces for resizes?** No. Observability is metrics + Pod events. You can instrument your app to trace a self-triggered resize, but the platform emits no resize spans. **Q17. What RBAC lets a user resize but not otherwise edit Pods?** `patch`/`update` on `pods/resize` (plus read on `pods`). Write on `pods` alone does not permit resource mutation. **Q18. Does resizing invoke the scheduler?** No. The Pod is already bound; the scheduler only accounts for the resize when placing *other* Pods. **Q19. What restart cost does a `RestartContainer` memory resize incur?** A container restart: process restart, dropped in-memory state/connections, re-run startup probes. Image is already present, so no pull. Plan it like a controlled restart. **Q20. Can I resize sidecars?** Yes — sidecars (restartable init containers) can be resized. Non-restartable init containers and ephemeral containers cannot. **Q21. What happens to a resize if the kubelet restarts mid-actuation?** The kubelet checkpoints the actuated state and resumes reconciliation on restart, re-issuing only if runtime-actual differs from checkpointed-actuated. **Q22. Is cgroup v1 supported?** cgroup v2 is the supported baseline. Memory-decrease semantics differ (v1 rejects the write; v2 OOM-kills), and the feature is exercised/validated on cgroup v2. --- ## 21. Key Takeaways - **The reservation is now decoupled from the process lifecycle.** In v1.35 you can change cpu/memory requests and limits on a running Pod without recreating it — GA, on by default, no feature gate. - **The mental model is DESIRED (`spec`) vs ACTUAL (`status`).** A resize is converged when they match; everything else is a state in the kubelet's Desired→Allocated→Actuated→Actual machine. - **Use the `resize` subresource.** Resources are immutable on the main Pod resource; `kubectl patch pod --subresource resize` is the only path, and it doubles as a clean `pods/resize` RBAC seam. - **`resizePolicy` is a runtime decision.** CPU applies live (`NotRequired`); memory often needs `RestartContainer` (a container restart in place — not a Pod recreation). - **Status is conditions, not the old string field.** Read `PodResizePending` (`Deferred`/`Infeasible`) and `PodResizeInProgress`; alert on them. - **GA added memory-limit decreases (best-effort OOM check) and prioritized/deferred resizes** ordered by PriorityClass → QoS → deferred duration. - **Know the boundaries.** Only cpu/memory; QoS invariant; no swap/static CPU/Memory managers; no Windows; a scheduler↔kubelet race exists; resize stays within a node's capacity. - **VPA `InPlaceOrRecreate` is the automation path** (GA in VPA v1.6.0), emitting `InPlaceResizedByVPA` and preserving the limit:request ratio. - **Observability is metrics + events, not traces** — six `[ALPHA]` kubelet resize metrics plus Pod events; correlate desired (kube-state-metrics) with actual (cAdvisor). - **Everything here is reproducible.** Pinned kind v1.35 cluster, pinned charts, and a Taskfile drive the whole demo; the example outputs in Section 10 come from a Kubernetes v1.35 kind cluster. --- ## 22. References Official and primary sources. **Kubernetes — In-Place Pod Resize (v1.35 GA)** - GA blog (2025-12-19): https://kubernetes.io/blog/2025/12/19/kubernetes-v1-35-in-place-pod-resize-ga/ - Task: Resize CPU and Memory Resources assigned to Containers: https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/ - Task: Resize CPU and Memory Resources assigned to Pods: https://kubernetes.io/docs/tasks/configure-pod-container/resize-pod-resources/ - Pod Conditions concept: https://kubernetes.io/docs/concepts/workloads/pods/pod-condition/ - KEP sig-node/1287 (In-Place Update of Pod Resources): https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/1287-in-place-update-pod-resources/README.md - Kubernetes Metrics Reference: https://kubernetes.io/docs/reference/instrumentation/metrics/ - System traces (API server / kubelet tracing): https://kubernetes.io/docs/concepts/cluster-administration/system-traces/ - KEP sig-node/2831 (kubelet tracing): https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2831-kubelet-tracing - About cgroup v2 (node cgroups): https://kubernetes.io/docs/concepts/architecture/cgroups/ **Vertical Pod Autoscaler** - Features (InPlaceOrRecreate, InPlace, gate, limitations, metrics, CPU boost): https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/docs/features.md - Quickstart (update modes, example): https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/docs/quickstart.md - Components (recommender/updater/admission-controller): https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/docs/components.md - Known limitations (HPA conflict): https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/docs/known-limitations.md - AEP-4016 (in-place updates support): https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support - AEP-7862 (CPU startup boost): https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler/enhancements/7862-cpu-startup-boost - AEP-8818 (eviction-free InPlace): https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler/enhancements/8818-in-place-only - Fairwinds charts (stable/vpa): https://github.com/FairwindsOps/charts **Observability stack** - kube-prometheus-stack chart: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/README.md - Prometheus Operator API reference: https://prometheus-operator.dev/docs/api-reference/api/ - kube-state-metrics pod metrics: https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/pod-metrics.md - Grafana dashboard JSON model: https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/view-dashboard-json-model/ - Grafana import dashboards: https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/import-dashboards/ - OpenTelemetry Collector on Kubernetes (Helm): https://opentelemetry.io/docs/platforms/kubernetes/helm/collector/ - OTel kubeletstats receiver: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/kubeletstatsreceiver/documentation.md - OTel k8scluster receiver: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver **Tooling** - kind: https://kind.sigs.k8s.io/ - metrics-server: https://kubernetes-sigs.github.io/metrics-server/ - Helm: https://helm.sh/ - Task (go-task): https://taskfile.dev/ --- ## 23. SEO Block **Meta Description (≤160 chars):** In-Place Pod Vertical Scaling is GA in Kubernetes 1.35. A production deep dive: resize subresource, resizePolicy, status conditions, VPA InPlaceOrRecreate, metrics — with verified output. **Keywords:** `in-place pod resize`, `Kubernetes 1.35`, `InPlacePodVerticalScaling`, `pod resize subresource`, `resizePolicy`, `RestartContainer`, `PodResizePending`, `PodResizeInProgress`, `Vertical Pod Autoscaler`, `InPlaceOrRecreate`, `VPA in-place`, `kubelet resize metrics`, `vertical scaling Kubernetes`, `right-sizing pods`, `cgroups v2 resize`, `kube-prometheus-stack`, `OpenTelemetry Collector kubelet`. **Slug:** `in-place-pod-vertical-scaling-kubernetes-1-35-production-deep-dive` **Twitter/X Title:** In-Place Pod Vertical Scaling in Kubernetes 1.35: the production deep dive (GA, no feature gate) — resize subresource, VPA InPlaceOrRecreate, real verified output. **Social Media Summary:** Kubernetes 1.35 makes cpu/memory requests & limits mutable on running Pods — no recreation. This deep dive covers the desired-vs-actual model, the resize subresource, resizePolicy (NotRequired vs RestartContainer), the new status conditions, VPA InPlaceOrRecreate, RBAC, and observability — all reproducible with pinned IaC and real verified command output. --- ## 24. LinkedIn Post > Kubernetes 1.35 quietly changes how we think about resource management: **In-Place > Pod Vertical Scaling is now GA — on by default, no feature gate.** > > For years, changing a Pod's CPU or memory meant deleting and recreating it. That's > fine for stateless web apps and painful for everything else: JVMs with long warm-ups, > game servers holding live sessions, pre-warmed workers, anything stateful. You either > over-provisioned "to be safe" or paid a restart to right-size. > > Now you patch the Pod's `resize` subresource and the kubelet reconfigures the running > container's cgroups in place. CPU scales with no restart. Memory can restart the > container in place (not the Pod) for runtimes that need it. > > I wrote a production deep dive — not a feature tour — covering: > • The desired-vs-actual model and why convergence ≠ "the patch returned 200" > • resizePolicy: NotRequired vs RestartContainer, and the QoS invariant > • The new PodResizePending / PodResizeInProgress conditions (the old string field is > gone) > • VPA InPlaceOrRecreate driving resizes automatically — with real verified output > • RBAC on pods/resize as a least-privilege seam > • Observability: the six kubelet resize metrics + events (and why there are no > traces) > • Every failure mode: Infeasible, Deferred, memory-decrease OOM risk, the > scheduler↔kubelet race > > Fully reproducible: pinned kind v1.35 cluster, pinned Helm charts, a Taskfile, and > command output captured on a live cluster. > > This is Article 1 of a 5-part series on next-gen Kubernetes resource management. > Next up: DRA and GPU scheduling. > > #Kubernetes #CloudNative #DevOps #PlatformEngineering #CNCF #SRE --- ## 25. 10 Advanced Topics That Build on This Article In-place resize is the vertical dimension of resource management: change *how much* a running Pod gets. The rest of this 5-part series moves outward — to *what kind* of resource, *when* Pods run together, and *how* large models are served. Ten topics that build on this foundation: 1. **Dynamic Resource Allocation (DRA) and GPU scheduling** — *(Article 2)* Requesting and sharing structured devices (GPUs, accelerators) via `ResourceClaim`/ `ResourceSlice`, and where DRA's device model meets (and differs from) the cpu/memory resize model in this article. 2. **In-place resize for accelerator-adjacent resources** — why GPUs are *not* resizable in place today, what would have to change, and how DRA reshapes that question. 3. **Gang scheduling and co-scheduling** — *(Article 3)* All-or-nothing placement for tightly-coupled Pods, and how vertical resize interacts with gangs whose members must grow together. 4. **Kueue and quota-aware batch admission** — how in-place `Deferred`/`Infeasible` signals should feed a job-queueing layer, and admission-time vs runtime right-sizing. 5. **Cluster Autoscaler / Karpenter awareness of resize pending signals** — turning `Deferred`/`Infeasible` resizes into node-provisioning decisions instead of stuck Pods. 6. **LLM inference serving and vertical scaling** — *(Article 4)* Right-sizing CPU/memory around model load, KV-cache growth, and batch dynamics for large-model serving, where recreation is especially expensive. 7. **llm-d and disaggregated serving** — *(Article 5)* Prefill/decode disaggregation and how per-phase resource profiles could exploit in-place resize for the CPU/host side of an accelerator-heavy serving stack. 8. **CPU Startup Boost (AEP-7862) in anger** — automating the startup-vs-steady-state CPU pattern from Section 14.2 across a fleet, built directly on the in-place scale-down. 9. **Pod-level resources resize** — moving from per-container to pod-level (alpha in v1.35) and what it means for sidecar-heavy and mesh workloads. 10. **Closing the scheduler↔kubelet consistency gap** — the design work behind #126891, and what a race-free resize/schedule interaction would enable for aggressive bin-packing. Article 2 picks up the thread with **DRA and GPU scheduling** — moving from "how much cpu/memory" to "which device, and how shared."