Same Orchestrator, Different Rules
Teams that have run Kubernetes for years often assume AI workloads are just another deployment — write a manifest, set a few resource limits, let the scheduler handle the rest. That assumption breaks quickly. A typical microservice is stateless, cheap to replicate, and forgiving if the scheduler makes an imperfect placement decision. A model-serving pod is none of those things. It's expensive to run, slow to start, and a single bad scheduling decision can waste GPU hours that a web app equivalent would never notice. Kubernetes still works as the orchestration layer — most teams running AI in production have converged on it rather than a specialist alternative — but almost everything about how you configure it changes once GPUs and models enter the picture.
GPU Scheduling Is Not Pod Scheduling
The default Kubernetes scheduler works at the level of an individual pod, deciding one placement at a time. That's fine for a web app. It's a real problem for anything involving multiple GPU pods that need to run together — a distributed training job with eight worker pods, for instance. The scheduler might successfully place seven of them and get stuck on the eighth because that node's GPU is occupied elsewhere. The seven placed pods now sit idle, holding GPU resources while contributing nothing, because the job as a whole can't start without all eight. This is called partial admission, and it's one of the more expensive mistakes a GPU cluster can make silently, day after day, if nobody's specifically defending against it.
The fix is gang scheduling — treating the group of pods as a single unit that's either fully admitted or not admitted at all — handled through tools purpose-built for this, like Kueue, rather than the default scheduler's pod-by-pod logic. Kueue also does something else standard Kubernetes scheduling can't: fair-share GPU allocation across teams sharing a cluster, with quotas, priority-based preemption, and the ability for one team to temporarily borrow another's unused capacity. Done well, this is the difference between GPU utilization sitting in the 25–35% range and pushing past 60–85% on the same hardware.
Nodes Need to Know What They Are — And Guard It
GPU nodes are expensive, and an unprotected one will accept whatever the scheduler throws at it — including workloads that have no business being there. Without a node taint explicitly reserving GPU nodes for GPU workloads, it's entirely possible for a logging sidecar or a static-file server to land on a node with an H100 attached, quietly burning premium compute on something that could run anywhere. Tainting GPU nodes and tolerating only genuine GPU workloads onto them is a small piece of configuration with an outsized effect on cost. The same logic applies to model placement: GPU feature discovery can label nodes with hardware specifics — model, memory, CUDA version — so a 70-billion-parameter model gets scheduled onto the high-memory nodes it actually needs, while a smaller model runs on cheaper hardware, instead of everything competing for the same pool by default.
Autoscaling Breaks in a Specific Way
The standard Horizontal Pod Autoscaler scales on CPU and memory. For GPU inference, that signal is close to useless — GPU utilization spikes to 100% during an active batch and drops immediately after, in a pattern CPU-based autoscaling was never designed to read. Effective autoscaling for model serving needs to key off different signals entirely: request queue depth, KV cache utilization, or GPU-specific metrics rather than generic pod resource consumption.
Scale-to-zero is where this gets genuinely difficult. It's an appealing cost lever — a cold GPU node costs nothing — but the sequence to bring a scaled-to-zero model back online involves provisioning a new node, installing GPU drivers, pulling a large inference image, and loading the model into GPU memory. End to end, that's commonly five to ten minutes before the first request can be served. For a real-time, user-facing API, that's not a minor delay — it's an outage from the user's perspective. The workable middle ground most teams land on is keeping a minimum of one GPU node warm during business hours and only scaling to zero overnight or on weekends, which still captures most of the cost saving without exposing live traffic to a cold start.
Consolidation policy matters here too, and it's an easy thing to get backwards. Node consolidation settings that evict pods from underutilized nodes to save cost need to be configured carefully for inference workloads — a policy that consolidates based on utilization, rather than strictly on an empty node, can terminate a node mid-inference to save cost, killing in-flight requests in the process. For anything serving live traffic, consolidation should only happen when a node is genuinely idle, not merely underused.
Rollouts Need an Extra Dimension
For a typical microservice, a rolling update is usually enough — replace pods gradually, watch for basic health check failures, and you're done. Model rollouts need to check something rolling updates were never built to check: whether the new version's actual outputs are still good. A model can pass every standard health check — the pod is running, the endpoint responds, latency looks normal — while quietly producing worse answers, hallucinating more often, or costing significantly more per request than the version it's replacing. None of that shows up in a liveness probe.
This is where canary rollouts earn their keep for AI specifically. Route a small slice of real production traffic to the new model version, hold it there, and evaluate it against metrics a normal deployment would never track: response quality, hallucination rate, token cost, task success rate — not just error rate and response time. If those hold up, increase the traffic share in steps; if they don't, route everything back to the stable version immediately. Blue-green deployment is the right call for a different situation — sweeping changes where you want an instant, clean cutover with a fast rollback path, at the cost of running double the resources during the transition. On expensive GPU infrastructure, that doubled resource cost is a real consideration, which is part of why canary rollouts tend to be the default for model updates specifically, with blue-green reserved for larger structural changes.
Cold Starts Deserve Their Own Line Item
Large model images are big, and pulling one from scratch on every new node is slow — a 70-billion-parameter model can take twenty minutes or more just to download before it's even loaded into memory. Teams that don't plan for this end up with GPU nodes finishing provisioning and then sitting idle for the better part of that time. Pre-warming — pre-pulling model weights to a persistent volume via an init container rather than downloading from scratch on every cold start — turns a ten-plus-minute wait into something manageable, and it's the kind of detail that separates a demo deployment from one that survives real production traffic.
The Common Thread
None of this replaces standard Kubernetes practice — it extends it. The core skills still apply: manifests, health checks, namespaces, network policies. What changes is that every one of those primitives now has to account for a workload that's expensive to run, slow to start, and capable of failing in ways a status code will never reveal. Treating a model-serving deployment like any other microservice is how teams end up with idle GPU spend, partial-admission jobs that never start, and rollouts that "succeed" while quietly serving worse answers.
How We Approach This at Rakri AI
When we run production workloads on Kubernetes — including the AKS clusters underneath the Suri ecosystem — we operate the cluster directly: pod scaling, rollouts, and cluster-level operations included, not handed off to a managed black box. That means GPU scheduling, autoscaling behavior, and rollout strategy get configured for what the workload actually is, rather than left on defaults that were designed for stateless web services. Treating a model deployment as if it were interchangeable with any other pod is usually where AI infrastructure costs quietly get out of control — and where quality regressions slip through unnoticed until a user reports them.
