Local Kubernetes Dev — Part 14: Preparing for deployment and CI

One set of manifests, three environments: dev, staging, prod

The temptation is obvious: copy the k8s/ folder into k8s-prod/, change the replica count, the domain, and the image tag. Do it once — tolerable. A month later you have three nearly identical copies, in one of which someone forgot to fix the memory limit, and production's behavior quietly drifts away from what you tested. This is called configuration drift, and you fight it with tools that keep a single base and layer only the differences on top.

Environments usually differ in predictable places:

• replica count (replicas);
• the image tag;
• resources (requests/limits);
• the database address and other settings;
• the Ingress domain.

There are two popular approaches to factoring out these differences. The basics of configMapGenerator (Kustomize) and values files (Helm) were covered in chapter 10 — here we look at them specifically through the lens of "multiple environments and CI," without repeating the fundamentals.

Helm: templates + values

Helm is a templating engine for Kubernetes. Manifests become templates (Go templates), and the concrete values are substituted from values.yaml. Each environment gets its own values file that overrides the defaults.

1# chart/values.yaml — defaults (as for dev)
2replicaCount: 1
3image:
4  repository: k3d-registry.localhost:5000/myapp
5  tag: dev
6resources:
7  requests: { cpu: 50m, memory: 128Mi }
8ingress:
9  host: myapp.localhost

1# values-prod.yaml — only what differs in production
2replicaCount: 3
3image:
4  repository: ghcr.io/acme/myapp
5resources:
6  requests: { cpu: 250m, memory: 256Mi }
7  limits:   { cpu: 500m, memory: 512Mi }
8ingress:
9  host: api.myapp.com

Here you can clearly see the transition from the running example to production: locally the image lives at k3d-registry.localhost:5000/myapp:dev, while in production it's in a real registry, ghcr.io/acme/myapp, with an immutable tag. More on exactly what changes during the move in the section on local versus production below.

Deploying to production means layering one file on top of another:

1helm upgrade --install myapp ./chart \
2  -f chart/values.yaml \
3  -f values-prod.yaml \
4  --set image.tag=sha-abc123   # the last source overrides the previous ones

An important point about precedence. The Helm documentation describes the order like this: values.yaml (default) → parent chart's values → a user file via -f → --set. Each subsequent one overrides the previous, and when you use multiple -f flags, order matters — the last file wins. This is a common pitfall: swap the -f flags around and you get an unexpected replica count in production.

Kustomize: patches on top of a base

Kustomize comes at it from the other end: no templates, just plain YAML and patches on top of it. It's built into kubectl, so there's nothing extra to install. The structure is a shared base/ plus overlays/ on top:

1k8s/
2  base/
3    deployment.yaml
4    service.yaml
5    kustomization.yaml
6  overlays/
7    dev/
8      kustomization.yaml
9    prod/
10      kustomization.yaml
11      replicas-patch.yaml

1# overlays/prod/kustomization.yaml
2resources:
3  - ../../base
4namePrefix: prod-
5commonLabels:
6  env: prod
7images:
8  - name: myapp            # swaps the image without editing the Deployment
9    newName: ghcr.io/acme/myapp
10    newTag: sha-abc123
11patches:
12  - path: replicas-patch.yaml

It's applied like this:

1kubectl apply -k overlays/prod/
2# or build it and see what you'll get before applying:
3kustomize build overlays/prod | kubectl apply -f -

Note the images field — it's a clean way to swap the image tag without touching the Deployment itself. It comes in handy in CI (see the sections on image tags and the CI pipeline). Besides it, kustomization.yaml also holds resources, patches, namePrefix, commonLabels, configMapGenerator, and other transformers.

Which to choose? Helm is more convenient when you need parameterization and reuse (especially for third-party applications — the databases and caches from chapter 9 are often installed precisely as Helm charts). Kustomize is simpler and more transparent for your own services: you see exactly the YAML that will go to the cluster. A hybrid is common too: install third-party things with Helm, describe your own with Kustomize overlays. Both approaches are understood by Argo CD and Flux alike — more on them in the section on GitOps.

Image tags: why latest is painful

The most common beginner mistake is the latest tag. It seems convenient: always the "freshest" image. In reality, in production it's a source of hard-to-track problems, and the Kubernetes documentation explicitly recommends avoiding it: with :latest it's "harder to track which version is running and harder to roll back."

Let's break down the mechanics of the pain:

1. latest is mutable. A tag's contents change silently — today latest is one set of code, tomorrow it's another. Zero reproducibility.
2. Re-running apply doesn't trigger a rollout. Kubernetes triggers a rollout only if the pod template changed. The line image: myapp:latest didn't change — so as far as the cluster is concerned there's nothing to deploy, even though the image in the registry is already different.
3. A pod restart may pull a different image. If a pod crashes and comes back up, it may pull a new (possibly broken) version of latest — without any deploy on your part. Production changes its version "by itself."

Related to this is imagePullPolicy. Its default depends on the tag: for latest and untagged images it's Always, for a specific tag it's IfNotPresent, and for a digest it's IfNotPresent as well. With local images, IfNotPresent can do the opposite and "stick" on an old version — more on that in the chapter on troubleshooting.

The right way. Immutable tags that unambiguously point to a specific build:

• by short commit SHA: sha-abc1234;
• by semver: v1.2.3;
• for maximum reproducibility — pin by digest:

1image: ghcr.io/acme/myapp@sha256:45b23dee...

A digest guarantees that "the exact same code runs every time." In CI you don't need to generate any of this by hand — there's docker/metadata-action, which produces tags from the git context (type=sha, type=ref, type=semver) and also writes OCI labels like org.opencontainers.image.revision (commit SHA), .source, .version. This ties the image to a specific commit — so later it's easy to figure out exactly what's running in production.

A basic CI pipeline: build → push → apply

The simplest deployment model is called push-based: a commit lands in the repository → CI builds the image → pushes it to the registry → CI then updates the manifest and applies it to the cluster. No magic, all linear.

On GitHub Actions, this is assembled from the official docker actions. Here's a working skeleton for myapp:

1# .github/workflows/deploy.yml
2name: build-and-deploy
3on:
4  push:
5    branches: [main]
6
7jobs:
8  deploy:
9    runs-on: ubuntu-latest
10    permissions:
11      contents: read
12      packages: write
13    steps:
14      - uses: actions/checkout@v4
15
16      - uses: docker/setup-buildx-action@v3   # BuildKit: cache, multi-arch
17
18      - uses: docker/login-action@v3          # log in to the registry
19        with:
20          registry: ghcr.io
21          username: ${{ github.actor }}
22          password: ${{ secrets.GITHUB_TOKEN }}
23
24      - id: meta
25        uses: docker/metadata-action@v5       # immutable tags from git
26        with:
27          images: ghcr.io/acme/myapp
28          tags: |
29            type=sha
30            type=ref,event=branch
31            type=semver,pattern={{version}}
32
33      - uses: docker/build-push-action@v6      # build + push
34        with:
35          push: true
36          tags: ${{ steps.meta.outputs.tags }}
37          labels: ${{ steps.meta.outputs.labels }}
38
39      - name: Deploy
40        run: |
41          echo "${{ secrets.KUBECONFIG }}" > kubeconfig
42          export KUBECONFIG=kubeconfig
43          kubectl set image deployment/myapp \
44            myapp=ghcr.io/acme/myapp:sha-${GITHUB_SHA::7}

In practice, the deploy step is done in different ways:

1# the Kustomize variant
2kustomize build overlays/prod | kubectl apply -f -
3
4# the Helm variant
5helm upgrade --install myapp ./chart -f values-prod.yaml \
6  --set image.tag=sha-abc123
7
8# update only the image, surgically
9kubectl set image deployment/myapp myapp=ghcr.io/acme/myapp:sha-$GIT_SHA

The key thing is that the deploy updates the image to an immutable tag, not to latest. Then the new pod template differs from the old one, and a rollout honestly happens.

The push model has built-in weaknesses worth knowing in advance:

• CI holds the keys to the cluster (a kubeconfig or cloud credentials). This widens the attack surface: anyone with access to the CI secrets gets access to production.
• Drift between Git and the cluster is possible. Someone ran kubectl edit by hand — and the cluster's state diverged from what's recorded in the repository. CI never finds out.
• No automatic rollback. If a deploy ships a broken version, you'll have to revert everything by hand.

It's precisely these three points that nudge you toward GitOps (see the next section).

What moves over from local, and what's production-only

The good news: most of your work from the previous chapters gets reused. That's the whole point of a production-like local setup (chapter 2) — environment parity.

Reused as is:

• the same Dockerfile and the same image (chapter 6) — the built artifact is the same across all environments;
• the same Helm charts or Kustomize base — only the values/overlays change;
• liveness/readiness probes (chapter 13) — needed both locally and in production;
• if you use Skaffold — a single skaffold.yaml with profiles. Skaffold profiles are layered on top of the base section (by replacing fields or with a JSON patch) and activated via -p, an environment variable, or the kube-context. The authors' idea is straightforward: "use the same commands locally as for a remote deploy."

Added only for production:

• a real registry and immutable tags instead of k3d-registry.localhost:5000 and dev;
• production resources: real requests/limits, HPA (horizontal autoscaling), PodDisruptionBudget, anti-affinity, multiple replicas;
• real Secrets, TLS (for example, via cert-manager), a production Ingress and DNS (chapter 11);
• monitoring, NetworkPolicy, RBAC;
• production probe timings (local values are usually more aggressive).

And separately — what you must NOT carry into production. All the dev accelerators live only in the local setup:

• Live Update and file-sync from Tilt (chapter 8);
• hot reload (uvicorn --reload) — in production uvicorn runs without it;
• exposed debug ports and dev tooling.

These things are handy in the inner dev loop, but in production they're a security hole and a source of instability. Skaffold profiles, a separate values-prod, or a prod overlay are exactly what you need to make sure dev-only mechanisms physically don't end up in the production manifests.

GitOps as the next step

The push model from the previous section works, but its weaknesses (keys in CI, drift, manual rollback) haven't gone anywhere. The next rung on the evolutionary ladder is GitOps.

The idea is simple: Git becomes the single source of truth. You don't push changes to the cluster from CI — instead, an agent lives inside the cluster (Argo CD or Flux) that itself pulls changes from Git and continuously brings the cluster into the state described in the repository.

Argo CD describes itself as "a declarative GitOps continuous delivery tool for Kubernetes." It works as a controller: it "continuously monitors running applications and compares the current, live state against the desired target state." If someone fixed the cluster by hand, the agent will see the discrepancy (drift) and bring everything back to what's recorded in Git. It understands Helm, Kustomize, and plain YAML alike.

Push (CI) versus pull (GitOps):

The chain ends up looking like this: CI builds an image with an immutable tag and pushes it to the registry → a bot (or the same CI) updates the tag in the Git manifest (image.tag in Helm values or newTag in Kustomize's images) → the GitOps agent notices the commit and applies the change to the cluster. CI no longer holds a kubeconfig — its job ends at pushing the image and committing to the manifests.

Argo CD and Flux are both graduated CNCF projects. Argo CD gives you a rich web UI and an app-centric model; Flux is more k8s-native, built from a set of CRD controllers. For a first encounter, Argo CD is usually easier thanks to its visual interface.

We won't be setting up GitOps in this article — that's a topic for a separate piece. But it's worth knowing about already: when your push pipeline starts running into drift and manual rollbacks, you'll understand where to head next.

In summary: a single manifest base + per-environment overlays (Helm values or Kustomize overlays), immutable tags instead of latest, a simple build → push → apply pipeline, and a deliberate split between "what moves over from local and what stays production-only." And beyond that — GitOps, once you grow into it. In the next chapter we'll go through the common problems you run into along the way.

Sources

• Kubernetes Docs — Images (the latest tag, imagePullPolicy defaults, digests)
• Helm Docs — Values Files
• Kustomize — Kustomization File Reference
• docker/build-push-action (GitHub)
• docker/metadata-action (GitHub)
• Argo CD — Declarative GitOps CD for Kubernetes
• Skaffold Docs — Profiles