Local Kubernetes Dev — Part 10: Configuration and secrets

ConfigMap: non-sensitive settings and environment variables

A ConfigMap is a Kubernetes API object for storing non-sensitive data as key-value pairs. Its job is to make the container image portable: all environment-dependent configuration lives outside the image rather than inside it (ConfigMaps, kubernetes.io).

A few things to know about ConfigMaps right away:

• It does not provide secrecy. The docs say this directly: for confidential data you need a Secret, not a ConfigMap.
• The size limit is 1 MiB. If your configuration is larger, you'll have to mount a volume or use external storage.
• It has two fields for data: data (UTF-8 text) and binaryData (binary data in base64).
• A ConfigMap and the Pod that uses it must be in the same namespace (a namespace is a logical partition of the cluster; for us that's myapp).
• You can make a ConfigMap immutable with the immutable: true field (available since Kubernetes v1.19). This protects against accidental edits and reduces load on the kube-apiserver.

For myapp, let's move everything non-sensitive into a ConfigMap: the log level, the database name, and the PostgreSQL host and port.

1# configmap.yaml
2apiVersion: v1
3kind: ConfigMap
4metadata:
5  name: myapp-config
6  namespace: myapp
7data:
8  LOG_LEVEL: "debug"
9  DB_HOST: "postgres"
10  DB_PORT: "5432"
11  DB_NAME: "myapp"

Apply it and check:

1kubectl apply -f configmap.yaml
2kubectl -n myapp get configmap myapp-config -o yaml

You can create the same thing imperatively, without writing a single line of YAML — handy for quick checks:

1kubectl -n myapp create configmap myapp-config \
2  --from-literal=LOG_LEVEL=debug \
3  --from-file=config.properties

The --from-literal flag adds a single key-value pair; --from-file takes the contents of a file (the file name becomes the key, the contents become the value).

Secret: passwords/keys/tokens and why base64 ≠ encryption

A ConfigMap won't do for the PostgreSQL password — you need a Secret. Structurally it's almost the same (key-value pairs), but it's meant for sensitive data and is handled a little more carefully inside the cluster.

And right away — the biggest misconception among beginners. Values in a Secret are stored in base64, and many people think that means they're "encrypted." They are not. base64 is encoding, not encryption: it's reversible with a single command and gives you no confidentiality whatsoever.

1echo 'czNjcjN0' | base64 -d   # prints: s3cr3t

What's more, the Kubernetes docs say it plainly: "Kubernetes Secrets are, by default, stored unencrypted in the API server's underlying data store (etcd)" — meaning that by default secrets sit in the cluster's data store (etcd) without encryption, just base64-encoded (Secrets, kubernetes.io). Their contents are accessible to anyone with read permissions through the API, access to etcd, or the ability to create Pods in that namespace.

Let's create a Secret for the database password. To avoid fiddling with manual base64 encoding, we'll use the stringData field — the API encodes the values itself when it saves them:

1# secret.yaml
2apiVersion: v1
3kind: Secret
4metadata:
5  name: myapp-db-secret
6  namespace: myapp
7type: Opaque
8stringData:
9  DB_USER: "myapp"
10  DB_PASSWORD: "s3cr3t"

type: Opaque is the default type for arbitrary user-defined data. Besides it, there are specialized types: kubernetes.io/dockerconfigjson (for access to private registries), kubernetes.io/tls (TLS certificates), kubernetes.io/basic-auth, kubernetes.io/ssh-auth, and others — Kubernetes validates their structure.

The same Secret imperatively (this will come in handy later for sealed secrets):

1kubectl -n myapp create secret generic myapp-db-secret \
2  --from-literal=DB_USER=myapp \
3  --from-literal=DB_PASSWORD='s3cr3t' \
4  --dry-run=client -o yaml

The --dry-run=client -o yaml flag doesn't create anything in the cluster; it prints a ready-made manifest to stdout — handy when you want the YAML but don't want to apply it right away.

Since base64 doesn't protect anything, what measures do you actually need (especially in prod)? The official good practices for Secrets recommend:

1. Enable Encryption at Rest — encryption of secrets in etcd. Without it, in prod they sit in plaintext (well, base64).
2. Configure RBAC following the principle of least privilege. An important subtlety: the list permission on secrets implicitly exposes their contents, so you can't hand it out left and right.
3. Restrict access to a Secret to the specific containers that actually need it.
4. For serious workloads, move secrets into external stores (HashiCorp Vault, cloud key vaults) via the Secrets Store CSI Driver.

Feeding values into a Pod: env, envFrom, mounting as files

Creating a ConfigMap and a Secret is only half the job. Now you need to deliver their values into the container. There are three main ways.

Way 1: a single variable from a single key (env + valueFrom)

When you need to feed a specific key in under a specific variable name:

1env:
2  - name: LOG_LEVEL
3    valueFrom:
4      configMapKeyRef:
5        name: myapp-config
6        key: LOG_LEVEL
7  - name: DB_PASSWORD
8    valueFrom:
9      secretKeyRef:
10        name: myapp-db-secret
11        key: DB_PASSWORD

Way 2: all keys at once (envFrom)

If you want to feed an entire ConfigMap/Secret at once, without listing every key:

1envFrom:
2  - configMapRef:
3      name: myapp-config
4  - secretRef:
5      name: myapp-db-secret
6      optional: true
7    prefix: DB_

Here prefix prepends a prefix to all variable names from this source, and optional: true means "don't fail if the source isn't there." By default optional is false — and if a Pod references a non-existent ConfigMap or Secret, it simply won't start (Configure a Pod to Use a ConfigMap, kubernetes.io).

Way 3: mounting as files (volume)

Sometimes it's more convenient for an application to read config from a file rather than from an environment variable (a TLS certificate, for example). In that case you mount the ConfigMap or Secret as a volume: each key becomes a file, the key name becomes the file name, and the value becomes the contents.

1volumes:
2  - name: db-secret-vol
3    secret:
4      secretName: myapp-db-secret
5volumeMounts:
6  - name: db-secret-vol
7    mountPath: /etc/secrets
8    readOnly: true

After this, the files /etc/secrets/DB_USER and /etc/secrets/DB_PASSWORD will appear inside the container. With the items[].path and items[].mode fields you can override the file names and permissions.

The critical difference: env doesn't update, a volume does

This is the trap almost everyone runs into. Environment variables are fixed at Pod startup and are not updated if you change a ConfigMap or Secret. The Pod will keep running with the old values until you restart it:

1kubectl -n myapp rollout restart deployment/myapp

A volume mounted from a ConfigMap/Secret, on the other hand, updates automatically (with a small delay for the kubelet to sync) — but only if the mount is done without subPath. There's a caveat here too: the application itself has to be able to re-read the file; k8s only updates its contents on disk.

For myapp on FastAPI this means: we feed the log level and connection string via env (that's fine — they change rarely, and the Pod gets restarted anyway when a new image is deployed). And when you want a "hot" config swap without a restart, you use a volume mount and read the file at runtime.

How not to commit a secret into git

The most common and most painful blunder is committing a real Secret into the repository. As a reminder: base64 is not protection, so a Secret manifest with a real password in git is equivalent to leaking the password. And git remembers everything: even if you delete the file later, the value stays in the history. The official good practices explicitly advise not to store Secret manifests in version control (good practices, kubernetes.io).

The minimum for a beginner: templates + .gitignore

Commit only templates without real values into git, and keep the actual files outside git.

1# .gitignore
2secret.yaml
3*.secret.yaml
4.env

1# secret.example.yaml — we commit this file; it has no secrets
2apiVersion: v1
3kind: Secret
4metadata:
5  name: myapp-db-secret
6  namespace: myapp
7type: Opaque
8stringData:
9  DB_USER: "CHANGE_ME"
10  DB_PASSWORD: "CHANGE_ME"

A colleague clones the repository, copies secret.example.yaml to secret.yaml, fills in the real values — and git never sees them.

For prod: Sealed Secrets

But what if you genuinely want to store secrets in git — for example, for GitOps, where the entire desired state of the cluster lives in the repository? Then the secret needs to be encrypted before committing. A common solution is Sealed Secrets from Bitnami (the good practices mention it alongside the External Secrets Operator and SOPS).

It works on asymmetric cryptography. A controller with a key pair is installed into the cluster. The kubeseal CLI utility encrypts your Secret with the public key and produces a new object — a SealedSecret. Only the controller can decrypt it, using its private key, already inside the cluster. That's why a SealedSecret is safe to commit even to a public repository: no one can decrypt it, including you yourself.

A typical workflow:

1# create a regular Secret as a manifest, but instead of applying it, encrypt it
2kubectl -n myapp create secret generic myapp-db-secret \
3  --from-literal=DB_PASSWORD='s3cr3t' \
4  --dry-run=client -o yaml \
5  | kubeseal -o yaml > sealed-db-secret.yaml
6
7# sealed-db-secret.yaml is safe to commit
8git add sealed-db-secret.yaml && git commit -m "add sealed db secret"
9
10# in the cluster, the controller will unfold it into a real Secret on its own
11kubectl apply -f sealed-db-secret.yaml

A few things to keep in mind:

• The encryption is tied to the namespace + name pair: a SealedSecret created for one namespace/name won't decrypt under another.
• The controller's private key is just a regular Secret in its namespace. Keys are rotated (every 30 days by default), and the old ones are kept. Be sure to back up the private key: lose it, and your already-committed SealedSecrets turn into unreadable garbage.

For local dev on k3d, Sealed Secrets are usually overkill — templates and .gitignore are enough. But it's useful to know the direction: this very same repository will go to CI and to prod (see the chapter on preparing for deployment and CI).

Different values for different environments without duplication

Locally, myapp talks to PostgreSQL inside the cluster with a stub password and LOG_LEVEL=debug, while in prod it talks to a managed database with a real password and LOG_LEVEL=info. Copying all the manifests into two folders just for that and keeping them in sync is a straight path to divergence. There are two approaches to avoid this. Here we'll focus on what's unique to configs — the configMapGenerator/secretGenerator generators; the general mechanics of splitting things across environments in tandem with CI are covered in more detail in the chapter on preparing for deployment.

Kustomize (built into kubectl)

The idea behind Kustomize: there's a base with shared resources and overlays for each environment, which contain only the differences, in the form of patches.

1k8s/
2  base/
3    kustomization.yaml
4    deployment.yaml
5    configmap.yaml
6  overlays/
7    dev/
8      kustomization.yaml
9    prod/
10      kustomization.yaml

1# overlays/dev/kustomization.yaml
2resources:
3  - ../../base
4configMapGenerator:
5  - name: myapp-config
6    behavior: merge
7    literals:
8      - LOG_LEVEL=debug

This is where configMapGenerator and secretGenerator come into play — they generate a ConfigMap/Secret from literals, files, or env files. Their killer feature is a content-based hash in the name suffix: when the contents change, the object's name changes (for example, myapp-config-7c8f...), and Kustomize automatically updates the references in the Deployment. Name changed → Deployment changes → k8s does a rolling update on its own. In other words, the "I changed the ConfigMap but the Pod is running on the old env" problem (see the section above) is solved for you by Kustomize (configMapGenerator, Kustomize reference).

1kubectl kustomize overlays/dev | kubectl apply -f -
2# or, more concisely:
3kubectl apply -k overlays/dev

If the hash suffix gets in your way (for example, the name is referenced from outside), you disable it with disableNameSuffixHash: true — but then there's no automatic restart when the config changes. The behavior field controls what the generator does with a same-named resource from the base: create (the default), replace, or merge.

Helm (if you use charts)

If you package myapp into a Helm chart (packaging myapp into a chart and per-environment values), the same principle is implemented through values files: one chart + a values.yaml with defaults + one override file per environment, containing only the differences.

1# values.yaml — defaults
2logLevel: info
3image:
4  tag: latest

1# values-dev.yaml — only what differs
2logLevel: debug

1helm install myapp ./chart -f values.yaml -f values-dev.yaml
2helm upgrade myapp ./chart -f values-dev.yaml --set image.tag=abc123

You can specify the -f flag multiple times — the files are merged top to bottom, and on a conflict the last one wins. The full order of precedence (from lowest to highest): the chart's default values.yaml → parent chart values → files from -f → --set flags (Values Files, Helm Docs). To remove a key from the defaults, assign it null.

Which approach to choose is a matter of taste and of what's already adopted on your team. Kustomize doesn't require templates and comes "in the box" with kubectl; Helm gives you full templating and packaging. Both solve the same problem: one description of the service, different values for different places — without copy-paste.

What to remember

• ConfigMap is for non-sensitive data, Secret is for passwords and keys. A ConfigMap provides no secrecy, has a 1 MiB limit, and must be in the same namespace as the Pod.
• base64 in a Secret is encoding, not encryption. By default, secrets sit in etcd unencrypted. In prod, enable Encryption at Rest and configure RBAC.
• Three ways to deliver: env (a single key), envFrom (all keys), volume mount (files). env doesn't update without a Pod restart; a volume does (without subPath).
• Never commit real secrets. At a minimum — templates + .gitignore; for prod — Sealed Secrets / External Secrets / SOPS.
• Different environments without duplication — Kustomize (base + overlays, a generator with a hash suffix) or Helm (one chart + values-*.yaml).

In the next chapter we'll tackle networking: how to reach myapp from outside the cluster and set up Ingress.

Sources

• ConfigMaps | Kubernetes
• Secrets | Kubernetes
• Good practices for Kubernetes Secrets | Kubernetes
• Configure a Pod to Use a ConfigMap | Kubernetes
• configMapGenerator | Kustomize Reference
• Values Files | Helm Docs
• bitnami-labs/sealed-secrets | GitHub