Keeping Docker Containers Running When a Deployment Worker Restarts

When you first build a deployment system, it's easy to accidentally tie the lifetime of a workload to the lifetime of the process managing it.

A worker starts. It receives a deployment job, builds the application, launches a Docker container, records that the deployment is running, and continues monitoring the system.

Then the worker restarts.

Now you have an interesting problem: the worker's in-memory knowledge is gone, but the container it launched may still be running perfectly well.

What should happen next?

Killing everything and recreating it would be simple, but it would also turn a harmless worker restart into an outage for every workload that worker manages. A better approach is to treat the running containers as independent runtime state that the worker needs to rediscover.

That's where reconciliation and container adoption become important.

The worker and the workload are different things

A deployment worker is responsible for orchestration. It receives work, builds applications, launches containers, updates deployment state, streams information back to the control plane, and performs lifecycle operations.

The application container is the actual workload.

Those two things shouldn't necessarily share the same lifetime.

Docker already separates container lifecycle from the shell or process that originally launched it, and restart policies can determine whether containers restart after an exit or Docker daemon restart. (Docker Documentation)

But keeping a container alive solves only part of the problem.

Suppose a worker restarts and Docker reports three application containers still running. The new worker process doesn't automatically know which deployments those containers belong to, whether they're supposed to remain running, or whether the control plane agrees with what exists on the machine.

The workloads survived. The worker's knowledge of them didn't.

Rediscovering reality after startup

A useful way to think about recovery is to separate recorded state from runtime state.

Recorded state might say:

Deployment A → running → Worker 1

Runtime state might say:

Container X → currently running on Worker 1

After a worker restart, those two views need to be compared.

The worker can inspect the containers that already exist on its host, identify the ones belonging to the deployment platform, determine which deployment each represents, and rebuild its understanding of the workloads it's responsible for.

Instead of immediately creating replacements, it adopts the containers that are already there.

This general idea isn't unique to deployment platforms. Reconciliation is fundamental to systems such as Kubernetes, whose controllers continually compare desired state with current state and act to bring them closer together. (Kubernetes)

The important part isn't copying Kubernetes. It's adopting the same useful principle:

Don't assume reality matches your database. Observe reality and reconcile the difference.

Container identity matters

Adoption only works if you can reliably determine what a container represents.

A random Docker container ID isn't enough.

The runtime needs some durable relationship between the container and the deployment that created it. That might involve deterministic naming, labels, deployment identifiers, project identifiers, revision information, or other metadata attached when the container is created.

Then recovery becomes much safer.

Instead of seeing:

container 91a34... is running

the worker can effectively understand:

this container belongs to deployment abc123 for project xyz

Now it has something it can compare against the control plane.

That identity becomes useful for much more than restarts. It also helps with lifecycle operations, cleanup, debugging, resource accounting, and detecting containers that no longer correspond to valid deployments.

Not every surviving container should be adopted

This is where recovery gets more complicated.

Finding a container doesn't automatically mean the platform should keep it.

Imagine the database says a deployment was stopped, but the container is still running. Or a deployment was replaced by a newer revision while an older container somehow survived. Or the control plane no longer recognizes the deployment at all.

Blindly adopting everything would preserve stale workloads.

Blindly deleting everything would destroy legitimate workloads.

So recovery needs rules.

For every discovered container, the worker needs enough information to answer questions such as: does this deployment still exist? Is this the worker that's supposed to own it? Is the deployment supposed to be running? Does the container correspond to the expected workload?

Only then can the worker decide whether the container should be adopted, ignored, stopped, or eventually cleaned up.

Recovery is really a state reconciliation problem

This was one of the more important lessons from building Deploy Hatch.

Initially, worker recovery sounds like a process-management problem: how do I restart the worker without killing the containers?

But that's only the first layer.

The deeper problem is:

How do I restore trustworthy agreement between the control plane and the runtime after one side temporarily disappears?

Once you frame it that way, the architecture starts looking different.

The worker shouldn't depend exclusively on whatever happened to be in memory before it restarted. It should be capable of rebuilding enough state from durable records and the actual runtime environment to continue operating.

That also means recovery should be idempotent. Restarting the worker once or five times shouldn't create five copies of the same application.

The goal is convergence, not replay.

Restart policies aren't the same as reconciliation

Docker provides restart policies such as always, unless-stopped, and on-failure, which are extremely useful for controlling what Docker does when a container exits or the daemon restarts. (Docker Documentation)

But a restart policy answers a narrower question:

Should Docker restart this container?

A deployment platform has to answer a different set of questions:

Should this workload exist at all? Which deployment owns it? Is its recorded state correct? Should the platform still expose it? Should it count against resource capacity? Should lifecycle controls operate on it?

Docker can keep the process alive. The orchestration layer still has to understand what that process means.

That's why container restart behavior and platform reconciliation complement each other rather than replace each other.

Test recovery by deliberately breaking things

Recovery logic is difficult to trust if you've only observed it during normal operation.

One of the most useful things we've done while building Deploy Hatch is intentionally interrupt pieces of the system and inspect what happens afterward.

Start workloads. Restart the worker. Check whether the containers survived. Verify that they're rediscovered. Verify that the platform still associates them with the correct deployments. Check resource accounting. Exercise lifecycle operations afterward. Look for duplicate containers and stale state.

Then do it again.

The interesting bugs tend to appear on the second and third recovery cycle, not necessarily the first.

It's easy to build software that reaches the correct state once. Infrastructure gets much more interesting when it has to repeatedly find its way back to that state after interruptions.

Reliability isn't the absence of restarts

Workers will restart. Hosts will eventually reboot. Processes will crash. Networks will temporarily disappear. Deployments will fail halfway through operations.

Trying to design a system where none of those things ever happen isn't realistic.

The better goal is to make individual interruptions unsurprising.

For Deploy Hatch, that has meant treating customer workloads as something the worker can rediscover and reconcile, rather than something that exists only because the current worker process remembers creating it.

The result is a simple principle that applies well beyond container hosting:

Don't make uptime depend on one process remembering everything. Make the system capable of reconstructing the truth.