The Only Reliable Way to Control Anything Is a Loop

2026-09-24 · Based on a talk at Cloud Native Aarhus, "Bridging the Gap between OT and IT", September 2026

Look around you. Everything you see was produced somehow.

The apple juice in your fridge.

If you have a baby, you have probably learned which companies make baby food.

If you have a car, there are small fasteners from Böllhoff holding it together.

We at UMH build software for the companies that run the lines behind all of it. I am Jeremy, co-founder and CTO of UMH, and I have been shipping industrial software for nearly a decade.

So how do these production lines actually work?

A production line is built entirely from control loops

A production line is dozens of machines from dozens of vendors, wired together, all of them controlling each other. It is a distributed system, but not like any distributed system you have worked on. And it consists entirely of control loops.

Control loops. But what are control loops? I know it looks academic. Don’t worry, we will make it more practical in a second.

You have a setpoint. You have a sensor measuring whatever you have. You make the difference. It goes into a controller doing some very advanced mathematics. Then you control some plant or system, where you can have disturbances. And then you have the controlled variable, which again gets measured by the sensor, and you have a loop.

For example, a room thermostat. You set a target temperature of 21 degrees, that is the setpoint. The thermometer in the room measures 18 degrees, that is the controlled variable. You subtract: 21 minus 18, that is an error of 3. Then, based on some mathematics, P, PI, PID, or totally different ones, the controller says: we need to heat more, set the heating to 80 percent. And the heating influences the room. And other things are happening, someone opens the window, whatever, so you have disturbances. Then you measure again with the thermometer, and you have a loop.

And fun fact, if you have ever run Kubernetes, it is also based on exactly these control loops. But we will get to that further down.

A factory is those loops stacked on top of each other. Below you see the model the industry uses for it, the automation pyramid. Every factory is different, but it does not matter which factory you go to, it roughly looks like this.

A customer orders 3,000 bolts for Friday. That lands in the ERP, enterprise resource planning, the system where the invoices are written and the warehouse is managed. Maybe you have seen some Odoo commercials, which is an ERP for small companies, most of the larger ones just use SAP.

The ERP hands it down to the MES, the fine planning for manufacturing: line three, 500 per shift. Below that sits the SCADA, one per line, which turns that into a machine speed, 1,200 RPM. And at the bottom, the PLC (programmable logic controller) controls the actual physical components.

Then everything runs in reverse. The PLC reports the actual speed, the layers report upward, the MES may reschedule to another line if line three has a mechanical failure, and the ERP sees 2,400 done, on track, without knowing anything about how a bolt gets made. Every layer is a control loop wrapped around the one beneath.

Let’s take a closer look at the one at the bottom, the PLC.

Above is a Siemens S7-1200 PLC, a popular model in Europe. At the top the inputs come in (e.g., AI0, or via extra IO modules), inside runs the program, at the bottom the outputs control something (e.g., DQ0), a heating element, a motor.

The same loop as before, now on the PLC: the setpoint comes from the SCADA, the temperature sensor hangs on the analog input, the heating element on the digital output.

Every few milliseconds it reads all its inputs, runs the whole program from top to bottom, writes all its outputs, and starts again, which is called the scan cycle.

It even has its own programming languages. You cannot just pick up a PLC and program it the way you program anything else. You program it the way you would draw an electrical circuit diagram, because it is safety-critical, and you do not want to give the programmer all the options.

And all of these loops in the factory run at once, in real time, and a lot of them are safety-critical. If you are ever in the need to press this button, you want it to work. Always. Even if someone is updating a virtual machine, or Cloudflare has its next outage.

Software here has to hold to the same safety standards you know from mechanical engineering. If Cloudflare goes down in the IT world, you lose some revenue and it is annoying. If this button does not work, people can die.

So here is the question I find most useful for explaining our world. Who keeps this distributed system running?

Option A: a PhD in distributed systems, with a full team of IT engineers behind them. Option B: two electricians from the next town over.

Pick one before you read on.

It might already be in the way I asked the question. It is B, two electricians from the next town over. Maintaining this whole system is a three-and-a-half-year apprenticeship. You can even study it, but even then the term distributed system never comes up.

And how are they able to manage it? Because there is a whole ecosystem around OT (operational technology), around the electricians, the mechanics and the PLC programmers. In that ecosystem they have everything they need. Their own computers, the PLCs. Their own trainings. Their own programming languages, as we just talked about.

And because the ecosystem is so large, changing it is expensive. So it has stayed close to where it was when it was developed in the 1990s. Below you see two of the more modern dashboards you find on a production line today.

If you come from UX design, that looks old. We know. It is just the way it is.

Kubernetes is built entirely from control loops

Who knows Star Trek? I hope you are a Star Trek super nerd. If not, I’ll explain it anyway. This is going to be important here.

The people who built Kubernetes wrote down exactly why they made each design decision, and those decisions are still running in clusters today.

Before Kubernetes, Google built Borg and Omega. And there is this nice paper from Burns that explains their thought process on developing Kubernetes.¹ Let’s go through that thought process.

The first cluster management system from Google was called Borg, like the Star Trek Borg.² For non-nerds: in Star Trek, the Borg are the collective hive mind, one entity that controls all the drones. Every drone does exactly what the collective says.

Google’s Borg worked the same way. One “monolithic component”¹, the Borgmaster. One single program that managed the entire cluster. It kept its own internal store. The Borgmaster read from that store to understand what was happening, and told every machine what to do. And inside the master, every job and every task was an “edge-triggered state machine”³: it changed state when an event arrived. We’ll come to that again further down.

Borg worked. It was extremely robust. But now imagine: every team at Google who wants to change anything about cluster management has to modify one monolithic component.

So Google built Omega. In Star Trek, the Borg’s idea of perfection was the Omega particle. Omega was supposed to be the perfect Borg. It is not publicly confirmed that this is where they got the name from, but in my opinion it is pretty obvious once you see “Borg”², which is confirmed, and know some Star Trek lore.⁴

Omega broke the monolithic component into “separate components”¹, so that multiple teams could work independently of each other. It also put the state into a shared database. Each component read and wrote that database directly. And each component worked from its own copy of the state and resynced it. This means it is “level-based”⁵ in practice, and we’ll come to that further down.

So they went further. The founders of Kubernetes originally called it Project Seven, like Seven of Nine, after the Borg drone who broke free from the collective and became autonomous.⁶ But then, trademarks.⁷ You cannot just use this name if you want to make it public. So they instead went for Kubernetes, Greek for helmsman. A little bit of Kubernetes trivia now: as you can see, the logo still has these seven spokes, to reference the original name.

And in Kubernetes they went for, I am quoting here, many “small control loops”¹.

In Omega, every component wrote the database directly, so they “all had to be very highly-trusted, and the releases had to be tightly synchronized”⁸. Kubernetes wanted “a much wider array of clients”, including “less-trusted ones”⁸ (meaning OSS and scaling up).⁹ So they put the state behind one “centralized API server”¹.

And the design principles deliberately say that everything needs to be “level-based”¹⁰. And here they actually used the word “level-based”¹⁰, not only the principle like in Omega.

What “level-based”¹⁰ and “edge-triggered”³ mean, we will look at further down. But first, let’s look at a real Kubernetes object to remind ourselves.

For those of you who are not that familiar, this is how Kubernetes objects look.¹¹ The API conventions say: “All objects that represent a physical resource whose state may vary from the user’s desired intent SHOULD have a spec and a status.”¹² Below, a Deployment, a resource that spawns multiple containers.

apiVersion: apps/v1
kind: Deployment

spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: app
        [...]

status:
  replicas: 2
  conditions:
    [...]

The spec says replicas 3, based on this template. And the status reports: we only have two replicas. So the controller detected drift, and it will start spawning one more.

Every other object with a spec and a status works the same. Below, a Horizontal Pod Autoscaler.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler

spec:
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

status:
    currentMetrics:
    - type: Resource
      resource:
        name: cpu
        current:
          averageUtilization: 85

The spec says the average CPU utilization needs to be at 50 percent. The status says the current utilization is 85 percent. So it will start scaling up replicas.

The spec/status symmetry is one of the rules the API server enforces. The API server for example ensures that an ordinary update cannot change the status, and a status update cannot change the spec.¹³ In Omega, desired state and observed state were kept in separate records, which made it hard to see what was going on. In Kubernetes they sit in the same object.³

And if you draw this as a diagram, you have seen it before.

A control loop is the same thing as the thermostat, the same thing as the PLC on the production line. Spec where the setpoint was. Status where the sensor was. The controller where the PLC was.

You have the spec: replicas 3. The API server tells you the status: replicas 2. You calculate the error, one replica. That goes into the Kubernetes controller, which says: one pending pod. Then you have your cluster, with node failures and other disturbances. And then you have two pods running, which get measured again, and so on. It is way more complex than this in reality, but this is how Kubernetes in its core works.

On the PLC it is called the scan cycle. In Kubernetes it is called the reconcile loop.¹

Back to the table from before.

Now with the three parts of the loop on top. The controller: who is in charge. The sensor and the feedback: where is the state. The trigger: when does it react. Let’s go through these three mechanisms from left to right.

So, who is in charge?

In the figure above, on the left, Borg. In Borg it was all the single Borgmaster. At the top you give it “replicas: 3”, and the master does everything. As the paper puts it, a centralized system like this “tends to become brittle and rigid over time, especially in the presence of unanticipated errors or state changes.”¹⁴

On the right, Kubernetes. In Kubernetes it is “control through choreography”¹. At the top you give it “replicas: 3”. The Deployment controller reads the Deployment spec, does its own control loop, and sets the ReplicaSet spec. The ReplicaSet controller reads that spec, does its own control loop, and sets Pod objects. And so on.

When the Deployment controller crashes, the rest of the system keeps going. Nobody needs the Deployment controller to be alive. Everyone just reads their spec and reconciles.

If you remember, this looks exactly like a factory, and it behaves exactly like one. In the factory you can also take the MES offline, and the line keeps running. For a while at least. Same as in Kubernetes: it keeps running as long as you do not need the logic of the loop that crashed because you have changed something (e.g., the number of replicas).

Where is the state?

Or, as the paper says, don’t expose raw state.¹ In Omega, everything was in the raw cluster state. A shared database, flat and unstructured, and every component read and wrote it directly. That worked for scheduling. But a database only stores what you write into it. It cannot check that an object is well formed, fill in defaults, translate between versions, or refuse a change that breaks a rule for the whole cluster. Every component had to do that itself, and had to be trusted to do it right.

Kubernetes added the API server as a gatekeeper. Everything goes through the API server before it lands up in the database (which is in this case etcd). The API server will do validation, defaulting, versioning, and enforce the rules of the cluster.

When does it react? This explanation about edge-triggered and level-based is what I promised you above. Burns et al. write: “all action is based on observation rather than a state diagram”.¹ A state diagram is the Borgmaster’s kind of state machine: an event arrives, and the machine takes the transition for that event.

Remember the Kubernetes logo with the seven spokes? Tim Hockin, the guy who designed it,¹⁵ explained this with CPU interrupts.¹⁶ In electrical engineering, when you want to transmit a signal, you have a low signal, let’s say zero volt, and a high signal, something like five volt. In practice it is way more complex, but this should suffice.

Edge-triggered means you fire on the transition. You want to transmit information, so the line goes from low to high for some time, and then back to low. But what happens if the CPU misses this pulse, for whatever reason, a deadlock, a mutex, whatever? Then this whole interrupt is lost.

In the figure below, on the right we have level-based, or level-triggered. There the line stays high until the receiver has confirmed it. It does not matter how busy the CPU is, the signal remains high, so whenever it looks, the information is still there. It cannot be missed. Hockin calls it level-triggered; the Kubernetes design principles call it level-based.¹⁰ Same thing.

Let me show you what that means.

Let’s take the thermostat again, the one from the beginning. The code below is event-driven, edge-triggered. On a thermostat this looks quite stupid, but I have seen this logic pop up in a lot of programs. You take incoming events, and for each event you have a rule: if the window opened, set the heating to 80 percent; if the person left, set it to 60 percent (or plus 5 percent, it does not matter). Every new cause needs its own rule. But should one of those functions fail, the event is lost and we get a permanent drift.

events := make(chan Event)

for event := range events {
	switch event {
		case WindowOpened:
			setHeating(80)
		case PersonLeft:
			setHeating(60)
		case SunCameOut:
			setHeating(30)
	}
}

Below, how a normal thermostat works. A loop. You read the current temperature, you get the desired temperature, and then a very simple logic: if current is below desired, increase the heating, otherwise decrease it. And you can have a lot of CPU contention or whatever disruptions. It will only delay, but it will eventually reconcile.

for {
	current := readTemperature()
	desired := getDesiredTemperature()

	if current < desired {
		increaseHeating()
	} else {
		decreaseHeating()
	}
}

This is exactly what happened in Borg. The Borgmaster’s built-in state machines were edge-triggered. If a task crashed, that was an event, and Borg rescheduled the task. But if something went wrong in between, a panic, a programming error, whatever, and the change was never observed, the state machine could end up in the wrong state. As one of the authors of the Borg, Omega, and Kubernetes paper later wrote: “Unobserved changes could cause unexpected state transitions.”³

Kubernetes goes through cycles. Every cycle it compares spec against status. Cycle two: spec 3, status 2, so it emits the command, create one pending pod. Cycle three: two running, one pending, nothing to do. If you miss a cycle, it does not matter, the next one fixes it.

The design principles say it in one sentence: the system must work correctly “regardless of how many intermediate state updates may have been missed”¹⁰. So in reality it is actually implemented differently (watchers, etc.),¹⁷ but as it says in the guidelines, edge-triggered behaviour “must be just an optimization”¹⁰, so in its core it is a control loop.

The world is built on top of control loops

A Kubernetes controller works exactly like a PLC. And Kubernetes itself works exactly like a factory.

Because the world is built on top of control loops.

At UMH we use the same loop principle as well. We extract data from production lines.

We ran natively on Kubernetes, we always brought our own Kubernetes distribution. But it was always very painful, because Kubernetes distributions vary a lot between customers, everyone has their own admission policies, and we always had these weird discussions.

So we stopped using Kubernetes as our core and put everything in a single Docker container, so it can run on any Kubernetes platform. And with this we are actually more cloud native than before.¹⁸ And we ended up building the same loop as Kubernetes and PLCs ourselves.

You give our tool the desired state in a config file: the data is flowing from this IP address. Then we observe it: no flow exists inside the Docker container, the flow is at zero messages per second. So we calculate the difference, which means we need to add a data flow as its own process under S6, a process supervisor. Then you have disturbances: restarts, hardware failures, network failures, process failures. The observed state gets measured through CollectObservedState, and it goes in a loop.¹⁹

Besides that we also measure metrics and logs and way more parameters. So we can bring a lot of application logic in there. It is not only about starting a process, it is also about the application logic around it. If we detect issues with the data flows, we automatically reconcile there. Kind of a mini Kubernetes, including application logic, within a single Docker container.

But control loops are used everywhere else, in all types of applications, when there is something that reliably needs to be controlled. So not only manufacturing, not only Kubernetes, not only we at UMH.

From aerospace, where the flight control computers run the control laws on a fixed schedule, every cycle.²⁰

To insulin pumps, which check every couple of minutes the current blood sugar and emit insulin if needed, based on algorithms.²¹

To automotive, where the engine controller reads its sensors and sets injection and ignition as cyclic tasks with a fixed period.²²

Up to the moon landing.²³

Coming to the Apollo Guidance Computer. This is the real code that flew on Apollo 11, and it is public on GitHub.²⁴ It is assembler, of course a little bit weird to read, so I took some time going through it.

In automatic mode, Apollo 11 would land by looping every 2 seconds and reading the sensors, the accelerometers (PIPAs).²³ Then it calls the second block, SERVICER, which after a couple of internal function calls lands up at P65VERT, the automatic landing,²⁵ which takes the setpoint, the target descent velocity, measures the current velocity, makes the difference, and sends the signal to the throttle and the autopilot. Then it waits two seconds and repeats.

READACCS   CS    OCT37771  # <- loop body, every 2 s
           [...]
           TC    PIPASR    # <- sensor: read the accelerometers
           [...]
           2CADR SERVICER  # <- SERVICER reaches P65VERT through the guidance dispatch
           [...]
[WAIT FOR 2 SECONDS AND RESUME AT READACCS IN ASSEMBLER]

P65VERT    TC    INTPRET
           VLOAD VSU      # <- subtract:
                 V2FG     # <- setpoint: target descent velocity
                 VGU      # <- measured: current velocity
           V/SC  GOTO
                 TAUVERT  # <- gain: divide by a time constant
                 AFCCALC1 # <- actuator: hand to throttle and autopilot

Again, a control loop.

You do not have loops everywhere, and you do not need them everywhere.

For example a toaster. It is an extremely simple timer, just a fixed timer that cannot deviate a lot. And even if it would deviate a little bit, you only get some burnt toast. So no control loop here.²⁶

Let’s take in contrast an airbag: in its core a very simple logic. If the accelerometers in the car measure a very hard stop, it should open. But a failure of this is severe: either it opens while you are still driving, or it does not open in a crash. So even there, the airbag control chip works in a control loop and samples its crash sensors every few hundred microseconds and only arms after three consecutive readings over the threshold.²⁷ Again a control loop.

Conclusion

So it is safe to say:

The only reliable way to control anything is a loop.

The juice in your fridge was made on a line like this. A production line is built entirely from control loops, and two electricians from the next town over keep it running.

The Star Trek nerds at Google needed three attempts to get to Kubernetes. Kubernetes, Greek for helmsman. And the helmsman on a boat does one thing: look where the ship is, where it should go, correct, and repeat. Exactly like control loops.

The world is built on top of control loops, which is also what we are doing at UMH. Factories still run on 1990s software, and we at UMH are fixing that. If you have any questions, reach out on LinkedIn or Bluesky. Also, we are hiring Product Engineers in Germany.