Most writing on reinforcement learning vs PID control frames it as a replacement question: is a trained policy now good enough to retire the controller that has held industrial loops for a century? The measured record answers a narrower question. In several hardware experiments, RL does beat a rule-tuned PID on tracking metrics. In the one field deployment we found, it improved one loop and made another worse. And in nearly every design that reaches a real process, the learned policy ends up placed relative to the PID rather than instead of it.
So the useful question is not which one wins. It is where the learned policy sits, and what it is allowed to touch.
The question is where the learned policy sits
There are three placements, and they carry very different risk.
- Tuner. RL searches the gain space and hands back a set of PID parameters. The loop the plant runs is still a PID, fully inspectable.
- Supervisor. RL writes setpoints or biases to existing PID loops. The commercial guide currently ranking for this query describes exactly this: the agent outputs dynamic setpoints to the PID controllers, which hold the last valid setpoint or revert to a safe default if the agent or its computer fails.
- Replacement. The policy drives the actuator directly. This is what most academic comparisons test, and what almost nobody deploys without a fallback.
Keep those three apart while reading the numbers below. Most apparent contradictions in the literature are results from different placements being compared as if they were one.
What the head to head experiments measured
RL tuning a PID on real tanks
The cleanest hardware result is a two-tank level rig where a deep RL agent was trained to tune the PID itself, then evaluated against SIMC model-based tuning and a commercial Accutune III autotuner. Normalised integral absolute error came out at 39.34 for the RL-tuned loop, 47.74 for SIMC and 54.06 for Accutune. Settling time was 127 against 215.25 and 159.75. Every RL run reached peak performance in under 40 minutes of operation.
Two details matter more than the headline. First, the RL advantage was on IAE, ISE and settling time; overshoot and total variation were similar to SIMC, and two SIMC configurations beat RL on those. Second, the commercial autotuner produced its parameters in roughly 10 minutes. RL bought better tracking with four times the commissioning time and a closed-loop disturbance to the process while it learned.
RL gains in production actuators
The strongest production result is not a chemical plant but an optical switch. On piezoelectric actuators in a 384 by 384 port switch, a DRL method that generates PID gains per device got 87.5 percent of switching events to settle within the 20 ms target, against 17.5 percent for the default parameters, and cut mean settling time by 22.2 percent. Overshoot got worse, though every event stayed under the acceptable limit. The authors report hours to train and milliseconds to produce parameters for an unseen device, where manual tuning took on the order of a week.
Note the placement: the RL output is still a PID, running on an FPGA at 10 kHz. The learning happened offline.
A direct replacement on unstable hardware
Where RL replaces the controller outright, the best hardware evidence we found is a double inverted pendulum on a two-wheeled platform. Across 15 trials, the PPO policy averaged 0.762 s rise time against 0.886 s for the tuned PID, with 2.944 degrees of overshoot against 4.25. The PID still held a slightly lower average steady-state error, 0.2053 against 0.2253.
Simulation studies push further. A nuclear plant study compared a soft actor-critic agent with LSTM memory against Ziegler-Nichols-tuned PID controllers for cold shutdown operation, with the stated aim of showing that a learned controller can reach operational goals with minimal manipulation. It was validated on a simulator, which is the caveat that applies to most of this category.
Where PID still holds
The one field deployment in our source set is the most instructive. A PPO agent trained to tune PI loops, using domain randomization to generalise, was deployed at an industrial comminution circuit on two flow control loops. It improved one. It degraded the other. That is not a failure of the method so much as an honest picture of what sim-to-real costs when the training distribution misses something the plant does. We cover the mechanics in domain randomization for sim to real transfer.
PID also keeps three structural advantages that no tracking metric captures:
- Steady-state behaviour. Integral action drives offset to zero by construction. The pendulum result above, where PID kept the lower steady-state error while losing on speed, is what you should expect.
- Commissioning cost. Ten minutes of autotuning against 40 minutes of closed-loop learning on a single loop, and that gap grows with every loop in a cascade.
- Auditability. Three gains are something an operator, an insurer and a certifier can all reason about. A neural policy is not, which is why the replacement placement needs a separate safety argument (see disadvantages of reinforcement learning).
A practitioner thread makes the same point from the floor. Someone planning to put RL on a food production chamber run by cascaded PIDs got two answers: one that a plain PID beat their RL agent on cartpole, and one warning that an agent trained on a learned simulator will exploit wherever the model is wrong, with the advice to keep the PID running in the background and switch over when the agent leaves bounds. Anecdote, not evidence, but it is the instinct of people who have to answer for the plant.
Why most working designs keep the PID
Supervisor and fallback
The recurring pattern in the credible work is RL with a PID behind it. A stability-preserving tuning scheme starts from a conservative but stable baseline PID and uses a supervisor that swaps it back in the moment the running tracking error exceeds a benchmark, so the loop stays stable throughout exploration. A grinding mill study, on a model verified against industrial data, goes further and constrains the agent's actions analytically so it cannot push the loop into instability during training, and explicitly chooses to tune PIDs rather than replace them to keep interpretability.
Those are the same idea as a control barrier function safety filter or safe reinforcement learning via shielding: the learned component proposes, something with a guarantee disposes.
RL as a discovery tool, PID as the executor
A newer framing drops the ambition of letting RL act at all. On the Industrial Benchmark, the authors note that DRL's final policies can be unstable, so they use a TD3 agent only to discover good operating settings and hand those to a tuned PID, reporting that the hybrid matches the best agent's performance with the reliability of the classical loop. In hydraulics, a DDPG agent is integrated with the PID structure instead of outputting control signals directly, with the authors arguing that direct-output DRL is what makes industrial deployment hard. And for strongly nonlinear biotechnological processes, TD3 has been used to tune PID gains against classical tuning methods, in simulation.
Read together, the literature is not telling you RL lost. It is telling you the PID became the actuator interface, and RL moved upstream.
A decision rule you can apply to your own loop
Sort your loop before you pick a method.
- Single variable, roughly linear, well understood. A well-tuned PID. If tuning is the pain, try an autotuner first and RL tuning second.
- Many similar loops or many similar devices. RL tuning pays back, because the training cost is amortised. The optical switch result is this case.
- Coupled variables, competing objectives, slow drift. Supervisory RL writing setpoints to PIDs, with the PIDs as the fallback. If you are also weighing MPC for that layer, the trade is different and we sort it in reinforcement learning vs MPC for process control.
- Fast, unstable or strongly nonlinear dynamics where PID genuinely cannot hold the loop. Direct RL, but only with a safety layer that has a guarantee of its own and a validation record long enough to argue from.
In every case, compare worst-case behaviour, not averages, and do it on disturbances the agent did not see in training.
What we measure when the controller is ours
Equation Labs builds controllers for live physical and biological processes, so we sit in the last row of that list more often than we would like. Our design choices follow the literature above: the learned policy never acts without a separate safety layer, and the case for it is made with measured numbers rather than a demo. On our current stack, the safety layer runs in under 2 ms, end-to-end edge latency came down from 1.2 s to 285 ms, and the controller was validated on a 500 L pilot basin with zero safety violations across 400 simulated years, on a growth model with R-squared above 0.95.
Those figures are not an RL vs PID benchmark and we do not present them as one. They are what we think any replacement-placement controller owes the people running the plant before it touches a valve. Getting latency down enough to close the loop at the process itself is its own engineering problem, covered in our edge AI deployment pipeline.
FAQ
Can reinforcement learning replace PID control?
On some tracking metrics, in some experiments, yes. In deployed designs it almost always tunes or supervises the PID instead, with the PID kept as the fallback. The one field deployment in our sources improved one loop and degraded another.
How long does it take to tune a PID with reinforcement learning?
On the two-tank hardware rig, under 40 minutes of closed-loop operation per loop, against roughly 10 minutes for a commercial autotuner. Offline approaches trained on device parameters can take hours to train once and then milliseconds per new device.
Is it safe to train reinforcement learning on a live process?
Only with a guard. The credible schemes either revert to a stable baseline PID when tracking error crosses a threshold, or constrain the agent's actions analytically so it cannot destabilise the loop. Otherwise, train in simulation and deploy a frozen policy.
Which reinforcement learning algorithms are used for continuous control loops?
Actor-critic methods built for continuous actions. Across the studies cited here: TD3, DDPG, PPO and soft actor-critic.