The Simbox Lives

Bringing first hardware online to de-risk Powerplant control & sim

Welcome to “The Atom Mill” the hub for written updates from Applied Atomics. We’re developing co-located, economical fission power to unlock the next 1,000 years of advanced industry.

First Hardware Online!

Last week, we brought our Simbox online. This box consists of four computers which work together to replicate our Powerplant control computer setup. Crucially, it also serves as the home for our Fast Sim which will provide real-time simulation of our entire coupled Powerplant. As we develop the plant, we will use a technique known as “Hardware in the Loop” or HITL to pull elements out of simulation and into real hardware (valves, pumps, regulators, fuel rods) to derisk our approach.

What does all that mean? Read on to learn more!

The Philosophy of Controls, or Controls as a Philosophy

Build the Foundation

It’s natural to wonder why this is the first thing we built. The answer is simple:

This is the work that ties all the other work together.

Our entire design philosophy is based on using modular design patterns and generating a wholistic understanding of the entire power plant in simulation as soon as possible. This unites the team across disciplines and helps them understand the impact of their work at every interface.

We applied many of the same modular design patterns we’ve used in previous hardware deployments to this plant’s control systems, and the Simbox is the test bed for those choices. It also allows us to begin employing HITL as we progress. A HITL system is not the power plant per se, but it is the connective tissue that enables the power plant to be defined, built, and operated.

By starting with the fundamental control system, we can ask ourselves “Where are the errors? Where are the unknowns?” and track and mitigate each of those via tweaking the controls responses of the power plant. We can then define the characteristic time response of systems individually and as a whole in the design by exploring these edges. Then that information is fed into our system design process iteratively, run through, and the real-world solution is put back into the sim to check its behavior and the impact on other systems.

In essence, this is a digital twin but one that we can actually manipulate, calculation within, and observe actual physics based on our real-world components. We will spend the next year validating our designs while also starting to pull them out of the digital world into the physical world via HITL.¹

Where the System Isn’t

When designing a system we need to understand:

1. What does it need to do? (i.e.: what’s the mission?)

2. Can we define an approach to accomplish that? (i.e.: requirements & constraints)

3. Quantify how much margin is needed (i.e.: controls, or how wrong can I be?)

What we don’t need to know is where the system is aka the exact operating point. We only need to know that it is safely within design bounds. The load demanded by the off taker is what defines where the system is any given moment, because they are our customer. Our job is to design a power plant which is both responsive to those demands and also stays well within safety boundaries. Or put another way, the system knows where it is at all times because it knows where it isn't, which might sound like something from the Tao but is actually a hallmark of active system control.

The next step was applying these principles, so built our Simbox with four off-the-shelf Raspberry Pi’s.

The Result

We know what you’re thinking — you can’t use four Raspberry Pi’s to control a nuclear power plant.² While we are not suggesting that our final hardware will be R-Pi’s, we have also shown over the last week that you absolutely could do it if so inclined, and we did so on purpose.

We chose the R-Pi for the Simbox because it helps us quickly, cheaply, and effectively bound our control system:

• Allows us to test the full stack of ethernet, serial connections, and input sensors.

• R-Pi’s are slow, but industrial hardware is also relatively low processor speed, so we are forced to make disciplined choices around data throughput and control rate that’s relevant to the final design.

• The R-Pi + Python’s nature makes error stack-up more obvious. We can learn how to characterize all the jitter and latency throughout the system, and design around it. The errors are larger, and if we’re able to find reliable, consistent solutions here in theory then it should be even better on the actual hardware.³

The standard approach in nuclear power has been using a expensive, long-lead, special purpose computer, or manual controls backed by analogue systems which creates as a single failure point.

We are instead using a quad-computer architecture which lets us address the Byzantine General’s Problem, exchanging information between computers reliably while complementing the plant hardware design to make the right decision without degrading performance even if one computer goes down. Much more on this soon but suffice to say it increases our reliability significantly while also dropping cost, complexity, and lead-time (i.e.: our mission in a nutshell).

Here’s how they’re running after a few days:

Four R-Pi’s coming into sync.

Four computers moving data in sync with a jitter around 30 microseconds is pretty solid for R-Pi’s and Python.

A quick note on Python. We chose it for software on this build for the same reason we chose the R-Pi’s for hardware. Ultimately, we’ll employ a lower-level language like C, an RTOS, and different hardware for guarantees of consistency and longevity but right now, to move quickly we’re isolating the need to run very fast from proving out our theory of operation. Those other options are a lot of work to implement but are known quantities. In this early phase of the design, it’s all about quantifying bounds, reducing risk, and eliminating the unknown as quickly and efficiently as possible.

Expect a similar approach to standard hardware and quick validation all based redundancy over single-point reliability with our other systems as we progress. It’s this thinking which allows us to expand the supply chain, standardize the esoteric, and bring efficiency to the fore.

1. We will have a lot more to say about this approach and HITL, but in the meantime think of it this way: If you don’t want to get stuck running in design circles, then focus on interfaces between systems and bound those interactions with controls. Or put another way, if you have a group of 10 opinionated friends and you ask them “What do we want for dinner?” you’ll get 10 answers. If you take the time to instead understand everyone’s dietary needs and see what’s open nearby, you can instead ask, “Do we want Option A or Option B?”

2. tHaNkS, dAd

3. aka if you can play through a full Primus set on a $100 Dean bass guitar, you can definitely play it on a Carl Thompson Piccolo Custom.