AI trains itself match fit on a digital twin

With the ever-growing complexity of high-tech systems, it’s increasingly desirable that they can optimize themselves. The European Asimov project, co-initiated by Thermo Fisher Scientific, aims to achieve this through the combination of digital twins and artificial intelligence. With the help of TNO’s ESI as one of the partners, this should result in a generic approach for fully virtual training of AI algorithms.

Artificial intelligence continues to best people in all kinds of games. It all started in 1997 when IBM’s supercomputer Deep Blue defeated chess champion Garry Kasparov. Another well-known example is Alphago, developed by Google Deepmind, which even the best Go players can’t outsmart since 2017. Recently, the card game bridge also fell prey to the advance of artificial intelligence.

“Wait a minute,” thought Remco Schoenmakers, director Digital Science Technologies at Thermo Fisher Scientific. “Those deep learning algorithms could be interesting for electron microscopes as well. After all, you can compare the alignment of such a system to a game. You win if you move the microscope from an unknown state to the correct configuration the fastest and the most effective. I saw a big similarity.”

Together with his colleagues, Schoenmakers started an investigation into whether and how he could apply advanced AI techniques to the calibration of Thermo Fisher’s electron microscopes. “The adjustment differs all the time,” says Schoenmakers. “We want to keep the beam precisely on the right spot, within the nanometer. The alignment and calibration have to be meticulous for our customers to obtain the correct metrology results when researching new energy-efficient transistors in the semiconductor industry or determining the structure of proteins and viruses in the development of new drugs and vaccines. There are all kinds of influences that you have to take into account. Think of temperature variations, pressure changes and wear – the sources are plentiful. Some things are quite stable and things run smoothly for months. Other things you have to adjust almost every day.”

Digital twins

Thermo Fisher Scientific first contacted ESI. Soon after, Eindhoven University of Technology (TUE) and consultancy firm CQM joined. “Together, we looked at where the opportunities and challenges lay,” says Schoenmakers. “We realized pretty soon that we wouldn’t be able to train the AI algorithm on a real microscope. The amount of variations required to understand what’s going on is just way too large. There may be billions.”

The idea arose to build a digital twin of the electron microscope. Schoenmakers: “Let’s model the relevant aspects of the instrument. By applying variations on those models, you can create synthetic data and use it to train your neural network.”

A virtual copy of a complex system like an electron microscope is easier said than made. “It would take us five years before we would have developed a complete digital twin,” Schoenmakers laughs. “There are so many aspects to it, from the controls and physics to the system modeling. Ultimately, all of this must be captured in a digital twin to ensure that it responds in the same way as the real system. We’ve opted for an incremental approach.”

“On the one hand, you want to make an effort to get the link between the digital twin and the real instrument as close as possible,” TUE professor Maurice Heemels adds. “On the other hand, you don’t want to lose yourself in this because the learning algorithms must also be able to deal with the differences. Do we expect the trained AI to work on the microscope in one go? Or will it go in steps and does the algorithm get better with every iteration? Can we adjust our digital twin based on measurements on the real system to go through a faster learning curve? All these questions are still open, especially if you look at the problem more generically. One specific approach may work best for an electron microscope, but for other use cases, it might be wiser to take a different route.”

Broadly applicable

The AI exploration of Thermo Fisher Scientific isn’t a solo trip. Partly on the initiative of the Eindhoven company, a European project has been set up. Within Asimov (“AI training using simulated instruments for machine optimization and verification”), it’s all about the combination of digital twinning and artificial intelligence for the optimization of system performance. A consortium has joined in Germany that’s applying the approach to the control of autonomous vehicles. In Finland, a cluster of companies is working on process optimization for the paper and pulp industry.

This generalization is crucial for the other three parties that are affiliated with the Dutch arm of Asimov. “When we stepped in, we immediately asked ourselves how we could apply this approach more broadly,” says Jacco Wesselius, a project manager at ESI and the project leader at Asimov. “We already cover a broad spectrum with the German and Finnish cases, but we also approached the Dutch high-tech industry, explained our idea and asked for input. Not everyone has to join right away, but our role is to ensure that the ecosystem benefits from this development. So not only are we supporting Thermo Fisher in building the digital twin and evaluating the technology, but we’re also closely following the other use cases and looking at the approach in a broader context. For example, we map the wishes from the industry to this project, so that we gain better insight into how we can serve other parties and sectors well. At the same time, we validate our plans and ideas against a broader background. That’s an important objective for us in the project.”

CQM also became interested in the Asimov ideas because of their relevance and applicability in other applications. “We’ve been building digital twins for forty years, although they weren’t always called that,” says Jan Willem Bikker, a consultant at CQM. “The logistics world is one of our most important sectors. To give a practical example: in distribution centers, roll containers for supermarkets are filled, but there’s often a little air left at the top. That’s not efficient of course. By smart optimization, you can minimize that space. As there are many trucks with containers driving around every day, you can save a lot of money. It would be really cool if we could apply the combination of digital twins and artificial intelligence in logistics.”

Trained AI

TUE also has a more generic view of the development. The Asimov project uses the AI strategy of reinforcement learning. “That in itself is nothing new; the approach has been around for a long time,” says Heemels. “But we’re now applying it to a real instrument, where you encounter completely different issues in terms of complexity and where the real-time aspects are very different. In the application at Thermo Fisher, but also in the other cases within Asimov, all kinds of problems come to light. We’re trying to develop methodologies to deal with that.”

The TUE researchers are looking to translate this to other devices and distill the generic knowledge. “I notice in the interaction with Thermo Fisher that the engineers know their system very well and often know exactly what to do. A lot of things happen automatically,” observes Heemels. “Our role is to make it explicit. What’s the exact process that you go through to achieve a successful implementation? What steps are you taking? And why? I want us to be able to explain this method in a very detailed way to other parties in the future.”

Wesselius: “I strongly believe that given enough time, you can train AI algorithms with which you can perfectly calibrate a digital twin. But what if you do that on a real machine? How robust is that solution in practice? How big can the difference between the digital twin and reality be? The hypothesis is that AI trained on a virtual copy can also do its job effectively and efficiently on a real device, without having to go through all kinds of training sessions.”

Demonstrator

Asimov took off in mid-2021 and will run for three years. “We started to describe all use cases a little more clearly, to put requirements on paper and to record deliverables,” says Wesselius. “Where’s the overlap between the different cases and what are the application-specific properties? In general terms, it corresponded of course; otherwise, the parties wouldn’t have stepped in, but it’s good to make very concrete what you all have in mind. A conceptual architecture has emerged from this exercise. That’s an excellent basis for discussing the approach with outside parties as well.”

The next step is to fill in all the details. Schoenmakers: “We’ve already made a first physical model of our electron microscope with which we can view a certain calibration within the system. We ran it to generate a lot of training data. The question then arises: what’s right and what’s wrong? Once you figure out which picture is better or worse, you’re not there yet. The world isn’t that simple. Sometimes you have to go in the wrong direction twice to get to the optimal point. So how often are you allowed to go the wrong way? And how big should the steps be in the first place? We’re investigating that now.”

Wesselius again: “In Thermo Fisher’s case, we noticed that you’re deep in the machine very quickly. This makes it more difficult to go public with the result because it becomes too specialized or because it contains confidential knowledge. That’s why at ESI, we’re building a demonstrator that from a distance resembles an electron microscope. It’s a more general setup where we’ve defined another problem, including knobs you can turn to make an image better or worse. It’s a nice public deliverable without specialist knowledge of Thermo Fisher. The demonstrator is a perfect platform for us to experiment on a scale that’s much less complex than a real electron microscope, but on which we can master the combination of digital twinning and reinforcement learning. We can show that the technology works and where there are still hurdles to take.”

Asimov’s ambitious end goal is to have a system that calibrates itself and never crashes. Schoenmakers doubts whether this is feasible within the timeframe of the project, but “Asimov must in any case provide the technological direction and define the preconditions within which we can maneuver. On the one hand, it’s still very explorative while on the other hand, I can already see the first successes emerging. I’m confident that we’ll lay the technological foundation for self-optimizing high-tech systems.”

This article was written in close collaboration with the Asimov partners and ESI (TNO).