Helmholtz AI consultants @ FZJ

Project SunGAN

In the Project "Distributed GANs for synthetic Solar images" we train Generative Adversarial Networks (GANs) that create realistic images of the sun in extreme ultraviolet.

• Challenge: The Solar Dynamic Observatory (SDO) takes fascinating, beautiful images of the sun, and allows scientists to generate deep insights into dynamic processes within the sun. Imaging generates excellent qualitative insights, but a quantitative understanding is very difficult, as the space of all possible images forms a very complex distribution. With generative models, it is possible to study and understand this distribution in image space.
• Approach: In this project our team develops machine learning models that can create realistic high resolution solar images. By exploring the latent space of the model we can systematically study how the model's hidden variables control qualitative and quantitative feature outputs. We combine different state-of-the art methods to obtain high quality images with multiple consistent channels. We study different GAN architectures (StyleGAN, StyleGAN2, StyleALAE, BIGGAN, Unet BIGGAN) and combine these methods with differentiable data augmentation to increase the data-efficiency. We train models highly parallel on many nodes of the JUWELS Cluster at the Jülich Supercomputing Centre.
• Collaborators: Ruggero Vasile, Frederic Effenberger (GFZ Potsdam)

MLWater - Delta Learning of water potential energy surfaces

In the Project MLWater we explore different techniques for Learning a Neural Network Potential that corrects a classical water model to obtain accurate potential energy surfaces for water molecules in bulk and at interfaces.

• Challenge: The simulation of thermodynamic and structural properties of liquids requires accurate model potentials. In the last decade, machine learning based potentials, including the so called Neural Network (NN) potentials from Behler and Parrinello are gaining influence. Here, the intermolecular potential is constructed from so called Atom-Centered Symmetry Functions that obey translational and rotational symmetry that are fed into a multi-layer Neural Network. Training samples are obtained from quantum mechanical simulations. Our collaborators have recently trained a NN potential for water that recovers experimental thermodynamic properties with high accuracy indicating a high quality result.
• Approach: In this project, we jointly continue this investigation under two aspects: (a) How can a NN model be combined with a traditional water model? (b) How can the training be implemented in a differentiable framework so that experimental data can be taken into account? In question (a) we would like to learn only the difference between a classical force field and quantum mechanical simulations. This would allow us to add new atom types on the force field level without retraining and to ensure that the electrostatic force is treated on the classical level, including its long range part. For question (b) we would like to change our training and simulation approach to unlock the potential of differentiable simulation. Recently, two very interesting packages that combine training and execution were released: TorchMD and JAX MD. In this project we will provide a first implementation of a training in each framework as a starting point for future developments.

CovidNetX Initiative

In the CovidNetX Intiative, we contribute to the global fight against the COVID pandemic by developing a medical imaging technique based on large scale transfer learning.

• Challenge: At the time of writing this project the Sars-CoV-2 virus and the associated disease COVID-19, poses a great threat to mankind. The main diagnosis tool, the PCR test, has proven extremely helpful, but requires an appropriate lab and trained personnel. A complimentary tool could be medical imaging techniques that are widely available, even in remote areas. If COVID-19 could be diagnosed with a simple X-Ray image, the time between suspecting COVID-19 and entering quarantine could be substantially reduced. However, as in many medical imaging applications of machine learning, generating large dataset is challenging, especially under privacy aspects. Therefore, we focus on data efficiency and how to learn from large bodies of only remotely related data.
• Approach: It has been shown recently that models trained on large-scale scale corpora are excellent learners. For computer vision as well as natural language processing, models can be pre-trained even without available labels using techniques such as self-supervision, as in the GPT-3 model for natural language processing and the SimCLRv2 model for computer vision. We would like to apply these insights to identify medical images of patients with COVID-19. With extensive supervised or self-supervised pre-training, we would like to reduce the number of samples required to train a model that reliably performs this task. We will systematically study the role of pre-training and extend to newer, larger datasets where we make full use of the Juwels Booster module.

Understanding Spin-Tune Variations at COSY

In the project "Understanding Spin-Tune Variations at COSY" we help understanding interesting measurements of the "Spin Tune" with Machine Learning methods at the particle accelerator COSY in Jülich.

• Challenge: The COoler SYnchrtron COSY is a particle accelerator at the Institute for Nuclear Physics at Reserach Center Jülich. It generates proton beams in the energy range between 45 and 2700 MeV or deuteron beams between 90 and 2100 MeV for fundamental science experiments. COSY can create spin-polarized beams, where the spin axis of all particles are aligned. The quantity spin tune is defined as the number of spin precessions per turn in the accelerator. Under ideal conditions the spin tune gives a simple relation for the particle speed and a fundamental property of the particle and can be measured with high accuracy at COSY. However, in recent measurements it was apparent that the spin tune does not obey the simple relationship, but deviates slightly as the measurement conditions deviated from ideal conditions. Measurements show that this deviation has a complex dependency on many other measured quantities and can depend on the operation mode of the accelerator.
• Approach: In this project, our team applies machine learning methods to contribute to the understanding of these phenomena. We apply ML pipelines based on dimensionality reduction, regularized linear regressions, and Echo State Networks. A key challenge is that the number of experimental data points is very limited. Therefore, the selection of training and test sets must be done very carefully and regularization is a top priority.
• Collaborators: Jörg Pretz (FZ Jülich)

Atmospheric Machine learning Benchmarking System (AMBS)

In the project "Atmospheric Machine learning Benchmarking System", we develop video prediction methods for weather forecast that can be trained efficiently on the supercomputers.

• Challenge: The Atmospheric Machine learning Benchmarking System (AMBS) aims to use deep learning architectures for weather forecasting. It applies methods, developed originally for video prediction, to the prediction of atmospheric quantities such as temperatures. The AMBS is flexible with respect to the deep learning architecture that is used in training and testing. But, due to big data volumes and the complexity of the deep learning models, AMBS requires parallel data handling and parallel training methods on High Performance Computing (HPC) systems.
• Approach: Two challenges must be solved to handle large deep learning models and large data volumes. On the one hand the implementation must be parallelized such that it can work efficiently on large-scale parallel machines. Furthermore, data parallel training alters the training mechanism itself. It requires to increase the batch size of the optimization procedure to grow proportionally to the number of involved accelerators. This can lead to a performance degradation. Our team supports both of these aspects. We parallelize the model training using the horovod library. Furthermore, we apply advanced algorithms to mitigate performance loss, such as the LAMB and LARS algorithms.

AlphaNumerics Zero

In the project AlphaNumerics Zero, we work on Reinforcement Learning methods that accelerate the convergence of a numerics method.

• Challenge: Recent successes in Reinforcement Learning, such as AlphaGoZero reaching superhuman Go skills or the OpenAI Five playing Dota 2 on world class level have been very inspiring. Many researchers try to transfer the success to other fields hoping that computers can act as agents with skills superior to humans. For AlphaNumerics Zero this idea is applied to numerical linear algebra. Many numerical algorithms can be optimized by picking the right “magic” parameters. This can dramatically accelerate the convergence and hence reduce the computational effort. In the project AlphaNumerics Zero (αN0), Reinforcement Learning (RL) is used so that for a specified simulation problem, the computer learns to determine the “optimal” numerical solution method and its parameters by itself. The project focuses on iterative time-stepping schemes, i.e. specially spectral deferred correction (SDC) methods which are especially suited for supercomputers. This class of methods serves as a prototype for many different areas: stationary iterative solvers, preconditioning, parallel multigrid techniques, time integrators, and resilient numerical methods. Progress made here can be converted into progress in these other fields, with a very broad impact.
• Approach: We support this project with RL expertise and technical support. In intense and fruitful discussions, we discuss options how to formulate the problem as an ML task, directions like reward shaping and how to discretize the action space. We provided a first implementation of the framework in JAX and the PPG algorithm, and support the project in exploiting the possibilities of a fully differentiable formulation of the problem.

Jülich Challenges

In the Jülich Challenges project, we support the creation, deployment and launch of a data challenges website for Forschungszentrum Jülich.

• Challenge: Scientists from Forschungszentrum Jülich have had great success in organizing data challenges. Hanno Scharr's group from IBG-2 has attracted a lot of attention and very interesting solutions to a plant phenotyping challenge. Combined effort of several researchers from FZJ is built on the idea of creating a platform that can serve as host for challenges in the future. Researchers from fields as diverse as biology, neuroscience and material science join their efforts for creating the platform. In this way, they aim to increase both the visibility of FZJ's research and the visibility of FZJ as a location of great AI research.
• Approach: Our team contributes to the Jülich Challenges in many different ways. We have created the website for Jülich Challenges based on the code of the great EvalAI Website. If you are interested, please visit data-challenges.fz-juelich.de. We support scientists in preparing their dataset for the challenges. Finally, we contribute baseline solutions to some of the challenges and sometimes we try to improve the state of the art.