Helmholtz AI Conference 2026: a World Café Wrap

Introduction

Coming from science management and sitting between researchers, administration, IT, and leadership we wanted to explore a tension that we see every day: new tools arrive faster than policies, training often does not reach the right people, and nobody is quite sure who owns AI governance.

Key ideas

We started by asking three open questions:

• How to regulate without over-restricting?
• How can we build skills across very different roles?
• Who is actually responsible for what?

Key outcomes

People don't want more rules but better mechanisms: enhanced review (also by putting humans in the loop), critical reflection, and the question of whether mandatory courses are the right lever at all. The table seems to have agreed that an ad-hoc/unmanaged approach is unsustainable, but that heavy-handed top-down regulation risks similar outcomes.

Skills transfer & enablement was the most discussed topic. Participants named basic AI literacy as a foundation, then immediately complicated it: "How can we build skills?" leads to "scalability of training" and the gap between one-off workshops and something that actually sticks. Participants reported a high diversity between live training vs. recorded/online formats, and between using external/internal resources. Student feedback was flagged as a key input. The overall impression: training needs to be modular, scalable, and draw on what already exists within and beyond an institution.

The question "Who is actually responsible?" led to a key diagnosis: fragmentation of responsibilities. Proposed responses included a dedicated institutional AI office, ethics specialists, and the library as an existing trusted infrastructure that could take on an AI role. Living documents over static policies, weekly AI updates as a lightweight continuous mechanism, and role models as a cultural lever rather than a structural one. 

The underlying tension: institutions want someone to own this, but in some instances nobody has the mandate and/or capacity to do so yet.

Further discussion

Looking ahead, institutions need to do more than just define who’s responsible. They have to build the real-world structures and cultural habits that make AI stewardship work over time. That means creating spaces where people across departments can actually collaborate, where decisions are made openly, and where learning about AI is an ongoing part of the job, not a one-time checkbox.

Our World-Cafe discussed the challenges of developing and certifying AI systems for safety-critical applications. A central question was how to determine when an AI system is “safe enough” and who should be responsible for making that decision.

Participants agreed that domain experts are best positioned to assess acceptable safety levels through empirical testing, while regulators and other stakeholders must address legal and ethical aspects. However, many of these stakeholders lack a deep understanding of AI systems. Therefore, AI engineers must be able to explain model behavior, uncertainties, and limitations transparently and understandably.

A key topic was the importance of explainability, uncertainty estimation, and high-quality, bias-free data. Participants emphasized that evidence of data quality and model performance is essential to support certification. Existing standards can provide guidance, but they struggle to keep pace with modern AI developments and are often formulated in quite general terms.

The discussion further highlighted that established engineering practices, such as requirements-driven engineering and classical verification and validation (V&V), remain highly relevant for AI systems. Since extensive real-world testing is often costly, future approaches may use foundation models to generate edge cases and test scenarios. Especially the combination with neurosymbolic methods can help to ensure physically valid and trustworthy outputs.

The main conclusion was that certifying an AI system requires more than validating a final model. The entire AI engineering pipeline, including development processes, data, and design decisions, must be considered. Future work should focus on methods that more effectively connect AI engineering practices to safety and certification requirements.

The discussion brought together educators, researchers, and practitioners to explore how generative AI is transforming educational resource creation, delivery, and adaptation.

Participants shared their current uses of AI in teaching, which primarily include structuring lectures, drafting scripts and teaching materials, creating visualizations, and identifying jargon or unclear explanations to improve accessibility and clarity.

A broad consensus emerged that students should be allowed to use Large Language Models (LLMs) as part of their learning process, provided they receive a foundational “LLM 101” education covering also the limitations, and responsible use of these tools. The Helmholtz AI Consultant Team is already contributing to this effort through dedicated training. Participants also emphasized that the timing of AI adoption is critical: while AI can accelerate concept acquisition, skill development still requires time, practice, and productive struggle. As one participant noted, “Would we give a calculator to a child who is still learning basic arithmetic?”

Another recurring theme was the need for greater adaptation and personalization in education. Creating personalized learning experiences at scale requires significant effort, and participants saw strong potential for AI-powered assistants or agents that could tailor explanations, exercises, and learning pathways to individual learners' needs.

The challenge of designing robust assessments in the age of AI was also discussed extensively. Participants agreed that current assessment approaches are increasingly under pressure, but no clear consensus emerged regarding effective solutions for AI-resilient or AI-integrated assessment.

A further insight was that many educators, particularly in higher education and research environments, have received little formal training in pedagogy. Participants highlighted the importance of supporting educators in applying established teaching and learning principles when developing AI-assisted educational materials. Rather than simply prompting AI to “create content on topic X,” educators could be encouraged to incorporate pedagogical objectives and learning theories into their prompts, for example including retrieval practice, scaffolding, worked examples, and constructive alignment.

This discussion led to the idea of developing a practical booklet or guide for educators, containing example prompts and workflows for common educational tasks such as audience adaptation, exercise generation, accessibility improvements, and pedagogically informed content creation.

Finally, participants agreed that a centralized catalogue of educational resources and courses across Helmholtz would facilitate sharing, reuse, and collaboration among educators. As a next step, they suggested working with HIDA to include a contact person for each course of the Data Science Course Portfolio, making it easier to connect and exchange experiences.

To continue this work, an interest group will develop a practical booklet for educators and explore an AI assistant for creating pedagogically sound, personalized learning materials. The group aims to present its progress and initial outcomes at the next TEACH6 conference.

If you are interested in joining this interest group or contributing to these activities, please contact us at donatella.cea@helmholtz-munich.de , mueller3@hzdr.de, s.starke@hzdr.de , j.goshi@hzdr.de .

Our discussion began with a simple but important question: Can AI help make cities not only smarter, but also healthier, cleaner, and more pleasant places to live? While urban AI is often associated with efficiency, automation, and smart infrastructure, we wanted to explore how these technologies can directly improve people's everyday lives and contribute to more habitable cities.

A significant part of the conversation focused on air pollution, which many participants identified as one of the most visible environmental challenges in urban areas. The group discussed how AI can help us better understand pollution patterns by combining data from air-quality sensors, weather observations, traffic systems, and satellite measurements. Participants agreed that AI has great potential not only for monitoring pollution but also for predicting high-risk situations and supporting more informed environmental decisions.

One of the most interesting points of discussion was the balance between technological capability and human trust. While everyone recognized the value of advanced AI models, there was broad agreement that transparency and explainability are essential. Participants emphasized that urban AI systems should help people understand why decisions are being made, rather than functioning as black boxes. In this sense, AI should support human decision-making rather than replace it.

Several promising ideas emerged during the discussion. These included the use of urban digital twins, explainable AI, and integrated environmental intelligence systems that can provide real-time insights into urban conditions. Participants also highlighted the importance of linking environmental information with public health outcomes, enabling cities to take preventive actions rather than simply reacting to problems after they occur.

Looking ahead, the discussion revealed strong interest in further collaboration across disciplines, including AI, environmental science, urban planning, and public health. A key takeaway was that the future of urban AI should be human-centered. Success should not be measured solely by technological advancement, but by whether AI helps create cleaner air, healthier communities, and more sustainable urban environments. Ultimately, participants agreed that the goal is not simply to build smarter cities, but to build cities that are better places for people to live.

Motivation: Fast code, hidden errors

Generative AI writes syntax faster than any of us, but it optimizes for code that compiles rather than code that is fundamentally true. Pointed at a hard, domain-specific problem, an unguided model will sometimes hallucinate a rule or scientific law, or silently discard an established principle just to keep the code working. The output can look entirely convincing while being fundamentally invalid, a failure that is easy to miss precisely because nothing breaks. The table brought together people from a wide range of disciplines to ask how scientists can use these tools without letting them quietly overwrite hard-won domain expertise. The answer the room kept arriving at was a clear division of labor and boundary: the scientist stays in charge of the logic, and the AI is held to the syntax. Most of the discussion was about why that line is needed and how to hold it.

Discussion: Logic, syntax, and the line between

Much of the conversation began with concrete failures, and they were similar from one field to the next. The example that recurred most was an AI or LLM deleting working functionality in the middle of an edit, overwriting code that had already been validated, or ignoring fundamental scientific principles (e.g., energy conservation). People also described the slow accumulation of "AI slop," where a project keeps adding to the code until it grows past the point where the model can track its own structure, or it keeps creating numerous libraries and files. The unclear licensing of generated code came up as a separate, more practical worry. Across all examples, the common feature was how nothing looked wrong from the outside while quietly failing: the code compiled and ran, and the mistake stayed hidden until someone went looking for it.

From these failures the discussion moved, round after round, toward the same conclusion. The trouble starts when there is no clear division of labor, and the remedy is to keep the scientist firmly in charge of the scientific logic and boundaries of the project, while restricting the AI to implementation and optimization. The researcher dictates the concepts and constraints, ultimately defining what correct behavior looks like through their domain expertise. The model then writes the corresponding syntax. The agreement was close to unanimous across the disciplines at the table, and most participants turned out to already work in a similar way without having an explicit name for it.

On guardrails, the table had more open questions than settled answers, with standardized tests, scalability, maintenance, and review time all raised as ongoing problems. The practices that came up were modest but useful: committing frequently and keeping track of the prompts that generated working code, such that there is always a route back when the model removes something; fixing random seeds so that genuine errors can be separated from ordinary statistical variation; reviewing changes one feature at a time; separating the scientific logic into different files so that an unrelated change cannot silently corrupt it; and carrying these habits forward as work moves into agentic coding with multi-file setups.

Running under the whole conversation was a sense that the scientist's job is changing, moving from writing code toward reviewing it. The optimistic reading is that the work simply moves up a level, from producing syntax to overseeing concepts. The harder question, which this discussion did not settle, is whether that oversight still holds once it is the only thing you practice. It works only as long as you have the expertise to identify a silent error, and that expertise may be exactly what fades when the AI writes everything. That worry came up only briefly, though. What people kept returning to was how much these generative tools now make possible, including the freedom to sandbox speculative ideas that were never worth the coding time before.

Follow-up potential: from instinct to tested practice

The convergence was the part worth following up. A table with scientists from a wide range of disciplines still arrived at the same boundary between logic and syntax, and named the same failure modes, while working on completely different problems. A focused follow-up could test which guardrails are universal and which ones actually hold up in each field, and turn an instinct that clearly travels across disciplines into practice that has been checked against real cases.

The discussion focused on whether healthcare AI should move beyond predicting patient outcomes and instead support individualized treatment decisions. Central to the debate was the proposed iPITE  (integrated Predicted Individual Treatment Effects) framework, which combines clinical evidence, molecular data, and Bayesian decision theory  to estimate patient-specific treatment responses under uncertainty.

What was most controversial, or about what did all participants agree on?

There was broad agreement that AI should not replace existing clinical decision-making processes. Participants emphasized that AI systems must demonstrate clinical value  beyond predictive accuracy and should support, rather than substitute, clinician judgment. Explainability and interpretability were considered essential prerequisites for adoption, since clinicians need to understand the reasoning behind treatment recommendations.

The most controversial topic concerned the definition of a “good” treatment decision and who should ultimately determine it. Participants discussed the challenges of integrating AI recommendations with clinician expertise, patient preferences, and clinical context. Questions of trust, legal accountability, and professional responsibility generated the most disagreement.

What were the key ideas?

A central idea was that prediction and decision-making are not the same. Participants argued that a model can accurately predict outcomes while still producing poor treatment recommendations. Therefore, healthcare AI should be evaluated not only by predictive performance but also by the quality of the treatment decisions it enables.

Another important theme was the role of molecular data. Genomic, transcriptomic, and epigenomic information should not merely serve as additional predictors but should be incorporated as fundamental components of model design, particularly for diseases with strong biological drivers.

The discussion also highlighted the importance of uncertainty. Rather than reducing uncertainty to a single prediction, AI systems should propagate uncertainty through the decision-making process and communicate confidence levels to clinicians. This would allow treatment recommendations to better reflect the limitations of available data.

Finally, participants addressed the challenge of validation. There was strong agreement that predictive accuracy alone is insufficient to assess clinical usefulness. While randomized controlled trials remain the ideal validation strategy, alternative approaches such as simulation studies, emulator-based validation, and external cohort testing may provide valuable evidence before deployment in clinical settings.

Do you see any follow-up on this discussion?

No specific follow-up activities or future actions were proposed during the session. However, participants acknowledged that several challenges remain unresolved, particularly regarding trustworthy validation, uncertainty-aware decision support, and the long-term integration of AI-assisted recommendations into clinical practice.

At this table, we discussed how to make ML/DL models developed by researchers easier to use. The experience of the attendees showed that bad usability is wide spread across fields. While this especially affects less technical users, everyone struggles with models that are difficult to deploy. One suggestion was to change the incentives in academia to provide better installation instructions together with your published model, such as guidelines from the scientific institutions or requirements from the publishers.

While unified access to models is still difficult, there are several steps that can simplify usage of models. Creators of new models can upload the weights to Hugging Face to easily share them and follow good software engineering practices to help with dependency management (e.g., use an environment.yaml file or provide conda or container environments).