At the heart of modern AI systems is a deceptively simple idea: predict the next thing.

Predict the next word in a sentence. Predict the next action in a sequence. Predict the next token in a conversation. From this humble objective, we get astonishing capabilities: translation, summarization, reasoning, code generation.

In the process, models build something like an internal map of the world—a world model. The way they build and use that map is one of the most important questions in AI today, not just for technical progress, but for society.

What is a world model?

A world model, loosely speaking, is any internal representation that captures structure about the environment: regularities, causal relationships, constraints. It allows a system to simulate consequences and plan.

Humans build world models through perception, experience, and culture. AI systems build them through training data and optimization: they compress massive corpora into parameters that can generate plausible continuations.

This is not a philosophical nicety. A world model shapes:

• How an AI system generalizes.
• Which patterns it amplifies or ignores.
• How it fails.

The data is not the world

Models trained on internet text, code, and images are not learning from the world directly; they are learning from representations of the world—filtered through human language, media, and platforms.

That has consequences:

• Over-represented voices shape the model’s sense of what is “normal.”
• Under-represented communities may become barely visible in its internal map.
• Historical biases—racism, sexism, colonial narratives—become part of the priors.

We should not be surprised when models reflect and reinforce these distortions. They are not hallucinating them; they are reproducing patterns they were optimized to compress.

World models as institutional mirrors

When institutions rely on models to summarize information, draft strategy, or analyze risk, they are not just using a neutral tool. They are importing an external world model into their own.

Consider a regulator using an AI system to triage incident reports. If the model’s internal representation underestimates certain kinds of harm—because they are less documented in its training data—those harms may receive less scrutiny.

Or a bank using models to analyze market scenarios: if the model’s world model was shaped largely by a decade of low interest rates and certain geopolitical assumptions, its extrapolations might fail precisely when reality shifts.

The risk is not just that models “make mistakes.” It is that institutions adopt those mistakes as part of their own worldview.

Boundary problems

Unlike human experts, models do not have clear boundaries: they happily mix domains, genres, and perspectives. This can be powerful—creative analogies, cross-domain synthesis—but also dangerous.

• A model that confidently extrapolates from tech policy debates in the US to global regulation may ignore radically different legal and cultural contexts.
• A model that blends speculative fiction and factual reporting might generate plausible, but entirely false, narratives about the future.

In human discourse, we enforce boundaries: academic vs opinion, fiction vs news. Models blur them.

Alignment with whose world?

Efforts to align models often focus on behaviours: refusal to generate harmful content, avoidance of illegal advice, adherence to content policies.

But alignment also involves which world the model takes for granted:

• Does it treat certain political systems as more legitimate?
• Does it encode specific economic ideologies as default?
• Does it represent certain families, cultures, or identities as “normal” and others as deviations?

These questions are uncomfortable because they expose AI development as fundamentally normative work.

Institutional counter-measures

If institutions are going to rely on systems with powerful world models, they need their own defences:

1. Diverse evaluation
   Test models against a wide range of perspectives, geographies, and scenarios. Do not assume that passing a benchmark implies robust understanding.
2. Hybrid world models
   Use models as one input into decision-making, not the sole source. Maintain internal expertise, local data, and human deliberation.
3. Domain-specific training and guardrails
   Fine-tune or adapt models to institutional contexts, but do so transparently: document what data you added, what values you encoded, and what trade-offs you made.
4. Reflexive institutions
   Regularly examine how reliance on models is changing your own world model. Are you seeing fewer edge cases? Relying on summaries instead of primary sources? Narrowing your sense of what is possible?

The reality loop

We should also recognize a feedback loop: models trained on human data shape human behaviour, which generates new data, which train the next generation of models.

If models become major sources of text and media, they will begin training, at least in part, on their own outputs. If they become major sources of policy ideas, risk analysis, or legal drafts, they will shape the very institutions that set their constraints.

World models without boundaries can swallow their creators.

The challenge for the next decade is not just to make models more capable, but to ensure that their representations of the world do not quietly replace our own.