AI Does Not Remember the World — Approximation, Control, and Responsibility

Related: Generation Is Cheap; Judgment Is Expensive

One of the most common requests in generative image editing sounds almost trivial: “Keep everything else exactly the same and change only this.” Remove the glasses without changing the expression. Replace the color of a product without touching the rest of the scene. Swap one object in a video while preserving everything around it. To a person, the boundary between what should change and what should remain is obvious. For a generative model, however, that request can be harder than making the entire image again.

The prompt can be rewritten, the seed fixed, and the mask boundary adjusted, yet something outside the requested area still moves. The face feels subtly different. Letters in the background collapse. A texture that looked fine in one frame flickers in the next. The same procedure worked last time but fails now. It is tempting to treat these cases as temporary shortcomings: the model is not large enough, the dataset is not broad enough, and another generation of models will eventually make the problem disappear.

I do not think this is only a matter of current models being unfinished. It is tied more closely to the way generative models learn the world and produce an output.

The World Is Not Stored in the Parameters

Training a model is not the same as storing the world. Training compresses recurring shapes, relationships, concepts, and conditional possibilities from a large dataset into parameters. Individual examples may be partly memorized, but the model does not hold an orderly archive of original reality. What we call a generative prior is closer to a compressed expectation of what is plausible, formed from what was observed in the past.

$$\theta^\ast \approx \operatorname{Compress}(D_{\text{past}})$$

Making the dataset larger does not change that relationship. Broader data expands the range a model can handle and improves its approximations, but the model has still learned a distribution of observed reality rather than reality itself. Events that were never recorded, concepts that do not yet exist, and exceptions that the data treated as unimportant remain outside it. Nor can a finite set of observations uniquely determine the unobserved world. Many hypotheses can explain the same past while producing different futures.

It is more precise to call a new input new evidence or a new observation than a new prior. A model that has finished training interprets an unfamiliar image or sentence inside its existing representation space. It can condition an answer on that input, but in most deployed models the parameters remain unchanged when inference ends. This unfamiliar piece of the world does not harden into knowledge for the next decision. The model can form something like a temporary posterior from new evidence, yet it does not continually turn that posterior into the next prior.

$$y \sim p_{\theta^\ast}(y \mid x_{\text{new}}), \qquad \theta^\ast_{\text{after}} = \theta^\ast_{\text{before}}$$

This distinction separates novelty in an output from novelty in knowledge. Producing an image that never appeared in the training set does not mean that the model has learned a new way to understand the world. It can combine familiar elements into a configuration that did not exist before. Confirming what that result means in reality, revising an internal rule when it is wrong, and retaining the correction for later generations are different abilities.

Abstraction itself is not a defect. People do not remember the world as an exact original either. We preserve what mattered more than every detail we saw, and we infer unfamiliar situations from rules we already know. The loss of information is both a weakness of intelligence and one of the conditions that makes intelligence possible. If understanding required storing every pixel and every moment, it would be hard to say that people understand the world at all.

The deeper problem is not that a model abstracts, but that it does not know what its abstraction discarded. It struggles to revise its representation when a new observation contradicts an old rule, and it mixes knowledge supported by training with approximations used to fill the gaps. The model draws a map of the world, then continues drawing plausible roads through the blank parts.

Why “Change Only This” Is Hard

Image and video editing make this limitation visible. A finished photograph does not contain the process that produced it. The subject's 3D structure, occluded regions, lighting arrangement, lens and camera motion, editing layers, and masks have all been flattened into one array of pixels. To change just one element, a model first has to infer that invisible production state. But countless production processes can lead to the same image, and there is no guarantee that the recovered state matches the original one.

This remains true whether the input is a real photograph or an image made by another model. The system does not retrieve a perfectly mirrored pair and apply the requested change. It finds a state that can plausibly explain the input under its learned prior, then renders the scene again with the new condition. If glasses, eyes, identity, and lighting are not fully separated in the representation space, removing the glasses can move the eyes and skin texture along with them. In video, those small differences are sampled again from frame to frame and grow into flicker and shape drift.

Masks, ControlNet, reference images, fixed latents, and feature injection can reduce these failures considerably. They are useful production techniques and will continue to improve. Yet their role also reveals the current limitation. The model often does not preserve the rest because it understands every relationship; an external constraint holds still whatever must not change. The more freedom a model has to generate, the more firmly a production system may need to constrain what must be preserved.

That is why the two extremes are often easier in content production. A team can make the entire piece with conventional tools, or it can let AI generate the whole result while accepting some variation. The difficult work lies between them: change one part of an already approved asset while preserving identity, art direction, brand rules, continuity, and delivery specifications. Much of real post-production lives in precisely this middle ground.

Traditional production tools preserve more than the final output. Layers, tracks, keyframes, object IDs, color management, and version history remain available. A team can trace what changed at which stage and return to an earlier state. A generative model is usually optimized to produce a plausible finished sample rather than a recoverable production state. A generated image may look complete while remaining remarkably flat when someone needs to edit it again.

A PoC Can Succeed While a Service Fails

The important production question, then, is not whether the system can create one excellent sample. It is whether it can repeat the promised level of quality, preserve everything outside a revision request, and return to a known state after failure. The capability to succeed once is different from the reliability to keep a promise under real operating conditions.

$$\text{Capability}: \exists x,\ \operatorname{Success}(x) \qquad \text{Reliability}: P(\operatorname{Success} \mid x \sim D_{\text{real}})$$

A PoC or demo mostly demonstrates capability. The team can choose favorable inputs and select the best result from many generations. A service does not receive that protection. Users bring inputs the model has never seen, ask for conditions that conflict with one another, and want to know why the system failed. Average quality matters less than the long tail of failures and what each failure costs.

Generative AI makes this distinction easy to miss. An interface that accepts almost any request in natural language looks like a general intelligence by design. An open text box is confused with a system capable of fulfilling everything written in it. After seeing a polished demo and a few startling successes, users can begin to treat AI not as one tool among many but as something that should solve any task if only it is asked correctly.

When AI Is Treated as a Universal Tool

An artist can work in oil paint or in Illustrator. If one medium is unsuitable, choosing another does not mean that the paint or software broke a promise. We have long accepted that tools have appropriate and inappropriate uses.

Our tolerance seems lower around AI. When option A does not fit, we are less willing to move to option B. We rewrite the same request, tune the prompt, and wait for the model to eventually comply. If a service was sold as a universal solution, it is difficult to blame this entirely on unreasonable users. When possibility is promised broadly but failure is later explained as probability, expectation and responsibility no longer share the same boundary.

“It worked last time. Why does it fail now?” The developer explains that the model has reached the edge of its current capability, generation is stochastic, and the system is sufficiently black-box that the exact cause of an individual output cannot be stated with confidence. Technically, this is often true. The same prompt can change with the seed and sampling conditions, and it is difficult to reconstruct a human-readable causal account of which internal representation created a particular error.

But this is not the answer the customer needs. The customer is not asking for a lecture on probabilistic models. They want to know why the work they entrusted to the service failed and what happens next. Saying that the inside of the model cannot be explained describes the situation without recovering the job. The developer ends up speaking for the model and absorbing complaints in the gap between what the service promised and what the technology can guarantee.

A Quasi-Agent That Cannot Be Held Accountable

AI occupies a peculiar place in this exchange. It enters work between people as though it were another worker, yet it cannot explain its own inability. When it succeeds, it appears to be the author of a decision. When it fails, it retreats into the status of a probabilistic tool. It performs part of the job but cannot admit fault, bear a loss, or promise a corrective action. It is something like a quasi-agent with delegated authority and no delegated accountability.

Responsibility for the output therefore returns to people: the designer who approves the generated image, the QC operator who watches the final frame, the developer who connects the model to a service, and the organization that chose to deploy it. AI may be a cause inside the chain of responsibility, but it cannot be a responsible party. It can generate a sentence that resembles an explanation of its decision, but that sentence is not necessarily a trace of the process that produced the decision.

Adding Human QC does not resolve this automatically. A person being last in the loop is not proof that a person exercised meaningful judgment. Faster generation can produce more material to inspect while leaving less time for inspection. If the final approver lacks the expertise to find errors, the authority to reject the output, or the budget to remake it, Human-in-the-loop becomes an appearance of accountability rather than a safety mechanism.

There is also a verification cost that is easy to hide. If AI reduces a ten-minute generation task to ten seconds but a person still has to inspect every frame, fact, right, and piece of context, the cost of the entire workflow does not fall by the same ratio. Generation becomes cheaper, while verification and recovery remain expensive. As AI produces more candidates, it may also increase the amount of human judgment required. I think of this as a verification tax imposed by generative systems.

None of this means that developers and service providers can explain nothing because the model is a black box. Even when the exact path to an individual output is unavailable, teams can measure the conditions under which failure rates rise. They can record the model, seed, input, settings, and edit history; add rules and evaluators for unacceptable outputs; and decide whether a failure should be handled by a person or routed back to a conventional workflow.

The cause of a model failure may remain unclear. The failure boundary of the product should not. A service should describe what it does not guarantee, how much variation to expect, and which tasks should not use it, rather than showing only its best outputs. A path to option B when AI fails should be part of the product. This is not a lack of confidence in the technology. It is the minimum contract required to turn probabilistic technology into a service.

If a human is made responsible for approval, that person needs matching authority. They must be able to reject the result, stop using the model, switch methods, and receive enough time and information to make the decision. When the goal of AI adoption is merely to place Human QC at the end and make ownership of failure legible, the work has been automated while only the responsibility remains human.

A Revisable Map, Not a Perfect Mirror

The same question reaches deeper into research on AGI and world models. We train on more data, attach longer context and memory, and build agents that act in environments in an effort to close gaps in world understanding. These efforts matter. Still, no model of any size becomes the world itself. The world keeps changing, observations remain partial, and every finite representation has to leave something out.

If the goal is a perfect mirror, we are working on a problem that never ends. Intelligence, however, may not need to be a perfect mirror. A more realistic goal is a model that can live with the incompleteness of its map: revise representations with new observations, expose what it does not know, learn from the consequences of action, and distinguish confirmed facts from approximation.

Today's generative models are strong at abstraction and prediction but weak at closing the loop between an error in the world and the prior used next time. People fill that gap. We rewrite prompts, draw masks, compare outputs, repair mistakes, explain failures, and accept responsibility. Human QC may be more than an inspection stage attached outside the model. It may be the world-update process that the model itself still lacks.

This is not an argument to stop using AI. I study it and intend to keep using it. But generative AI should be selected according to purpose and failure cost, just as an artist chooses between oil paint and Illustrator. It can be used boldly when an entire result may be generated anew. Work in which a tiny change carries a large responsibility still needs structured tools and people. At times, not using AI may be the most technically mature choice.

AI is neither a god nor a target that can absorb responsibility for failure. If a model cannot explain its own inability, the people building the service must explain the conditions that expose that inability and the choices available afterward. If a person is accountable for approving the result, that person must have the power to reject and reverse it. The danger is not simply that we use models that cannot remember the world perfectly. It is that we believe they do, then entrust them with the responsibilities that exist between people.