Alfred: a Steerable Model with Emergent Capabilities

Alfred is our new general-purpose robot model. Out of the box, it matches the speed and robustness of specialist models fine-tuned for a single task, across a wide range of dexterous work. More importantly, it can follow language commands it has never seen and carry out tasks that never appeared in its training data.

The reason is a kind of generalization robots haven't shown before: composition. Alfred recombines skills learned from different tasks to solve new problems, like cooking a sweet potato in an air fryer it was never trained to use.

Why composition matters

Language models compose all the time. If an LLM knows how to translate English to French, and separately knows how to produce JSON, it can return a French translation formatted as JSON without ever having seen that combination. That ability is a large part of what makes them useful.

Robot models have not shown it yet. Vision-language-action models understand a wide range of semantic concepts, but they have not been able to combine skills in new ways, such as using an unfamiliar tool or appliance. Even on skills they have seen, the best results usually come from fine-tuning the model for that one skill, much as early language models were fine-tuned for each problem domain.

A true generalist should perform every skill well without fine-tuning, and recombine those skills when it meets something new. Alfred is our first model to show early signs of both. It performs dexterous skills with the same speed and robustness as our RL fine-tuned specialists, composes what it has learned to solve new tasks, and generalizes across robots, scenes, and tasks better than our prior models.

One model, straight out of the box

Everything below is the same model with no task-specific fine-tuning, from fine manipulation to long-horizon household work.

Telling the model how, not just what

Foundation models generalize when they train on broad, diverse data. For Alfred, that means data from many different robots and control modes, videos of people, and autonomous episodes collected by running earlier policies. Simply pooling all of it does not work well: the sources differ in strategy, speed, and skill.

The fix is to give the model diverse context in its prompt. We train Alfred with many kinds of multimodal prompts that describe not only what the robot should do, but how it should do it:

• Diverse language that describes the task and individual sub-steps.

• Metadata that describes how the task was performed, such as speed and quality.

• Control modality labels that indicate whether to use joint or end-effector control.

• Visual subgoal images that show what the end of the current sub-step should look like. These images can be generated at test time by a world model that provides for visual generalization.

Each of these disambiguates the behavior. A visual subgoal pins down exactly where objects should end up; a target episode length says how quickly to work. Because every example is labeled with how it was done, data with different strategies and levels of skill can all go into training. Even suboptimal autonomous data, which would ordinarily teach the model lower-quality actions, becomes useful once it is labeled as lower quality or lower speed.

At test time, Alfred takes an ordinary language instruction and, when we want, guidance about strategy and visual subgoals generated on the fly by a lightweight world model.

Continued in Part 2: how language coaching teaches Alfred a task it was never shown.