Alfred, Part 2: Coaching New Tasks

Part 1 introduced Alfred and how diverse prompting lets it learn from more kinds of data. This part follows a single task from start to finish: teaching Alfred to use an air fryer it was never trained on.

A task Alfred was never shown

The hardest test for a robot model is a new task: one it was never shown, described only in words. We first saw Alfred handle this when we asked it to operate kitchen appliances. We never collected demonstrations of these appliance tasks. Instead, we tried to talk the model through them.

Asked cold to cook a sweet potato in an air fryer, Alfred makes a reasonable attempt. After a few false starts it gets part of the way there, but doesn't finish:

Coaching it through, step by step

Then we coached it the way you might talk a person through an unfamiliar appliance: one short instruction at a time. With this language coaching, Alfred performs the task far more effectively. That is harder than it sounds. The model has to understand each fine-grained instruction and ground it correctly in what it sees:

From coaching to autonomy

After a few rounds of coaching, we use those instructions to fine-tune a high-level policy that writes the step-by-step instructions itself. Alfred then runs the task fully autonomously, without any additional teleoperation. In effect, it has learned the task from being coached:

Where did it learn about air fryers?

We wanted to know where Alfred learned what an air fryer even is. Our training set is too large and varied to trace the behavior back to specific episodes, and the knowledge probably comes from a mix of robot data and web-scale vision-language pre-training.

After a lot of searching, the closest matches we found were two episodes from a home where a robot closes an air fryer (labeled “push the frying basket into the airfryer” and “put the basket of the airfryer on the leftmost side of the counter”), plus Franka robot data from the open-source DROID dataset. None of them look much like what our mobile robot does here. Alfred appears to have composed what it needed from these pieces, much as an LLM composes text from fragments it has seen across the web:

Teaching robots with words

The air fryer is one example of a broader pattern. With step-by-step coaching, Alfred can carry out a task we never collected demonstrations for. After a few rounds, a high-level policy can write those instructions itself and run the task fully autonomously, with no additional teleoperation.

Steerable models like this let people teach robots new behaviors with words rather than demonstrations. If you would like to work on these problems, or apply models like Alfred to real-world robotics, you can contact us at research@blinklaboratory.xyz.