EN
LearningAdvancedVLAVLMOpenVLApi0SmolVLAGR00Taction chunk

Vision-language-action models (VLA)

In 30 secondsA VLA starts from a model that already links images and text and adds an action output: it takes camera images, an instruction and the robot's state, and returns a chunk of movements.

Also available in: Català · Castellano

The idea

A VLM (vision-language model) already links images with text. A VLA adds an output: robot actions.

  • In: camera images, an instruction (“put the cube in the box”) and the robot’s state.
  • Out: a chunk of joint targets, executed while the next one is being computed.

How it’s trained

You start from a model trained on huge amounts of images and text and fine-tune it with robot demonstrations. Models to look up: OpenVLA, pi0, SmolVLA, GR00T.

What it doesn’t replace

Low-level control and kinematics are still there underneath. The model decides targets; something has to drive the joints there.

About the demo

It illustrates the data flow; it is not a real model.

Interactive demo

Related concepts

Learning

Imitation learning

A person teleoperates the robot, observation-action pairs are recorded and a policy is trained to copy them. It fails in states it has never seen; the fix is more varied demos and predicting several actions in a row.

Learning

Reinforcement learning

Nobody tells the robot how to do it, only whether it did well: the agent tries actions, gets a reward and learns a policy that maximises it. In simulation, thousands of copies train at once.