Researchers at Delft University of Technology have developed an approach that allows passengers to personalise how an autonomous vehicle drives using natural-language commands. Instead of allowing a large language model (LLM) to control the vehicle directly, the system uses the LLM to interpret requests or indications that the passenger feels dizzy. These preferences are translated into adjustments to parameters used by a conventional, safety-aware motion planner, affecting characteristics such as speed, acceleration, steering smoothness and collision avoidance. The system also keeps the passenger in the loop by explaining the proposed changes in everyday language and asking for confirmation before applying them. Tests using the nuPlan autonomous-driving simulator showed that the system could successfully adapt driving behaviour to eight different passenger prompts, producing smoother driving when comfort was prioritised and higher speeds when urgency was expressed.
The approach is significant because it combines the flexibility and natural-language understanding of LLMs with the predictability of traditional autonomous-driving controllers. Rather than relying on an LLM for real-time driving decisions—which can suffer from latency, hallucinations and a lack of performance guarantees—the researchers use GPT-4o-mini only to translate subjective passenger preferences into changes within predefined safe limits. This separation means the underlying motion planner remains responsible for vehicle control, reducing the potential consequences of incorrect LLM outputs. However, researchers note that maintaining safe operating constraints still requires substantial engineering, and the Delft approach does not yet provide mathematically provable safety. Other work is therefore exploring additional verification layers that mathematically check AI-generated driving decisions against traffic rules and predicted road-user behaviour before they are executed.
More information:
https://spectrum.ieee.org/autonomous-vehicles-motion-planner-llm