Multimodal Agents: From Automation toward Open-Ended Self-Improvement

Overview

Recent progress in foundation models is enabling a new generation of agents that can perceive, reason, and interact with environments. Yet moving from narrow automation to open-ended self-improvement remains challenging.

This thesis develops methods for scalable multimodal agents and outlines a practical path from automation toward open-ended self-improvement:

i) Societies of agents.

We motivate natural-language societies of mind, where specialized agents communicate through a shared language interface to integrate knowledge, tools, and multimodal observations.

ii) Standardized workflows.

We present MetaGPT, which distills human standardized operating procedures into role-based collaboration with explicit intermediate artifacts for software development.

iii) Graph-based optimization.

We introduce GPTSwarm, a graph representation of agent systems that supports node-level improvement and system-level optimization.

iv) Scalable evaluation.

We propose Agent-as-a-Judge and the DevAI benchmark to assess developer agents using intermediate evidence and task dependencies, reducing reliance on purely human scoring.

v) Interactive environment modeling.

We study neural computers, action-conditioned generative models that learn command-line and desktop dynamics, enabling more controllable interfaces for agents.

Together, these contributions establish practical methodologies and evaluation frameworks for building multimodal agent systems that move beyond one-off automation toward continual self-improvement.