Maple - Model and Policy Learning Evaluation

Maple Logo

A unified CLI daemon for evaluating robotics policies across diverse simulation environments

GitHub PyPI Documentation License: MIT

Why Maple?

The Problem

Evaluating robotics policies—whether Vision-Language-Action (VLA) models, foundation models, imitation learning policies, or reinforcement learning agents—across different simulation environments is painful:

  • Environment chaos: Every simulator has its own observation format, action space, and API quirks

  • Dependency hell: MuJoCo, PyBullet, Isaac Gym, LIBERO—each with conflicting dependencies

  • Integration tax: Each policy-environment combination requires custom glue code

  • No standardization: Comparing policies across environments means rewriting evaluation scripts

The Solution

Maple provides a daemon-based architecture that:

  1. Containerizes everything — Policies and environments run in isolated Docker containers

  2. Standardizes the interface — One CLI to rule them all

  3. Handles the translation — Adapters automatically convert between policy outputs and environment inputs

  4. Scales evaluation — Batch evaluation across tasks, seeds, and configurations

# Start the daemon
maple serve

# Pull and serve a policy
maple pull policy openvla:7b
maple serve policy openvla:7b

# Pull and serve an environment
maple pull env libero
maple serve env libero

# Run evaluation
maple eval openvla-7b-xxx libero-yyy --tasks libero_10 --seeds 0,1,2

That’s it. No dependency conflicts. No custom scripts. Just results.

Demo

Demo video coming soon

Key Features

🐳 Docker-First Architecture

Every policy and environment runs in its own container. No more dependency conflicts.

🔌 Adapter System

Automatic translation between policy outputs and environment inputs. Write once, evaluate everywhere.

📊 Batch Evaluation

Run evaluations across multiple tasks, seeds, and configurations with a single command. Get aggregated metrics and per-task breakdowns.

⚙️ Flexible Configuration

YAML config files, environment variables, or CLI flags—use what works for you.

🏥 Health Monitoring

Background health checks keep your containers running. Auto-restart on failure.

💾 Persistent State

SQLite-backed state storage. Resume evaluations, track history, query results.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                        maple CLI                            │
└─────────────────────────┬───────────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────────┐
│                     Maple Daemon                            │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │ Policy      │  │ Environment │  │ Adapter             │  │
│  │ Backends    │  │ Backends    │  │ Registry            │  │
│  │             │  │             │  │                     │  │
│  │ • OpenVLA   │  │ • LIBERO    │  │ • openvla:libero    │  │
│  │ • SmolVLA   │  │ • SimplerEnv│  │ • smolvla:libero    │  │
│  │ • ...       │  │ • ...       │  │ • ...               │  │
│  └──────┬──────┘  └──────┬──────┘  └─────────────────────┘  │
└─────────┼────────────────┼──────────────────────────────────┘
          │                │
          ▼                ▼
┌─────────────────-┐  ┌─────────────────┐
│ Policy Container │  │ Env Container   │
│ (Docker + GPU)   │  │ (Docker + X11)  │
│                  │  │                 │
│ HTTP: /act       │  │ HTTP: /step     │
│       /load      │  │       /reset    │
│       /health    │  │       /setup    │
└────────────────-─┘  └─────────────────┘

Supported Policies and Envs

All the list of supported environments and policies can be Found - Policies & Environments Reference. Feel free to suggest any specific policy or environment you would like to be added in the repository.