maple.utils.eval.BatchEvaluator
- class maple.utils.eval.BatchEvaluator(daemon_url: str = 'http://127.0.0.1:8000')
Orchestrator for batch evaluations.
Manages the execution of multiple evaluation episodes across tasks and seeds, communicating with the MAPLE daemon to run individual episodes. Supports both sequential and parallel execution with progress tracking.
The evaluator: - Builds episode lists from task/seed combinations - Executes episodes via daemon HTTP API - Aggregates results with statistics - Persists results to database - Provides progress callbacks
Methods
__init__([daemon_url])Initialize the batch evaluator.
run(policy_id, env_id, tasks[, seeds, ...])Run batch evaluation across multiple tasks and seeds.
run_single(policy_id, env_id, task[, ...])Run a single evaluation episode.
Attributes
Lazy-initialized requests session.