Project

MAVEN

Asks several language models to review another model's answer and flags likely made-up facts and invented sources.

What it does

Sends a question and an answer to several reviewer models, and each votes RELIABLE, QUESTIONABLE or UNRELIABLE.
One model runs a fact check and another a citation check for sources that look invented.
Fixed rules turn the votes and flags into a LOW, MEDIUM, HIGH or CRITICAL risk level with a full trace, from Python or from a CLI that exits with code 3 on HIGH or CRITICAL.

How it works

Checking one answer with N reviewers costs N + 2 API calls. Replies are plain text, so MAVEN reads the labelled fields (VERDICT, FACTS_FAILED, SUSPICIOUS) with regular expressions, and a reply without a verdict counts as QUESTIONABLE. An async detector sends the calls at once under a semaphore and retries rate limits. Adapters cover Together AI, OpenAI, Anthropic and Google, plus mock models for tests.

The honest bit

Evaluation pending: a balanced TruthfulQA evaluation is written but has not been run, so there are no numbers here. Earlier accuracy claims were withdrawn in public after their labelling turned out to be broken. It started as a multi-model debate, which did worse than a single model in informal tests, so it became a panel of reviewers instead. The reviewers only read the question and the answer, so models with a shared blind spot will agree.

Python
asyncio
OpenAI SDK
Anthropic SDK
tenacity
pytest
GitHub Actions