Back to work RAG COMPLIANCE ASSISTANT, MULTI-AGENT EVAL

AdClaim

A retrieval-augmented marketing-compliance assistant that answers questions like whether influencers have to disclose paid posts with a grounded, cited answer, or a clean refusal when a question falls outside the corpus. Built with CrewAI over official US FTC advertising guidance (Endorsement Guides, dot-com Disclosures, Health Products Compliance Guidance, Green Guides, Made in USA, CAN-SPAM). The portfolio point isn't the assistant, it's the eval: a faithfulness-verifier agent is measured against a plain retrieve-then-answer baseline, and the README reports the result plainly rather than favorably.

View repo

Architecture & Trade-offs

What was built, and what was deliberately chosen against.

pipeline.txt

question
    │
    ▼
planner agent       reframes into retrieval queries
    │
    ▼
retriever agent      Chroma similarity search, top-k chunks
    │
    ▼
answerer agent       drafts cited answer from retrieved chunks only
    │
    ▼
verifier agent       ablatable: use_verifier true or false
  checks every claim against retrieved chunks
  forces refusal on any unsupported claim or out-of-corpus question
    │
    ▼
final answer: answer, citations, refused

The verifier is ablatable by design

The four-agent crew (planner, retriever, answerer, verifier) runs as a CrewAI sequential pipeline, but the verifier stage is a single flag (use_verifier), not baked into the answerer. That's what makes the eval possible: the same pipeline runs with or without the verifier stage, so its effect can be measured in isolation instead of assumed.

Structured output, not re-parsed free text

The answerer and verifier tasks emit validated Pydantic objects rather than free text to re-parse. The refusal contract itself is enforced in plain Python on that structured verdict, not by trusting the model's self-report of whether it verified anything.

The eval reports what happened, not what was hoped for

Measured over the same 29 answerable and 11 unanswerable questions, the verifier improved retrieval-facing Ragas metrics (context precision and context recall both up) but not generation-facing ones (faithfulness and answer relevancy both down) against the baseline. Refusal accuracy tied at a perfect score in both configurations, but the README is explicit that this isn't evidence the verifier adds refusal value: the unanswerable set is all obviously out-of-scope questions the baseline's own nothing-retrieved-so-refuse path already catches, before the verifier stage would ever matter.

A named next step, not a hidden gap

The current unanswerable set can't actually differentiate the two configurations on refusal judgment, since it has no near-miss questions plausible enough to test whether the verifier catches something the baseline's simpler heuristic would miss. That's documented as the explicit next step, not smoothed over.

Stack

Python, CrewAI, Chroma, Ragas, OpenAI

Status

Working prototype with a documented eval, not a deployed service. Public repo with a runnable quickstart: ingest, ask, eval, and a network-free test suite.

Public repo. View the repo on GitHub.

Back to work