AskMyGuru · AI Quality
AI Evaluation Framework
How do you test a system when there isn't always one correct answer?
Context
AskMyGuru's product is an AI astrology chatbot. It answers in natural language, and no two answers to the same question are worded the same way. Functional tests were already in place and were passing. That did not mean the answers were right.
The Problem
A functional test can confirm that the bot responded, that it responded in time, and that the response matches a shape. None of that checks whether the response is true, whether it stays grounded in what the system actually knows, or whether it contradicts itself two turns later. Those are the failures a user notices and a green test suite does not.
How It Worked Before
The suite asserted structure, not content. A response could be well-formed, confident, and wrong, and nothing would fail.
Catching that kind of failure meant a person reading transcripts, which does not scale past a handful of conversations and catches whatever the reader happens to notice.
The Engineering Question
How do you score correctness in a system that has many right answers, many wrong answers, and no fixed string to compare against?
Architecture
An evaluation dataset supplies prompts to the application model. Its responses, together with the source context, go to a judge model, which scores them across named dimensions to produce a report.
The pipeline runs the application model the same way production does, then hands its output, together with the context that output should be grounded in, to a separate judge model rather than to a string comparison.
Technical Decisions
Why an LLM judge rather than assertions?
A deterministic assertion needs one correct string. This system has many correct answers and many wrong ones that share vocabulary with the right one — an assertion can't tell them apart, but a model reading for meaning can.
How do you stop the judge being wrong?
The judge scores a response against supplied source context rather than against its own knowledge of astrology. That turns an open question — "is this true" — into a checkable one — "does this claim trace back to the material it was given."
Why these five dimensions?
Each one maps to a failure mode I had actually seen: confident invention, drift from the source material, answers that are on-topic but not responsive, self-contradiction across a session, and plain factual error. They are not a generic checklist.
Outcomes
What the test suite could detect
The suite found grounding and factual-accuracy failures that had been shipping under a fully green functional run. The functional tests were not wrong about what they measured — they were just measuring the wrong thing for this kind of system.
What I'd Change Today
The evaluation dataset was assembled from failures I already knew about. That makes it good at catching regressions of known problems and weak at catching the failure mode nobody has thought of yet — it under-samples the unknown case by construction.
I would also add a regression baseline for the judge itself. Right now a judge score is only meaningful against the judge model version that produced it. If that model changes underneath the framework, scores can drift for reasons that have nothing to do with the application getting better or worse, and nothing in the pipeline today would catch that.