AskMyGuru · AI Quality · Automation
AI Test-Case Generation
Can a system read a PRD and a Figma file and produce test cases a human would have written?
Context
AskMyGuru is a seed-stage AI product. Features ship weekly, and every feature arrives as a product requirement in Notion plus a Figma file. Before any of it could be tested, someone had to read both documents end to end and translate them into test cases.
That translation was the bottleneck. It took a day to a day and a half per feature, it was almost entirely mechanical, and it scaled linearly with the number of features — which is the shape of a problem worth automating.
The Problem
Test design is repetitive in a specific way. Roughly eighty percent of the work is transcription: reading a requirement, identifying the states it implies, and writing out the obvious permutations. The remaining twenty percent is judgment — knowing which edge cases actually matter for this product, at this stage, with these users.
Automating the eighty percent is valuable. Automating the twenty percent is dangerous, because a plausible-looking test case that tests the wrong thing is worse than no test case at all: it produces the appearance of coverage.
How It Worked Before
Read the PRD. Read the Figma. Write cases in a document. Review them. Move them into the test repository. Repeat next week.
Every step was manual, and nothing carried forward. Two features with nearly identical authentication flows produced two independently hand-written sets of authentication test cases.
The Engineering Question
Can a system read a product requirement and a design file and produce test cases a human would have written — while keeping the human as the one who decides what ships?
Architecture
Product requirements from Notion and design files from Figma feed a context extraction layer. The context layer feeds a generation engine. Generated cases go to human review, and only reviewed cases reach the test repository.
The important structural decision is that human review sits inside the pipeline, not beside it. Nothing reaches the test repository without passing through it.
Technical Decisions
Why a context layer instead of pasting the requirement into a prompt?
Pasting a raw requirement produces cases about the document. Extracting entities, states and flows first produces cases about the system the document describes. The extraction step is also where domain constraints get applied — it is the part that knows what this product considers an edge case.
Why keep a human in the loop when the goal was to remove manual work?
Because the goal was never to remove the human, it was to remove transcription. A generated case that looks correct and tests the wrong behaviour is more expensive than no case, because it survives review by looking plausible. Keeping the accept decision with the engineer preserves the twenty percent that is actually judgment.
Why extend the context layer with HLD and LLD later, rather than starting there?
The first version worked from product requirements alone and produced good functional cases and poor integration cases — it could not see the seams between services because the PRD does not describe them. Adding design documents was a response to an observed failure, not a guess.
Outcomes
Test design per feature, including human review
The measured number includes review time, which matters: a generation step that is instant but produces cases requiring two hours of correction has not saved anything. The pipeline is only worth its cost because the review pass is short.
What I'd Change Today
The pipeline generates test cases but never evaluates them, which is an obvious gap given that I built an LLM evaluation framework for the same product. Generated cases should be scored for grounding — does this case trace back to something the requirement actually says — before a human ever sees them. That would shorten review further and, more importantly, would make the failure mode visible instead of leaving it to be caught by whoever is reviewing that day.
I would also version the context extraction separately from the generation step. Today a change to either one changes the output, and it is harder than it should be to tell which change caused a regression in case quality.