I’ve been building Rehearsal at Gradeless AI — a voice-powered interview practice platform serving 2,000+ candidates across IIMs, Engineering interview prep, and campus placements. Here’s what nobody tells you about scaling AI simulations.
The Problem Nobody Talks About
Most AI interview tools treat each session like a random Q&A generator. Throw some questions at the candidate, get answers, done.
Three things break at scale:
1. Repetition kills learning. Users practicing multiple times start recognizing patterns. Your “practice platform” becomes a memorization tool.
2. Randomness misses the point. Pure random selection means some users get three behavioral questions and zero technical. That’s not how real interviews work.
3. Surface answers go unchallenged. Real interviewers don’t let you off with “I had a conflict with my manager.” They dig deeper. Most AI tools don’t.
I learned this watching early users game the system. Practice 10 times, memorize the patterns, think you’re ready. Spoiler: they weren’t.
Building DeepProbe: Rethinking Adaptive Questioning
The insight that changed everything: real interviews have structure AND unpredictability.
Think about it. A good interviewer covers multiple dimensions — your experience, problem-solving ability, domain knowledge, cultural fit. But the order varies. The specific questions vary. And they always follow up when your answer is thin.
DeepProbe was built to mimic this behavior.
Coverage Without Predictability
The system ensures every interview touches multiple competency areas, but the selection and sequencing changes each time. You can’t game it by memorizing question order.
The hard part wasn’t the randomization — it was defining what “comprehensive coverage” means across different interview types. A finance role needs different dimensions than a product role. We had to map this carefully.
Forced Depth
Here’s what most AI interview tools get wrong: they accept surface answers.
Q: “Tell me about a challenge at work”
A: “I had a conflict with my manager...”
→ Move to next topic.That’s not interviewing. That’s a checklist.
DeepProbe forces follow-up questions. If you give a thin answer, the AI probes deeper. Not randomly — intelligently, based on what’s missing from your response.
This single change had the biggest impact on user feedback. People said it finally felt like talking to a real interviewer, not a chatbot reading from a script.
The Freshness Problem
Even with good coverage, users practicing 15+ times will eventually see familiar questions.
First instinct: track what’s been asked and exclude it.
The problem? Context matters. Someone practicing for sales roles shouldn’t have their question pool affected by finance practice sessions.
What worked: domain-aware rotation that treats each practice context separately. Questions cycle back in eventually, but with enough gap that it feels fresh.
The specific parameters took experimentation. Too aggressive and users complained about repetition. Too loose and the question pool got exhausted too fast. Finding the balance required watching actual usage patterns, not guessing.
The Latency vs. Accuracy Trade-off
Voice AI has a constraint that text chat doesn’t have: time pressure.
In a text chat, 2-3 second response times are fine. Users are typing anyway.
In voice? Anything over 500ms feels broken. Users think the system crashed. They start talking again. The conversation derails.
But faster models are less accurate. They hallucinate more. They miss context from earlier in the conversation.
My solution: not all parts of the system need the same model.
Real-time conversation needs speed above all. Post-interview analysis can take longer — accuracy matters more there. Report generation? Use the best available model. Users will wait 30 seconds for a good report.
This seems obvious now. But my first version used the same model everywhere. Latency was terrible. Users hated it.
What Actually Moved the Needle
After months of iteration, three things made the biggest difference:
1. Automatic context detection. Instead of asking users to configure their practice session, the system infers context from their resume and job description. Less friction, better targeting.
2. Graceful degradation. When the AI can’t determine context confidently, it falls back to general-purpose questions rather than failing. Users never see an error — they just get a slightly less personalized experience.
3. Recording decisions immediately. This is a boring infrastructure detail, but it matters. The system logs what questions were selected at selection time, not after the interview. This prevents weird edge cases in the rotation logic.
The Uncomfortable Truth
Building AI products that work in production is 80% engineering, 20% AI.
The prompts matter. Model selection matters. But the real work is handling edge cases, building feedback loops, and making systems that degrade gracefully when things go wrong.
DeepProbe isn’t a magical AI breakthrough. It’s careful system design solving specific user problems: repetition, coverage gaps, shallow questioning, latency.
What’s Next
Currently working on:
- Cross-session memory (AI adapts based on your improvement patterns)
- Cohort analytics (how do you compare to others targeting similar roles)
- Smarter caching (reducing costs without sacrificing quality)
If you’re building AI applications and fighting similar problems, I’d love to hear what’s working for you.
Building AI systems that actually work in production. More practitioner notes coming soon