I'm sure you've heard about the mystery of AI models being "black boxes", where even the smartest engineers can't tell with certainty how the models learn and evolve.
That's scary and fascinating at the same time, and for someone who loves to understand how things work, frustrating as well.
It also got me thinking about what would make it easier (and cool) for virtualX users and future clients to understand how their virtual expert really works behind the scenes.
They already get the logs. Under every answer the expert sees which entries it stood on, which ones it looked at and didn't use, how long the client waited and what the answer cost.
But that's not enough, because I wanted them to really see what the brain looks like and how it connects the dots.
1/ Connecting the dots
I would never put an AI in front of someone's clients if the expert couldn't see what it was doing in their name.
So this is what the inside of one looks like. Every dot is something the expert wrote and approved.
Lit dots, are things person asked about it during chat conversation. Dark dots represent what nobody has asked about yet.
Red dots are interesting, and they represent what virtual expert refused and sent the question back to a real human.
If an answer doesn't sound like him, he fixes one entry and it's fixed everywhere - it takes about a minute.
I'll admit, I wanted it to look cool, but the point was to make it useful when you're fine-tuning how your virtual expert communicates:
- What does it look like when it's leading an excellent conversation, and what does it look like when it still can't model your flow?
- Why does some of your advice show up in answers more often than the rest?
- Which boundaries are doing the work, and which have never been needed?
- What have your clients never asked about?
Each of those is one view of the map. At a glance, what's alive, what people come for, where it falls through, the fence, and this week.
You click through them the way you'd flip through pages.
2/ 20% is the problem
Over the past six months of conversations I've learned that even the AI sceptics want to find a way to use at least some form of AI in their client work, but they want as much control as possible when building these tools.
No one is worried about a tool that answers correctly 80% of the time. The problem is the remaining 20%, when it sometimes invent stuff.
And don't get me wrong, a 100% hallucination-free AI tool is impossible.
I went through the numbers in Letter 034: plain ChatGPT or Claude made up legal answers 58% to 88% of the time in a Stanford test, the professional "trained on your documents" tools were still wrong 17% to 33% of the time, and even a model locked down to only restate a document you hand it is wrong about 1% of the time.
OpenAI's own researchers explained why. These models learn the way students learn for exams: a blank answer scores zero, a confident guess sometimes lands full marks, so guessing wins and the habit gets baked in.
The fix was never more intelligence. It was permission to not answer.
That's why, with virtualX, I built one where you can see:
- what it answered from, under every single reply
- what it looked at and chose not to use, so you can tell "it picked the wrong entry" from "it had nothing"
- where it said "I don't know" and sent the question to you
- where it refused, in your own words
- which of your entries nobody has asked about yet
3/ It will get things wrong
Your virtual expert may from time to time find the right knowledge and then combine it in a way that doesn't make sense. When that happens the system logs it, and the human behind it fixes the logic.
We caught exactly this last week. A check we built read every answer on two live brains and found steps inside answers that no entry had. The fix was one rule for the writer: every step comes from the entries it was shown, nothing added in between. The entries didn't change, the behaviour did.
And for those of you who care more about the nerdy technical side than a pretty map, this is what that check prints for one answer:
Every step the virtual expert gave is quoted, and under it the sentence from my entry it came from. Step four has nothing under it. That's the one it made up, and that's the one that gets flagged. No scores, no magic, just quotes checked against quotes.
Plus, here are a few more of the questions I get, and what actually happens.
"What if a client asks about something I never wrote?"
Then there's nothing for it to answer from, so it says "I don't know" and sends the question to you. On the map it sits under "where it falls through", in the client's own words. You answer it once, in one box, the client gets your answer under their question, and your answer becomes a new entry. The next person who asks gets it from the start.
"What if someone asks it something it should never answer?"
That's what the boundaries are for. You write them in your own words, and when a question crosses one, your virtual expert says no in your words and sends the question to you. These are the red dots on the ring around the brain. The map counts how many times each one held, and tells you which ones have never been needed.
"What if it finds the right entry and still sounds nothing like me?"
The centre of every map is one cell called "How Virtual [your name] sounds": your voice and your rules. Every lit dot links back to it. When an answer is off, you don't retrain anything. You open the entry it stood on, fix the sentence, and it's fixed everywhere. That's the minute I mentioned above.
Next part of the series: I share my screen, add the first entries to Virtual Filip's brain and draw its boundaries, so you see where every dot on a map like this comes from.
"Filip, you're overcomplicating this"
If you're thinking: "I just need a simple chatbox that answers questions from the materials I've uploaded"
...then the way I'm approaching AI in client work is not a good fit for us.
But if you teach, coach or consult, and your clients are already asking ChatGPT about your materials with zero clue what's correct and what isn't, then this is for you.
Two spots left in October
The way in is the same as last time: I build a free demo from what you've already published, so you can see what your virtual expert would answer, where it would say "I don't know", and what it would refuse.
Then we talk about whether one of the two October founding spots makes sense.
Apply for a free demo
-Filip "lit, dark and red" Sardi
Frequently Asked Questions
What do the lit, dark and red dots on the brain map mean?
Every dot is something the expert wrote and approved. Lit dots are entries people asked about in conversations. Dark dots are what nobody has asked about yet. Red dots are what the virtual expert refused, sending the question back to a real human. The centre cell is the expert's voice and rules, and every lit dot links back to it.
Can an AI tool be 100% free of hallucination?
No. In Letter 034 Filip went through the numbers: plain ChatGPT or Claude made up legal answers 58% to 88% of the time in a Stanford test, professional 'trained on your documents' tools were still wrong 17% to 33% of the time, and even a model locked down to only restate a document is wrong about 1% of the time. The fix was never more intelligence but permission to not answer.
What happens when a client asks about something the expert never wrote?
There is nothing to answer from, so the virtual expert says 'I don't know' and sends the question to the expert. On the map it sits under 'where it falls through', in the client's own words. The expert answers once, in one box, the client gets the answer under their question, and it becomes a new entry for the next person who asks.
How are wrong answers caught and fixed?
A check reads every answer and quotes each step beside the entry sentence it came from. A step with nothing under it is the one that was made up, and it gets flagged. The fix that came out of this was one rule for the writer: every step comes from the entries it was shown, nothing added in between. When an answer sounds off, the expert fixes one entry and it's fixed everywhere, in about a minute.
What are boundaries in a virtual expert?
Rules the expert writes in their own words for questions the AI should never answer. When a question crosses one, the virtual expert says no in those words and sends the question to the expert. They are the red dots on the ring around the brain, and the map counts how often each one held and which have never been needed.
Client Flow Letter
If this was useful, the next one will be too.
Retention strategy for coaches and founders — every week. No filler.
