Ziv Seker from Microsoft shows how to build synthetic users in ChatGPT based on Stanford research that achieved 85% accuracy predicting real user responses.
The Stanford study started with two-hour audio interviews from over 1,000 participants discussing topics like the economy and COVID-19. Researchers left out some questions, then used AI to generate synthetic responses and compared them to actual answers - achieving 85% accuracy.
Hereâs Zivâs demonstration of their method:
First, avoid the naive approach. Simply prompting âYou are a 38-year-old American Jewish product manager, please respond as if you are realâ wonât work - itâs completely out of context and produces poor results.
Second, provide the full interview transcript to ChatGPT. Ziv uses a free dataset from Princeton University and gives it to the AI.
Third, generate expert perspectives. Prompt the AI: âPlease make observations as you would as each one of these expertsâ (listing experts like psychologists, behavioral economists, etc.). The AI produces 5-20 bullet points from each expertâs perspective analyzing the interview.
Fourth, select the right expert for each question. Ask: âPlease let me know which expert would be bestâ for the specific question youâre testing. Different questions benefit from different expert lenses.
Fifth, combine everything for the synthetic response. Using the expert reflection, the interview transcript, and the new question, generate the synthetic userâs answer.
When Ziv tests this with a question left out of the original interview, the synthetic response isnât exactly what the real person said, but gives âa pretty good idea where we are.â
Zivâs key insight: âSynthetic users can help us test assumptions and understand if weâre completely off-track, but they canât tell us definitively what will work. They wonât replace user research, but they can help us come more prepared to interviews and maybe help us understand general directions for smaller questions.â
âĄď¸ Use synthetic users to pressure-test assumptions before real interviews, not to replace them. The 85% accuracy is good enough to catch when youâre completely off-base, saving time and improving real research quality.
Put LLMs to work
Synthetic users
Build synthetic users that predict real user responses with 85% accuracy