Skip to main content
If you’re finding it hard to get an AI agent working, you’re not alone. I ran a usability test with 5 AI-forward PMs and they all had a hard time. TLDR: The blocker isn’t AI, it’s all the mechanical stuff around it that makes it hard. I ran a usability test with 5 of the most AI-forward PMs at one public tech company. These were PMs that were using multiple AI tools daily at work (one had built a custom GitHub documentation scraper for AI context). To increase chances of success, they were all from the same company, had enterprise contract to an AI automation platform, and we delayed the workshop a few times so that all the IT permissions and approvals could be in place. After 3 hours only one got an automated AI workflow going: a Slack automation that extracted three pain points from user interview transcripts As one PM put it: “it sounds like a magic but while most of us all of us are pretty much advanced, and we are all struggling in creating, like, a really, really small thing.” Since these early adopting PMs are my yardstick, that says a lot more about the state of agents than their abilities.

​
Here’s what they did right:

  • They identified use cases that could really help them in their job (using AI to brainstorm use cases relevant to their roles)
  • They chose tasks that were “event-driven” rather than batch tasks.
  • They were able to scope them down to something that should have been simple to build

​
What blocked them:

  • IT permissions for simple tools like Drive and Slack, e.g. “it finds the Google Doc. It finds the Google Drive. But it can’t read the content of the file”
  • Not being able to send details that might identify customers through their automation platform
  • Missing integrations or API calls (One PM discovered they needed to use a manual “Google Docs API request” as a workaround to the platform integration)
  • AI agent platform UX was confusing (remember, these are super AI-forward operators)
In real life, off-LinkedIn, beyond the demo clips, this stuff is hard. A lot of what we did together was lowering expectations. Even if you get all the right IT permissions, even if you’re comfortable hacking together systems, even if you’ve done your homework and come up with a great use case (and scoped it down!) 
there’s no promise the hours you invested will be worthwhile. I usually like to share things that are more practical and optimistic. I wanted to share this one because I hope it’s reassuring and letting people know they’re not alone. PS A glimmer of excitement: recently agentic browsers (like Perplexity Comet) have shown a completely different approach to reaching the same goals
 local native desktop software that basically “sits over your shoulder” and uses the same UI and permissions you do—all without IT knowing the difference (mua ha ha ha ha). I’m excited to see where that line of innovation goes.

​
Let’s build together

If you’re reading this, you’re in the top 10% of people pushing the limits of AI at work. To get your entire team on board, I offer build sprints for individuals and companies who want hands-on implementation and adoption.

​
Get notified about future posts

Subscribe