Can you elaborate on how your mental model of how to structure projects has shifted with working through individual copilots/agents per initiative?
Tal: One thing I’ve noticed that changed is before the copilot, I would have these really messy, long Notion documents where all this new information would come in, or we’d change the direction or the solution. That would just be like a monologue with myself, basically. A lot of that has moved into the chat, so I can take the best parts of that, and my documents are a lot cleaner.I still think in terms of initiatives. Maybe that would be changing in the future, because right now each thread in the project doesn’t really know about the other threads unless you add it to the project knowledge. It’d be really interesting to see how that evolves and how that affects me too.
What are your best practices for establishing robust custom instructions?
Tal: I basically thought, what would I actually want if there’s like a Jiminy Cricket on my shoulder, or a person sitting next to me? What would I want that person constantly reminding me of? And I thought, what are the PMs I most admire? What are the values in the company? I didn’t use any special magic words or anything – I just in natural language said, “Please constantly push me to bias towards action, make decisions, try to get value to customers as soon as possible, involve stakeholders.” All this stuff. So I just kind of thought about the people that I admire or the principles I admire.
Have you found ways to have Claude unlearn previous context that might have changed as new information is added, decisions are made, external events happen?
Tal: That’s a trade-off. I don’t have a great answer there, because in some cases I really want it not to bias to the latest messages, and in some cases you do. I think one thing you could do is have it summarize all the new information and new things that happened in that thread and use that to start a new thread. You can edit that a little bit.If it’s something that happened really recently, and everything that happened after that was not relevant, there’s on every chat bubble that you write a little pencil icon you can click, and you can change what’s in there. Everything after that will just get erased and start all over. So it’s kind of like a rewind feature as well. It just depends on the situation, but it’s definitely not perfect. It’s difficult.
How do you find time to tinker or explore with AI? Do you use what you need, or test with random use cases to see how the processes and how it handles those requests?
Tal: At first I was really like, is this the irony of working on generative AI products but not using it in my workday? What got me to start using it was I had no choice. Once I opened up to “wait, I could use this,” I should try to keep asking, “How can I use LLMs?” And LLMs really improved over that time.It’s usually just like, let me try it, let me open the window and see if it can do it. Usually the first time it’s not what I wanted, and then I’ll edit that message and try again. Maybe add another sentence: “Oh, I should have told it not to do this,” or “be more succinct,” or “I should have told it to ignore that.” I go back, edit, and put it in. At some point I was like, “This works really well.” It actually takes very few iterations – surprisingly few iterations.Follow-up: How often are you surprised?Tal: I’m constantly surprised. I’ve been using both OpenAI and Claude and switch back and forth, mainly sticking to Claude because of the nice UI. I think at this point it’s not about being surprised by the technology. It’s kind of like, “Wow, this is… it just keeps thinking about things I wouldn’t have thought of.” It’s just a great thinking partner. It’s the same feeling when you work with somebody who’s really awesome at what they do, and they’re constantly mentioning that thing you would have never thought about on your own. You’re not surprised that they’re doing it, but it’s still this feeling of delight and surprise.
What blind spots, if any, have you encountered throughout this process? In other words, is there anything we should not rely on the copilot for?
Tal: It’s the same mental framework as for a coworker. If you bring up an idea and they say a bunch of stuff, you want to take the good and separate out the stuff you don’t agree with. So think of it as inspiration where you only get the good parts, and stuff that’s not inspiring you or doesn’t feel right – hit the brakes on it.The best analogy here – the term “blind spots” is a really good term – if you view it that way, it’s like Mobileye, the driving assistant. It makes mistakes a lot of times – it beeps when it’s not supposed to, but that doesn’t send my car into a tree. I just look at it and I’m like, “What are you talking about? I ignore you.” But there are a lot of moments where it really helps, and the upside is really significant. The downside is very lightweight, so the blind spots of Mobileye don’t bother me as much because I come at it with that mindset of just taking the good.
How do you overcome AI’s positivity? It seems to agree with everything, and even is incorrect rather than risk being straightforward or harsh.
Tal: I found Claude to be a little bit better about that than ChatGPT, although I think yesterday or the day before OpenAI announced that they improved the personality. The personality matters – when I started using all this material on ChatGPT, on a whim one morning I was like, “Let me just see how that would be on Claude,” and it really just felt like somebody I’d rather work with.The other part of it is in the custom instructions. I also just tell it to challenge my assumptions, ask me hard questions. I basically say, “Here’s what I need.”
Are you afraid that this will make you less creative?
Tal: I thought about that a lot. My mental model for that is kind of like, well, if I managed more people, if I had a business and I hired people to help me – hired really creative, really talented people – would that make me less creative? Or would that just give me more people to bounce things off of? Maybe it would make me more creative because it would inspire me and provoke me in really great ways. It’s still a conversation.That thought always goes through my mind – what if I was working with more people on my team? Would that make me less creative? Intuitively, the answer is no. Think of CEOs of really big companies – you’d expect them to be the least creative people, but they’re not. So that’s how I look at it.
Tal: I basically walk around the office, have conversations, sit back down at my desk, hit the dictation button, and share what just happened if I find it important. That’s basically also what would happen before I had a copilot. I’d sit next to my engineering manager or product designer, and I’d sit back down at my desk and be like, “You won’t believe the conversation I just had.” So it’s really similar to that.
What strategies have you seen succeed to overcome confirmation bias?
Tal: What I’ve done that helps me is I will ask it to be a devil’s advocate. I have a section in the playbook for that too. After it’s already made a lot of recommendations, asked me a lot of questions, and converged on a conclusion, I’ll say, “Okay, great. Now I want you to play devil’s advocate on the other options,” and it’ll do a really good job of that. Then after all that, I say, “Okay, now with all that, please make your recommendation.”The fact that you asked that question means you’re already aware of it. It’s just basically asking it to argue against itself, which is really fascinating to watch. It’s funny because I know it’s going to happen – I don’t want it because I just want it to agree with me and tell me I’m smart – but that’s exactly what you’ve got to do.
How do you balance between sharing the company-level data and context versus what’s only for that particular project?
Tal: It’s a good question. It’s true that they mix. I think if you think about it chronologically, the company-level context is basically everything you know up until that point. The initiative-level context is kind of what you’re figuring out. So there’s actually when you start a project, there’s not a whole lot of context, and you’re figuring out. At the very end, you say, “Okay, let’s take everything we learned from here and bring it back.”So I think in the beginning it’s kind of chronological – if you know it, it should probably be in the company-level context. If it’s uncertain and you’re figuring it out, work that through in a thread.
Do you structure your day with AI? Do you block off time to talk to it, or do you just sort of find free time wherever you can grab it?
Tal: I haven’t found myself blocking off time. It’s kind of like the two-minute rule – most of these things are two minutes once the ball is rolling. It’s kind of like I don’t block off time to shoot a message or text somebody or grab somebody in the hallway. These are all really short interactions, even dictation – a minute can be a lot of dictation.So I usually don’t plan it to be a thing I sit down and do. It’s more like I just give an update and then, “Oh, that’s a good question,” and I kind of get into a conversation with it, and time might fly by, but it wasn’t planned that way.
Given that context is king, I can imagine there’s a period as you’re getting started where the output of the copilot is not as useful. What strategies did you use to stick with it, and how long did it take until it was useful?
Tal: It’s pretty fast. I think with context, it’s very 80/20. If you just invest a little bit, you’re already going to see major improvements in the quality of the answers and the usefulness. That just makes you want to share more and give it more context.You get a lot of value really fast. What I’ve seen is the reason a lot of people don’t integrate AI more into their work is they’re not even doing that initial 20% because it’s still a little bit of work – but I’m talking like 5 minutes of work. Export a few docs, upload them, dictate a little bit. You’re already going to see a huge improvement that’s like 80% of the benefit, and it just makes you want to give more context. So it’s not like there’s a trough of sorrow or anything – it’s not very big, not much time.
So what about for a new PM? Could use of an AI copilot at this early stage be detrimental to my growth and development of PM craft?
Tal: As a new PM, the downside of this approach is that it works really well when you already kind of know what a successful product process looks like, you’ve been through it a few times, you have a good intuition, and you feel comfortable supervising something that needs to be supervised. So it might not be the best option.But it could be a very good option if, as somebody already answered, you tell it that you want it to help guide you. Maybe tailor the custom instructions. Again, it’s about discernment between the good and the bad. If you feel comfortable saying, “I know enough to tell you that this is good output or not good output” – if you feel you’re past that threshold where you can take the good and leave the bad, I think that’s all you need to make this valuable.
Have you ever shared its output without editing it?
Tal: No, I don’t just copy and paste, but what usually ends up happening is I’ll take pieces of the documents that it creates and put them into my document. I find that maybe it’s because I want that control, or maybe because I enjoy the high level – like I want the narrative of this document or what I’m saying. Let me just take the best: “Oh, that’s a really good line. That’s a great line. That’s a really good thought.” That’s usually what I converge on doing.
How much time did it take initially to build out your context?
Tal: The step that took the longest was finding it, getting in there, exporting it. It took me a while to be like, “Wait, all this… there’s a lot of stuff in slide decks. Oh wait, I can export that as a PDF.” So it’s basically the mechanics and finding it.In beta groups when we’ve tested this, it’s funny – people are like, “Oh, we really should have that doc. We don’t even have that for people.” One idea that came up was: if you don’t have that doc, use the AI as an excuse and have a meeting and talk about it. “What is our strategy? What is our…? It could be implicit – we know we’ve been operating with it, but let’s put it explicitly.” Take that transcript of that meeting, have Claude summarize it, and put it in as project knowledge.So there is that stone soup fable – the AI could be an excuse to go gather all those things.
Can you touch on the optimal formats for context docs that you feed it?
Tal: I know that LLMs do really well with Markdown. I remember that same conversation with the heads of product for Anthropic and OpenAI – they mentioned Markdown is just a very natural format. That’s how things were fed to it, so it works really naturally in that markup language.I know Claude is particularly good at XML, but who does that? I think it’s less critical at this point – it’ll do fine with everything, especially the latest models. So over time, these optimizations are going to be even less important.Take what you can get – it’s better to upload than not. If you have it in PDF form, upload in PDF form, it’s fine. It’ll read it. The Claude demo for this is the Apollo 11 Manual – it’s terribly scanned with tons of scratches all over it. I trust that if it’s not equal already, it’s going to be a negligible difference on the format.
Have you found a clever way to integrate either audio recording or transcripts from meetings into your context?
Tal: I’ll take the transcript of a meeting, and I won’t add it to the project knowledge as is. There’s a prompt included that works really well for me. In just a random thread, say, “Hey, can you distill this in the following ways? Just summarize these things. Maybe some exact quotes that were really good.” Then take that output and add it to the project knowledge. I find that keeps things cleaned up and saves space.
Because your emphasis is on dictation, why is dictation the best medium for communicating with your copilot?
Tal: It just comes down to how much context I can give and how much time. I have a friend who doesn’t like dictation and really likes typing, so that’s the better answer for him.I personally can talk forever. I can talk a lot – I’m a PM, that’s most of what I do. So I feel it’s very natural for me to talk the same way that I would give context to a person on my team or an engineer I’m going to start working with on a particular initiative, or in a kickoff, or somebody in marketing. For me, it’s very natural to tell the story of something.I also noticed that while I talk out loud, it helps me think better, just like writing helps you think. We’ve all had that experience where we’re working on a hard problem, and we’re stuck. Then we go ask for help, and we explain the problem, and suddenly the answer comes to us. That’s because speaking or writing uses different cognitive processes than just thinking something. That’s where new ideas and realizations come from – hearing yourself talk out loud or seeing yourself type or write.But really, the bottom line is I ask myself: What is going to get the most out of my head and into the text box? If that’s typing, go ahead and type. If that’s talking, dictate. I think for a lot of people, it’s talking.
Are raw transcripts generally fine, or do you edit things or curate little pieces of things to put together for context?
Tal: I recommend first summarizing it, having another side where you clean it up and summarize the main takeaways. You can also ask it for the best quotes, exact quotes, things like that. If there are particular things that you remember from the meeting that you want to add, you can add those, and then add it to the project knowledge.Meeting transcripts on their own can get pretty big and use up a lot of space. Hopefully in the future, we won’t talk about context window size anymore – it’ll just be negligible. But my gut is that with a meeting transcript, there’s a lot of noise there that’s not valuable. So just pre-process it – fancy word – just clean it up a little bit.
How do you determine what additional items to upload into the project knowledge versus upload at the chat level?
Tal: That’s the beautiful part – I struggle with this all the time, but then I realized it just always works out. Why? Because you just go into a thread, do the thing, talk to it, have a conversation, have some realizations. At the end, say, “You know what, this was great. Let’s take all the stuff that we decided on, all the new information, all the stuff we worked through. Summarize it into a doc,” and I’ll use that button and add it back to the project knowledge. So you don’t have to decide right away.
