Building PM Practice: Extracting Real Product Decisions from Lenny's Podcast Transcripts

Over Christmas break, I started working on a way for product managers to practice actual PM work. Programmers have LeetCode. Designers can show a portfolio. PMs spend a lot of time talking about frameworks and hoping someone believes they can use them.

I wanted to give someone a product scenario, ask what they'd do, and let them compare their reasoning with a real decision. So I started building PM Practice.

I worked through the product with Claude and built the Laravel scaffolding: scenarios, grading, and an admin interface in Filament. Getting the app running was the part I knew how to do.

Then I got stuck.

An app without the scenarios

Where do you get enough good product decisions to practice on? I could write from my own experience, but that would lean heavily toward Laravel consulting and nonprofit tech. I wanted a wider range of situations.

Grading was another problem. A programming exercise can have a right answer. A product decision depends on the context, constraints, team, and options. Matching an answer key wouldn't tell me much about someone's judgment.

And would people actually write a thoughtful response? Multiple choice didn't feel right for this. But asking someone to write a few hundred words could easily turn the whole thing into homework.

I had the app. I didn't have those pieces worked out. I shelved it.

Then the transcripts showed up

Two weeks later, Lenny Rachitsky shared the podcast transcripts on GitHub. The release I started from had 269 episodes.

I'd been listening for months. The useful part for this project was the specific stories: what someone was trying to do, the decision they faced, what they chose, and what happened afterward.

That's raw material for an exercise. Stop the story before the decision, ask the reader what they'd do, then let them see how it played out.

I started with Bret Taylor's episode. Google Maps, the Facebook Like button, FriendFeed. Plenty to work with in one conversation.

Extract first, curate afterward

Each scenario needs three pieces:

1. Enough context to understand the situation without giving away the answer.
2. An open-ended decision question.
3. The expert's actual choice, reasoning, and outcome.

I worked with Claude to build an extraction prompt using Sonnet 4.5. The first pass on Bret Taylor produced 15 candidates. I kept 10 after curation.

That curation matters. An interesting passage isn't automatically a good exercise. The context needs to stand on its own, the question needs a real decision, and the expert answer needs to be supported by the transcript. Otherwise you're just asking someone to guess what happened in an interview they haven't heard.

The transcripts gave me a content source and a way to reveal the original decision. Whether people would enjoy writing the answers was still something the product had to prove.

Grade the reasoning

The grading model doesn't see the expert answer. It gets the scenario context and the user's response.

That was an important choice. If it had the answer in front of it, agreement could easily become the thing it rewarded. I want someone to be able to choose a different approach and still explain it well.

The rubric adds up to 100 points:

• Problem understanding: 20
• Strategic thinking: 25
• Stakeholder awareness: 20
• Execution realism: 20
• Communication: 15

GPT-5-nano does the grading through PrismPHP. Laravel queues run that work in the background. After grading, the user can read what the expert actually did and compare the approaches.

I'm interested in the feedback behind the score: which tradeoff you missed, whether your plan is executable, and how clearly you explained it. A number alone doesn't tell you much.

Making room to think

I wanted the interface to feel like reading and working through a problem. Newsreader for the body text, IBM Plex Mono for the UI, forest green, plenty of whitespace. No badges or multiple-choice quiz format.

Browse scenarios by category, difficulty, or leader. Read the context. Write your answer. Get feedback. Then see the original decision.

The stack is Laravel 12, Tailwind 4, Alpine, Filament 4, MySQL, and Laravel Cloud. Sonnet handles extraction; GPT-5-nano handles grading. Those are separate jobs with separate inputs.

The first set

At launch I had 10 Bret Taylor scenarios, including decisions around Google Maps, the Like button, and FriendFeed. More leaders and scenarios were planned, with a goal of 50 by the end of Q1 and 100 by the end of the year.

The interesting experiment is whether taking time to answer before seeing the original decision changes what you get out of the story. Listening to someone explain their choice is useful. Having to make your own first gives you something to compare it with.

You can try PM Practice. And if you're building something with transcript extraction or AI grading, I'm happy to talk through the approach. Find me on X or email andy@pmprompt.com.