How to assess Product Managers
My most recent post on how to hire your first PM hopefully helped some teams hiring PMs for the first time. However, it left many questions about the hiring process unanswered. This second post will focus on one of the core areas of interest during the hiring process: how to assess a product manager?
A note before we start: anything I mention below is a recommendation based on my experience. The fact that it worked well for the specific teams and companies I was part of doesn't mean that this is the best possible approach; there are other ways that might be better for you depending on your circumstances.
Thanks for reading Rubik's Cube! Subscribe for free to receive new posts and support my work.
A Simple Model for Hiring
A simplified view of the hiring process classifies a candidate as a good (high performing) or a bad fit (low performing) for the role in this specific company. The “for the role” part is important as a bad candidate for a PM position could be an exceptional candidate for a Sales role or vice versa. Similarly, a PM candidate who would be a great hire for a big tech company, could be a bad hire for an early stage startup.
As a result, the goal of a hiring process is to assess candidates and classify them in the good/bad buckets (again for the specific role/level/company). The decision matrix looks something like the table below.
Table 1 - Hiring decision matrix
The ideal states are hiring a good candidate (True Positive - TP) and not hiring a bad candidate (True Negative - TN). The other two states are not ideal (errors) but there is a huge cost difference between the two. Not hiring a good candidate (False Negative - FN) can be costly as you are missing out on good talent and potentially delaying some initiatives, but you will eventually find another candidate for the role. However, hiring a bad candidate (False Positive - FP) is orders of magnitude more costly as this impacts the whole team, delays the domain progress, frustrates high performing coworkers, etc. Given the cost difference between not hiring a good candidate and hiring a bad one, a good hiring process is focusing on minimizing the risk for a bad hire, potentially missing out on some good candidates. Assessing risk is a key element which we will refer to in the next sections. Now that we took this out of the way, let’s get back on the PM hiring question.
What does a good PM look like?
The Amazon approach for product development is working backwards: Amazon starts by defining an end vision and then decides on what to work on today based on this vision. Similarly, we can work backwards to answer the “how to assess a PM” question by first defining the characteristics of a good PM.
Fundamentally a PM defines product strategy and drives execution. So by definition a good PM can define a good strategy and execute it successfully. To accomplish this a PM would need several skills:
Strategy: A good PM should be able to create a product vision which sets the direction, inspires the team, and helps secure funding (raising VC money for a startup or allocating headcount and other resources in big tech). To set this vision, it is key that a PM combines user empathy, to ensure they deeply understand customer needs and the way to address them, with a solid market understanding. After defining the vision, a PM needs to create a roadmap by defining and prioritizing a set of initiatives.
Execution: In the execution step, the PM needs to define the right metrics for measuring progress towards their goal. The key elements to execution are a sense of ownership, an attitude of getting things done in a timely manner, while keeping a high quality bar.
Communication: For both areas, PMs are required to communicate effectively and collaborate with several stakeholders.
Special skills: There might be other special skills required based on tenure, domain, and track (IC vs. management). Depending on the product, PMs might be required to have domain knowledge (e.g. ads, fintech) or be familiar with specific technologies (e.g. machine learning, cloud computing). I find these a bit overrated as a good PM can usually ramp up on a new domain quicklypretty fast but they may matter for very specific roles.
Hiring and developing teams: This is one of the top skills for people managers (Group PMs and above) as they are as good as the team they build.
Fit with values and principles: Last but not least, the process should also include an assessment of the fit with the organization’s values and principles. It is critical to extend the assessment beyond functional skills. At the end of the day, a PM can develop functional skills but it is much harder to acquire behaviors that align with an organization’s values and principles. If you don’t do that, you will end up with a corporate culture which is not intentional.
A word of caution: “fit” should mean a candidate’s compatibility with the values and principles of a company (i.e. the culture). It should not mean that the candidate gets extra points for having a similar educational background or the same hobbies with the interview team. This distinction is important to keep clear to help you avoid bringing bias to the hiring process.
Assessing a PM
Defining the relevant skills is very important as it ensures: (a) a fair process as all candidates are assessed against the same criteria, and (b) a complete and efficient process as candidates are assessed against all key areas (no blindspots) while minimizing overlap. Before starting the interviews, interviewers should sync and each get assigned some of the above dimensions. This could look like the below table.
Table 2 - Assignment of assessment areas
The two main types of questions used to assess these dimensions are hypotheticals/case studies and behaviorals. Case studies allow you to dive deep on analytical questions and understand a candidate’s way of thinking. Many interviewers use a mini consulting case to assess strategic thinking. Behavioral questions (tell me about a time when …) focus on past examples where the candidate displayed the same qualities. Of course, similarly to mutual funds, past performance does not guarantee future returns. As a result, the focus should not be on whether a past example was successful but on how the candidate approached the situation, as well as the level of introspection and acquired learnings they displayed. I haven’t found most other types of questions (e.g. tell me about yourself, or what are your strengths/weaknesses) to be very helpful.
Each of the dimensions of a good PM can usually be assessed by both types of questions so in most cases it comes down to preference. For example, you can assess strategy through both hypotheticals or through behaviorals; it is equally as effective to ask a hypothetical (what would you do if you were the PM for Product X) or a behavioral (tell me about a time you defined strategy for a product).
While we have so far focused on interviews, written assessments are often also included as part of the process. This requires some additional work and might deter busy candidates, but since writing is one of the most important (and underrated) PM skills, I would recommend getting a writing sample from the candidate in the form of a short assignment.
How many interviewers should we do?
Luckily this is a question where we don’t need to speculate much about as Google and other companies have done the work for us. Given there is a huge cost for every bad hire and a much smaller cost for every interview, we want to find the lowest possible number of interviews that accurately identify a good hire. Intuitively there are diminishing returns as we add interview steps (a process with 9 or 10 interviews will most likely lead to the same recommendation). Based on Google, the magic number is four.
To maximize the success rate, these four interviewers should ideally include both PMs (e.g. hiring manager and peer) as well as main stakeholders (e.g. engineering, design lead, functional leads). Interviewers should coordinate in advance to ensure they cover all areas that need to be covered, while minimizing overlapping questions.
Raising the Bar
Fast growing companies have aggressive goals and they need to hire quickly to meet them. Teams in fast growing companies are often over stretched and this creates a huge motivation for growing companies to prioritize hiring quickly above hiring well. As discussed above, it is critical to avoid poor hires (false positives). Furthermore, the temptation to hire ‘good enough’ candidates quickly can create a long term issue as the company’s average staff quality drops as more below average employees join the ranks.
To address this risk of diminishing team quality, Amazon introduced the bar raiser program. A bar raiser is an experienced interviewer, who is an independent party in the hiring process. The bar raiser is part of the company but not a direct member of the hiring team. Not being part of the team (so not accountable for the same goals), increases the focus on quality and reduces the influence of hiring urgency. Their intent is to ensure that the quality bar is raised with every hire so that the average quality in staff is actually increased over time. Many companies have adopted this model and it is a highly recommended best practice for fast growing startups.
Written Feedback & Debriefs
Each interviewer should submit written feedback ideally within 1-2 hours after the interview and never in more than 24 hours. The feedback should assess the candidate across assigned areas (e.g. above the bar on strategy, at the bar on execution) and include a clear hire/no hire recommendation. All candidates have some risk so the art of hiring is to take calculated risks. If you have not identified a risk for a candidate, then you haven’t done a good job with your assessment.
After all the written feedback is completed, all interviewers should read the submitted feedback and then discuss if they agree to hire or not. Interviewers are required to present their thinking and assess candidate risks. While the voting does not need to be unanimous, a good process usually brings all interviewers on the same page.
Closing a Candidate
After spending so much time assessing a candidate, don’t drop the ball after extending an offer. Changing jobs is a very stressful period for many candidates so this is the time to be empathetic. Offer them the ability to sync again with the hiring manager or other members of the team to ensure they get all the information they need. Connecting the candidate with members of your team that previously worked at the same company or attended the same school with the candidate could be a powerful closer.
Hiring is one of the most important things you will do so take the time to do it right. It will be surprisingly time consuming but well worth the investment. Best of luck and let me know if you have any questions.
We covered several of these topics in a Fireside chat with the Product360 team which you can watch on YouTube.