How to Answer Metrics Questions
Metrics questions are asked in product manager job interviews for a particular feature of a product (e.g. How would you measure the success of the Meta (Facebook) Like button?) or a particular product itself (e.g. How would you measure the success of Amazon Echo?). This article is applicable to metrics questions in relation to both features of a product and a product itself as the steps to answer both questions are the same. For a product metrics question, follow the same guidelines for the product instead of the feature.
Metrics questions assess your ability to:
Here is how you can answer a Metrics question:
1. Describe the feature
Explain your understanding of the feature, what problem it solves, and how it solves it. Provide any additional information you may have about the product's lifecycle, user groups, and pain points. Ask the interviewer any questions that you may have in order to clarify your judgement of the feature.
Before moving on to the next step, make sure that you and the interviewer have the same understanding of what you are going to be evaluating.
2. Choose a goal
In a metrics question you are measuring the success of a feature in achieving a specific goal; hence, it is crucial to have clarity on what that goal is.
It is important to keep the company’s mission in mind when you think about the goal you are focusing on.
Note that there might be multiple goals per feature. For example, Meta (Facebook) Groups can drive engagement with the user and also generate more revenue for Meta (Facebook).
If the feature goal is explained as part of the interview question (e.g. How do you measure the success of Reactions in driving engagement in Meta (Facebook)?), you can skip step 2.
3. Walk through the customer journey
Walk through the customer journey from the beginning to the end of the interaction with the feature to demonstrate your understanding of the customer experience. This step also helps you think about potential behaviours that can impact the success of the feature, which you’ll be discussing in step 4.
At the end of this step, you can ask the interviewer if they think you’ve covered all the key steps of the customer journey.
4. Map and quantify (through metrics) user behaviours in the customer journey
Step 4 involves mapping user behaviours that are part of the customer journey and impact the goal of the feature.
Below is a list of all the customer journey phases a user can go through:
For each phase relevant to the feature you are evaluating, you will want to address two things:
i. discuss various scenarios in each step of the journey that impact the goal of the product/feature
Review the phase by highlighting various scenarios / behaviours and relevant outcomes that can have an impact on the goal(s) and explain why measuring the behaviours and (or) their outcomes is relevant. This step helps you determine what needs to be measured.
ii. highlight metrics that quantify the user behaviours & their outcomes and estimate their impact
As you go over the relevant behaviours and outcomes of each step of the user journey, describe the metrics that can help measure the behaviours and their outcomes. Simply put, think about the outcomes that you would like to see at each stage of the customer journey to achieve the feature’s goal and come up with metrics that indicate if a user behaviour is helping achieve the goals or not.
Given the limited time of the interview, try to come up with at least 1 and at most 3 metric per relevant customer journey phase. If you are looking at only one user journey phase (e.g. evaluating the Meta (Facebook) Like example above), come up with at least 5 metrics for that particular phase.
Let’s look at each step of the customer journey in more detail.
Awareness
Awareness is the first step of a customer journey. If users are not aware that a feature exists, they will not use it.
You want to use metrics that help you measure the level of awareness of a feature among users and get an idea of the level of knowledge and awareness of the feature. Looking at the right awareness metrics also helps you determine how users are becoming aware of the feature (e.g. how channels are performing in driving awareness) and may suggest how you can drive awareness.
For example, if you are evaluating the success of Meta (Facebook) Groups, it is important to measure the level of awareness of Meta (Facebook) Groups among Meta (Facebook) users (i.e. Do users know about Meta (Facebook) Groups, how they work, and what a user flow for joining a Meta (Facebook) Group is?)
Acquisition
Acquisition represents users that are interested in using a feature and are willing to take an action towards starting to use the feature (e.g. sign up, clicking, download of an app). It is an indication of the level of success of the feature in gaining the attention of its users. It hints at the level of interest in the feature among those users.
The metrics chosen in the acquisition step should be meaningful in a way that measure the size of the opportunity.
In the Meta (Facebook) Group example, the number of new Meta (Facebook) Group users as a stand-alone number will not reveal how large the interest among users is. However, number of new Meta (Facebook) Group users along with % of growth in number of new users and % of aware users who are acquired can however indicate the success of acquisition for Meta (Facebook) Groups.
Activation
If you can’t convert acquired users into activated users, it is an indication that the highly interested users are either getting discouraged from activating their accounts or there are barriers to activating them.
Activation generally has a unique meaning that is specific to the feature or product that you are dealing with. For example, activating a new online bank account can mean first transaction, completion of identification verification, and (or) setting up bill payment. However, activation for a mobile accounting app can be defined as a simple behaviour such as user registration.
Note that in some cases, activation and acquisition can be combined into one step. In the Meta (Facebook) Group example above, as soon as a user joins a Group, they are immediately activated.
The context of the user behaviour determines which phase of the user journey a particular user behaviour belongs to. For example, sign up might be an acquisition or activation activity, depending on the context of the activity. In the online banking example above, sign up is an acquisition activity. In the case of an online accounting website, sign up can be thought of as an activation activity.
Engagement
User engagement is what makes new users valuable to the organization and ecosystem of the product.
Similar to activation, engagement metrics will be specific to the feature and its goal.
In the Meta (Facebook) Group example above, here are a few metrics that one can consider to measure the metrics:
Engagement can have a direct impact on retention since higher engagement generally results in higher retention. For example, if you are using your mobile banking app every day and are engaged with the services that your bank provides via their mobile app, you will most likely stay on as a customer of the bank.
Retention
Retention is very important as it is often harder to obtain a new customer than it is to retain an existing customer. The relevance of this phase depends on the feature. In some cases, retention is a key driver of revenue. For a service-based product such as an online travel website (e.g. Expedia) or a car sharing app (e.g. Uber), retention and repeat business is critical.
Some metrics to consider in retention would be % or number of 30-day active users, % of new users who repurchase, distribution of ratings and user reviews, and product return rates.
Revenue
Monetization metrics are particularly important for eCommerce websites (e.g. online travel booking websites), however, new features are not always driving revenue and might be serving a purpose other than growing revenue. The purpose of certain features can be an increase in customer satisfaction, engagement of the core product, less number of calls for assistance, or many other purposes that help a product or business become better and are not monetized features.
If this phase is relevant, some metrics to consider for monetization are average revenue per user, % of paid users, average spending per certain time period, frequency of purchases and revenue generated via ads.
Referral
If users like a product, they are willing to refer it to their friends and family. A referral is an indication of success and usefulness of the feature for the user.
A few metrics that can help with measuring referrals are # and % of users who are referring the product or feature, # and % of converted referrals.
5. Evaluate your metrics
Now that you have gathered a list of metrics that impact the goal, evaluate them based on some meaningful criteria such as reach, impact, confidence, and effort. Reach represents the number of people that are affected by the new feature. Impact represents the size of the impact the feature has on an individual user. Confidence represents your level of confidence in your estimates. And effort represents the level of effort required to measure the metric.
6. Prioritize the metrics
Prioritize your metrics based on the results of your evaluation in the step 6.
7. Summarize your answer
Complete your answer by providing a quick summary of the feature’s goal and the key metrics that you will use to measure the success of the feature that you are evaluating.
Metrics questions assess your ability to:
- Measure the performance of the feature from various perspectives such as the impact on the user, the company, and the ecosystem.
- Identify user behaviours that significantly impact the performance of the feature.
- Identify metrics that help you evaluate the performance of the feature.
- Use data analytics to make and explain your decisions.
Here is how you can answer a Metrics question:
1. Describe the feature
Explain your understanding of the feature, what problem it solves, and how it solves it. Provide any additional information you may have about the product's lifecycle, user groups, and pain points. Ask the interviewer any questions that you may have in order to clarify your judgement of the feature.
Before moving on to the next step, make sure that you and the interviewer have the same understanding of what you are going to be evaluating.
2. Choose a goal
In a metrics question you are measuring the success of a feature in achieving a specific goal; hence, it is crucial to have clarity on what that goal is.
It is important to keep the company’s mission in mind when you think about the goal you are focusing on.
Note that there might be multiple goals per feature. For example, Meta (Facebook) Groups can drive engagement with the user and also generate more revenue for Meta (Facebook).
If the feature goal is explained as part of the interview question (e.g. How do you measure the success of Reactions in driving engagement in Meta (Facebook)?), you can skip step 2.
3. Walk through the customer journey
Walk through the customer journey from the beginning to the end of the interaction with the feature to demonstrate your understanding of the customer experience. This step also helps you think about potential behaviours that can impact the success of the feature, which you’ll be discussing in step 4.
At the end of this step, you can ask the interviewer if they think you’ve covered all the key steps of the customer journey.
4. Map and quantify (through metrics) user behaviours in the customer journey
Step 4 involves mapping user behaviours that are part of the customer journey and impact the goal of the feature.
Below is a list of all the customer journey phases a user can go through:
- Awareness
- Acquisition
- Activation
- Engagement
- Retention
- Monetization
- Referral
For each phase relevant to the feature you are evaluating, you will want to address two things:
i. discuss various scenarios in each step of the journey that impact the goal of the product/feature
Review the phase by highlighting various scenarios / behaviours and relevant outcomes that can have an impact on the goal(s) and explain why measuring the behaviours and (or) their outcomes is relevant. This step helps you determine what needs to be measured.
ii. highlight metrics that quantify the user behaviours & their outcomes and estimate their impact
As you go over the relevant behaviours and outcomes of each step of the user journey, describe the metrics that can help measure the behaviours and their outcomes. Simply put, think about the outcomes that you would like to see at each stage of the customer journey to achieve the feature’s goal and come up with metrics that indicate if a user behaviour is helping achieve the goals or not.
Given the limited time of the interview, try to come up with at least 1 and at most 3 metric per relevant customer journey phase. If you are looking at only one user journey phase (e.g. evaluating the Meta (Facebook) Like example above), come up with at least 5 metrics for that particular phase.
Let’s look at each step of the customer journey in more detail.
Awareness
Awareness is the first step of a customer journey. If users are not aware that a feature exists, they will not use it.
You want to use metrics that help you measure the level of awareness of a feature among users and get an idea of the level of knowledge and awareness of the feature. Looking at the right awareness metrics also helps you determine how users are becoming aware of the feature (e.g. how channels are performing in driving awareness) and may suggest how you can drive awareness.
For example, if you are evaluating the success of Meta (Facebook) Groups, it is important to measure the level of awareness of Meta (Facebook) Groups among Meta (Facebook) users (i.e. Do users know about Meta (Facebook) Groups, how they work, and what a user flow for joining a Meta (Facebook) Group is?)
Acquisition
Acquisition represents users that are interested in using a feature and are willing to take an action towards starting to use the feature (e.g. sign up, clicking, download of an app). It is an indication of the level of success of the feature in gaining the attention of its users. It hints at the level of interest in the feature among those users.
The metrics chosen in the acquisition step should be meaningful in a way that measure the size of the opportunity.
In the Meta (Facebook) Group example, the number of new Meta (Facebook) Group users as a stand-alone number will not reveal how large the interest among users is. However, number of new Meta (Facebook) Group users along with % of growth in number of new users and % of aware users who are acquired can however indicate the success of acquisition for Meta (Facebook) Groups.
Activation
If you can’t convert acquired users into activated users, it is an indication that the highly interested users are either getting discouraged from activating their accounts or there are barriers to activating them.
Activation generally has a unique meaning that is specific to the feature or product that you are dealing with. For example, activating a new online bank account can mean first transaction, completion of identification verification, and (or) setting up bill payment. However, activation for a mobile accounting app can be defined as a simple behaviour such as user registration.
Note that in some cases, activation and acquisition can be combined into one step. In the Meta (Facebook) Group example above, as soon as a user joins a Group, they are immediately activated.
The context of the user behaviour determines which phase of the user journey a particular user behaviour belongs to. For example, sign up might be an acquisition or activation activity, depending on the context of the activity. In the online banking example above, sign up is an acquisition activity. In the case of an online accounting website, sign up can be thought of as an activation activity.
Engagement
User engagement is what makes new users valuable to the organization and ecosystem of the product.
Similar to activation, engagement metrics will be specific to the feature and its goal.
In the Meta (Facebook) Group example above, here are a few metrics that one can consider to measure the metrics:
- % of Meta (Facebook) Group users that like posts in Meta (Facebook) Groups
- % of Meta (Facebook) Group users who comment on Meta (Facebook) Group posts
- Average # of groups a Meta (Facebook) user is a member of and any changes in that number
- % of users that visit their Meta (Facebook) Group pages within the last 30 days
- Average number of comments per Meta (Facebook) Group post per 1000 views – compare to user comments in other posts
Engagement can have a direct impact on retention since higher engagement generally results in higher retention. For example, if you are using your mobile banking app every day and are engaged with the services that your bank provides via their mobile app, you will most likely stay on as a customer of the bank.
Retention
Retention is very important as it is often harder to obtain a new customer than it is to retain an existing customer. The relevance of this phase depends on the feature. In some cases, retention is a key driver of revenue. For a service-based product such as an online travel website (e.g. Expedia) or a car sharing app (e.g. Uber), retention and repeat business is critical.
Some metrics to consider in retention would be % or number of 30-day active users, % of new users who repurchase, distribution of ratings and user reviews, and product return rates.
Revenue
Monetization metrics are particularly important for eCommerce websites (e.g. online travel booking websites), however, new features are not always driving revenue and might be serving a purpose other than growing revenue. The purpose of certain features can be an increase in customer satisfaction, engagement of the core product, less number of calls for assistance, or many other purposes that help a product or business become better and are not monetized features.
If this phase is relevant, some metrics to consider for monetization are average revenue per user, % of paid users, average spending per certain time period, frequency of purchases and revenue generated via ads.
Referral
If users like a product, they are willing to refer it to their friends and family. A referral is an indication of success and usefulness of the feature for the user.
A few metrics that can help with measuring referrals are # and % of users who are referring the product or feature, # and % of converted referrals.
5. Evaluate your metrics
Now that you have gathered a list of metrics that impact the goal, evaluate them based on some meaningful criteria such as reach, impact, confidence, and effort. Reach represents the number of people that are affected by the new feature. Impact represents the size of the impact the feature has on an individual user. Confidence represents your level of confidence in your estimates. And effort represents the level of effort required to measure the metric.
6. Prioritize the metrics
Prioritize your metrics based on the results of your evaluation in the step 6.
7. Summarize your answer
Complete your answer by providing a quick summary of the feature’s goal and the key metrics that you will use to measure the success of the feature that you are evaluating.
Like
Dislike