+9 likes
801 views
asked in Metrics by (1.1k points) | 801 views

2 Answers

+6 likes

This is a classic Facebook – execution question.
For this answer, I would follow the framework below:
1. Goal and success metrics of product/feature in question (In this case, the feature is Facebook reactions)
2. Goal and success metrics/impact on eco-system because of feature in question (In this case, the eco-system is the Facebook news feed of which Facebook reactions is a sub product or feature. So I will go through the metrics that I need to monitor on the Facebook news feed to understand the overall impact of Facebook reactions)
3. Test and measurement methodology – How will you A/B test, what metrics will you track? How will you determine cohorts in each group? How will you make sure that the cohorts don’t get mixed up during the test (avoid test corruption?)
So, following this framework:
1. When I first launch, reactions, I will state my goal/purpose as:
a. Customer goal: Human emotion has a wide range and variety. The Like button limits the range of emotions that someone can express while engaging with content on their news feed – provide customers with more options in addition to “Like” to express themselves.
b. Customer goal: Increasing social validation – By making it easier to engage with the news feed through reactions, you increase the likelihood of activity across friends and increase the feeling of social validation.
c. Business goal: Increase engagement of users on the news feed- By engagement I mean – average number of shares, posts, likes, reactions, comments per visitor during a 30 day period
d. Business goal: Increase retention: Increase the rate of repeat visits (daily, weekly, monthly) per user

2. These are the metrics I will test specifically for this feature:
a. Discoverability/Acquisition: Are people able to discover Reactions? It is currently hidden under the Like button- is it easily discoverable?
b. Usage: For the visitors that discover Reactions – How many people are clicking on the reactions? What is the total number of clicks? I would measure this for at least a week.
c. Usage by reaction type: How many clicks per reaction type? This would help me understand if I even have the right set of reactions in there or do I need different reactions?
3. These are the metrics I will test for in the eco-system:
a. Usage/Engagement/Content Creation: For the user group that is exposed to reactions (Test Group)– does their average activity during a 30 day period increase due to reactions – Are they liking/reacting/commenting/sharing more as compared to people (Control group) who do not see reactions? Are they creating more content than before the test (pre-post analysis) and as compared to the control group (A/B)?
b. Retention: For the test group that is exposed to reactions – do we see higher repeat visits (to see how their friends have reacted to their posts/shares or if others have liked/reacted to posts that they have liked/reacted)
c. Engagement: Does it increase time spent and length of scroll on news feed? Test A/B and see if there is statistically significant difference?
d. Monetization – Is there higher click thru rate on ads for visitors who were exposed to reactions? (Probably least priority as FB does not like to talk about monetization much).
4. A/B test tactics
a. I already mentioned what I would test in my A/B test across test and control groups. I also talked about pre-post analysis a little bit
b. You may be asked follow up questions on how you would determine incrementality? How would you prevent control group from seeing reactions in their feed if their friends are in the test group?
c. The way to answer the questions above might be:
i. Make sure that test and control group are very apart in the Facebook social graph.
ii. If for some reason, someone on the test group and someone on the control group are part of the same social graph, have an engineering solution to convert a “reaction” from the test group in to a “like” when seen by the control group.
iii. Also, to truly measure incremental improvement in engagement, take people in test and control groups with similar levels of prior engagement. How do you determine prior engagement – You could look at average activity rate per user (average number of likes/shares/posts/visits/friends) etc and make sure that test and control groups have people with similar activity rates. This way you can test to see if “passive users” became more active? And if “active users or highly socially engaged people” also had a marked improvement in their social activity.

answered by (285 points)
0
Thanks for submitting your answer, Nabspm! I think it’s well written and covers the key areas that matter when measuring the success of Reactions. Here are some of my feedback:

– 2a I would list specific metrics that help you measure discoverability

– for 2b, I agree the total number is important. I think you’ll also want to include a couple metrics that help you determine the size of the usage (e.g. % of users who used to only like and now like & react, % of increase in # of reactions posted, etc). You touch on them in section 3 and maybe, that’s enough but I would have still highlighted them in 2b as I think the % of users who use the new Reaction feature is an indication of the feature’s success)

Hope it helps. Overall, great answer. Thanks.
+3
Hi Nabs,
Thanks for this, your answer is very thorough.

what is a classic facebook execution question? is there a proof that your framework is the right one to use for such questions?

in my eyes its a basic metrics questions that tests if the candidate knows how to define and analyze specific metrics.

I would have taken a different approach here and just list out specific metrics and explain why would they help me define the success of the feature.

In that regard, Discoverability and acquisition are too ‘high level’. I would dive deeper and use the following:
number of reactions per day? reactions per user per day, ratio between reactions and likes (because like is what users are used to), % of DAU that used reactions? I would also define a sub metric called engaged users for people who comment at least once a day/use “like” once a day and see how it affected on that cohort – did they change their behavior overtime?

What do you think?
0 likes

I explain why we have reactions: Because like is too limited. The human feelings, reactions is more varied than a simple like.

How does it work:

  1. I click on reactions, I see a list of reactions. I select one reaction or I close the list.
  2. I can click to see who has reacted how to a certain item (post, comment, etc).
  3. Without need to click, I can see the list of reactions a post has received.

What is the BIZ goal to have reaction: To increase user engagement. FB needs users to spend more time on FB, so that the revenue which is based on CPM (impression ads) and hopefully CTR (click through rate, click ads) increase.

What does engagement mean in FB:

#of posts, comments, shares, likes, scrolls in news feed, videos watched, reactions to comments.

How do we expect reaction to change the engagement:

  1. Hypothesis: People see more meaningful reactions to their posts, e.g. 5 people smiled, 2 people showed anger, 3 people cried. Seeing different reactions is more rewarding than seeing 10 likes ===> People will be more likely to post more.
  2. Hypothesis: All users, especially passive users, will have a better way to express their feelings toward an image, text, video, comment. They feel that FB is a place were they can express what they think with one click. 

In terms of design, our hypothesis is that reaction is a reward to both poster and viewer, and it is also very easy to interact with.

Summary: 

  • BIZ goal: increase engagement
  • User goal: Increase the feeling of validation and engagement in what is happening is the social network

Since we are considering the reaction as a significant improvement to like, it is very important to do A/B testing. 

Metrics:

  1. Users can see who reacted how to an item:
    1. %users who saw a post and clicked to see who had what reaction to this post (
    2. The number of seconds users spend checking who reacted how
  2. #of times like was chosen vs. reaction
  3. #of times reactions button was clicked to see the list of reactions
  4. %of times reactions button was clicked to see list of reactions but no reaction was chosen (could mean that the right reaction was not there)
  5. distribution of the number of times each reaction was used. If a reaction was too few times, possibly it should be replaced with something else.
  6. avg # of comments written for posts with reaction vs. posts without reaction (does seeing reaction of other people encourage the user to write a comment)
  7. %increase in the engagement of passive users after introducing reactions. (see below the blue metrics for calculating engagement in this case)
  8. %increase in user engagement after he gets a lot of reactions for his posts.

A/B tests:

We need to choose similar control and test groups. They need to have the same amount of activities before going live with reactions in test group. We can measure activities using:

  • #hours spent daily on FB
  • #posts daily
  • #likes daily
  • #comments daily
  • #share daily
  • Type of posts should have the same disribution, e.g. 50% of posts are text, 10% video, 40% images
List of metrics for A/B testing
  1. #posts that get reaction vs. #posts that get like
  2. total # of reactions for 1000 posts vs total # of likes for the same amount of posts (posts should be chosen with the same nature, e.g. 100 text posts, 100 videos, etc)
  3. #of times 1000 posts were shared in A vs. B
  4. total #of comments for 1000 posts for A vs. B
  5. total #of times X videos were played in A vs. B (the best is if we test it for videos of the same nature, e.g. videos of cute animals)
  6. avg #hours spent on FB in A vs. B
  7. avg #posts daily by A vs. B
  8. avg total #reaction in A vs. total #like in B
  9. avg #of comments written by A vs. B
  10. avg #of shares in A vs. B

Prioritisation:

To prioritise the metrics, I will concentrate on 

  • BIZ goal, engagement
  • User Goal, being validated and that they can express their exact feeling.

(M, H) means BIZ Goal: Medium, User Goal: High.

  1. Users can see who reacted how to an item:
    1. %users who saw a post and clicked to see who had what reaction to this post (L, H) --> user wants to validate himself socially
    2. The number of seconds users spend checking who reacted how (L, H) --> It matters for user who has reacted how
  2. #of times like was chosen vs. reaction (L, H) --> user has been able to express what he thinks
  3. #of times reactions button was clicked to see the list of reactions (L,L) --> The user is just discovering what is there
  4. %of times reactions button was clicked to see list of reactions but no reaction was chosen (could mean that the right reaction was not there) --> (H,H) User has not been able to find the reaction he wants, BIZ Goal, user has not been able to engage with the feature ---> Requires a change from our side on the list of reactions we offer.
  5. distribution of the number of times each reaction was used. If a reaction was too few times, possibly it should be replaced with something else.---> (H, L) Helps us to refine our list of reactions.
  6. avg # of comments written for posts with reaction vs. posts without reaction (does seeing reaction of other people encourage the user to write a comment) ---> (L,L) does not tell us directly if we need to act, because the reason could be that posts have a different nature
  7. %increase in the engagement of passive users after introducing reactions. (see below the blue metrics for calculating engagement in this case) ---> (H, H) Passive users now find it more rewarding to use FB, We have increased engagement for users which were passive earlier
  8. %increase in user engagement after he gets a lot of reactions for his posts. ---> (H,H) Retention : user feels validated (rewarded) and wants to return to the cycle. 

A/B tests:

We need to choose similar control and test groups. They need to have the same amount of activities before going live with reactions in test group. We can measure activities using:

  • #hours spent daily on FB 
  • #posts daily
  • #likes/reactions daily
  • #comments daily
  • #share daily
  • Type of posts should have the same disribution, e.g. 50% of posts are text, 10% video, 40% images
List of metrics for A/B testing
  1. #posts that get reaction vs. #posts that get like ---> (H, H) People feel validated and use reactions more. We have high engagement
  2. total #of comments for 1000 posts for A vs. B (H, L)
  3. total #of times X videos were played in A vs. B (the best is if we test it for videos of the same nature, e.g. videos of cute animals) (L,L)
  4. avg #hours spent on FB in A vs. B ---> (H, H) 
  5. avg #posts daily by A vs. B ---> (H, L)
  6. avg total #reaction in A vs. total #like in B ---> (H, H) People feel validated and use reactions more. We have high engagement, 
  7. avg #of comments written by A vs. B  ---> (H, L)
  8. avg #of shares in A vs. B ---> (H, L)

In case of A/B testing we need to calculate if the increase between two groups is statistically significant or not. p-value of 5% could be a good indication.

I will choose all metrics with (H, H) and (H,L)

Summary:

We defined why we introduced Reactions, and we explained what success means form the eyes of our business and from the eyes of user. BIZ wants to increase engagement and User wants to feel validated and able to express what he thinks fast and easy.

Based on this we defined two sets of metrics. 

1) measures for the success of reactions among users for which reaction is launched.

2) The measures for A/B testing.

We prioritised metrics based on BIZ goal and User Goals and chose metrics that maximise both Goals and then metrics that maximise BIZ Goal. Because I believe a high engagement (BIZ Goal) means people have felt rewarded (User's Goal).

answered by (60 points)
0
Thanks for posting your answer. It’s great that you listed various metrics and prioritized them based on criteria. I’ve got some feedback on this. Hope you find it valuable:
- I would simplify the structure a bit. Start with the goal (as you did), describe the user journey and behaviour in each phase of the journey, then jump right into metrics using AB testing. Right now, you have two groups of metrics which might make it a bit hard for the interviewer to follow. Interviewers really want to see you easily communicate the metrics in a simple language and chronological manner to people across the organization.
- I think you could combine the two metrics of engagement and user expressing their feelings and just say goal is to drive engagement. You can explain that Facebook wants user to express themselves better so that they engage more with Facebook. An example is if I see something that makes me sad, I won’t like it (reduction in engagement) because a like does not represent my emotion.
- consider various phases of user journey and include metrics for each phase of the journey to ensure the increase in engagement does not impact other phases such as retention in a negative way
- I would have included impact as a criteria for evaluating the metrics and prioritizing them.
I really like the structure I’ve described in the below article.
https://productmanagementexercises.com/how-to-answer-a-metrics-question

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
Anti-spam verification:
To avoid this verification in future, please log in or register.

Post a question

About this site

Product Management Exercises is the best place to get help preparing for product manager interview questions. Any member of Product Management Exercises can post product manager interview questions, submit answers to the questions, and give feedback to other members' answers.

Follow PM Exercises



Report bug

Privacy - Cookies - Terms - Contact - Twitter - Facebook - Slack - RSS