+9 votes
Post and review answers and feedback to answers in the comments section of this post.
asked in Metrics by (567 points)
edited by | 468 views

1 Answer

+6 votes

This is a classic Facebook – execution question.
For this answer, I would follow the framework below:
1. Goal and success metrics of product/feature in question (In this case, the feature is Facebook reactions)
2. Goal and success metrics/impact on eco-system because of feature in question (In this case, the eco-system is the Facebook news feed of which Facebook reactions is a sub product or feature. So I will go through the metrics that I need to monitor on the Facebook news feed to understand the overall impact of Facebook reactions)
3. Test and measurement methodology – How will you A/B test, what metrics will you track? How will you determine cohorts in each group? How will you make sure that the cohorts don’t get mixed up during the test (avoid test corruption?)
So, following this framework:
1. When I first launch, reactions, I will state my goal/purpose as:
a. Customer goal: Human emotion has a wide range and variety. The Like button limits the range of emotions that someone can express while engaging with content on their news feed – provide customers with more options in addition to “Like” to express themselves.
b. Customer goal: Increasing social validation – By making it easier to engage with the news feed through reactions, you increase the likelihood of activity across friends and increase the feeling of social validation.
c. Business goal: Increase engagement of users on the news feed- By engagement I mean – average number of shares, posts, likes, reactions, comments per visitor during a 30 day period
d. Business goal: Increase retention: Increase the rate of repeat visits (daily, weekly, monthly) per user

2. These are the metrics I will test specifically for this feature:
a. Discoverability/Acquisition: Are people able to discover Reactions? It is currently hidden under the Like button- is it easily discoverable?
b. Usage: For the visitors that discover Reactions – How many people are clicking on the reactions? What is the total number of clicks? I would measure this for at least a week.
c. Usage by reaction type: How many clicks per reaction type? This would help me understand if I even have the right set of reactions in there or do I need different reactions?
3. These are the metrics I will test for in the eco-system:
a. Usage/Engagement/Content Creation: For the user group that is exposed to reactions (Test Group)– does their average activity during a 30 day period increase due to reactions – Are they liking/reacting/commenting/sharing more as compared to people (Control group) who do not see reactions? Are they creating more content than before the test (pre-post analysis) and as compared to the control group (A/B)?
b. Retention: For the test group that is exposed to reactions – do we see higher repeat visits (to see how their friends have reacted to their posts/shares or if others have liked/reacted to posts that they have liked/reacted)
c. Engagement: Does it increase time spent and length of scroll on news feed? Test A/B and see if there is statistically significant difference?
d. Monetization – Is there higher click thru rate on ads for visitors who were exposed to reactions? (Probably least priority as FB does not like to talk about monetization much).
4. A/B test tactics
a. I already mentioned what I would test in my A/B test across test and control groups. I also talked about pre-post analysis a little bit
b. You may be asked follow up questions on how you would determine incrementality? How would you prevent control group from seeing reactions in their feed if their friends are in the test group?
c. The way to answer the questions above might be:
i. Make sure that test and control group are very apart in the Facebook social graph.
ii. If for some reason, someone on the test group and someone on the control group are part of the same social graph, have an engineering solution to convert a “reaction” from the test group in to a “like” when seen by the control group.
iii. Also, to truly measure incremental improvement in engagement, take people in test and control groups with similar levels of prior engagement. How do you determine prior engagement – You could look at average activity rate per user (average number of likes/shares/posts/visits/friends) etc and make sure that test and control groups have people with similar activity rates. This way you can test to see if “passive users” became more active? And if “active users or highly socially engaged people” also had a marked improvement in their social activity.

answered by (149 points)
Thanks for submitting your answer, Nabspm! I think it’s well written and covers the key areas that matter when measuring the success of Reactions. Here are some of my feedback:

– 2a I would list specific metrics that help you measure discoverability

– for 2b, I agree the total number is important. I think you’ll also want to include a couple metrics that help you determine the size of the usage (e.g. % of users who used to only like and now like & react, % of increase in # of reactions posted, etc). You touch on them in section 3 and maybe, that’s enough but I would have still highlighted them in 2b as I think the % of users who use the new Reaction feature is an indication of the feature’s success)

Hope it helps. Overall, great answer. Thanks.
Hi Nabs,
Thanks for this, your answer is very thorough.

what is a classic facebook execution question? is there a proof that your framework is the right one to use for such questions?

in my eyes its a basic metrics questions that tests if the candidate knows how to define and analyze specific metrics.

I would have taken a different approach here and just list out specific metrics and explain why would they help me define the success of the feature.

In that regard, Discoverability and acquisition are too ‘high level’. I would dive deeper and use the following:
number of reactions per day? reactions per user per day, ratio between reactions and likes (because like is what users are used to), % of DAU that used reactions? I would also define a sub metric called engaged users for people who comment at least once a day/use “like” once a day and see how it affected on that cohort – did they change their behavior overtime?

What do you think?

Please log in or register to answer this question.

Privacy - Cookies - Terms - Contact