This is a classic Facebook – execution question.
For this answer, I would follow the framework below:
1. Goal and success metrics of product/feature in question (In this case, the feature is Facebook reactions)
2. Goal and success metrics/impact on eco-system because of feature in question (In this case, the eco-system is the Facebook news feed of which Facebook reactions is a sub product or feature. So I will go through the metrics that I need to monitor on the Facebook news feed to understand the overall impact of Facebook reactions)
3. Test and measurement methodology – How will you A/B test, what metrics will you track? How will you determine cohorts in each group? How will you make sure that the cohorts don’t get mixed up during the test (avoid test corruption?)
So, following this framework:
1. When I first launch, reactions, I will state my goal/purpose as:
a. Customer goal: Human emotion has a wide range and variety. The Like button limits the range of emotions that someone can express while engaging with content on their news feed – provide customers with more options in addition to “Like” to express themselves.
b. Customer goal: Increasing social validation – By making it easier to engage with the news feed through reactions, you increase the likelihood of activity across friends and increase the feeling of social validation.
c. Business goal: Increase engagement of users on the news feed- By engagement I mean – average number of shares, posts, likes, reactions, comments per visitor during a 30 day period
d. Business goal: Increase retention: Increase the rate of repeat visits (daily, weekly, monthly) per user
2. These are the metrics I will test specifically for this feature:
a. Discoverability/Acquisition: Are people able to discover Reactions? It is currently hidden under the Like button- is it easily discoverable?
b. Usage: For the visitors that discover Reactions – How many people are clicking on the reactions? What is the total number of clicks? I would measure this for at least a week.
c. Usage by reaction type: How many clicks per reaction type? This would help me understand if I even have the right set of reactions in there or do I need different reactions?
3. These are the metrics I will test for in the eco-system:
a. Usage/Engagement/Content Creation: For the user group that is exposed to reactions (Test Group)– does their average activity during a 30 day period increase due to reactions – Are they liking/reacting/commenting/sharing more as compared to people (Control group) who do not see reactions? Are they creating more content than before the test (pre-post analysis) and as compared to the control group (A/B)?
b. Retention: For the test group that is exposed to reactions – do we see higher repeat visits (to see how their friends have reacted to their posts/shares or if others have liked/reacted to posts that they have liked/reacted)
c. Engagement: Does it increase time spent and length of scroll on news feed? Test A/B and see if there is statistically significant difference?
d. Monetization – Is there higher click thru rate on ads for visitors who were exposed to reactions? (Probably least priority as FB does not like to talk about monetization much).
4. A/B test tactics
a. I already mentioned what I would test in my A/B test across test and control groups. I also talked about pre-post analysis a little bit
b. You may be asked follow up questions on how you would determine incrementality? How would you prevent control group from seeing reactions in their feed if their friends are in the test group?
c. The way to answer the questions above might be:
i. Make sure that test and control group are very apart in the Facebook social graph.
ii. If for some reason, someone on the test group and someone on the control group are part of the same social graph, have an engineering solution to convert a “reaction” from the test group in to a “like” when seen by the control group.
iii. Also, to truly measure incremental improvement in engagement, take people in test and control groups with similar levels of prior engagement. How do you determine prior engagement – You could look at average activity rate per user (average number of likes/shares/posts/visits/friends) etc and make sure that test and control groups have people with similar activity rates. This way you can test to see if “passive users” became more active? And if “active users or highly socially engaged people” also had a marked improvement in their social activity.