+1 vote
152 views
Post and review answers and feedback to answers in the comments section of this post.
asked in Metrics by (549 points)
edited by | 152 views

4 Answers

+1 vote

The first question I would ask is: what does “success” mean? Is it aimed at increasing user satisfaction, or usage/engagement/increased ad revenue? My assumption is that the answer is “yes” – at the core of a successful feature is the personal and social value for the users which ideally also generates profitability for Facebook and its advertisers. (A win/win/win).

A lot of telemetry can be measured, but in order to gain the most valuable insights, let’s start with the value for the users. Here are some reasons why they might want to make use of the save function:

-1- Ability to return to a certain post at a later time to consume or share/react/comment
-2- Ability to browse more content faster, while marking (=saving) a subset of interesting posts for more focused consumption/interaction
-3- Ability to collect posts related to certain themes/topics
-4- Ability to collect posts from certain friends or groups (what happens when a friend deletes a saved post from their wall??)

The assumption is that the save function is successful if it enables the above listed abilities. In the absence of actual user feedback, the combination of the following measurements are good proxies to assume that the save function is successful:

a. #saves per user per Facebook visit
b. #of returns to saved content per user
c. #interactions with saved content per user (or per saved piece of content)
d. #saves per amount of content scrolled through per user and FB visit

High numbers in a. plus any of b/c/d are a good indicator that a user is likely satisfied with the feature.

However, we also need to look at how many Facebook users (with regular activity on FB) have never used the save function. A high percentage would indicate a possible discoverability (awareness) issue.
If such an issue had been identified and addressed, then a decrease in the population which has never used the function is another measure of success.

If, furthermore, good analysis of what individual users tend to save helps surface more similar content in their newsfeed, we might be able to measure increased engagement with the newsfeed on content which had been better personalized based on previous save actions. Any such engagement can also be considered an (at least indirect) success metric for the save feature. Relevant measurements would be:
– #reactions/shares/comments/active consumption (e.g. play video) for content suggested based on a user’s saved list (even if it does not lead to added saves, which would be counted in a. above)

Other success metrics could be derived from non-instrumented data sources:
– actual user-feedback (e.g. one-question pulse queries on individual FB features)
– counterfactual testing (what if for a population of FB users the save function would be removed; how many complaints would we get?)
– Mining of any mention of the FB save feature in FB posts, comments, or FB-related user forums

The cost of deriving the last 3 suggested metrics is not likely justifying the result. Especially the first two suggestions have the potential to irritate the impacted users, while the ROI for the third is likely low.

I would therefore focus primarily on the readily available, yet relevant, logged usage telemetry described under a.-d. above, secondarily on the “indirect” success metrics, and furthermore measure if we can confirm any correlation between engagement with the save feature and increased # of visible and/or clicked ads. The latter measurement would be a metric related to the successful generation of financial profitability for the advertisers.

answered by (15 points)
0 votes

To my understanding, the way the Save feature works is – It let’s me save posts I want so I can view them later under my “Saved” label to the left of the Facebook News Feed.

I would track the following metrics to define success:
– Total saved posts per day
– Average/Median saved posts per user
– Number of saved posts revisited per day/week/month
– Number of webpage views for the “Saved” feature (and number of DAUs who also visited the saved webpage)
– Number of unsaved posts (the act of going to the saved webpage and clicking “unsave” to remove from the post from the list)
– I would also look at different posts to understand what users save overtime, do they save posts with/without videos? just regular posts? recommendation posts? posts with picture?
– amount of time spent on the saved webpage
– lastly, we can track the retention of the feature. The only tricky part here is to define what’s good retention. Unlike retention for the FB app (D1, D7, D14 etc…) as users who save posts might return to them even after a few weeks and that’s not necessarily a bad thing. Once we have a good benchmark of how retention for this feature would look like, we can align it to all players to help define its success

answered by (13 points)
+2
You might want to look at the scroll length. Do people scroll more per session because they are saving links that are hard to consume right away? Has time spent overall increased or decreased? What is the impact to videos watched? Because people are now saving – has the time spent watching videos and # of videos watched gone down? Are people saving and not coming back to view these videos?
Save also lets Facebook get more signals about what a user is interested in. So is the feed relevancy improving overall for users with more saves vs less saves? Has overall engagement with newsfeed gone up – more posts viewed, more active time spent, longer scroll length?
0 votes

Think there r about 5 categories of metrics to look at for most feature rollouts:
1) who is using it – persona / segment / type (novice, proficient, expert)
2) when – what do they do immediately before / after, etc.
3) usage – number of times in a session / week, duration between 1st-2nd, 2nd-3rd usages, etc.
4) impact – aarrr (short-term & long-term), funnel, etc.
5) cannibalization – did usage of some other feature decrease

answered by (13 points)
+1
Hi Roy
Thank you for submitting your answer. I Th ibk your answer is too short and needs to be more detailed. Have a look at the article I wrote about answering Metrics questions. https://productmanagementexercises.com/how-to-answer-a-metrics-question-in-a-product-manager-job-interview/
Good luck!
0 votes

primary:
– # unique users that clicked ‘Save’ on post
– # unique users that clicked the “Saved” section
– # unique users that re-opened at least one saved post
– # items “saved”
– # times “Saved” section got clicked
– # times saved item re-opened

secondary:
– Overall DAU, WAU, MAU
– Stickiness (DAU/MAU)
– Average Mins per user spent on facebook before and after (platform, watching videos..)
– Monthly revenue from ads

answered by (206 points)
0
Hi there
Thank you for submitting your answer. I Th ibk your answer is too short and needs to be more detailed. Have a look at the article I wrote about answering Metrics questions. https://productmanagementexercises.com/how-to-answer-a-metrics-question-in-a-product-manager-job-interview/
Good luck!

Please log in or register to answer this question.

Privacy - Cookies - Terms - Contact