The first question I would ask is: what does “success” mean? Is it aimed at increasing user satisfaction, or usage/engagement/increased ad revenue? My assumption is that the answer is “yes” – at the core of a successful feature is the personal and social value for the users which ideally also generates profitability for Facebook and its advertisers. (A win/win/win).
A lot of telemetry can be measured, but in order to gain the most valuable insights, let’s start with the value for the users. Here are some reasons why they might want to make use of the save function:
-1- Ability to return to a certain post at a later time to consume or share/react/comment
-2- Ability to browse more content faster, while marking (=saving) a subset of interesting posts for more focused consumption/interaction
-3- Ability to collect posts related to certain themes/topics
-4- Ability to collect posts from certain friends or groups (what happens when a friend deletes a saved post from their wall??)
The assumption is that the save function is successful if it enables the above listed abilities. In the absence of actual user feedback, the combination of the following measurements are good proxies to assume that the save function is successful:
a. #saves per user per Facebook visit
b. #of returns to saved content per user
c. #interactions with saved content per user (or per saved piece of content)
d. #saves per amount of content scrolled through per user and FB visit
High numbers in a. plus any of b/c/d are a good indicator that a user is likely satisfied with the feature.
However, we also need to look at how many Facebook users (with regular activity on FB) have never used the save function. A high percentage would indicate a possible discoverability (awareness) issue.
If such an issue had been identified and addressed, then a decrease in the population which has never used the function is another measure of success.
If, furthermore, good analysis of what individual users tend to save helps surface more similar content in their newsfeed, we might be able to measure increased engagement with the newsfeed on content which had been better personalized based on previous save actions. Any such engagement can also be considered an (at least indirect) success metric for the save feature. Relevant measurements would be:
– #reactions/shares/comments/active consumption (e.g. play video) for content suggested based on a user’s saved list (even if it does not lead to added saves, which would be counted in a. above)
Other success metrics could be derived from non-instrumented data sources:
– actual user-feedback (e.g. one-question pulse queries on individual FB features)
– counterfactual testing (what if for a population of FB users the save function would be removed; how many complaints would we get?)
– Mining of any mention of the FB save feature in FB posts, comments, or FB-related user forums
The cost of deriving the last 3 suggested metrics is not likely justifying the result. Especially the first two suggestions have the potential to irritate the impacted users, while the ROI for the third is likely low.
I would therefore focus primarily on the readily available, yet relevant, logged usage telemetry described under a.-d. above, secondarily on the “indirect” success metrics, and furthermore measure if we can confirm any correlation between engagement with the save feature and increased # of visible and/or clicked ads. The latter measurement would be a metric related to the successful generation of financial profitability for the advertisers.