Thanks for the question. I have a couple of follow-up questions:
1. WHY? Why do we want to measure retention - are we concerned that people might be leaving the platform? Or is this part of a bigger strategy for something else?
Answer: We just want to understand our user behavior and make them spend more time on FB.
2. Thanks. Please give me a couple of minutes to think thru this question.
What is the question trying to test?
- Knowledge of Metrics and Measurement
- North Star Metric and Guardian Metrics
- Critical User Journeys
- Product Intiution
- Come up with multiple top line metrics and decompose general ones into specifics
- More than one way to solve the problem
Define what we are trying to measure?
Connect CUJ with measure
Multiple ways to measure
Isolate to one way - decompose it into sub metrics
Provide guardian metrics
We'll start off by thinking about the feature itself - i.e. FB timeline, then we'll define retention and what it means to retain users. After that we'll connect the user journey and try to come up with ways to measure retention.
The feature - FB Timeline. The FB timeline is a rolling wall of updates where each user can post something (image, video, text etc.) and other users can see it. There are two primary modes for the timeline - your personal timeline and the social timeline (what everyone else sees). I think we should first focus on the social timeline for measuring retention. Is that okay? (Answer: Yes).
Okay now let's try to think about what retention is. I'd define retention as the total time that a user spends on a feature. i.e. the point in time in between them first interacting with the feature and last interacting with the feature.
Before moving forward let's take a step back and think whether there are any other ways to define retention. Another way to define retention would be the total number of users who are active at any point in time. i.e. the total users that are being retained due to the timeline feature.
Or at a higher level, it could be the time that a user spends browsing (not actively interacting) with the timeline.
I'm going to stick to the first definition for now i.e. time between user's first interacting with the feature and last interacting with the feature.
Let's apply this definition to the FB timeline. So we need to measure two things - First interaction and last interaction.
First interaction can be - first time user sees another user's post. first time user likes or does an action on the post. first time the user posts something. To be broad, I'll take the first time the user performs any action on the timeline - like, click, comment etc.
Last interaction - this is tricky. Last interaction can be the user unsubscribing from FB. But that's too harsh. Let's define last interaction as a period of inactivity from a user for 1 week.
We can use this metric Retention as the North Star metric - i.e. a metric that guides us. But we should also try to have some guardian metrics to make sure we are not optimizing too much on one behavior.
What are the things that are important for a robust timeline? People should post, people should share others' posts, people should post things that increase the retention time of others, people should click on ads shown in the timeline. All of these could be our guardian metrics. i.e. when we optimize for retention, we should also make sure the guardian metrics don't suffer dispropotionately.