1. Describe the product:
Before jumping into the analysis, I want to clarify my understanding of the product. As per my understanding, fb messenger is a messaging or chatting app and helps users connect more directly with their contacts (even contacts who are not fb users) or SMBs.
- Users can create groups with their contacts, text, video call.
- Users can create a poll, share location, add reminders
- Users can play games, use fb camera to share instant snaps/videos
- Can securely transfer money and split bills between friends
- Can get in touch with businesses to do reservation, place orders etc.
- Is integrated with Uber and lyft so users can hail a ride
Some user groups that use messenger are:
- businesses, school/college kids, working professionals
Here we will assume it is for regular people (not businesses)
<verify with interviewer if he/she is onboard with this description and user group>
2. Scope of question
Now to clarify the scope of question a little more, are we talking about improvement in terms of :
- number of new users, increased engagement(in terms of time spent, features used etc.) , retention%, number of active users
- also what type of platform we are concerned with : mobile app vs. website
<we will assume its the mobile app and we are looking to improve engagement>
3. Pain Areas and metrics to measure
With an aim to improve the engagement of the messenger app, we want to understand how each of these features are being used by the users. Hence, some metrics that I would like to track are:
a. Call experience : Voice calls and group video chats are one of the widely used features of messengers. Can we track and improve quality of video calling?
metrics: % voice calls that ended abruptly (without the user having to hang up). similarly for video calls.
For users who have experienced abrupt call drops, verify the % distribution between android and ios to analyze if this is limited to one os.
Also verify the version of the app used for the users
Tracking method - A/B testing to see whether bug fixes actually improved call quality between control and test groups. Also see if that increase Avg. # of video calls initiated or received by user over 30 days. Also do a pre-post change analysis of this metric. We can also capture direct customer feedback in terms of call rating to evaluate the improvement
b. Text quality: Are users spending more time typing responses? Can it be improved by using predictive responses?
metrics: Avg. # of times user responds in a conversation. Again we
c. B2C: One unique feature of messenger is that it lets users connect directly with businesses easily. Can the service around business to customer (B2C) be improved? User generally use this to make reservations, place orders etc. We can implement enhancements like integrating this with ubereats or seamless or any food ordering apps for restaurants.
Metrics: %of users using this feature, Avg. # of chats initiated with businesses per user, Avg. # of orders placed. Again implement A/B test and measure change in these metrics over 1 month between control and test groups