+1 vote
1k views
asked in Metrics by (71 points) | 1k views

2 Answers

+3 votes

Objective

I will assume that the feature is for Facebook Messenger. 

The objective of this product is to enable "users to have private conversations to drive strong social connections"

The read feature has an objective of "Helping Senders of message understand the conversation engagement"

Users and Use Cases

There are now different of user personas who are in the messenger eco-systems

  1. Senders of messages
    1. individuals to individuals
    2. Individuals to Groups
    3. Individuals to Businesses
    4. Businesses to Individuals
  2. Receiver of messages
This feature is applicable to Group #1 directly and Group #2 indirectly. The feature is described as 
  1. Senders sends a message
  2. Message gets notified once receiver logs in and sees the message
The outcome it wants to drive directly via the feature is-
  • Let senders decide on ways to forward the conversation (or not)  based on read receipts in order to drive richer more engaged conversations
  • Make receivers aware that senders know that they have seen it in order to drive engagement
Long term this aims to drive engagement for facebook ( which is usually measured as DAUs)
The way all users interact with the feature is via inside the messenger

Metrics

We can start looking at the metrics in a number of ways

  1. Usage & adoption
    1. Senders: % of senders seeing this is not very relevant as this is always visible - what is interesting is what happens after they see it
    2. Senders: % of senders logging back to see message status even without response
    3. Senders : Chain length ( Average number of messages sent by one party before other responds)
    4. Receivers : % of conversations immediately started by receivers 
    5. Receivers: Average time between visit screen and next message
    6. Businesses : Response rate of business queries ( Eg % of messages responded in 1 Hour)
  2. Quality of conversations: The hypothesis is that this should improve so we should look at
    1. Average number of messages / conversation ( it can also be median ) : Conversation = Fixed time frame
    2. Average Message distribution in a 2 party conversation ( Messages by A / Total Messages)
    3. Average Time to respond between messages in an conversation OR % messages responded in X hours
  3. Retention
    1. % of users reactivated due to this feature compared to base
  4. Overall Impact on social value
    1. Impact of feature on DAU for Messenger ( attributed)
    2. Total messages exchanged in a 1:1 conversation ( as a proxy for strong social connections)
Prioritization
If i had to pick one North star metric that would be "Average messages exchanged in Messenger / 1x1 conversation" since that could be a proxy for strong social connection. Ways to measure this would be
  • A/B with control group at launch
  • Blackout feature to group and test / Hold-out
  • Pre-Post feature analysis
Can this be gamed
This metric is based on a few assumptions which may not hold true and thus there are risk
  1. More messages exchanged May not necessarily lead to stronger social connections - this belief needs to be tested with data
  2. Its a high level metric and changes in it over time may not be directly attributable to the feature
  3. There is a risk that the metric may go up due to more one-sided conversations and thus we need to check message distribution also
Summary
The feature aims at enriching conversation by mimicking how a personal 1:1 conversation will happen where the sender knows whether  reciever has got the message and can appropriately change the communication strategy. Thus we measure the feature by its impact on communication strategy leading to richer conversations mainly average number of messages / conversation
answered by
+1 vote
The core purpose of this feature is to help improve engagement in Facebook messenger.

Read Receipts are used to foster the social obligation of responding to a message which you , the recipient have read and responding back to the message sender.

If we were to segment the messenger customer base by usage - Active(use at least once a day) and Passive users(use once a week) , this feature will help improve engagement for passive users and will help improve the engagement for that user group. I dont' see this improving the engagement metrics significantly for active users who are already using the messenger application actively.

So , the metrics I will be looking at are the following -

Feature Success

Definition of Conversation- A session where interaction happens between two users or a group of users with a maximum delay of 12 hours.

a. Number of messages per conversation (Applicable to both 1:1 as well as Group Conversations)
b. Increase in number of conversations(Applicable to both 1:1 as well as Group Conversations)

c. Time spent in the messenger per user

d. Conversation Duration (Time per conversation)

e. Number of view receipts read for a message/user

Amongst the above metrics, I would prioritize c (ultimate goal of this feature),c,b,d and e.

The way to test this hypothesis will be to launch this feature to a test group and a control group with a similar mix of Active to Passive User base and measure the above metrics and see if we see a statistical significant improvement in engagement metrics mentioned above.

One important thing is also to measure if this feature is actually leading to decrease in engagement of a segment of the user base.
answered by (44 points)
+1
I think another goal of the feature is to inform user if their message has been read.
Having said that, there are a few new metrics come to mind:
- # of times user logs back in to check their message before they receive a respond
- % of first messages that go without response (does the new feature help reduce this?)
- average response time (does it go down?)
There are a few more I would list.
After you list a few more, you'll have to evaluate and prioritize your metrics as described in the article. I think this step is currently missed in your answer. https://productmanagementexercises.com/how-to-answer-a-metrics-question
0
Thanks for your feedback. I have updated my answer and added a few more metrics and my prioritized list as well.

Post answer and get feedback

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
To avoid this verification in future, please log in or register.