0 votes
asked in Problem Solving by (12 points) | 555 views

3 Answers

+1 vote

Assuming no other change was implemented that could have affected these metrics, lets start by breaking down the question. First piece of information is that additional profile information increased by 8%, which is desirable. We do not have information about what exactly we are asking the user to provide, so lets assume that the additional profile information consists of user's interests and experiences. And second info we have is that 7 day retention decreased by 2% because of this change.

The objective of getting more profile information from the user is to find better content and connections related to the user's interests and experiences. If we are not able to use the additional user info to find more relevant and engaging content for the user, then this effort is a waste. So, our primary focus should be to make the best use of this info to improve the experience for the user. This would have an effect on the churn rate too and generate positive word-of-mouth marketing for the product.

Now lets try to understand the possible reasons for the decrease in user retention. A thought that would come to our mind is that probably we have made the process so elaborate and complicated that user did not even complete the process and chose to leave the app. But since retention period metric tracks the user after they have completed the signup process, we can rule out this reason. We ought to be more specific about what kind of information we are asking from the user. For privacy reasons, many users will not be comfortable in sharing more personal info. Make sure that users are satisfied with your product's security and privacy policies and convinced that they have full control over how your product uses their data.

In summary, we need to do the following to improve retention -

  1. Create a more engaging experience for the user by finding and displaying relevant content based on user's interests.
  2. Communicate your product's compliance and privacy standards and make sure they trust you with their data.
answered by (29 points)
0 votes

Clarifying questions: 

  1. What was the new sign up the flow and how is it different than the old sign up flow. My assumption is the new sign up flow asks for more user information, thus the increase in the % of profile information. 
  2. What was our goal? Was it to increase profile information? What was the acceptable counter metric decrease? Is the result within range? 
List pros + cons of both metric
  1. Increase in profile information - the more data we have, the better our recommendation and personalization system and the network becomes more valuable to the users. Cons, depending on the types of information we ask and require of the user, the user may have a different level of comfort and privacy concerns
  2. Decrease in 7 day retention - if a user does not come back in 7 days, this is not great for the platform, however, I want to also consider whether 14 day or monthly retention has decreased, perhaps the new user no longer needs to come back on 7th day and add a profile pic or other information. 
What would I do to validate my hypothesis
1. One hypothesis is 7th-day retention may decrease due to the new user no longer come back to fill out additional information, I would then compare 14 day and 30 day retention to see if it decreased. 
answered by (24 points)
I agree with the clarifying questions here, it is a must before making assumptions. I also like the structure and approach of the answer.

Merging both answers above would be ideal in a way tho.
0 votes

You launched a new signup flow to encourage new users to add more profile information. A/B test results indicate that % of people that add addtl. profile information increased by 8%. However, 7 day retention decreased by 2%. What do you do?

Let's start from the WHY behind the change- likely that you were implementing this change to improve retention ; let's proceed with the assumption that this is a social app where profiles play a critical role.

An increase of 8% for sign up flows is a significant increase in number of people completing profile information - I would ask how was this implemented because from experience I know that every time we add a step it introduces a 3-4% drop off. 

My assumption here is that this is an added pop up screen during sign up which is causing a drop off in total number of users completing sign up successfully since they drop off on the profile info screen.

*Interviewers nods yes*

Also I would recommend changing the way we measure success of the A/B test , let me tell you why - consider the following scenarios

In case A for every 100 people signing up 50 people ended up signing up of which 20 completed profile information

In case B ( winning variant ) for 100 people signing up 48 people ended up signing up of which 22 complete their profile 

So the blocking screen is causing an overall drop off which reduces D7 while increase number of people completing their profile

I would re-configure the experiment hypothesis to " users with better profile information have higher retention than users who don't - how can i increase in the number of users filling their profile information on D0 ( primary metric) while increasing/not affecting the successful sign up rate ( secondary metric that doubles up as a kill metric) "

if my goal is to increase profile information of new sign up I would focus on passive methods  ( push notif, in app pop ups, incentives)  POST users successfully signing up so as to negate this drop off

if my larger business goal is to increase retention I would reduce the steps of sign to increase successful signup and focus on passive methods ( push notif, in app pop ups, incentives) POST users successfully signing up so as to increase total user successfully signing up and help improve D7

answered by (22 points)

Please log in or register to answer this question.

Privacy - Cookies - Terms - Contact