You launched a new signup flow to encourage new users to add more profile information. A/B test results indicate that % of people that add addtl. profile information increased by 8%. However, 7 day retention decreased by 2%. What do you do?
Let's start from the WHY behind the change- likely that you were implementing this change to improve retention ; let's proceed with the assumption that this is a social app where profiles play a critical role.
An increase of 8% for sign up flows is a significant increase in number of people completing profile information - I would ask how was this implemented because from experience I know that every time we add a step it introduces a 3-4% drop off.
My assumption here is that this is an added pop up screen during sign up which is causing a drop off in total number of users completing sign up successfully since they drop off on the profile info screen.
*Interviewers nods yes*
Also I would recommend changing the way we measure success of the A/B test , let me tell you why - consider the following scenarios
In case A for every 100 people signing up 50 people ended up signing up of which 20 completed profile information
In case B ( winning variant ) for 100 people signing up 48 people ended up signing up of which 22 complete their profile
So the blocking screen is causing an overall drop off which reduces D7 while increase number of people completing their profile
I would re-configure the experiment hypothesis to " users with better profile information have higher retention than users who don't - how can i increase in the number of users filling their profile information on D0 ( primary metric) while increasing/not affecting the successful sign up rate ( secondary metric that doubles up as a kill metric) "
if my goal is to increase profile information of new sign up I would focus on passive methods ( push notif, in app pop ups, incentives) POST users successfully signing up so as to negate this drop off
if my larger business goal is to increase retention I would reduce the steps of sign to increase successful signup and focus on passive methods ( push notif, in app pop ups, incentives) POST users successfully signing up so as to increase total user successfully signing up and help improve D7