Exercise 12 – Utilization went down by 45% on app xyz in Italy during the month of August — why and draft a plan to fix.

Post and review answers and feedback to answers in the comments section of this post.

See also:

List of problem solving questions for product manager job interviews

6
Leave a Reply

avatar
3 Comment threads
3 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
2 Comment authors
DebbieKRmacBijan Recent comment authors

This site uses Akismet to reduce spam. Learn how your comment data is processed.

  Subscribe  
newest oldest most voted
Notify of
Bijan
Guest
Bijan

My initial guess is that this is too big of a drop in a short time to be caused by competitors and given that Apple and Google don’t launch their new OS in August, it’s unlikely that it’s caused by an update to an OS or launch of a product by major competitor such as Apple.

First, I’ll take a look at list of updates and changes recently made to the app to get a picture of what things has changed internally. Maybe one of the changes have broken a flow in the app or has impacted the user experience of the product. The app reviews can also help me see if there is a trending complaint about some aspect of the product. Let’s assume that there was’t many complaints and there weren’t any major releases or updates that could have caused this.

Then I’ll check to see if this is related to seasonality of the product usage. I’ll look at historical data to see if I see a similar drop in previous years. Assuming there wasn’t any drop in the previous years, I’ll look at other apps within Google to see if the issue was isolated or across the spectram. If it was across the platform, I would know that it is probably not related to my particular app and I would look for more macro-related issues. Examples would be telecom provider changing data plans or having outages. Now let’s assume the issue is isolated.

I’ll first look at top of the funnel to see if there have been any drops in the number of app daily opens. If there isn’t, I’ll look at the engagement rates to see if I get any hints on the issue. Numbers to look at could be time spent, number of certain tasks completed, etc. For example, if I find out the number of tasks completed is same as before but time spent has come down, it might be an indication that an update has made the app more efficient.

Le’s assume that the drop is in engagement. Then I will look at more metrics related to engagement and particular flows that are contributing the most to the user engagement and see if I see any changes in user behaviour in those flow. Hopefully, this will help me spot the problem.

Would love to get people’s feedback on this.

Debbie
Guest
Debbie

I think till spotting the problem, its great. But after you have spotted the problem, how do you fix it? What are the steps you would take to mitigate it?

mac
Guest
mac

Hi Bijan, a great detailed reply. However, dont you think you should have started by asking what is meant by utilization, because it could be engagement as you mentioned in your last paragraph. It would be good to start there and then hypothesize about each option.

Bijan
Guest
Bijan

That’s a good point. I think you’re right. Would have helped narrow down the problem if I asked clarifying questions at first, especially around the meaning of “utilization”

KR
Guest
KR

Let’s assume utilization means engagement. The problem statement is vague so I’d ask some questions to narrow down the root causes:

– Is it down 45% YoY or MoM?
– Is the same app available in other countries and how are they performing?
– Are there similar apps that are showing a decline in publicly available proxy metrics (eg: # of app ratings being submitted every week)
– What’s changed in the industry? Any new competitors?
– Is the decline from new customers or repeat customers?
– What source of traffic is causing the decline?
– Is the decline in select components of the app?
– Where in the conversion funnel do you see higher than expected drop offs?
– What do you see in customer feedback? What is the nature of customer service calls in August?

Bijan
Guest
Bijan

Really good clarifying questions. I think asking these clarifying questions help narrow down the problem significantly.