80% is a huge drop in any key metric.
I would try to first narrow down exactly what metric this is so I would ask the interviewer if they could tell me if the metric is new user retention, churn, monetization etc.
Second, I would try to understand if this was a sudden or gradual drop. for 80% definitely sounds like a sudden drop or else somebody would’ve said something already.
If it’s a sudden drop, I would try to pinpoint around what time this drop occured and figure out if there were any internal/external factors that could have caused it.
internal factors include: new feature was released, server went down, a new bug became prevalent. For the last two, you can segment it by region, browser/device type, and OS type. The issue could also be that the metrics we are grabbing is incorrect.
external factors include: a new competitor has joined into the market, bad PR, maybe a firmware was pushed outside of your control. It could also be due to seasonality or a major temporary event. If it’s a major temporary event, you should see KPIs begin to return to their normal state shortly.
Third, I would try to see if any other relational KPI drops. It’s easier to know what KPI it is before, but we can go along the user journey and see if any KPI before it dropped.
IE: A user signs up for the service -> enters in a credit card for payment (optional) -> clicks on a video to watch -> Watches the video -> chooses another video to watch
This is important in narrowing down exactly when the problem first starts. For example, if a key KPI is number of videos watched, perhaps the sign in is where most people are failing.
If the issue is a feature, I would try to clarify what the goal of the feature is. It could be possible that we started doing targeted ads and conversion dropped but the first time purchase after clickthrough increased. It would be important to understand if the goal of the feature change was met even with this big of a KPI drop.
If I can ascertain the exact issue, I would work with Sys Ops, Engineering, and other people on my team to try to address it. If the issue is a bug, we would have to issue a hotfix. If the issue is a server, than sys ops can look into it. If the issue was due to a feature release, we should probably look into either fixing it or reversing it quickly.
If the issue is external, this would be harder to solve immediately and would often require going through the normal cycle of product development to address them.
So in summary, I would first make sure we can ascertain if the drop was temporary or permanent, gradual or sudden and if the KPI drop may have occured elsewhere int he user funnel. I would look at internal and external factors to try to see if I can pinpoint the issue. Third, if the issue can be fixed immediately, I would contact my team to put out a hotfix or roll back a change that we may have made. If not, we should understand the issue thoroughly before acting and let people in the company know of our findings.