Problems could have multiple layers to them which adds more to the complexity, In this case study I am going to demonstrate, discovery phase and problem prioritization. I am going to walk you through my process of finding problems, building a hypothesis, and creating a road map.
Case-Study: Pip Pip Yalah booking process.
Pip Pip Yalah is a long-distance carpooling application, leader in mobility in Morocco, with a community of more than 320,000 members who are carpooling through their platforms every day.
While I was working in PipPip Yalah as a Product Manager, I was tasked to optimize the booking process and figure out what is happening. An email sent to me had the following problem statement:
We have recently noticed a huge drop in our Ride Completion rates. While, lots of people are searching for rides and booking them. But, the number of passengers finishing their rides by scanning their QR codes are still significantly lower than expected. On the other hand, looking at our analytics clearly the company is struggling with retention rates. Which, have caused the Stakeholders of the application some concerns.
This case study is going to be divided into 2 parts, in the first part I am going to focus more on the following topics:
- Product Analysis and Research
- Problem Redefinition
- Problem Prioritization
Product Analysis and Research
At the beginning of any project, I always like to hear more from the Customer Service Departments (CS) since they’re regularly interacting with customers they’re likely to know their top issues and concerns. CS always proves a new perspective on the product which will help find potential experience problems.
I always like to ask the CS team to provide me with a report with all the common problems they face. the report contained: Product aspect, affected Persona, problem statement, frequency, suggested solution by CS, and finally how does the CS team solve the problem.
Right from the bat, I started to notice lot’s of problems, one of the biggest pain points for the CS team was that lots of people booked multiple Rides and, they don’t show up or cancel which resulted in banning lots of those passengers without them knowing about it thus, they started calling the CS team to unblock their accounts. On the other hand lots of the tickets that the CS team is handling come from drivers who can’t contact passengers or the other way around. which also has increased the pressure on the CS team. a lot of the cancelation that was happening between the driver and the passengers were happening between of them without canceling it on the application which causes a lot of problems for everyone.
This report came very handy not only by having a birds-eye view for the most frequent problems faced by the CS team also, but it was also a building stone for our initial hypothesis and gave us a starting point. But, you have to take everything with a grant of salt because; sometimes people report the symptoms of a problem, instead of the root cause. this means we can’t only rely on what people tell us.
Therefore, the best approach is to use multiple research methods, so the limitations of one method are mitigated by data from another source. This approach of applying multiple research techniques is called triangulation. there is an article by the NN group talking about Triangulation in detail I would recommend you to read it.
Since we have started creating hypotheses in our minds, an outside opinion will shed light on problems we might have missed or not seen because of our biases. So, we have decided to do a heuristic analysis by inviting 5 professionals from outside of the company to evaluate the App. we have used the 10 heuristics by the NN group since they are the most known by everyone, this has helped us to align everyone on the same page from designers, product managers, etc … even with the differences from our backgrounds experiences.
As predicted most of the comments were about the Design and aesthetic of the application, which was obvious to everyone but, we have noticed an interesting comment which was related to Visibility of System Status, it did have to do with signifiers, where the evaluators have noticed that there is no signifier indicating that you can interact with the address. This insight made us think and correlate with what we have found from the CS report about the multiple bookings, that we started to hypothesize about and we thought this could be the cause of those multi-bookings that are happening because people couldn’t find the button that allows them to show the address on their map application.
Nonetheless, the heuristic analysis also has given us lots of insights but, all of them were mostly assumptions. therefore, we wanted to see what’s actually happening on the application with real users. So, we have decided to perform usability testing to validate some of those hypotheses that we have.
We invited 5 users for our usability test then, we have prepared a few tasks for our users to do while we are watching them. our tasks were tailored to help us understand our user mental model but, mainly :
- Are we delivering the promised added value to our customers?
- How do our users deal with the problems they’re facing?
The insights gathered from the usability testing were extremely eye-opening, when our users were tasked to check the address of their ride none of them knew that they can kick the button next to the address to open it on their map application. but, what also was intriguing is that sometimes users have been complaining that the address is not clear and they can’t understand them.
Besides that, we started to notice more problems in other aspects of the product but, still didn’t make any sense to us because we had way too much data to deal with. While we were doing our research on the whole booking process, I have asked the technical lead manager if he can do a quality assurance test for us if there are any bugs that we can find, considering that we have noticed a few of the tickets listed in the CS report talking about bugs or technical issues, and some of our usability test participants have pointed out some technical problems.
Problem Redefinition
at this stage, we’ve got piles of data gathered from our research activities and, It’s time to start synthesizing and clustering our data to make sense out of it. I have learned a technique called areas of interest from an agency called CXL they have written an article under the name of their research method called Research XL Model Framework I encourage you to read it.
before I explain what we have done, let’s talk about Areas of interest and what does it mean?
Area of Interest is when multiple research methods point to the same aspect of the product such as the user profile, search results, etc… which indicate that something is causing a problem for the user and it requires feather investigation.
In the beginning, I started organizing all the data collected by our research method into tables and key coding them so we can visualize the data. then, I have created a keymap where you can find all the icons and, what do they mean, and to which research method do they refer. as soon as I got this done I started by placing those icons on the wireframes of our application. then it would be easier for you to spot the areas of interest since they’re becoming more obvious.
what was so interesting for us was that some problems are caused because of other problems so some of those problems are a consequence of other problems. at this point, the data from google analytics made more sense to us because now we can ask the right questions first, and then you can seek the right data that might help you find an answer. Data is just there, it’s up to you to interpret it and to pull insights out of it.
at this point, it was time we start writing our problem statement so we can clearly start identifying our problems and prioritizing them.
Problem statements
While defining our hypothesis statement we would revisit this stage of the research process multiple times to reiterate every time we find a new insight. but for the sake of time, I am going to focus on one Hypothesis statement, which will be discussed in detail in the 2nd part of this case study.
After, creating our areas of interest some patterns were showing up consistently. so we decided to start clustering those problems into themes which will make it easier for us to tackle down and prioritize them later on. So, we came with 3 main themes for our hypothesis:
- Security and Safety Problems
- Technical Issues
- Community Guidelines and Polices
During our research, we have been hypothesizing that the unclear departure address pushed our users to book multiple rides to secure a ride to their distention. the reason why they behave like this is that they can’t chat with the driver to confirm the address. the only way to do so is throughout getting accepted by the driver to chat with him. what we have noticed throughout our data and analytics is that people tend to ask exchange phone numbers to connect outside the application. due to the basic chatting feature which has caused us to lose users, or when the user faces a security problem we can’t help him/her.
besides none of our passengers have recognized that there is a button that allows them to see the departure address due to lack of signifiers as we have discovered throughout our heuristic analysis. all of those factors lead the users to heavily rush booking rides so they can chat with the drivers as fast as they can since the chat feature is blocked by our system so they can discuss the ride details and make sure that the departure address is clear enough.
Besides that, we have unveiled technical issues throughout the quality assurance test. some of those problems have caused some trust issues and added more friction and frustration to the users. nonetheless, all of this has pushed the users to start finding workarounds that have exposed them to security problems such as scammers, were can’t help them since some users are taking the rides outside of the application.
Problem Prioritization
The reality is that not everything can be done at once. My goal here is to protect the team from getting overwhelmed or ignoring long lists of product-backlog items. my plan is to prioritize issues accordingly and add to them to the backlog only those problems that make the most sense for the product vision, users, and team workload. the way I like to start prioritizing the problems accumulated from the research process is by asking the following questions:
- Who is affected and how is it affecting them?
- Where does the problem in the user journey occur (awareness, consideration, conversion)?
- Frequency of occurrence?
then, we will score each issue by assigning a value to it based on how we answered the previous questions by silently voting on issues. Each team member gets the same number of votes, they should vote based only on criteria that fall within their domain of expertise. Use different colors for different areas of expertise. For example, Developers may have green, while designers may have orange dots. Of course, someone needs to assess the impact of the problem since certain problems can have a devastating effect on the product, even if they are “objectively” quite easy to overcome.
After that, I would use a prioritization matrix which serves to identify the most important problems. This structured, objective approach helps achieve collaborative consensus while satisfying the varied needs of the user and business. This visualization can help you rank issues and communicate progress back to stakeholders and leadership over time.
The aim here was to start working on the problem that will result in the highest value to the users while requiring the least effort, those quick wins could boost the team spirit since we are fixing things that matter to the user. then, we can start moving to tickets that require an effort but still yields high value. But, before we do that I and the UX team will start researching and investigating each ticket usually we are not sure what caused our users to act in a certain way, our goal here is to figure out a solution that would add value to the user before, implementing anything and wasting time or money. In other words, we are de-risking the assumptions and validating our hypothesis. In the meanwhile, the development team is working on easy-fix tickets or bugs.
Conclusion
Discovery is an important tool to research the problem space, framing the problem(s) to be solved, and gather enough evidence and initial direction on what to do next. Discoveries do not involve testing hypotheses or solutions. In order to be effective, discovery should be broad and technology- or solution-agnostic.
In this second part of this case study I am going to dive deeper into one of the Problems we found during the discovery phase and investigate furthermore to figure out what’s going on.