TESTING THE PRODUCT: NEXTBUS

Project Overview

Timeframe: 2 months
Role: UX Researcher
Target user: Bus dispatchers, large and small transit agencies
Platform: Desktop
Tools used: Affinity diagramming, Contextual inquiry, Heuristic analysis, In-depth user interviews, Subject-matter expert, User testing

NextBus offers a web-based platform to transit agencies that enables bus dispatchers to manage bus fleets, assign bus operators, and track schedule adherence and headway. In 2017, NextBus was redesigned and the version would be taking over the current system that customers have been using for more than two decades. After initial production, I traveled to eight cities in the US and Canada to visit with current NextBus customers. Visits included in-depth user interviews, contextual inquiry, and user testing of the current platform. Since these visits occurred only four months before product release, feedback and results were used to shape a long-term product strategy.

tEAM

As the only UX researcher on the project, building connections with other departments and key stakeholders was important.

I worked closely with a Subject-Matter Expert (SME)—a Project Manager (PM) who is actively responsible for dozens of current clients and has 10 years of experience in transportation—who I consulted with as I prepared my test plan.

A Business Analyst (BA) was assigned to travel and work with me on all testing. As the facilitator, I led each session, while the BA was the note taker. We worked through the synthesizing process together, and then presented our findings and recommendations to stakeholders.

OUTCOME

Overall, the survey and on-site visit allowed customers to have their voices heard and be a partner in the development of the product. Showing users NextBus 2 allowed them to see the new platform’s progress, building confidence in the NextBus organization and product. The survey allowed customers to share their experience with NextBus—the company and product—while testing the new features and overall experience of NextBus 2 revealed both critical and non-critical issues that needed to be addressed.

I thought out a range of recommended solutions for each issue and after working with design and engineering leads, presented the least costly solutions. The impact of these solutions will outweigh upfront cost; customer support and delayed engineering may be more costly than if these solutions are implemented prior to final product release.

PROCESS

HEURISTIC ANALYSIS

When first coming onto the NextBus project, I completed a deep-dive heuristic analysis of the NextBus 2 platform. Using Nielsen Norman Group's 10 UX/UI principles, I applied them to each section of the platform. I sorted and presented findings to all departments working on the platform to help highlight and address issues prior to testing.

TEST PLAN DEVELOPMENT

I worked with the SME to understand all known issues that users were facing in NextBus 1. We ran through all possible scenarios that a user may face, and I developed tasks for the test plan based on these scenarios. Participants jobs varied, however, a total of 15 dispatchers were a part of the planned testing sessions. This called for research into the 'who, what, why, and how' bus dispatchers perform their duties as this is a big part of NextBus. I needed to understand not only what features are available in NextBus 1 but also what need they were intended to fulfill. By understanding the day-to-day situations and circumstances that dispatchers encounter, I was able to create scenarios that would offer enough context during testing to determine task completion or failure.

PRETESTING AND ANALYSIS

I ran through the test with three different PMs (excluding my SME). I sorted 25 pages of notes onto 220 post-its and created an affinity diagram. The affinity diagram drew attention to key pain-points that were currently affecting the NextBus platform.

Stakeholders were moved by the pretesting results and wanted me to explore the issues further. I was tasked with visiting customers to complete test sessions to see if these patterns would repeat.

SURVEYS

Prior to testing with customers, I created and sent a 33-question survey to each participant. This survey provided an understanding of the customer's mindset, issues they were facing, and information regarding feature usage and desire for new features.

IN-DEPTH USER INTERVIEWS

While visiting eight different agencies, each session began with an in-depth interview to best understand that user and their needs. They each walked me through how they use NextBus 1, which allowed me to tailor the UX test and scenarios to best match how they would approach and encounter the NextBus 2 platform.

USER TESTING

During the visits, 25 users were tested at eight difference agencies. The test included nine tasks covering the dashboard, map, replay map, reports, navigation, and overall experience. Each testing session allowed the user to encounter and explore NextBus 2 in a hands-on way. The test allowed each participant to interact with NextBus 2 as they would with NextBus 1; the scenarios positioned the user in situations they would encounter in their day-to-day duties.

Based on preliminary conversations with participants, I chose specific tasks for them to do. As scenarios were tested, slight changes were made to the test plan in order to collect the most feedback and data. Not all participants tested each area of the portal. The main purpose of the test was to start a conversation around what the user was experiencing versus what they expected; this allowed for further examination of participants' needs and how best to meet them through design.

Being able to test participants in person at their operation centers offered a rare level of comfort for test participants. The situation was prime for collecting qualitative data; participants walked me through their expectations, thought processes, frustrations, and delights. This offered insight into how a large swath of users expect things to work and how they expect to go about using certain features.

DATA ANALYSIS + SOLUTIONS

I worked with a Business Analyst to help sort through the notes, audio recordings, and video recordings from these sessions. We performed rapid synthesis using a digital affinity diagram. I pulled out ideas, opinions, issues, and quotes to be written onto individual digital post-its.

I prioritized these findings based on how likely the product was to fail (i.e. lose customers) if the issue was not addressed. From this prioritization, I worked out solutions with estimates using story-point values. I presented all findings and the suggested solutions to the stakeholders.

Full detailed information regarding prompts and outcomes can be found in my presentation given to stakeholders.

TAKEAWAY

Meeting with, interviewing, and testing a variety of transit employees who interact with the NextBus system revealed that while users are passionate about transit, providing quality service to their customers, and taking care of their drivers, they are not technologically savvy. Users' focused not on figuring out how to make the system serve them, but on managing their fleet. The idea that these things may be related was not necessarily clear.

Testing revealed positive reviews of the newly designed platform, as well as many issues that needed to be fixed. Despite high success rates for some tasks, there was still a large margin of error. I found that many of the issues revealed during testing were critical to the product’s success.

Beyond specific fixes, I also learned that users need and want access to training materials that help with their specific tasks and are mindful of their busy schedules. Many users told us that they know they aren’t using NextBus 1 to its full capability because they don’t know all of the features they have access to and they are hoping to understand and use more features in NextBus2.

NEXT STEPS

UX Research was brought in to NextBus late into the project. This means costly fixes will need to be made. This entire process helped stakeholders understand that it is imperative to continue incorporating UX research and user testing into the design process. This research will inform designers and provide context regarding how users interact with the system