Lessons Learned: Launching Continuous User Research at Oscar
The key to creating high-value experiences is to build with our members and deeply understand their needs, barriers, motivations, and pain points. It is our priority to honor and defend the needs of our end users whether that means members, providers, insurance brokers, etc. We constantly challenge ourselves to think of new ways to live out our “user obsession,” which led us to explore a new process called continuous user research.
“Continuous user research” is the ongoing collection of user feedback and an overall commitment to regularly talking to your customers. The premise of continuous research is intriguing and positioned to address two primary UX Research challenges:
1. We don’t always know what we don’t know.
We can identify lots of problems from quantitative data, analytics, or other signals that surface research questions. However, sometimes, we don’t know what we don’t know.
2. We don’t always have time (or the need) for a dedicated, point-in-time study.
Dedicated, cross-sectional research is critical in many cases, but sometimes, we don’t have time to set-up, recruit, and properly execute a large study. Some questions may also not require a large, point-in-time study to explore. These small to medium-sized questions (more evaluative in nature) are no less important to address, but may not be the right candidates for full blown, dedicated studies.
That’s where continuous research programs shine. At Oscar, we call our continuous research program “Tuesday Testers.” Every other Tuesday, we interview Oscar members, and we make the most of the time we have with them. These thirty-minute sessions include:
A pulse check on the member, their experience with Oscar, where we excel, and where we fall short. This helps us address UX research challenge #1. We create a space where problems can surface that might otherwise be left in the shadows.
Gathering member reactions to new concepts and early design work. This helps us address UX Research challenge #2. Every week as our designers develop and refine new concepts, they stumble into questions that member feedback helps answer. Our Tuesday Testers program has made space for these explorations and helps us quickly decide which early concepts are worth further exploring (and which we should reevaluate).
While continuous research might seem easy and breezy, this program was hard work to bring to life and to maintain. From idea to execution, we’ve captured three major lessons learned about continuous research programs:
When we launched this program, we initially designed it for Friday testing — the original name was “Friday Five.” We’ve since had to pivot to Tuesdays because, like us, members have “Friday brain” too. We had significantly lower interest in feedback sessions and many who booked sessions never showed up. Thus, “Tuesday Testers” was born. We haven’t had a problem with engagement or no shows since.
Continuous research infrastructure is meant for exploring very specific research questions. We have strict requirements for what we can explore with Tuesday Testers versus what deserves a dedicated, point-in-time study. Our product design team helps us inform the questions we want to explore and which of those are most pressing, which helps maintain the integrity of the program. This stringency has been critical to preserving research integrity and methodologically sound study design. Continuous research doesn’t replace traditional UX research studies! There’s a time and place for both.
Pro Tip! If your study aims to explore questions like “what is the right thing to build,” “who is our user,” “what do they need in a given context” — you probably need to use a more traditional UX research approach with a point-in-time study. If you’re more curious about the usability, comprehension, or utility of an early stage design concept, then a continuous research approach may be appropriate. Read more about choosing the right study for your research questions here.
3. This approach requires dedicated days for user interviews that are always kept meeting-free. Our members deserve our undivided attention. It’s important to create a culture that supports that!
Our “Tuesday Testers” program has helped us keep our finger on the pulse of the Oscar member experience — along the way learning what’s worth further investment…and what’s not.
Quick Example:
Vetting Before Setting: As a part of the Oscar experience, members on certain plans are encouraged to “set” a PCP (Primary Care Provider). Most of our design flows prompt the user to set a PCP right after searching for one. However, as Tuesday Testers have pointed out, members don’t want to set a PCP until they’ve had a chance to “vet” the doctor with a visit. This has helped us reimagine where and when a user is prompted to formally set their PCP.
Oscar’s mission is to make a healthier life accessible and affordable for all. By maintaining this continuous feedback loop, we can effectively gather and respond to member needs, helping us fulfill our mission. Our calendars are marked for every other Tuesday: turning feedback into action!