Can Continuous Product Design Improve CPD Certifications?

You must have known this was coming. After releasing our Continuous Product Design (CPD) Certification we couldn’t not practice CPD on the certification itself. CPD is in our DNA and we were excited to understand how its application could improve our own user experience.

Continuous Product Design (CPD) is the cross-team approach to building better digital products faster — based on a shared, quantified, and continuous view of customer signals. If you’re unfamiliar with CPD you’ll want to check out the CPD ebook or certification site to learn more.

Understanding how a new digital product, product enhancement or a piece of content performs is a difficult but crucial part of the release process. At Quantum Metric, we had a list of questions: Were people able to get certified? How did they like the content? Did anyone experience usability issues?

To answer these questions, we used CPD in practice to discover, define, develop, and deliver an improvement to the CPD certification.

VOC feedback alerts us to user friction

A short time after the CPD certification launched, our product owner was alerted to a new response from an integrated VOC survey. From the survey, the product owner discovered that a user was confused about how to move to the next stage of the CPD certification course. To advance, you have to click “Next,” but this user initially missed the button. In the feedback they mentioned adding an auto-advance feature would improve the course.

Before exploring this potential improvement, the product owner wanted to understand the impact and scope of the underlying issue. He needed to determine if customers were failing to advance through the course because they couldn’t find the “Next” button and if this was causing many to abandon the course.

Quantifying the impact of digital friction

The product owner set out to define the issue. Working across multiple teams, our Director of Customer Enablement, Marketing Analyst, and Product Analyst met to discuss what data was necessary. We needed to determine if a large number of users were missing the “Next” button, lowering the certification completion rate.

The working group decided to look at the number of people who play the first video, and the number of people who click “Next” on the first video to understand if this issue is widespread. Fortunately, we use Quantum Metric on our CPD certification so we had great analytical tracking available.

When quantifying issues—speed matters.

The analysts used Quantum Metric’s out-of-the-box tracking to quickly create a dashboard that showed the percentage of people clicking “Next”. This simple analysis was complete in 25 minutes. The percentage of people successfully clicking “Next” was high. Again, working cross-team, we decided the high percentage indicates the issue is not widespread so we moved on to answer our next question.

We compared completion rates for CPD against other certifications we offer, as well industry benchmarks, to determine if the issue was causing a decrease in certification completion. The completion rate for the CPD course was above benchmarks, so we were comfortable with our students’ progress.

To truly validate our analysis, we watched replays of people experiencing the issue. With Quantum Metric, we can see pixel perfect replays of users’ sessions. We noticed that while some people didn’t initially find the “Next” button, they were able to after a few seconds.

An auto-advance feature would be really cool. However, we couldn’t justify focusing engineering resources on this feature request since the impact was negligible. We didn’t stop there. This insight started a discussion on other simple ways to improve the user experience.

Data downvotes the need for an auto-advanced feature. What’s next?

Improving user experience with small improvements

We began developing improvements to the user experience based on the customer signals we discovered and defined. In cross-team discussion, we agreed that adding text on the page inviting users to click “Next” might improve the user experience.

CPD allowed us to minimize the risk of introducing new problems by ensuring we quantified its scope and impact first. This change had no real potential negative effects to commensurate with the small business impact of the initial issue we discovered.

We agreed a small text change was appropriate for our “Next” button issue. Without following the Continuous Product Design methodology and using Quantum Metric to quantify the issue, we may have gone directly to the engineering team requesting a time-intensive auto-advance feature be developed. Our simple text change took 20 minutes to implement once all teams were aligned. Below is the before and after:

Before:

After: See the additional text added at the bottom of the screen

Monitoring releases to ensure success

The time came to push the change. CPD enabled us to deliver this update rapidly, with confidence. We weren’t finished just because the change was live. CPD emphasizes having analytics in place to monitor releases. As a team, we discussed what we wanted to measure and identified key metrics. Using Quantum Metric’s out-of-the-box tracking, we set up a dashboard accessible to everyone involved.

We thought about any other metrics that would allow us to see a more complete picture of the user journey. We decided we also wanted to know the number of users that complete the first video and click “Next.” This would allow us to see the actions of the most motivated users, the ones who completed the video. Within minutes, we configured an additional event in Quantum Metric to make this additional analysis possible. Next, we set up alerts and agreed on a monitoring cadence to catch any immediate issues and planned to convene for a review of the change.

A better CPD certification

One month later, we reconvened. If we had been alerted to a drop in activity or negative effects in our monitoring dashboard we would have met sooner. During the meeting, we looked over the data. After making the change, users were clicking “Next” to advance 1.25% more often – an exciting, measurable impact!

Additional event data showed that over 95% of users that completed the first video clicked “Next.” This signaled that the vast majority of people who complete the first video wanted to learn more and continue the certification and that the course content was engaging and intuitive. Our change had great results, which we were able to quantify using Quantum Metric. If our change had, had a negative impact, we would have caught it quickly through alerts and monitoring we put in place using the Continuous Product Design methodology.

Improve User Experience With Less Risk

CPD allowed us to score a quick win. Even better, we avoided a potentially months-long, engineering-heavy, change with little improvement to show. Without CPD and Quantum Metric, we may well have fallen into the trap of listening to the loudest customer voice.

One user proposed adding a new auto-advance feature. No doubt a cool feature, but one that would take time and money to develop. Following the CPD methodology, we reduced risk by ensuring we discovered the real issue, quantified the impact, and defined a solution to match. We determined the impact did not warrant a major change and identified a small design change could improve user experience.

As we’ve shown, issues don’t need to be large to warrant investigation and small changes can have measurable impact. With CPD, we were able to act quickly by developing changes to specifically address what we discovered and defined. We delivered the change with monitoring in place so we could be confident our change had a positive impact. Quantum Metric’s tracking gave us all the data we needed at every step in the CPD process allowing us to act with confidence. Together, Continuous Product Design and Quantum Metric empowered us to improve our digital product quickly, based on a shared, quantified, and continuous view of customer signals.

You can do the same. Begin your CPD journey today.