Measuring Usage Is Only the Beginning

This chapter is from the book

Bridging Intention to Impact: Transforming Digital Product Development through Evidence-Based Decision-Making

Learn More Buy

This chapter is from the book

This chapter is from the book 

Bridging Intention to Impact: Transforming Digital Product Development through Evidence-Based Decision-Making

Learn More Buy

Measurements of Success Beyond Usage and Usability

One company, the New York Times, has successfully bucked the trend of declining usage faced by the media industry. Their leadership resisted the digital tide for years, appearing as a laggard in the early internet era. However, when they finally embraced the digital frontier, they eschewed the prevailing trend of offering vast amounts of free content. Instead, they leaned into their century-old strength: producing premium content worthy of a paid subscription.

The Times’s editorial board remained unwavering in its commitment to delivering articles that resonated with readers, satisfying their needs for information, understanding, and entertainment. As the digital realm became saturated with fleeting, free content, readers gravitated back to the trusted source they had relied on pre-internet. Subscriptions to the Times continued to climb as people enjoyed the experience of accessing their expansive library of resources while finding that it fulfilled their desires. By 2023, the New York Times boasted the most extensive subscription base of any U.S. media entity, a testament to their dedication to quality over mere engagement.

It is no surprise that the company, coming out of the “growth at all costs” era of media, also stuck to its lofty mission of “We seek the truth and help people understand the world.” This ethos compelled them to look beyond superficial metrics, focusing instead on whether they were satisfying their readers’ thirst for knowledge and entertainment. Describing their success, the consulting company McKinsey highlighted how the Times’s product team works alongside their reporters to find the best display format for each article to maximize the content delivery. This approach has proven beneficial at encouraging app returns, subsequently improving subscription retention.

User and Behavioral Outcomes

Measuring impact based on outcome metrics, in addition to usage and usability, is the core tenet of this book. This lens is called the Impact Mindset and focuses on measuring success based on whether a product made an impact on the outcome a user is attempting to create. Throughout the subsequent chapters, we will discuss the five levels of metrics shown in TABLE 1.1, each providing a piece of the puzzle to identify whether a feature is successful. With an Impact Mindset, success means an enjoyable experience that is widely used while also accomplishing its desired impact. We’ve already covered the first level of success metrics in detail: usage. It is the easiest to measure and is good for understanding how much a product grows, along with whether it is sticky when measuring retention.

TABLE 1.1 Success Levels

Usability, the second level, captures a user’s sentiment toward their product experience. At its most straightforward, usability is measured as satisfaction and a single question such as, On a scale of 1–10, what level of satisfaction did you have with this experience? Another frequent approach is the net promotor score, commonly asked as How likely is it that you would recommend this company to a friend or colleague? Although both questions offer a glimpse into user sentiment, they remain surface-level indicators. For instance, a dip in NPS might signal a problem but doesn’t pinpoint the root cause.

Making usability a more valuable measurement requires breaking the overall experience into defined chunks. More granular usability metrics, such as Rate the level of difficulty that you had finding the Complete Purchase button, provide more actionable insights. The question targets a specific element of the product interaction. If the difficulty-finding-a-button metric spiked, it is likely due to a bug or a recent design change, and with that information, the team could remediate it more swiftly. Quantifying usability is a newer approach that teams are adopting and has much to offer on building satisfying products. It is a powerful tool for measuring whether people like and will continue using a product, which is essential for growth, but it still doesn’t determine if it fully satisfies user desires.

The third metric—the first novel one—in Table 1.1 introduces the behavioral outcome. This metric captures a user’s tangible actions within the product environment and in their real-world activities post-engagement. Consider a meditation app with a user aiming for stress reduction. Behavioral outcomes might encompass the number of guided mediations in-app that they completed or more emotional check-ins they performed throughout the week—specific actions linked to their overarching goal. Behavioral outcomes focus on what activities a user performs during and after usage.

The fourth metric—another novel one—is user outcomes, which assesses whether a product has catalyzed the desired change or impact for the user. If that sentence confuses you, don’t worry; the next chapter provides more detail. Following the previous example of a meditation app, the user outcome could be reduced stress, gauged by heart rate variability, or perceived stress levels. A team could measure both outcomes across time and in combination with the user’s behaviors to determine if the product is making an impact.

The fifth and final metric is one that is all too familiar to the financial side of the business but is often overlooked by research and even sometimes product teams, the business outcomes. These are the variables that are used to track the success of the company at selling products. Used to determine whether a product is a viable source of funding to continue sustaining operations, these metrics are critical to the ultimate impact of a product to the firm. While generally determined by the leadership of the company, impacting these metrics with the release of new features is an essential component to a successful product.

A comprehensive assessment of these five metric levels paints a vivid picture of user engagement, sentiment, and the tangible benefits caused by specific actions prompted by the product. Adopting an Impact Mindset is underpinned by the ability to measure the behavioral and user outcomes, and the next chapter further details the value of collecting both. For now, let’s explore two illustrative examples measuring all five success metric levels.

Case Studies: Grammarly and Ninjio

The first example focuses on a business-to-consumer (B2C) company, Grammarly, which offers a writing assistance tool. Consumers purchase the solution to have a significantly enhanced autocorrect that focuses not just on spelling but on the entire sentence structure. Grammarly doesn’t stop working after fixing errors; it improves the user’s writing by offering scores based on five metrics to measure a writer’s success (FIGURE 1.1): grammatical correctness, clarity, engagement, and overall delivery. It has taken abstract constructs likely learned in grade school and turned them into beautiful visuals. As the tool learns more about the writer’s style, it personalizes recommendations and the goals it attempts to improve through its recommendations. Grammarly’s success, as per their marketing, is developing better writers, and their product does so as shown through anecdotal account and academic study alike.⁴ TABLE 1.2 shows how Grammarly could hypothetically measure the five levels of success of its app.

FIGURE 1.1 Grammarly’s five metrics to measure the effectiveness of a user’s writing

TABLE 1.2 Grammarly Metric Level

Shifting to the business-to-business (B2B) space, Ninjio addresses the pressing issue of cybersecurity awareness. Recognizing that many cyber breaches are caused by human error, their solutions focus on training employees on best practices to reduce human-based cyber risk. They have developed a suite of programs intended to train employees on risk-reducing behaviors. One of their offerings, Ninjio Aware, comprises short, engaging videos that spotlight specific actions that employees can adopt to minimize errors. By creating memorable learning experiences, Ninjio aspires to reduce data breaches, a claim that a customer can test by using fake email phishing campaigns. TABLE 1.3 shows how Ninjio could hypothetically measure the five levels of success of its training.

TABLE 1.3 Ninjio Metric Level