The person saying “AI is incredible” is not hallucinating. The person saying “AI is unreliable” is not being a hater. They are likely just touching different parts of the frontier. — Corey Noles

A lot has been written recently about the concept of “jagged AI frontier”. If you aren’t familiar with this expression, MIT Sloan’s working definition and Tomas Pueyo’s viral image below can help get you up-to-speed:

We’ve all seen many examples of how AI, despite dramatically surpassing human cognition in many areas, can still falter in simple tasks. But there’s another layer to the problem of the jagged frontier that makes it even trickier to manage: its non-monotonicity.

The jaggedness of AI progress doesn’t move in a single direction. It’s not simply a matter of weak areas catching up over time — sometimes capabilities that were previously strong can deteriorate, even as others advance.

No one would expect a player who used to excel at both checkers and chess to continue to improve their chess game while starting to lose in games of checkers, right? But with AI models, it’s not uncommon for a replacement model (from the same AI lab, with better benchmark scores than the previous version), to start giving incorrect answers, or making inference mistakes, in contexts where it used to work well before.

People with deep knowledge in a domain where they use AI see examples of the “jagged AI frontier”all the time, even in interactions with the most powerful models.

AI labs regularly update their models, and this is when the dynamic nature of the jagged AI frontier may cause the most unexpected failures.

For example, a colleague recently told me about an issue he experienced with a paid model after its latest update. He asked the AI whether a higher RMSE is good or bad when evaluating a predictive model. (RMSE, or root mean square error, is a common measure of how far a model’s predicted values are from the actual values. A lower RMSE means the model fits better.) To his surprise, the model “confidently” said that a higher RMSE is better—a mistake the previous version of the same AI model would not have made.

This is why companies adopting AI agents to automate their workflows need to worry not just about the unexpected weaknesses their AI models may have today, but also about new potential weaknesses they may develop in the future.

For instance, imagine a healthcare company implementing an AI-based self-service solution that interacts directly with healthcare providers to help them get their medical claims approved.

The company does its due diligence, defining what good behavior looks like and turning domain expertise from its claims analysts into concrete, testable criteria that engineers and operations teams can use to determine whether the solution is ready to ship. As time passes, the AI service is given more autonomy based on its record of reliability.

At this point, if the work was still being done by experienced claims analyst, the company wouldn’t have to worry about them suddenly starting to make basic mistakes, such as divulging patient confidential data or making wrong calculations.

But with AI models, that risk is there, especially when a model update may suddenly change the existing jagged frontiers in a way that break parts of a tested workflow.

In this scenario, incorrect approvals could facilitate improper payments or fraud; a data breach involving medical claims data could expose the company to lawsuits or other causes of action; failure to meet claims-processing service levels could violate agreements, etc. And with thousands and thousands of claims being processed at a much higher speed than analysts can oversee, it might take time to understand what’s happening across an entire body of transactions.

So far I’ve been talking about the downsides of the jagged edge frontier, but in the title of this article I mention that I also see a benefit to it.

And the positive side I can see in the jagged AI frontier is that it helps explain why companies doing AI-driven layoffs are often proven wrong, with costly reversals already happening.

AI may be dramatically changing how work gets done, but we still need domain experts with deep knowledge of the workflows, policies, and edge cases the AI agents are tasked with handing; professionals who can turn this knowledge into prompts with precise instructions to be consistently followed; teams capable of developing sophisticated evaluation processes that run daily; and so forth.

Of course, this “positive side” that I’m celebrating will only benefit the professionals who have or can develop the skills that are becoming more and more valuable across industries.

Skills like technical depth, analytical fluency, human judgment, coordination, domain expertise, are going to remain in demand. But what does it mean for early-career professionals who didn’t yet have a chance to develop such capabilities? Time will tell if employers will find ways to help juniors become seniors in an environment where the opportunities to develop our intellect by working on problems that now AI can easily solve for us have become increasingly rare.

I gave Claude Code a real-life behavioural dataset and asked whether there were any interesting patterns. A few minutes and a few thousand tokens later, it had a clear answer for me:

“I ran an exploratory analysis across all pairwise correlations and group comparisons in the 100-person behavioural dataset. The strongest statistically significant finding was: Higher education level is associated with fewer monthly leisure activities (Spearman rho = -0.23, p = 0.019). […]”

As Kucharski says, at first glance, the fully automated data analysis and confident conclusion look impressive.

Yet, his dataset had been created by randomly simulating 100 individuals with 5 demographic characteristics and 6 behavioral indicators, which means that none of the AI conclusions had merit. As a statistically literate person, Kucharski wasn’t fooled, but the technically polished but substantively bogus AI output could have easily mislead an unsuspecting bystander.

A qualified analyst would have taken much longer than AI to complete the analysis, but they would have never made this statistical mistake when performing the task.

The fact that the dataset was based on “random noise” doesn’t even matter here (except as a way to efficiently prove the point about the AI analysis being wrong). Even when data collection is carefully designed and executed, issues like p-hacking can still occur during analysis and reporting. The problem here isn’t how the data was created, but what happens after the data exists.

AI enthusiasts will say that this kind of mistake can be prevented by giving AI “more context” and guardrails, as well as by reviewing the output for technical flaws and asking for correction.

Well, what they’re saying then is that generic AI tools can only be reliably used by experienced professionals leveraging it to accelerate their results without delegating the actual thinking.

But at the moment we have lots of CEOs (I know some of them around the world, including in my native country, Brazil) celebrating how they’re being able to reduce their workforce using AI to “scale productivity” by replacing seasoned (read: expensive) programmers and data analysts with a team of much cheaper junior professionals armed with generic AI tools.

I thank them when they boast publicly about adopting AI while going through rounds of layoffs involving senior staff. That means I can mitigate my investment exposure to businesses facing the elevated risks of delegating tasks like statistical analysis or programming to a generic AI tool without robust supervision. Sooner or later they’ll end up developing false confidence in incorrect data findings and/or having to deal with potentially disastrous bugs and security vulnerabilities in their internal software.

The risks can be significantly reduced by choosing dedicated AI tools that have not only been built with robust guardrails to perform a specific task, but fully tested and vetted for the specific context of the business, data, and codebase.

And on top of that, one has to make sure that there are enough experienced professionals around to proactively anticipate and detect errors before they cause any harm.

But who wants to go through such heavy investments in technology and human expertise when hyperbolic statements by tech leaders insist that giving novice users access to generic AI tools will aggregate into huge “productivity gains”? Perhaps it will happen once the damage of a flawed AI output that is public-facing or tied to sensitive decisions starts to cascade across reputation, legal exposure, and day-to-day operations.

In the first group are people who have done their own research and formulated a solid hypothesis about what will get them their “dream job”: skills to develop, projects to add to their portfolio to become a stronger candidate, mentors to seek. Only then they’ll reach out to get my opinion: does it look like the plan they put in place might work? Are there any tweaks I’d recommend?

Typically I have no notes or just a few pieces of advice to improve the strategy of these evidence-hunters. (For example, narrowing down their focus to one area of expertise to develop their unique “|” shape strength.) Because they’ve already done their homework before asking for help, they’re already in the right path to achieve their goal.

But an alarming number of messages comes from people in the second group, the ones with a scattered, unfocused approach to developing their data science skills. Those tend to reach out only after they’ve invested significant amounts of time studying topics or pursuing certifications that won’t help them become an attractive candidate for the kinds of jobs they seek. I hate having to give them the bad news: you’ve just wasted months moving in the wrong direction, and now need to course-correct.

The impulsive learners from group 2 perfectly illustrate the saying, “The cobbler’s children have no shoes.” I wonder if they realize the irony of failing to apply an evidence-based approach to drive their own career choices while in pursuit of a job that’s primarily focused on asking, “What does the data say?”

The solution to avoid this pitfall is to evaluate your strategy (the what) and tactics (the how) of skill building with the same discipline you’d use for analyzing trends and patterns in data. And then keep adjusting your learning goals based on hard evidence rather than habit or hype.

For example, rather than assuming that more certifications will help you land your target job, check job postings, employer feedback, and market data to base your conclusion.

As Seth Godin says,

If you are showing up with skill and effort and executing perfectly, all in support of a strategy that doesn’t make sense, you’ve wasted your effort.

(You can learn more about how to customize your learning path and avoid the “career confusion” that afflicts many impulsive learners here: Finding your T-shaped strength.)

—
You may also like the follow-up: Great Decisions Start with Challenging Your Own Assumptions

In parallel, larger organizations are quickly moving beyond the traditional turn-by-turn conversation of a chatbot and adopting AI agents that autonomously plan and chain together multiple steps to achieve various goals, from booking travel to completing research that requires logging into accounts, running code, and compiling results into spreadsheets or slides.

According to the KPMG’s latest AI Quarterly Pulse Survey conducted between May and June of 2025 with “130 top-tier U.S.-based executives and business leaders, all from organizations boasting annual revenues of $1 billion or more,”

Organizations are rapidly accelerating from experimentation to piloting AI agents.

Still, according to the Preliminary Findings from AI Implementation Research from Project NANDA, despite $30–40 billion in enterprise investment into GenAI, 95% of organizations are getting zero return.

Examples abound of failed trials, like the one by the UK government using Microsoft Copilot that resulted in “no discernible gain in productivity.”

What’s the secret to extracting real value out of GenAI?

First, to repeat the requirements you’ll find in most articles written by consulting firms or AI leaders describing how to move the needle with GenAI:

• Successful change management, with “mobilization of the C-suite leadership to effectively drive AI adoption and scaling.”

• Blended teams that integrate specialized external talent with full-time employees.

• An “assemble approach” that customizes solutions for specific business needs based on open-source building blocks that can be easily updated or swapped out (a key step when “the shelf life for state of the art AI is shorter than a jar of organic marinara sauce”).

To understand why all these elements may not suffice, let’s at a fictionalized case study in a domain in which I’ve spent two decades working: software development.

Company B hears about the excellent results a competitor, Company A, achieved using AI-assisted software development to increase the productivity of their developers. It hires a consulting firm to help with GenAI capability building, redesigns its software development process from the ground up, and establishes robust change management mechanisms to prevent resistance or training gaps. But despite doing everything “right” according to expert advice, at the end it can’t reproduce any of the productivity gains claimed by Company A.

What went wrong? Assuming that Company A wasn’t exaggerating its results and execution in Company B wasn’t flawed, the missing element is likely to be untested assumptions that led one company to copy what was done in another without understanding why it worked.

Clearly, this is not a problem limited to investments in GenAI (thus the “beyond” in the title reflecting the fact that the lessons here are applicable broadly to significant investments). At the core of the issue is the lack of a reliable foundation for making informed investment decisions.

Misunderstanding the “why” may lead to weak (or negative) results

In our example, the development teams in both Company A and Company B create software primarily in Java and React. However, in Company A, the developers were being slowed down by labor-intensive and easier-to-inspect tasks like documentation and test case generation (things the AI assistant solution excels at). In contrast, at Company B the bulk of the work is about optimization, defect fixing, and hard-to-inspect tasks like verifying system architectures. No one with knowledge of vehicles would assume that a car that performs well on smooth roads would necessarily work off-road, so it shouldn’t be a surprise that the solution that yielded productivity gains for Company A failed in Company B.

Humans are terrible predictors

There are numerous studies in behavioral science, psychology, and decision theory that explain why humans are generally poor at predicting the future, especially in complex, uncertain environments. With that in mind, it makes little sense to pay attention to what people '“expect” will become true in the next months or years, or what outcomes they “believe” will be obtained from an investment in technology or workflow redesign.

Take for example the results of a randomized controlled trial (RTC) used “to understand how AI tools at the February–June 2025 frontier affect the productivity of experienced open-source developers.”

In this experiment, the developers displayed an overoptimistic opinion of how AI affects their productivity, both before and after completing tasks. Before starting work, developers forecast that AI would reduce issue completion time by 24%. After the work, their estimate was a bit lower, 20% on average. In reality, the study measured a negative effect of AI assistance on their productivity.

The Bottom Line: Evidence-based decisions are the only reliable foundation for positive ROI in any significant investment

The saddest part of the current state of affairs is that many AI-assisted workflows that could achieve persistent value will be abandoned after millions are wasted in failed pilots or flawed implementations.

What’s missing for the organizations on the wrong side of the GenAI divide (high adoption, low transformation) is an evidence-based approach to investment decisions.

In its modern form, the “evidence-based approach” was popularized by the influential work of Professor Archie Cochrane, who in 1972 argued for the need for systematic reviews of clinical evidence, changing medical practices.

Applied to investments in digital business transformation, an evidence-based approach requires going beyond the veneer of credibility of vendors and their successful case studies. It calls for framing your assumptions as testable questions. It demands independent research that isn’t tainted by conflicts of interest or weak claims that only look like solid evidence. It avoids common pitfalls such as survivorship bias, where failed projects are excluded from the analysis, or inappropriate benchmarks that don’t reflect the context of how a solution will be used.

Lack of access to formal studies or reported data is not a valid excuse for failing to adopt an evidence-based approach to business innovation. Smart organizations protect their large investments by first identifying all relevant assumptions and creating testable hypotheses (“We believe that implementing [new tool/process] will reduce [X] by [Y]% in [Z] time.”). And from there, they use experiments, data, and feedback to develop their own evidence in a systematic, rigorous way.

However, despite today’s almost limitless options for visually interacting with tables on the fly, the problem of turning data into information that actually changes the way people think hasn’t been solved by data visualization software.

If your aim is to inform, change people’s minds (as opposed to dumb things down, make things faster and more average), there is no way around applying effort and workmanship in the service of this goal.

Knowing the basics of data visualization, including which charts tend to be best suited to convey different types of information, while useful, is hardly sufficient to produce actionable information.

Consider this example from Seth Godin. Both visualizations use bar charts to show the decline in reading among Americans during a period of time, but the one on the right is a thousand times more effective to describe what’s happening.

Questions to be answered before you start creating any chart

Whether you’re a statistician that believes that everything could be a bar chart, or a designer who takes pride in building stunning visuals, nothing will define your visualization’s resulting value more than answers to the following questions.

1) Why do we think we need data viz here?

It’s true that clear and engaging visualizations are often the most effective way to make complex data more accessible and understandable. When we’re trying to comprehend relationships, trends, or patterns contained within a complex data set, our brains tend to process pictures much more easily than rows and columns of data.

Still, when we need to communicate information, it’s always wise to ask first why we might want to do a visualization. Some people are so eager to jump into building a graphical representation that they skip the step of understanding whether they truly need data viz to communicate their data insights effectively.

2) Who is this visualization for?

Are you trying to convey information to executives with an urgent need to make a decision? Change public perception about a social issue? Entice casual browsers looking for entertainment?

The best starting point for good data visualization is clarity about who is it for. Decisions based on assumptions about who the viewers are and how they interpret information are not going to yield a high-impact visualization.

3) What change do we seek to make with this information?

Is the intent merely to get people to recognize your good taste, sign up, 'like', share? Turn data into actionable truth? Change someone’s mind about an important issue?

Clarity about the change you want to make helps establish the context for your visuals and produce specificity in terms of what to emphasize in the data: trends or relationships, differences or similarities, exact numbers or big picture.

4) What constraints do we need to design around?

Will your visuals be part of an in-person presentation in which you’ll be able to click on various parts of a chart to drill into different views of your data on the fly? Or will they be part of a report offering a static snapshot requiring things like explanatory text, annotations, and footnotes to be understood without help?

Will your audience have the opportunity to interact directly with the underlying data to build their own charts? If so, are they sophisticated consumers of visual analytics, or unfamiliar with the tool?

The constraints you’re facing should be used as an asset, an opportunity to ensure that your charts become a powerful narrative device for presenting information to its intended audience.

Nowadays there is no shortage of tools and libraries to generate plots in one click.

The hard part is choosing, among near infinite possibilities, the subset of graphical and pictorial representations that will deliver on expectations.

And just because ChatGPT can build you a chart that is full of bells and whistles and looks pretty, it doesn’t mean that it can do the work that needs to be done to communicate data insights effectively.

Specificity in terms of why, who and what is the key to using data viz to make people see something they weren’t expecting and compel them to act on it. While design matters, it’s easy to fall into the trap of thinking that beautiful visuals matter more than they do.

Between a fanciful graph that causes your audience to walk away wondering what they were supposed to get out of it, and a simple chart that produces an incredible “a-ha moment,” which one would you choose?

As Seth Godin says,

The purpose of a graph is to get someone to say "a-ha" and to see something the way you do.

Begin there and work backwards.

Mistake 2: Starting to evaluate model performance before you have a clear and specific business use case

As the AI arms race intensifies, the biggest names in tech keep rushing not only to offer new and improved versions of their LLMs, but also to facilitate comparisons between language models and their different versions via automatic evaluation tools with predefined metrics such as accuracy, robustness, and friendliness.

The reality, though, is that to successfully compare LLMs and identify the smartest LLM available to power a particular business chatbot, first you need to have a concrete use case in mind.

This is because each large language model has its own “personalities”, strengths, and weaknesses. While GPT-4 remained the dominant model for over a year, various comparative studies now show Google's Gemini Advanced outperforming it in some tasks, Antrophic’s Claude 3 Opus in others, and so forth.

In practice, even two tasks that look similar may lead to dramatic differences in model performance.

For instance, consider a conversational AI tool designed to retrieve information from large bodies of documents. In the surface, it looks like a fairly specific scenario. But when we look deeper, we can see how the underlying technical problem may diverge enough that the problem requires further refinement for us to be able to make the right comparison between LLMs and corresponding prompt strategies.

One of the key questions to ask here is, “What kind of information needs to be retrieved?”

Imagine that a chatbot is being designed to help legal professionals extract, from a large volume of files, content that meets a known requirement (e.g., “Give me a list of all the references to liability amounts greater than one million dollars in this set of contracts”.)

This problem is similar to the task described in an experiment performed by Arize AI and documented in the article The Needle In a Haystack Test. In this experiment, various LLMs were prompted to answer what the best thing to do in San Francisco was based on an essay on a different topic where the answer was “hidden” at different places. The experiment concluded that different models need different prompting strategies to perform well in the task, and small differences in prompts could lead to big changes in retrieval accuracy.

In the scenario above, the LLM can be informed in advance exactly what object is to be retrieved. But what if, instead of a haystack hiding a needle, you have an auto salvage yard filled with old cars and trucks where you’re hoping to find any kind of junkyard treasure?

A problem of this nature can be found in the HBR article Use GenAI to Uncover New Insights into Your Competitors. The article describes how an analysis using generative AI allowed a company to to flag a valuable detail that had been initially overlooked by a team of analysts pouring over a 200-page annual report of a rival manufacturer of heavy industry products.

The detail (the purchase of a plot of land in India) could be interpreted as a clear sign that the competitor manufacturer had the intention to built a factory to expand into the Indian market. That piece of information, contained in 14 of the 33,660 lines of text in the report, made it possible for the company to start making informed decisions to respond to the rival’s expansion months before it became a reality.

Trying to evaluate models for information retrieval not knowing whether we’re optimizing for a needle in a haystack challenge or a treasure in the junkyard competition is like trying to evaluate a runner not knowing if we’re looking for a sprinter and a marathoner. Each modality requires vastly different skill sets, and consequently demands distinct performance assessment tests.

To avoid mistakes when testing LLMs for an AI-powered business chatbot, start from understanding exactly what your use case is. To be trusted, the evaluation approach and metrics should be defined after the business application is understood at the “sprint or marathon” level, rather than just at the “running race” level.

It’s all about large language models (LLMs) these days.

With their innate ability to accept arbitrary inputs from users and handle digressions and follow-on questions, LLM-powered conversational bots are swiftly replacing traditional AI assistants and rules-based chatbots. And this makes sense, given their potential to enhance customer service, improve employee interactions, reduce response times, and cover a wider range of work-related tasks.

LLM creators, cloud providers, and software vendors are quick to promise AI tools that “comprehend and respond to your queries immediately, even those pertaining to internal, confidential data” and “provide exceptional service round the clock” with “virtually no initial setup time”. However, bringing risks down to an acceptable level and achieving positive ROI with an LLM-powered chatbot is much more complex (and costly) than they would have you believe.

I’m not going to focus on costs here, other than noting that most organizations using proprietary or even open source LLMs are currently relying on heavily subsidized services that cost much more than what they’re currently paying. (More on costs of both proprietary and open source LLMs can be found here.)

This and future installments of this series will center the discussion on common mistakes to avoid to prevent business initiatives involving LLM-powered chatbots from having disappointing or even disastrous results.

Mistake 1: Overestimating the abilities of LLMs

Hiding behind the awe-inspiring and fast-evolving natural conversation ability of the current breed of LLMs are some weaknesses that, combined with our tendency to anthropomorphize (i.e., attribute human characteristics or behavior to an object) broadens the scope for misuse of language models.

As well illustrated by Cal Newport in his article Can an A.I. Make Plans?, LLMs can at the same time ace SAT exams, beat us in chess, and falter on basic math or planning activities that an average person can easily handle.

The following fictitious example was adapted from a real-life scenario.

A vendor that provides email marketing services decided to offer a conversational chatbot to help customers automate marketing tasks. One of the tasks the chatbot is charged with is to help marketers compose marketing emails.

The solution passed numerous tests involving promotional offers and mathematical calculations. But after going live, the chatbot began to fail in tasks like the following:

You are a marketer in charge of communications for an e-commerce business. Write a message inviting customers to take advantage of a time-sensitive promotion: get 20% off the original price of $87.50 on a pair of high-end headphones when purchasing within the next two days. The promotion can be combined with another 20% discount applied to the already discounted price when they sign up to a free account during checkout. Include the total discount percentage the customer will get when taking advantage of the two promotional offers.

In early April 2024, I tried this prompt with the free versions of OpenAI’s ChatGPT (based on GPT-3.5), Microsoft’s Copilot (based on GPT-4) , and Anthropic’s Claude 3 Sonnet (Claude’s second-most intelligent model, behind Claude 3 Opus).

The only chatbot to get it right was Claude. Both ChatGPT and Copilot provided the wrong percentage (“Sign up for a free account, and we'll sweeten the deal by applying an additional 20% discount to the already discounted price. That means you'll get a total of 40% off relative to the original price!”) when in reality the total discount percentage is 36% when we apply the straightforward compounding discount rule as the prompt instructed.

You may be thinking, “The technology is getting better all the time; I wouldn’t be surprised if the paid versions of the models that failed the test are already getting it right.” And this wouldn’t surprise me either: the playing field is constantly changing as LLM creators fine-tune their models further.

But as LLMs’ behaviors change over time, it’s wrong to assume those changes always yield better outcomes. Studies like How Is ChatGPT’s Behavior Changing over Time? show how language models can experience performance degradation in a variety of tasks—including their ability to provide correct answers to math problems.

In particular, given how many experiments keep revealing LLMs’ surprising failure modes, it’s a bad idea to rely on their astonishing performance on bar exams, SAT math tests, and other tests designed for humans to execute on its own work involving things like quantitative analysis or strategic planning. In such cases, with the appropriate guardrails an LLM-powered chatbot can add value by introducing a user-friendly layer to the delivery of answers and self-service actions, but not successfully replace a computational system.

For instance, consider a provider of decision-support technologies for the aviation industry. Clearly, using an AI chatbot to produce on its own answers to questions like “When is this plane expected to land?” would make no sense. On the other hand, the same chatbot might become a great companion to a model that calculates flight arrival estimates based on data about weather, radar data, and other factors, helping users interpret complex analytics and answering follow-on questions in human-like language.

What I’m saying here may seem obvious for people working in organizations that have already made significant progress integrating data and analytics across their core functions. But in companies left behind in the quest to embrace advanced data analytics, it’s common to see executives being dazzled by the draw of a new favorite buzzword, “generative AI” and placing unwarranted trust in the “cognitive abilities” demonstrated by the newest versions of LLMs.

Whether AI chatbots are used in customer interactions or only in internal processes to help staff complete their tasks fasters, the risks arising from overestimating their impressive capabilities to respond to human prompts should not be ignored. (Given the many limitations and challenges of LLMs, it’s hardly surprising to see the biggest names in tech investing heavily in the next wave of computing beyond language models.)

This is not to say that LLM-powered chatbots can’t be put for good use in business environments.

The point here is that cognitive tests designed for humans are not a reliable measure of an LLM’s abilities and limitations, and while their “apparent reasoning” make them suitable for some specific real-world tasks, the technical and managerial barriers to making LLM-powered chatbots perform well in high-stakes business scenarios is hard to overstate.

Part II of this series can be found here:

LLM mistakes to avoid (Part II): The key for effective model evaluation is to start from a clear-cut use case

Looking for advice on using LLMs for personal productivity?

Recommended reading:

One Useful Thing

Which AI should I use? Superpowers and the State of Play

For over a year, GPT-4 was the dominant AI model, clearly much smarter than any of the other LLM systems available. That situation has changed in the last month, there are now three GPT-4 class models, all powering their own chatbots: GPT-4 (accessible through ChatGPT Plus or Microsoft’s CoPilot), Anthropic’s Claude 3 Opus, and Google’s Gemini Advanced…

Read more

3 years ago · 317 likes · 17 comments · Ethan Mollick

From the archives:

How to increase your individual productivity with free generative AI

On Modern Analyst:

2024 Trends in Business Analysis, and How not to Be Replaced by ChatGPT in the Age of AI

We’ll get to them, but first the good news: if you think your company is behind in its ability to compete in analytics, keep in mind that even organizations that dominate their fields and are known for their industrial-strength analytics from time to time suffer from the same weaknesses.1 Moreover, the competitive advantage these industry leaders developed due to the prohibitive cost of technology and difficulty consolidating structured and structured data is quickly disappearing. As a byproduct of technological advancements, now every firm in every industry has the ability to leverage data imbued with relevance and purpose to support better decision-making.

Here are the three mistakes to avoid:

1. You ask your data team to go on “fishing expeditions”

Imagine that sales are going down. If you ask your data analysts to go figure out what’s going on without additional context, chances are they’ll look at every variable they can possibly get their hands on at once. And they’ll probably find relationships that don’t really exist (the same principle of flipping a coin multiple times and incorrectly concluding that a run of consecutive heads or tails means the coin is biased).

Solution: Make sure you have at least one member of your data & analytics team that is a hybrid of statistician, analyst, communicator, and trusted advisor. Give this individual an incentive to speak up when a problem brought to the analytics team isn’t well formulated or need to be further distilled into a set of hypotheses that can be properly tested.

2. You ignore the issues caused by data silos

In one of my projects, I worked with a telecommunications company that had excellent data about subscriptions and cancellations, as well as data usage and quality of service. However, due to departmental silos, the business had no way of cross-referencing these two sources of data to allow an understanding of how usage behavior and signal strength affected customer churn.

As a result, the company was able to create wonderful dashboards with beautiful charts to display its vast collection of historical data, but failed to develop an understanding of what was causing the business to lose an alarming number of subscribers every month.

Solution: First, determine the root cause of the problem. It may be technical (data that is not naturally congruent or easy to integrate), or, as happened in many of my projects, primarily caused by politics (internal groups being overly protective of their data and finding excuses not to share it).

The technical problem is getting easier and easier to solve with cloud-based tools that facilitate integrating data across sources. The internal politics issue may require escalation to the C-suite. Be specific about the business problems that could be solved and opportunities that could be exploited if the barriers to consolidating information from various sources (e.g., online search, customer complaints, commercial transactions) are removed.

3. You only use the data when it agrees with your intuition

Many decision-makers only accept data that validate their own conclusions. In a project to identify fraud in a marketplace, an executive had his own intuition about what types of activity represented fraudulent behavior or not. After analyzing the data, it became clear that some of the activities that were considered legitimate also pertained to fraud. Rather than accepting the evidence, the executive kept asking for more and more analyses, constantly delaying important decisions necessary to curb the fraudulent behavior.

The solution: If you’re a senior executive, help change the decision-making culture by openly recognizing when data have disproved one of your hunches, and allowing your opinion to be overridden. lf you’re not part of the leadership team, see if you can pick a quant-friendly leader to help teach the organization the habit of asking, “What do the data say?”

In parallel, update processes and create simple, understandable tools for people on the frontlines to view analytics as central to their decision-making.2

As a data science consultant, I lost count of the times when I heard a customer say, “Oh, but we have such limited data!”—only for me to realize that they had better data that many of their competitors that are ahead in the use of sophisticated modeling to sharpen their marketing, risk management, or operations.

As I wrote in a past article, imperfect data is often good enough. Companies succeed in leveraging data to improve decision making not because they have more or better data, but because they have leadership teams that ask the right questions and maximize cross-functional cooperation to learn from the available data.

To better compete on analytics, focus on helping your leaders and employees become more evidence-based in their thinking, more open to sharing the mountains of data trapped in department silos, more knowledgeable on how to frame and test hypotheses, and more willing to conduct experiments and incorporate evidence-based insights into their decision-making.

It would be logical for retailers in the food space to be nervous. One would expect that having all that data about consumer habits and preferences and access to the most powerful cloud services—including processors designed for AI training— would make Amazon the big winner on physical retail.

Yet, the e-commerce titan has paused new openings of Amazon Fresh and is shutting down its clothing stores. Jason Del Rey, who was on the Six Pixels of Separation podcast to talk about his new book on the rivalry between Amazon and Walmart, points out that years after the acquisition, Amazon is still figuring out how to leverage Whole Foods. “It’s a fine business, but it’s been underwhelming to me.”

Amazon should be an inspiration for companies still using “we need more data” as an excuse for inaction. Should we try to use the best data possible to increase our decision-yield? Absolutely. However, as a data science consultant, I’ve been to all kinds of organizations (including big retailers) where the following sins continue to be repeated over and over:

• Treating quantitative data as a panacea the detriment of opportunities that can only be exploited if we go past the usual assumptions and conventional thinking.

• Using the existence of data gaps as an excuse for inaction.

It’s true that the right data can lower risks and improve decision-making. However, many companies that proudly declare to be “data-driven” tend to forget that their customers and employees are people.

The title of this post was inspired by this 2019 article by Alasdair Rae. UK-based Rae partnered with American scholar Garrett Nelson to write a paper on the “economic geography of the United States” based on open data. The paper ended up being useful for people on a multitude of fields, from epidemiology to renewable energy.

Rae writes,

Why do I think this story demonstrates that data needs people more than people need data? The first reason is that this wasn’t new data. It just needed people with the inclination and time to make it useful. The fact that it was open data wasn’t enough. The second reason is that it needed human input to make it appealing and accessible. A third reason is that it demonstrates how it needed human interaction to take it beyond the realm of data into information and knowledge. The knowledge came about through human interaction.

In the same podcast mentioned above, Jason Del Rey and Mitch Joel talk about the growth of Trader Joe’s (American chain of grocery stores).

Like me, both are Trade Joe’s shoppers, with Mitch having to cross the border from Canada to buy there. The retailer’s success is a great example of how “experience can still matter”, as Jason explains:

I walk into a Trader Joe’s in Clifton, New Jersey, and I’m going to like all the people that are working there, and they’re going to seem at least that they care about my day, and that I’m having a good experience. There’s obviously a need in the country and in the world for retailers that focus exclusively on getting you the price you need to afford to shop, but darn, it’s cool to go into a physical store and have someone actually care about how you’re gonna feel walking out thirty minutes later.

We can add all the data and technology in the world to a business, but in the end it’s people who build things that create connection and value that just isn’t possible when we try to optimize output at the expense of our humanity.

Instead of obsessing over perfect data to help you sell more average stuff to more average people, consider an alternative: leveraging human curiosity and creativity to build interesting and memorable experiences, magical products and services, or even just displays of generosity and delight.

I wrote in my previous post why I believe that’s the case:

To illustrate what I mean, consider the following example.

A company was struggling to meet service level agreements (SLAs) in contracts that guarantee certain levels of incident reaction time. To minimize losses from SLA complaints, management decided to invest in AI to scale ticket resolution. The company spent millions building a conversational AI application. While the solution achieved the stated goal, the operational costs—in particular those associated with regularly updating its knowledge library and conducting tests to ensure that AI continues to perform accurately over time—quickly become prohibitive.

What would have prevented this undesired outcome? It starts with knowing how to prioritize an automation initiative. Eighty percent of the reported incidents were outside the scope of an SLA. For those 80%, the company only had to confirm the condition and instruct the customer to troubleshoot on their side.

Armed with this information, the team responsible for the conversational AI solution could have achieved the desired outcome—minimize losses from SLA complaints— simply building a triage system to screen out the 80% false alarms and escalate to technicians the 20% of tickets that truly needed assistance. The simplified solution would have required a fraction of the content originally ingested to fine-tune the language model, made the technicians’ workload manageable, and achieved the desired service metrics at a much lower cost.

This may seem like an obvious mistake easily avoidable by checking the relevant metrics. After all, “only 20% of the support tickets opened by our customers can result in an SLA infringement and need to be addressed by one of our technicians” is information critical for the health of a company’s operations. Why wasn’t it considered when choosing appropriate targets for AI automation?

The truth is that few companies have developed the kind of “industrial-strength analytics” required to piece together the knowledge hidden in disparate, unclear, conflicting sources. Even in large organizations with big investments in technology it is common to see the IT department working hard to integrate internal data sources with slow and underwhelming results.

As noted in the prior article, the culprit is a weak data strategy that encourages rogue data sets to propagate in silos. Here is one of the most common situations I’ve faced in over a decade of data science consulting work:

Client: “We want to use AI to prevent or minimize X.” (Where X can be anything from billing exceptions to avoidable support calls or erroneous location readings from IoT devices.)

Me: “OK, can you tell me the frequency at which X happens?”

Client: Blank stare followed by an admission that a reliable answer to my question wouldn’t be ready in time to inform any decisions about their urgent AI project.

Game theory provided a formula for the value of information many decades ago. In an ideal world, we would be able to eliminate any uncertainty about some big investment decision by seeking all relevant information. In practice, we know that the cost of data acquisition may exceed its benefits. Still, if we are realistic about the uncertainty surrounding a business decision and the cost of making the wrong choice, we must care about collecting enough data to mitigate the risk of a bad investment.

Executives anxious to find proof cases that show that Gen AI can deliver the promise of outsize productivity gains may fall into the temptation to make swift decisions for fear of waiting too long and falling behind competitors. But this is exactly when slowing down and investing even in partial uncertainty reduction can dramatically increase the odds of success.

In a pinch, this may require using a labor-intensive ad hoc analysis to minimize the risk of erroneous allocation of limited resources. But hopefully this exercise will also lead to a better understanding of the value of taking raw data (such as customer support logs) and integrating it with other data to transform it into information to guide decision making. And, from there, maybe more C-level executives will start to appreciate how improvements to data management enable the strategic use of operating data to minimize investment uncertainty and make the organization more responsive to market changes.

One can hope.

From profit-making private enterprises to public agencies and third-sector entities, leaders seem fixated on the same question: How might we adopt Gen AI to gain productivity, reduce costs, increase time to market, improve customer experience, get ahead of the competition?

Hopefully soon we’ll get past the next phases of the hype cycle to arrive at Gen AI’s plateau of productivity. But that won’t solve a problem that for decades has plagued all kinds of organizations.

I’m talking about the well-known issue of accumulating big piles of data without achieving insights that lead to better choices and lower risks.

For a long time, studies have shown that companies that adopt a data-driven attitude toward decision making are on average more productive and profitable that their competitors. But too many organizations eager to follow that path fall into the trap of assuming they just need collect and store high volumes of data to get there, when this couldn’t be further from the truth. So they stop their data strategy initiatives prematurely, long before they have achieved the necessary agility to fullfill even a fraction of their emerging needs of data reporting and analytics.

Because I’m a consultant, my sample is naturally biased toward entities struggling to escape this “data rich, insight poor” state. But you can easily check if your organization is any different. Think of a question whose answer would drive smarter decisions and more accurate investments but isn’t routinely tracked in a dashboard.

Now, think about how fast you can get that answer:

• If you are an executive at a manufacturing company about to make a consequential decision that requires understanding lead times, can you effortlessly get a list of the top factors causing shipping delays for each product?

• If you are responsible for ensuring a safe working environment across multiple construction sites, can you quickly report how many workers got injured last year and where those injuries occurred?

• If you manage marketing campaigns, can you pull up a current list of active customers for which an upsell opportunity exists?

Consider yourself very lucky if reliable answers for questions like that are at your fingertips. Give your organization extra points if different teams searching for the same information within their own systems would arrive at the same results.

In my decades working with numerous entities in the private, public, and third sectors, I can’t think of one case where leaders weren’t constantly having to make critical decisions based on partial and often conflicting information. If a senior executive requires a complete answer and has the luxury of waiting for it to be produced, managers will reach out to the data team. Then they’ll impatiently wait for analysts to piece together information from various disparate systems with data stored in incompatible formats, often requiring “reconstructive surgery” to get to a state to be finally queried for answers.

With all attention now turned to Gen AI, the root causes of this problem continue to be left untouched:

• Ambiguous and mutable data definitions. Definitions of what constitutes the “truth” are missing or nonstandardized, causing stakeholders to squander time and resources trying to navigate disparate interpretations of the data.

• Vague or inconsistently applied data rules. Rules for aggregating, integrating, and transforming data are unclear, conflicting, or simply not followed, making it difficult or impossible to replicate transformations and leverage information across the organization.

• Needless duplication of work. Complex data analyses such as predictive modeling that is relevant across the organization must be redone by different groups due to lack of mechanisms to share the information through internal channels.

We all know the solution to this recurring problem. It requires investments in:

• processes to optimize the extraction, standardization, storage, transformation, enrichment, modeling, and visualization of data, and

• solutions to support the seamless information flow between departments and teams.

But with all eyes now on Gen AI, and other emerging technologies with “transformation potential” already vying for attention, why would a CIO or CTO feel motivated to allocate resources toward such mundane data management approaches?

And this is why I expect those unglamourous but vital to high performance data management initiatives to remain neglected. And ironically, as the hype subsides, the organizational impact of generative AI will be largely dependent of a robust data management strategy.

I’ll talk about why that’s the case in my next article.

Subscribe now

Photo by Joshua Woroniecki on Unsplash

Roles like mine, heavily reliant on programming, data analysis, and content creation, are estimated to have the highest exposure to generative AI tools. It’s not surprising that with the help of LLMs, about 20% of my tasks are being completed faster at the same level of quality.

And that’s practically just from using the free version of ChatGPT, which means I don’t have access yet to the updated model that can also browse the internet, or a tool approved to use with sensitive data.

What kinds of tasks am I automating or streamlining with ChatGPT?

For me, it’s primarily about efficiently retrieving pieces of publicly available information that fit these two conditions:

1. Comprise common knowledge that a LLM is likely to have seen numerous time during training, and therefore has low probability of being misrepresented.

2. Can be quickly checked for accuracy, or will have negligible impact if the information turns out to be wrong.

I’ve learned that tasks with these two characteristics are great for generative AI. Below are some examples that hopefully will inspire you to find similar opportunities to delegate repetitive work so you can use the time saved to focus on being a good leader, or producing higher-quality results for your creative work.

1. Explain technical concepts to a client, colleague, or mentee

Another day I was asked by a manager to urgently review a proposal document he was writing. The document had an image with the title “System State Diagram”, when in fact it depicted a process flow diagram. When preparing my feedback, instead of writing down the differences between the two diagrams, I asked ChatGPT to compare and contrast the two concepts. I then copied the answer and added a note recommending to keep the title but replace the image with one of a real state diagram, which was the right visualization for the document. Not having to search for or write down an explanation on my own made it possible for me to finish my review in the limited time I had between two client meetings.

Tasks like this are common in my job and in many other roles that involve giving feedback for other people’s technical work. For example, if it’s part of your responsibilities to teach employees how to use commercial software, approve power point presentations, or perform peer-based code reviews, it should be possible to delegate to ChatGPT many of your “explaining” tasks that don’t involve proprietary information. This is particularly true when you’re already familiar with the concepts that need to be explained, and thus can easily check the accuracy of AI-generated content.

2. Fix code

Yesterday a mentee was getting an error in a Python notebook. An online search for the error message showed that the issue was an incompatibility between the latest versions of two packages. But what if the recommended solution (downgrade one of the packages) isn’t feasible? In seconds, ChatGPT provided alternative code using a different library. We could immediately confirm that the proposed fix worked by simply running the new code in the notebook.

3. Help streamline random tasks

There are so many tasks I’m currently delegating to ChatGPT that it’s even hard to decide which examples to share.

Today I needed to write an email in Italian. Since the content didn’t include any sensitive information, I wrote it in English, then asked ChatGPT to translate for me.

(TRUE) 1. Comprise common knowledge that a LLM is likely to have seen numerous time during training, and thus have low probability of being misrepresented.

ChatGPT explains: “My strongest language is English, but I can also work reasonably well with languages such as Spanish, French, German, Italian, Portuguese, Dutch, Russian, Chinese, Japanese, Korean, and many others.”

(TRUE) 2. Can be quickly checked for accuracy, or will have negligible impact if the information turns out to be wrong.

I understand Italian reasonably well, so it was easy for me to confirm that the translation was in good shape before sending the email. If I had to write, say, in Korean, a language I don’t know, I’d find another way to validate the content. In that case I’d probably use Google Translate to revert the text to English.

The main benefit of using ChatGPT for writing content like this is that you can simulate a human conversation and ask for tweaks, like making the tone of the email less formal—something you can’t do with a tool like Google Translate.

In their current state, LLMs are great when all we need is a classic, tried-and-true answer to a question

The scenarios above were highly suitable for the current state of LLMs because none required a creative or innovative solution. What I needed was, “just the facts, ma'am.”

The same rules work with personal tasks as well. For instance, last week I wanted to make savory crepes for a quick lunch before my next meeting. Rather than search for a recipe, try to guess the best link to pick from the search results, close an annoying subscription pop-up, and scroll down a thousand words about the good memories that particular food blogger associates with eating crepes to check the ingredients, I simply asked ChatGPT.

(TRUE) 1. Comprise common knowledge that a LLM is likely to have seen numerous time during training, and thus have low probability of being misrepresented.

The model has certainly seen a large number of savory crepe recipes during training.

(TRUE) 2. Can be quickly checked for accuracy, or will have negligible impact if the information turns out to be wrong.

Since I’ve made crepes many times before, I could easily validate that the ingredients and proportions made sense, so the risk of trusting ChatGPT’s instructions was very low.

If, instead of a quick meal, I wanted a recipe that would expand my food horizons, rather than relying on a chatbot, I’d have looked for a recipe by Yotam Ottolenghi or another talented chef.

At least for the moment, generative AI is not the place to go to for things that require “the genius that is the domain of human beings.”

"I wish I could say that the advances in AI will make it easier to create hits, obviously it won't. Hits are created by genius. And data sets plus compute plus large language models does not equal genius. Genius is the domain of human beings and I believe will stay that way."And data sets plus compute plus large language models does not equal genius. Genius is the domain of human beings and I believe will stay that way.”

Take-Two's CEO Strauss Zelnick answering a question about how emerging AI tools may affect the development of games like Grand Theft Auto.

It’s possible that at some point models specifically trained on content from top chefs will become a good source for cooking exploration, but LLMs are not there yet. Take this quote from a food writer who wasn’t impressed with the recipe provided by ChatGPT for Moroccan Spiced Meatballs with Yogurt Dipping Sauce (emphasis mine):

It uses beef (something I see often in North American online recipes), not lamb, as in the cookbooks in my library. When I asked for a recipe with lamb it subbed out the beef for lamb, without adjusting any of the other seasonings. This is similar to the “one sauce to cover them all” mentality I keep running into online and in restaurants. Many of us from non-Western food cultures know that lamb and beef have different flavour profiles and adjust our spicing and cooking accordingly.

—Jasmine Mangalaseril, The Incredible Blandness of ChatGPT

This is also why, when I’m writing an opinion piece, I don’t use LLMs at all.

Wait, but isn’t creative content considered one of the primary use cases for generative AI?

It may be so, but while I’ve read about people reporting increased productivity using chatbots to generate marketing content or create article drafts from lists of talking points, that’s not the job I want to hire a chatbot to do for me.

Three main reasons for that:

• When creating content for work, I am not allowed (for good reasons) to submit proprietary information to a third-party chatbot API. This limits the usefulness of LLMs, preventing document summarization, classification, etc.

• When writing an article for general consumption like this one, I can’t stand the bland content I get when I ask GPT to argue a point I want to make. Moreover, trying to fact-check its plausible BS only slows my productivity.

• Creating content without the help of AI also makes it much easier to have it reflect my viewpoints and/or interpretation of recent data or findings, as opposed to merely follow the patterns and information present in ChatGPT’s training data.

Of course, when you’re tired and lacking ideas, LLMs can offer a quick solution. For example, I just asked it for ideas for the theme of a birthday party for a 40-year-old who loves the movie Apollo 13. The suggestions were pretty boring, but when in a pinch…

Last thought: the goal of productivity is not to maximize activity

My purpose for increasing individual productivity is to reduce how much time I spend on drudgery or repetitive work, not to maximize activity.

By asking ChatGPT to explain a technical concept, find the fix for a coding issue, or translate content to another language, I have protected my amount of “well spent” hours thinking through new ideas and projects that lead to my best work and, more importantly, increased my downtime.

Research on naps, meditation, nature walks and the habits of exceptional artists and athletes reveal that mental breaks not only replenish attention and increase creativity, but are also a key productivity booster. The next time you use generative AI to produce quality work faster, don’t forget to use the extra time to step away from your work or routine tasks and allow your body and mind to rest, repair, and rejuvenate.

Every few months, I get this question on my inbox: Would you be willing to design and teach a quick data literacy course for the executives at my company?

Typically the request comes from a senior manager from a traditional company hoping to get more buy-in for their goal to improve core operations using machine learning. And the requester is not alone in thinking that executives should become more fluent in data and analytics concepts. For example, in the book Keeping Up with the Quants: Your Guide to Understanding and Using Analytics, Thomas Davenport writes:

"Some of the concepts that any executive needs to understand include:

• Measures of central tendency (mean, median, mode)

• Probability and distributions

• Sampling

• The basics of correlation and regression analysis

• The rudiments of experimental design

• The interpretation of visual analytics

The methods for acquiring this knowledge can be the same as for more junior personnel, except that senior executives may have the resources to bring in professors or consultants for sessions with groups of executives, or even private one-on-one tutoring."

There are various schools of thought of what a “data literacy for executives” program should look like. Some offerings are more focused on traditional analytics, covering dashboards, KPIs, and data visualization; others concentrate on developing “AI fluency”, which is the primary goal of the people asking for my help.

Let’s review the two ideas separately: the need for analytical skills at the business executive level, and the efficacy of training to fulfill this need.

1. Every executive needs analytical skills.

I don’t think anyone will disagree that competencies like statistical literacy or the ability to contextualize data presented in visual formats can be valuable assets for any business decision-maker.

Still, I have worked with highly successful executives who couldn’t correctly read a bar chart to save their lives. Or who had trouble understanding why repeatedly running different statistical tests on the same dataset and only reporting the most interesting results isn’t a statistically-valid practice.

I’ve also worked with executives with the same knowledge gaps who made terrible decisions that ended up killing their startups or destroying value for their company.

That difference in outcomes can be explained by a set of behaviors that characterize the best decision-makers:

• Surround oneself with advisers that 1) have the very best analytical skills; 2) can explain things in plan language; and 3) can be trusted not to spin the numbers.

• Seek feedback (not consensus) from the people who are the most knowledgeable about the topic at hand—usually employees at a lower level in the organization doing the hands-on work.

• Ask questions about the reasoning or argument that connects the data to an insight, or a business problem to an AI/ML strategy.

• Push back when they don't understand why a recommendation was made or a conclusion was reached, if necessary asking for translations of highly technical terms into "English-speak" explanations.

In short, what someone needs in order to be a leader that creates value from data and analytics is the ability to think critically, apply first principles to problem-solving, and recognize the value of consulting with those who can contribute in a meaningful way to their decision-making process. Understanding the concept of p-hacking, or the difference between algorithms like Random Forest and CNN, is entirely optional.

2. Small group training sessions can be an effective method to make executives more “analytically-minded”.

The many providers of “data literacy” courses for executives will disagree with my stance on this, but let’s think about it for a moment. We’re talking about busy executives with severe demands for their time and attention. How likely is it that even the smartest of them will successfully tackle the cognitive work that understanding complex analytics concepts requires? Even under the best circumstances of extremely intelligent people being offered content sufficiently fresh, provocative, and relevant to their business, I’d estimate this probability as very low.

The one-on-one tutoring suggested in Davenport’s book is a slightly better approach, allowing for a more custom delivery of content that focuses on an individual’s specific needs. But it risks suffering from the same deficiency that group training does: attempting to squeeze too much learning into a few sessions in response to time and budgets constraints. 

My verdict is in

In my experience working alongside senior management and C-level executives with an anemic experience with analytics, there’s little to be gained from sending them to a “data literacy” program in hopes that they’ll come back more analytics-minded. It’s much more effective to surround those leaders with proficient analytics practitioners who are deeply curious about the business and can customize their analytics presentations to their audience’s needs.

And for an organization considering incorporating AI/ML in some of its processes or offerings, more important than “executive AI fluency” is to make sure business leaders understand and apply the elements of great decisions. That’s what will help them make wise decisions regarding when to use machine learning to automate operational decisions and how mitigate risks and optimize ROI. And that’s also what will help them ensure the organization has all the elements in place to make analytics part of the fabric of their daily operations, embracing not only the right data and technology, but also the right processes and ongoing acquisition of analytical skills across all organizational levels.

And if you’re a business leader looking to upgrade your analytical skills or become “AI fluent”, some good starting points are the books Behind Every Good Decision: How Anyone Can Use Business Analytics to Turn Data into Profitable Insight by Piyanka Jain and Puneet Sharmaand, and Digital Decisioning by James Taylor.

A problem arises when companies start to equate experimentation with split testing. As you probably already know, split testing, also called A/B test, has subjects randomly assigned to one treatment or another (patients prescribed drug A vs. drug B, consumers exposed to price A vs. price B, etc.). Its experimentation process starts with a hypothesis (“drug B is better than the existing standard drug A”, “price B is more profitable than the current price A”).

Such tests are common in web design and marketing, where results are readily measured. In fact, failing to run an A/B test before committing to a marketing change can cost a company millions in revenue, as Basecamp learned from a failed redesign that removed the signup form from their landing page.

However, an overreliance on split tests is a hindrance to innovation.

When split testing becomes an unhealthy obsession

Years ago I was working on a software-as-a-service (SaaS) application that had an obvious flaw that made it very difficult for users to get their job done. The solution to improve user experience was obvious to anyone who understood the “job” the software was hired to do. Still, because the startup relied on data generated from A/B tests for every product decision, it was hard at first to convince the decision-makers to approve the change without running a comparison experiment first.

A good analogy would be a restaurant chain with a floor plan where the kitchen is divided into two rooms separated by a large dining area. To grill some vegetables, the cook has to go back and forth, walking around tables to the other side to wash the produce, then carrying it back to chop it, then crossing the dining area again to finally start grilling.

Imagine setting up an experiment to test the hypothesis, “Bringing the whole kitchen to one side of the dining area (variation) will make the cooking process smoother and the cook happier compared to keeping it divided by a dining area (control).” That experiment would not only be needless, but also cause avoidable delays in fixing the issue across all restaurants in the chain.

In the case of the startup with the SaaS product, the situation was similar. The time required to set up a valid A/B test would only increase the frustration of the customers waiting for improvements. Across a subset of interviewed users and employees who knew the needs of the target audience well, there was full agreement that the proposed change would dramatically improve user experience. It took some effort to convince the leadership team that moving forward without a split test was the proper response, but in the end a quick improvement in customer satisfaction and renewal metrics spoke for itself.

Why split tests are unlikely to create a game-changer

Imagine that you’re in charge of improving user adoption of a dashboard offered an add-on to customers of a SaaS (software-as-a-service) product with the goal of increasing conversion after a free trial.

If you suggest a split test of bar charts vs. gauges to improve chart readability, your boringly rational, incremental idea is likely to be approved without question. Suggest instead experimenting with a no-charts, narrative-based analytics tool to tell a story with the data, and your attempt to “think outside the box” might put you through an endless process to even approve testing the idea.

The sad reality is that teams fixated on split testing often get stuck in a cycle of incremental changes that, at most, yield modest returns on the investment. A/B tests—or A/B/C/D experiments where you are testing multiple versions of feature, price, headlines, colors, and so on—are not a panacea. If they were, applications like Google+, a product made by a company with plenty of A/B test expertise and that welcomed 90 million users in its first year, would still be around rather than being featured in product graveyard collections.

Smart companies understand that true innovation happens outside the usual assumptions and conventional thinking. At times, as advertising executive Rory Sutherland likes to say, we need to “rewrite the brief.”

How to improve your experimentation process

Two key tactics can help organizations improve how they think about experimentation to avoid the pitfalls of split testing.

First, make sure you are working from first principles.

Radically superior solutions are rarely achieved by testing slight variations on the same theme. To quote from a Farnam Street Blog post that explains how first principles work,

When we take what already exists and improve on it, we are in the shadow of others. It’s only when we step back, ask ourselves what’s possible, and cut through the flawed analogies that we see what is possible.

For instance, in one of my projects, following nurses on the job to see how they used the software I was working on helped immensely in understanding the “real job to be done” by the product. This approach allowed us to “rewrite the brief” and consider alternative solutions: what if instead of requiring nurses to use a writing instrument to record patient notes on an iPad (as every competitor product did), we provided a voice-to-text functionality that automatically generated the written notes as they talked to patients?

Exploring multiple perspectives through first principles helps to expand the number of alternatives under consideration and mitigates the risk of getting stuck in a “local optimum”. In that project, rather than merely achieving incremental improvements by testing the efficacy of different writing tools via A/B experiments, we were able to envision what turned out to be a better solution: a voice-to-text feature that took us to product-market-fit much faster.

Second, avoid the tyranny of the spreadsheet

One one hand, we should never settle for a messy innovation process that ignores the experimentation process. On the other hand, it’s often possible to build and validate our knowledge of the problem and solution spaces without extensive data collection or endless A/B testing that in practice might simply create an unwarranted patina of scientific credibility.

As I’ve highlighted in a previous post,

Quantitative data is not a panacea. In two decades working for all kinds of businesses, from Fortune 500 companies to tiny startups, I’ve witnessed time and again successes that couldn’t be projected on a numeric spreadsheet. To strike gold, sometimes we need to rely on curiosity and qualitative insights extracted from talking to and observing our target audience.

For example, many data-driven companies fail to recognize that the more variables you add to an experiment, the easier it is to find support for some spurious, self-serving narrative. If you are comparing results across multiple treatment groups, you are asking multiple questions: Is A different from B? Is B different from C? Is A different From C? And with each question, you are increasing the chance of being “fooled by randomness”, as the more variables you add to an experiment, the greater the probability that something will emerge as “significant” just by chance.

Rory Sutherland points out in his book Alchemy: The Dark Art and Curious Science of Creating Magic in Brands, Business, and Life that for all we obsess about scientific methods, “it’s far more common for a mixture of luck, experimentation and instinct to provide the decisive breakthrough; reason only comes into play afterward.”

. . .

It pays to remember that oftentimes all the data we need to develop breakthrough products and services might be acquired from small experiments to validate our knowledge of the problem to be solved and the criteria our customers use to measure value. And that combination of small experiment efforts may create a roadmap for innovation and growth that’s far superior to the conventional data-driven, A/B test focused model that demands huge sample sizes to achieve statistical significance.

Sure, incrementally better decisions add up to a lot of value over time (probabily), but maybe we’re just stuck in a local optimum and getting many small changes right will never get us to where we want to go. A modification of the famous Henry Ford quote kind of works here: you can’t A/B test your way from selling horses to selling cars. And a corollary: if you’re testing a horse against a car, you definitely don’t need an A/B test.

— Locally Optimal from Causal Engineering

The hardest thing about Machine Learning is NOT about training models or the math behind the algorithms! In fact, that might the easiest thing and interestingly enough, that is the only thing that is taught in school or in textbooks. 

One of the most difficult pieces is first to be able to frame a business problem as a machine learning solution. (…)

I didn’t ask, but it’s possible that the person who brought the post to my attention expected me to agree with this statement. This is because framing the business problem is a big part of my job, and it would be nice to be able to advertise it as being a very difficult task :-).

But in reality, I had a different take:

Take a look at these descriptions of machine learning problems one can find in data science books and online challenges:

• Extract skill requirements from job posting data.

• Predict future friendships from social network data.

• Compute travel probabilities using graph theory techniques.

• Track disease outbreaks using a cluster algorithm.

• Assess online ad clicks for significance.

These, of course, are not good descriptions of business problems. These are challenges that have been conveniently framed to reflect what machine learning does best: find underlying patterns that may be difficult to identify by inspecting the data manually.

And because this is how machine learning challenges are typically presented to students and PhD candidates, many find it difficult to reframe their problems in business terms.

But this is a barrier that a bit of deliberate practice can solve. What is missing here is a link between the machine learning challenge you’re describing and a business solution that can produce a concrete business gain, such as maximize revenue yield or reduce the time to decide and the cost of decisions.

Finding the business purpose for a predictive model

With any predictive model, the business purpose needs to come first, explaining not only what you hope to predict, but also why.

Assess online ad clicks for significance is a bad description for a machine learning project because it doesn’t explain why.

Imagine that you’re working for a retailer using online ads to attract customers. The businesss is interested in increasing revenue and profitability. An executive heards that machine learning models are helping competitors achieve this goal, and hires a data scientist to start building some models.

Someone — either the data scientist or a savvy business analyst — has to make a connection between the large picture (increase revenue and profitability) and the contributions to operational decisions that a machine learning solution can provide.

During this process, a series of use cases may be identified:

• Determine the cross-sell offer with the highest likelihood to convert so that the best offer can be displayed to each customer during checkout.

• Identify customers who are ready to make a purchase in a specific product category within the next few days so that the marketing department can react accordingly (send them an email offer, display the appropriate ad).

• Distinguish customers who will only make a purchase if given a discount vs. others who will buy at full price so that only the former are given a discount to protect profitability.

• Calculate the likelihood of a paid ad improving the loyalty of an existing customer, measured by their willingness to buy other, more profitable, products so that the company only pays for ads that help the bottom line.

This list could go on and on, all focused on models to support high-volume operational decisions that help increase conversion, loyalty, or profitability.

As part of a ranking process, the team would be looking at the benefits of each solution in terms of speed, efficiency gains, cost redutions, adaptability, learning-improvement loops, as well as the costs to build and maintain the required technologies.

It is possible this analysis will show that machine learning can help increase the precision, consistency, and agility of all these operational decisions while reducing the time to decide and the cost of the decision.

If so, the next step would be to try to quantify the return expected from each model. For example, an accurate model to predict which cross-sell offer will convert more at checkout time may have the potential to generate $2M a month, while a near-perfect classification of customers who need a discount to buy may only have the ability to bring in $500K a month.

Assuming the same effort to build both models, the cross-sell model could then be prioritized ahead of the “customers who need a discount to buy” model.

Formulating your problem as a machine learning problem

Most real-world machine learning problems are about finding patterns in data. But such problems are only worth solving if they can produce a concrete business gain. In practicae that typically means building a model that generates an output that automates, expedites, or improves an operational decision.

If you are given a machine learning challenge to work on, try adding the “so that…” part to your ML problem. If you can’t answer how your solution will increase the precision, consistency, or agility of a business decision, and/or reduce the time to decide and the cost of the decision, you probably don’t have a well-formulated business problem yet.

Photo by Michiel Annaert on Unsplash

Ask employees what they think might help improve morale, and they might tell you that reducing the number of meetings they have to attend every week would be a great start.

But that’s not how the human brain typically works. Whenever a business experiences an obstacle, the most common reaction from its leaders is to think of something that can be added to make things better. An array of biological, cultural, and economic factors is constantly pushing us toward more, even when subtraction would bring more practical value:

“Our model to predict customer churn is no longer accurate? Let’s look for some new data sources we can use to increase the number of input variables!”

“The new employee engagement survey came back with low job satisfaction numbers? Let’s create a new employee recognition program!”

Fashion designer Coco Chanel is the author of the quote I used as this post’s title. I don’t know how helpful her literal advice is these days when many of us are already minimalist in the way we dress. My husband and I don’t wear jewelry or even a wedding ring; sunglasses, a mask, and our phone/wallet are the only accessories we typically carry. It’s not as if it would be easy to leave one thing behind without being either uncomfortable or indecent.

But when we extrapolate to other areas, like business, product design, and data analytics, this advice can be incredibly useful.

Product designer Mehek Kapoor writes about her experience failing an UX assignment when applying to a job at Microsoft:

Putting everything into any design or an app is always a bad idea. And that was when I learnt it! My FinTech app did everything, from mutual funds to smart deposits, to systematic investment plans to calls and puts, from share market evaluation to charts of various shares, from SEBI’s investment book to various articles and resource material on economics—you could find everything there. Every. Single. Damn. Thing. It was so complicated that while explaining it to the lead designer, even I got confused about what a particular feature did, and how another feature functioned! What could be the worst first-impression of a UX Designer than someone who can’t understand how her own app works?

According to Leidy Klotz,

Our mental preference for addition—for adding to what’s already there rather than thinking of taking away—is so wide-spread and strong that we would prefer to accommodate wrong ideas than simply remove them.

The first step to correct this problem is to recognize our bias toward addition and neglect of subtraction.

Then, every time we’re facing a challenge, we can get in the habit of metaphorically looking in the mirror and asking, “Before I leave, what can I take off?”

If you immediately thought of a fancy dashboard with sleek charts, you guessed right.

It was a big pet peeve of mine to see the word insight so blatantly misused. I was well aware of the fact that the dashboards in my SaaS products were data rich but insight poor.

On a rare occasion, one of our charts would identify a pattern for the first time, helping challenge or evolve the user’s current knowledge and beliefs. But 98% of the time, the dashboards delivered only information (data that had been processed, aggregated, organized, and displayed in a more human-friendly format).

Unsurprisingly, the adoption rate of the “Insights” module was invariably low.

The two classes of insight

Research psychologist Gary Klein has a definition for insight that’s much more useful than the ones we can find in dictionaries:

Insight: An unexpected shift in the way we understand things.
Gary Klein in Seeing What Others Don’t – The Remarkable Ways We Gain Insights

Expanding on his definition, I like to classify insights into two groups. A basic insight answers a question in a way that unexpectedly shifts how we understand something. An actionable insight, the most valuable kind, goes further, having the power to not only make us rethink something, but also push things in a new direction. Unfortunately, many dashboards are bad at the former, let alone the latter.

What a basic insight looks like

If I go to my Substack dashboard, I can see statistics like email open rate, number of subscribers, how many new visitors the newsletter had this week, etc.

I may have the intuition that the number of new visitors goes up on weeks when I publish a new article and down when I don’t. I just opened my dashboard and confirmed that understanding (i.e., no insight gained here).

I did glean an insight from visiting the dashboard though. Excluding email subscribers, I have the same amount of traffic arriving at my newsletter directly from opening datapoints.substack.com on a web browser as coming from LinkedIn. My first guess would be that the top source of traffic is LinkedIn, the only place where I alert non-subscribers when a new article is available here.

Substack statistics unexpectedly shifted my understanding of where traffic comes from. But there’s not much I can do with this insight, since I don’t know what causes visitors to come here directly through typing in the domain name or selecting a bookmark. This is an example of what I call a “basic insight”.

What an actionable insight looks like

Jerroyd Moore, web and mobile developer, wrote an article that serves to illustrate what the second category, “actionable insight”, looks like.

As happens with many businesses, Moore’s company was constantly struggling to make decisions when different groups expressed diverging opinions about the best course of action. Here’s one of various examples provided in his article:

Staff disagreed on the user experience of the website. Many believed customizing your software was too complicated for users to understand. They believed the users did not want to go through the customization process, and instead wanted to emphasize the ability to download all software packages (over six gigabytes). The discussion lasted for weeks and the arguments were being repeated in an echo chamber, without any data on the user’s behavior.

After implementing a monitoring tool based on Elastic Search (ES), Moore was able to end the endless discussion using evidence instead of opinions:

Collecting the data in ES, I demonstrated that 60% of users consistently customized their software, settling the discussion and keeping both options available to the user.

In this context, “60% of our users customize their software” is an actionable insight regardless of whether it’s presented via a fancy dashboard or simply stated verbally or through written words. Learning this fact allowed the team members who previously believed that users didn’t want to go through the customization process to rethink their opinion. Deploying the monitoring tool earlier and encouraging the team to look at the evidence could have avoided weeks of discussions and arguments about website design.

A real insight shakes our understanding of what is. The best kind is also actionable, allowing us to act on it or share it with an agent who can drive change.

Like I wrote in a previous article,

Contrary to what many vendors say, the best analytics tools are not the ones that present official-looking views into KPIs and let users ask questions and perform limitless data exploration. The best tools to drive action are the rare ones that deliver key insights through narration and visualization that answer a single question: What’s the story?

The next time a vendor tries to convince your company to purchase an analytics product they claim produces valuable insights, consider if it has the ability to provide unusual or expected findings that shed new light on a key subject area. If the answer is yes, does it prepare a pathway for action to occur, or is only capable of raising more questions than action?

Relevant, specific insights that encourage people to act can yield incredible returns for a business. The secret is to reset the meaning of the word “insight” so it’s no longer confused with more data, KPIs, or prettier interactive charts.

But the reality is that machine learning can be a high ROI solution for various problems in companies of all sizes. The investment required is getting lower and lower. For a machine learning solution to have the potential to move the business forward, the problem must be well defined and understood. The other condition is the existence of enough data that exemplifies the information necessary to make a decision--which doesn’t necessarily mean big data, or when it does, that it is generated internally.

Understanding customer behavior from small data

The famous Titanic competition offered by Kaggle is a good illustration of how machine learning can be successfully used on small data. In this competition, participants gain access to two datasets containing passenger information like name, age, gender, socio-economic class, etc. One of them has the details of a subset of the passengers on board the Titanic (891 to be exact) and, importantly, reveals whether they survived or not.

With data from 418 passengers, the other data set offers the same information but does not disclose the "ground truth". It's the participant's job to predict which of those passengers did survive.

It's true that this particular exercise doesn't translate into any actual business application. However, it helps showcase the effectiveness of predictions made by a machine model trained on fewer than 900 records. In the contest, the best scores were in the 83-84% range, which might be a perfectly reasonable accuracy for the intended purpose, as previously discussed here:

Data Points

When imperfect data is your friend

Despite how much emphasis is currently placed on algorithms and data analytics, we don’t see enough discussions about how much accuracy/precision is required of the data we use to inform our actions and decisions. In some cases, this is a non-issue. For example, with quantitative data like new customers, renewals, errors per 100 shipments, it’s typically…

Read more

6 years ago · 2 likes · Adriana Beal

We can apply the same approach to various business problems even in the most resource-constrained environments. For example, when an online boutique used a survey to learn about customer preferences, in minutes, using free resources, I was able to train a machine learning model to predict the answers for the customers who left a question unanswered using the data of customers who did respond.

Combining small data with models pre-trained in big data to support better decisions

A common application is fraud detection.

Armed with petabytes of data collected every hour from customer transactions, big e-commerce players like Target and Walmart can leverage sophisticated deep learning systems to minimize the incidence of fraudulent purchases.

While fraud has a more significant impact on mid/large businesses, the volume and sophistication of fraud attacks keep rising for small businesses as well. A small online store is unlikely to be in a position to acquire the same volume of proprietary data and perform the large-scale analyses made by large retailers. However, it doesn’t mean it can’t leverage a model pre-trained on an extensive third-party data set to achieve similar results.

Take, for instance, the AWS fraud detection system. A small e-commerce website concerned with fraud can use the system for several things, including spotting potential fraudsters among new customers to minimize fraud loss. By sending as little as two pieces of data from a guest checkout order (e.g., email, IP address), the business can get back a risk score and use it to automatically accept a transaction, place it under review, or collect more customer details.

Another great example can be found in precision agriculture.

Small farms are using agricultural technology that leverages ML to deliver recommendations that increase yield and reduce the emissions released into the atmosphere that add to global warming. For instance, growers now can avoid hit-and-miss results and save time and money using a service that takes local soil samples and feed their data into a pre-trained model to answer questions like, "Which seeds should I plant?" and "Where do I need to spread fertilizer?"

Even in the smallest of organizations, the quality of decisions can mean the difference between success and failure.

Do I give this customer a special price? Do we approve this transaction, or request more information to prevent fraud? What is the right quantity of herbicide to use during this application window to avoid crop damage?

By themselves, these individual decisions may have little impact on business performance. Taken together, they influence everything from profitability to reputation. For that reason, smart businesses of all sizes are leveraging machine learning to gain a competitive edge by achieving a decision yield that is better than the industry norm.

Not every business problem will benefit from machine learning or justify the investment. However, free tools and affordable third-party solutions make it increasingly easy for any organization to adopt ML when it can improve decision-making. Businesses will either embrace this fact or be replaced by others who do.

Consider a software gaming company looking to optimize its advertising spending. The head of marketing has an intuition: in-theater advertising will boost sales. The marketing analytics team proceeds to do an optimization analysis that suggests that shifting some ad spending from online-video (such as YouTube) to in-theater advertising will produce the desired outcome.

Right before the holidays, the company executes the new plan. Sales of new version of its flagship game increase in 23% compared to previous versions. The leadership team celebrates the victory and decides to continue with the new budget allocation that favors in-theater over online-video advertising. At the end of the next quarter, both the ROI on advertising and the company market share have significantly decreased.

What went wrong?

Once the boss came up with the idea to invest in in-theater advertising, the entire team started looking for data to confirm that hypothesis, ignoring any information that rejected it.

Instead of doing a local experiment and using the results to refine its calculations, the company forged ahead with the new strategy. Then, after the first alleged win, the team didn’t look beyond the increase in sales to weed out misleading or false explanations. If it did, it would’ve found that a) the main competitor had a 40% increase in sales during the same period, suggesting that the positive result was the consequence of “a rising tide lifts all boats” rather than the use of in-theater advertising; b) the effect on sales of online-video advertising (now significantly reduced due to budget reallocation) was much greater than believed. 

Three strategies to avoid confirmation bias

1) Focus on learning

It’s fine to start from an idea that is initially based purely on somebody’s hunch. The danger resides in cherry-picking data that reinforces that initial belief.

When working with big data sets, it’s easy to find statistically significant relationship between random variables. As the phrase famous in the statistical literature goes,

If you torture the data long enough, it will confess to anything.

This is why, after an intuition for the “next big thing”, the initial steps need to involve generating hypotheses and conducting experiments to validate them. The cycle of sensing, analyzing, and discovery may need to be repeated many times, ideally in cycles of short duration.

2) Look for disconfirming evidence

When we come up with what looks like a great idea, looking for supporting evidence is par for the course. Equally important, but often neglected, is the process of looking for evidence that proves us wrong.

Hypothesis maps are useful to capture both types of evidence and how they relate to each of our hypotheses, helping us reach appropriate conclusions.

3) Use premortem to learn from the future

Premortem exercises are a powerful tool to challenge assumptions and seek disconfirming evidence. In a premortem, instead of waiting until the end of a project to find out what went wrong and learn from the future, we go on an “imaginary time travel” to avert real failures.

To do a premortem, block some time at the beginning of a project to imagine a time in the future after it has been completed. Assume that the effort had the worst possible outcome. Write a story about why the failure occurred. Do the same with the opposite result: pretend the project was a roaring success, and write the story that explains why.

Evidence suggests that looking at what could go wrong using the premortem technique to think in terms if “did this, not that”, rather than “do this, not that” give us a different, superior vantage point.

Awareness of the problem is the first step to overcome it

Confirmation bias permeates our personal and professional lives, and can be extremely difficult to overcome. No one likes to admit they’re wrong, so our first impulse will always be to try to find evidence that justifies our beliefs.

The best decision-makers, however, are experts at avoiding confirmation bias. They focus on learning and acquiring new data. They look at both confirming and disconfirming evidence, balancing the persuasiveness of one against the other. They use premortem exercises to learn from the future.

By using the same three strategies for your own ideas, and ensuring that anyone defending a solution use them, you’ll dramatically reduce the risk of making a bad decision as a result of biased analyses.

Humans have a bias for simplicity, and for good reasons. Our brains weren’t designed to process large volumes of information in a short time span, and will do anything to avoid engaging higher-order brain functions when it has to reach a judgment and formulate a response.

Simplicity is a worthy goal to have. My previous post, Lowering uncertainty with limited data, include examples of how simple rules of thumb can support high-quality decision-making, in some cases even exceeding the results of much more complex models.

However, as the aphorism often attributed to Einstein goes,

Everything should be made as simple as possible, but not simpler.

So, how can we tell when simplification has gone too far?

Below are two signs that oversimplification is happening and needs to stop.

1) Optimizing for a single measure

Be it revenue per visitor, monthly active users, Net Promoter Score (The One Number You Need to Grow), many companies and teams tend to fixate on a single measure of success they hope will help them achieve a desired outcome (close a new round of funding, convince a prospect to sign up, etc.).

It would be great to find one performance measure capable of indicating when everything is not in fine shape and tell us what to do about it. Unfortunately, it can’t be done.

A classical example is measuring (and rewarding) software developers by number of bugs fixed. If a consequential bug exists, it’s of great value to have it fixed, but what happens if that’s the only dimension of performance being measured?  We can quickly devolve into The folly of rewarding A, while hoping for B. Companies using this approach quickly found out that developers were deliberately shipping buggy code so they could subsequently fix it and get their reward.  

2) Relying on averages for important decisions

Averages are inherently reductive and often misleading because they ignore the impact of the inevitable variations. 

For a long time, scientists believed that the truth of something could be determined by collecting and averaging a massive amount of data. However, with more research, it became clear that in domains with too much variability systems designed around averages do not work well. As the joke that first appeared in the 1950s goes, a statistician who put his head in an oven and his feet in a freezer may conclude, “On average, I feel fine.”

Around the same time, the US Air Force learned the lesson: If you design a cockpit to fit the average pilot, you’ve actually designed it to fit no one.

Decisions made based on average conditions usually go wrong:

• Segmenting customers based on their average spend and frequency of purchase may be useful in some scenarios, but create a costly oversimplification in others. For example, among your regular big spenders, a group may be willing to buy a new product without any discounts, while another will only buy if offered a discount. Treating both cohorts as part of the same homogeneous group may cause the business to lose significant money.

• Using a scoring system to assess the technical skills of job seekers may be useful to quickly filter out unqualified candidates. However, if the choice of five candidates to be brought to an interview is based only on a composite score, it’s possible that a candidate who scored exceptionally high in one skill but displayed significant weakness in another is kept in the pool while an attractive candidate with solid skills across all important domains is eliminated from the process.

What to do to avoid these negative effects of oversimplification?

• Don’t fixate on a single measure of performance to avoid developing a distorted perspective of reality and creating incentives for gaming the system.

• Don’t focus on measuring just what’s easily measurable in detriment of what’s important to know. A lot of what’s considered intangible and unmeasurable (like the performance of business analysts) can be measured with a useful degree of accuracy and repeatability to improve decision-making.

• Don’t rely on a composite score (e.g., overall code quality) without also reporting on its component parts. If everyone isn’t constantly reminded of what makes up the composite measure (e.g., defect rate, time-to-market, reusability), it soon becomes a meaningless abstraction.

• Don’t trust that averages will be a good reference standard. When the US Air Force discovered that measuring average pilot body dimensions wasn’t helping ensure better-fitting cockpits, it started demanding from airplane manufactures that all cockpits needed to fit pilots whose measurements fell within the 5% to 95% range on each dimension. As Sam Savage said in the 2002 HBR article The Flaw of Averages, rather than “Give me a number for my report,” what every executive should be saying is “Give me a distribution for my simulation.”