Connect the Dots or Be Replaced

Much has been made of ChatGPT’s ability to pass multiple graduate-level assessments — including MBA-level exams and the legal profession’s bar exam—but is the new AI-powered chatbot so impressive it will soon put swathes of human white-collar managers out of work?

It’s a question Professor Oded Netzer determined to find out when he opened up ChatGPT recently and plugged in some straight - forward facts about a company across two years of business: Sales had fallen from $10 million in 2019 to $8 million in 2020; call center complaints had increased from 100,000 to 150,000 during that time; and so on. “Explain what is going on with this company,” Netzer’s prompt instructed.

ChatGPT’s answer began, “Based on the information provided, it appears that the company experienced a decrease in sales between 2019 and 2020,” and continued in that vein, summa - rizing the information it had been given.

The response wouldn’t have earned a pass from Netzer. Why? Because the chatbot had merely summarized the prompt rather than offering any higher level insights. Netzer expects more from his CBS students because more will be expected of them as business leaders. He predicts the AI tool won’t be replacing business decision-makers anytime soon—as long as those decision-makers understand that the value they bring to their roles is changing.

“At the heart of management and leadership roles is the ability to see what everybody else is seeing and think what nobody else is thinking— to look at different pieces of information, apply critical thinking, and pour on some judgment to arrive at a synthesis and eventually a decision,” Netzer explains. “Synthesizing information means going beyond the “what” in the data to “So what?”—what does it mean—and “Now what?”—what should we do about it?”

Not long before the launch of ChatGPT, Netzer co-authored a book, Decisions Over Decimals: Striking the Balance Between Intuition and Information, with CBS adjunct professors Christopher J. Frank and Paul F. Magnone. The book zeroes in on the importance of business leaders focusing on synthesis rather than merely summarizing information, and explains where big data can fit in. When ChatGPT burst on the scene, the book’s themes took on a new urgency.

Netzer has devoted much of his academic career to exploring how unstructured data (that is, data not formatted for structured database storage, whether it’s audio, video, text, internet of things sensor data, and so on) can be used to support sound decision-making. In the following Q&A, he applies his expertise to some of the most pressing concerns for business leaders raised by ChatGPT: Is AI poised to encroach on the jobs of human decision-makers? What technological developments could lead us there? And how can MBA programs ensure their graduates maintain their relevance?

CBS: In your book, Decisions over Decimals, you offer a framework to help decision-makers use data without becoming overreliant on it. Can these ideas also apply to decision-makers wondering about how to use ChatGPT and other fast-improving AI tools?

Oded Netzer: The book is very much related to the way I’m thinking about ChatGPT. Part of what we are arguing in the book is that in decision-making today, the value comes from combining data (or analytics) with intuition, something we call Quantitative Intuition, or in short, QI.

When I say intuition, I truly mean human judgment. Human judgment could be in how I search for data, or it could be in how I evaluate the data. For example, if I find surprises in the data, that is, by definition, combining judgment and information because surprise is what happens when data clashes with an expectation.

Now, I think all of this very much relates to the discussion that came a few months after the book was published, when ChatGPT came out.

People are asking, “Is this going to replace human decision-makers? Are machines going to take away not only the analytics or quantitative part of management but also the intuition or judgment parts?”

I looked into this. In a few tests, I’ve tried to push ChatGPT to go beyond the data and the “What?” to the “So what?” It has done quite well in the “what” aspect, mainly summarizing information. It’s a very good tool for summarizing large amounts of information in a comprehensive manner (with a few pitfalls there, too).

However, what ChatGPT rarely does well is synthesize information, by which I mean apply judgment to say, “This is what the data implies we should do.” While it is possible that future generations of generative AI tools will learn to do that as well, at least for now, humans are still much better than machines at doing that.

CBS: What does the recent history of hype around big data reveal about the current excitement about ChatGPT?

Netzer: When big data came about, many people said, “Forget about human judgment. We are going to have data that makes decisions for us.” Then as a decision-maker, you realize, no, data doesn’t make decisions; that is still our domain. We can’t rely solely on data. We need some combination of data and judgment to use it well, and we may have shifted the pendulum too far in this whole big data revolution, believing that data and analytics will make decisions for us.

I think ChatGPT further highlights the point that what is left for humans is that we need to focus on the judgment aspect of data-driven decision-making. But at the end of the day, this is often the fun part of our jobs. We don’t want to get bogged down with creating tables; we want to spend our time interpreting them.

This could be true for people in many industries—take journalism as an example. Yesterday, I was talking to a journalist from a major media outlet about ChatGPT. At one point he said, “Wait, is this going to be writing articles for me soon?”

I believe the answer to that is no, but I do believe tools like ChatGPT can make journalists better. In other words, perhaps what that journalist should fear is not being replaced by AI but being replaced by a journalist who knows how to take advantage of AI.

Why not ask ChatGPT to create a first draft of a summary of the information and then apply your judgment—your writing acumen—to it? If, as a journalist, you feel very comfortable going to Google and searching for information, why wouldn’t you feel comfortable with Google creating a coherent summary of what’s in Google? Of course, what is currently missing from ChatGPT for that purpose is appropriate and validated sources for fact checking, but this is likely to be resolved in the near future.

CBS: Is the impressiveness of ChatGPT a sign that we’re getting closer to machines being able to make decisions, though? We’re not there yet, but is it likely that AI will begin to close in on that all-important differentiator—human judgment?

Netzer: Machines are starting to get better at doing things a little more complex than, “If X passes a certain threshold, then do Y.” An example might be helping us with the next word that we want to write in an email. One day, it may be able to get to a human-level judgment in certain domains.

What tools like ChatGPT will need to continue to improve are a lot of observations. Another way to put it: They’ll need much more data. When it comes to machine learning, most major technological innovations didn’t come from a fancier machine learning model; they came from much more data being available to train the model.

Now, we are very good, as humans, at putting information in the right context, whereas machines are not that good at it yet. When I talk to you, I have a lot of context, even though we don’t know each other. If I use the word model, you know that from a nerd like me, it’s got to be a mathematical model and not someone who walks on a runway.

Technology-wise, one of the reasons why ChatGPT is one of these true innovation breakthroughs is context. What the newest version of ChatGPT specifically did was create a context of about 4,000 to 8,000 tokens around every word—think of a token as a word and the context as the adjacent thousands of words used to understand the meaning of each word. That is way more than what previous language models had. The previous GPT version had a context window of only 2,000 tokens.

Part of why humans are good at judgment is because of this concept of context: We are able to connect information to all kinds of other things we observe. Now, to be honest, we don’t fully understand our own brains. Otherwise, we would’ve trained machines to be as good as humans, but we don’t know how we learn context. Before a child can even speak, you can show them an illustration of a cat in a book and tell them that it meows, and then show them a drawing of a dog and tell them that it barks. And after you show them three or four of these images, they’re going to meow or bark when they see a real cat or dog in the street. Machines need to train on millions of observations to show a similar level of recognition.

CBS: How will the maturation of AI tools like ChatGPT impact MBA classrooms—and job prospects for MBA graduates?

Netzer: As MBAs, where does your value come from? You are not getting paid to tell me what I can see in the data myself. You are getting paid to tell me, “So what? What does it mean? And now what? What should we do about it?” Generative AI tools like ChatGPT are becoming better and better at summarizing the “what” at a super-human level. Answering the “so what” and “now what” questions requires judgment, and it requires taking a risk—bringing in information from somewhere else, not from what you just showed me here.

For those of us who teach, this emphasizes the need to go beyond “just tell me what’s in the case” and “here’s how to make magic with Excel.” We must push students to interpret things, to make recommendations.

MBAs are in demand because they act as translators between the data team and the C-suite. They’re able to translate what has been done within data analytics into the business problem because they have the context of the business, which often, data analysts don’t have. And these skills will not easily be replaced by something like ChatGPT any time soon.

If anything, we are in a good situation in business schools because of the role of judgment in our education. The first question in every case study is, “What is the business problem that they’re trying to solve?” There is now higher value than ever on pouring the judgment into information, and not just learning information and technical skills. This has been, and will continue to be, our focus in training future leaders.