The Science of Mistakes

The following sections are included:

• Dedication
• About the Author
• Acknowledgments
• Contents

This introductory lecture has four goals:

• (1) To lay out the operational challenge in identifying mistakes;
• (2) To present the “ideal data” method of responding to it;
• (3) To explain the interwoven themes of the course:
  • (a) the science of mistakes,
  • (b) the economics of attention,
  • (c) cognitive economics,
  • (d) economic data engineering;
• (4) To pique your interest in the lectures to come and in the many exciting research paths forward.

The goals of this lecture are:

• 1. To formally introduce the Blackwell model of experimentation, expected utility maximization, and the space of strategies as suited to purpose.
• 2. To formalize the Block and Marschak identification challenge and to understand why it calls for data engineering with new basic observations.
• 3. To introduce state-dependent stochastic choice data (SDSC), the basic observations that form the analytic core of the book, as well as revealed posteriors, experiments, and strategies.
• 4. To precisely pose the question of what it means for SDSC to be rationalized by the Blackwell model and to illustrate data sets that can and cannot be so rationalized.
• 5. To introduce and illustrate the no improving action switch (NIAS) characterization of such rationalizable data sets.
• 6. To prove the resulting characterization theorem.

Delivering all of this material in a single lecture is excessive. So, the real goal of this lecture is to permit students to review the material ex post and accomplish the above goals by themselves. This comment applies to all of the lectures that follow.

The goals of this lecture are:

• 1. To formally define costs of learning and rational inattention theory.
• 2. To precisely specify what it means for SDSC from various decision problems to be consistent with a particular cost function and hence to have a costly information representation (CIR).
• 3. To illustrate cases in which such a CIR exists and cases in which no non-trivial CIR exists and to make intuitive progress in understanding the underlying logic.
• 4. To introduce the counterfactual attention switches that play a key role in the examples and to construct matrices of the resulting direct and indirect value switches.

In this lecture, I follow Caplin and Dean (2015) and Caplin et al. (2023c) not only to characterize CIRs but also to precisely show how to recover all rationalizing cost functions for any such utility function. All results are organized around matrices introduced at the end of the previous lecture. Before plunging into the general results, I open by restating the key objects and providing a number of illustrative examples that point the path forward…

The following sections are included:

• Introduction
• Three Bayesian Model Classes
• Example 5.1
• The CCR Utility Cone
• The CIR Utility Cone
• Utility Cones in Example 5.1
• Cost of Revealed Learning in Example 5.1
• The Variational Lower Bound and Posterior Separability
• Blackwell’s Comparison of Experiments
• Generating the Data
• Rationalizability: Mean- and Optimality-Preserving Spreads
• Possible Learning in Example 5.1
• Full Characterizations
• Fixed Information Representations
• FIR in the Example

In terms of how to do applied research, the central idea in these lectures is that rich data about decisions in a fixed learning environment with very different incentives to learn allows rich inference about utility, learning, and the costs of learning. In this lecture, I take this point to its limit and consider a researcher who is able to vary incentives widely and otherwise to design the decision-making environment. In addition to laying out some particularly insightful variations in the learning environment and decision problem, I present an experiment designed around the main recovery result. The attentive reader will note that the results in this section are illustrative rather than fully definitive. There are many ways to design experimental choice environments that generate insightful SDSC. Many of these have, in fact, been pioneered in psychometrics, which routinely generates such data. I focus on the few cases that have been formally studied in detail. The result is partial rather than complete recovery. It is clear that one can drive far further forward on this road, and I close the lecture by pointing out a few valid directions forward…

As with the material on recovering rationalizing learning in Lecture 5, this lecture takes as its starting point the wonderful approach and results of Blackwell (1953) on ranking experiments by their information content. Given that it was not central to Lecture 5, I did not there note the third equivalent definition of one experiment being more informative than another, which is that it offers at least as high maximum utility function regardless of the particulars of the utility function. While Blackwell’s characterization is definitive, it reveals that many experiments cannot be unequivocally ranked. This has given rise to a sub-literature whose goal is to find methods of comparing experiments that are less demanding than Blackwell’s…

In this lecture, I introduce and analyze behavioral patterns associated with a large and important family of attention cost functions. Costs based on Shannon entropy introduced into economics by Sims opened the door. In this lecture, I introduce this cost function and indicate some of the many reasons why generalizations are being explored. In a nutshell, the model has properties that are often contradicted in data. With that in mind, I introduce some seemingly natural generalizations that share a key property of the Shannon model, which is posterior separability: Costs are additive in nature in a particular sense. These are now well studied due to their allowing for most observed empirical failings of the Shannon model…

The following sections are included:

• Two New Domains of Choice
• The SDSC-Based Solution
• The Posterior-Based Solution
• Safe vs. Risky: The SDSC-Based Conditions
• Safe vs. Risky: The Posterior-Based Approach
• Choosing Among Options with IID Priors
• Rationally Inattentive Nash Equilibrium

Caplin et al. (2019) focus on the fact that many options are unchosen. In this section, I analyze this in a few special cases and introduce the ILR hyperplanes, a general-purpose analytic structure for mapping from prior beliefs in a given decision problem to the structure of the consideration set and optimal unconditional choice probabilities. There is much more to be done with these hyperplanes, both computationally and in terms of economic analysis. A few openings and pointers are provided. I open by providing a very standard model of type I and type II errors that illustrates the value of understanding the conditions under which inattentive choice is optimal. I then present two examples from CDL, in both of which the key issue is to identify the optimal consideration set. The ILR hyperplanes are then introduced and discussed. The lecture closes by pointing toward dynamics, in particular to the work of Miao and Xing (2020), which is posterior based and applies to the broad class of uniformly posterior-separable cost functions.

This section opens with two simple models of how rational inattention plays out in market settings. The first makes one simple point: In models with free entry, equilibrium conditions can be used to pin down beliefs about market composition. For that purpose, it is particularly useful to use the posterior-based formulation of optimal strategies, which show many features of the solution to survive variation in prior beliefs, aiding in equilibrium analysis. The second example is dynamic and makes a distinct point: Past market outcomes reveal information that causes priors to be updated. Social learning from market outcomes may, in some cases, be less onerous than private learning. This points to the direction of dynamics. Again, knowing how to solve the model for all priors is of particular value…

This lecture applies the methods of Parts 1 and 2 to model, better understand, and better apply machine learning methods in decision-making. All of the work is jointly conducted with Daniel Martin and Philip Marx and much is taken directly from Caplin et al. (2022b). Algorithms are the most important “decision makers” in the modern world today and will become ever increasingly so. They predict if a driving route has a low expected travel time, an eye scan shows physical damage, a manufactured product has a defect, a house for sale is a likely match, internet activity is a security threat, an email is spam, and so on. Virtually all industries, jobs, and consumer experiences have been impacted in some way by the rapid rise in automation brought about by this technology. Economically important applications of machine learning include what ads to serve, what content to show, what coupons to provide, facial recognition, translation, voice assist, credit scoring and loan decisions, medical decisions, product recommendations, driving routes, spam filters, fraud detection, and so on…

In this lecture, I outline applications of the ideas and methods of the book to teaching and testing protocols for humans rather than for machines. I pick up where the last lecture left off, with the use of proper scoring rules to better understand and to impact learning. Both de Finetti (1965) and Savage (1971) proposed the use of just such scoring rules in multiple-choice tests. To date, this approach has found traction only within decision analysis, where it appears to have proven its worth (Bickel, 2010). I outline some of the powerful arguments in favor of broader experimentation and implementation…

We don’t stop making mistakes when we leave school. We just stop measuring them. The few cases alluded to above, such as sporting calls, are exceptional in this respect because ideal SDSC is available. While there is value in finding other settings in which this is true, at least to a first approximation, there is also value in developing entirely different research strategies in bringing the science of mistakes to the field. My focus in this lecture is on development and deployment in the field of cognitive instruments. Many mistakes can be seen as reflecting cognitive skills that are required, for example, to make a correct sporting call or identify the appropriate medical procedure and implement that procedure effectively. This suggests a psychometric approach of developing cognitive skill measures that predict particular mistakes and implementing them in field settings. If one identifies differences in skill using cognitive instruments and finds that these differences are strongly correlated with clearly related bad outcomes in the field, it strengthens the plausibility of a causal channel running from cognitive skill to behavioral mistakes…

Decision-making skills matter in all phases of life. While being a manager of a fixed group of workers is a particularly salient example of this, there is no realm of behavior in which such skills are irrelevant. In this lecture, I focus on a particularly important set of decisions that are known to have massive impact on lifetime income: decisions on whether and how to search for jobs, including when to quit, what to do in the face of an impending layoff, when to take time out of the labor force and retool, and when to retire. These transitions are important events that appear to have a large effect on income dynamics over the life cycle. When, why, and how such transitions are made is much studied in administrative data. However, the facts alone are not rich enough to determine whether or not there were significant gaps in knowledge and/or illusions that rationalized erroneous decisions. Decision quality is as much about what was not known as what was and about actions that were not taken as much as those that were. In this lecture, I outline research in which measurements are enriched to distinguish between those who make successful labor market transitions and those who do not, thereby to gain insight into the role that decision-making skills play in lifetime income…

Traditional policy tools operate through prices (e.g., taxes) or quantities (e.g., quotas). The theories guiding policy design are based on an essentially error-free understanding of these instruments. Yet, in a world of attentional constraints, thinking in these traditional terms is far too narrow. In fact, the rational inattention revolution was launched, in part, to capture the cognitive constraints relevant to monetary policy design and the perfect information fiction, suggesting that prices and wages would instantaneously adjust to increases in policy…

The following sections are included:

• Epilogue
• Bibliography
• Index