From the Tail: Big Fat Regret

With the recent indictment and resignation  of I. Lewis "Scooter" Libby over the Valerie Plame affair there have been repeated calls from the obvious quarters  for the President to apologize over the CIA leak. Whether Mr. Bush feels any regret over the incident or not, it seems unlikely that he will express any, at least until he absolutely has to.

Regret (the  feeling of disappointment or distress about something that one wishes could be different) is a strange beast. It is, of course quite possible to feel regret without guilt, or even without any acknowledgement of personal responsibility. Also, regret is sometimes the inevitable biproduct of actually making a choice (I'll take the granola bar over the chocolate cake). Regret is not a rare commodity.  During any given day there are dozens of moments when one expresses some inconsequential level of regret to oneself. So what exactly do people mean when they say (often with a flourish) that they "have no regrets"? Perhaps what they mean to say is that they don't really regret anything enough, i.e. it all comes down to the degree of regret experienced.  The question about regret that I find interesting is: How is the degree of regret distributed? Is it a Bell Curve? That would be nice. A Bell Curve is conveniently symmetric around its mean (average), that is to say, its median value is the same same as its average. Let's do the following thought experiment to find out!

1. Try to remember the things that you have regretted over the past day and count those which rise to the level worthy of reflection in bed tonight. Chances are you will only be able to come up with a couple.
2. Go back a week and do the same thing. What regretful things have stuck in your craw? Strangely, there are likely to still be a similar number, maybe two or three. How did that happen? Surely, it should be more like seven times your daily regret count. Your threshold for craw stickiness has gone up.
3. Now do the retrospective on your life and try to catalog your regrets. Again, the number is about the same.

To be more quantitative, if your could go back in time and correctly catalog the list of your regretful incidents over a day, week and month, and graph the degree of regret versus the sequence of incidents, you would probably come up with something like this for the daily , weekly and monthly lists. (The "Real Regret" Zone is is the threshold of regret degree above which you would count it while doing the three step experiment I outlined above.)

The week graph is a blown up version of the month one, and the day graph is a blown up version of the week one, but they all have the same structure. Each graph has  many tiny incidents with a few much bigger ones, and only those incidents above the threshold register as being truly regretful.  A strange "self similarity" or scale invariance is observed.

Now if your regret were distributed like a bell curve it would not look like this. The big events would not be so big, and the scale invariance would not be present. So what is this kind of distribution?

This kind of distribution is Fat Tailed. The reason is that the infrequent events are huge. Most of us are trained to think in terms of Bell Curves, but this is a very different animal. A Bell Curve has the same median and mean. In the Fat Tailed distributions the median is extremely small relative to the mean  since the rare events are so huge.  This skew in the distribution leads to various odd properties, for example, that the variance is infinite!

It turns out that a very large number of things are fat tailed:

1. The frequency of words in a book. This is what captured the imagination of a Harvard Lingusitics lecturer by the name of George Zipf who discovered that if the most frequently mentioned word in a book was used N times, the Kth most frequent word in that book would be used about N/K times. This relationship holds (with some minor fudge factors) for most books written in the English Language and is known as the Zipfian distribution.
2. The distribution of population in a city: Remarkably this turns out to be Zipfian as well. Take any developed or developing country and rank the cities. A terrific source of data for a bunch of different countries is here. You will find that the distribution is very close to being Zipfian. The number of people who have scratched their heads about why this happens reads like a who's who of economics: Herb Simon, Paul Krugman, Benoit Mandelbrot (who isn't regarded by economists as an economist but actually is), and most recently Xavier Gabaix among lots of others. Even those who don't think the distribution is Zipfian agree that it is fat tailed.
3. The number of web links pointing to a web page. Various popular books have been written on this and since it has been covered extensively elsewhere I'll resist the temptation to expound. 
4. The number of subscribers to a blog feed. Most blogs are barely read, but a few blogs get a huge amount of subscribers. An interesting set of graphs relating to this has been provided from the folks at Ask Jeeves.
5. The distribution of income. Ever since Pareto, it has been observed that the rich are few and much richer than the rest. The scale invariance of the distribution results in very rich people actually feeling quite poor! If you think that you'd feel rich with $10 million in the bank, think again!

Fat tailed distributions are more frequently called Long Tailed. (The term "The Long Tail" yields 1.57 million google results as opposed to a paltry 12,700 for the "The Fat Tail".) In statistics, the rare events of a distribution are said to be in the tail. When these rare events are large, we say that the tail is FAT. Even a bell curve, can have a very long tail, so why the terminology Long Tail? Well, if  you arrange the values in descending order and graph them, i.e. the big ones on the left, and the smallest values to the extreme right, the picture looks like a long tailed beast. Of course, the head of this beast is the tail of the distribution! Also, it seems more natural to say, for example, that the distribution of blog readership has a big fat head and a long tail, but that isn't really accurate from a statistical point of view. So take your pick of terminology.

Now the question of WHY things are fat/long tailed is still somewhat unclear, but of course,  a multitude of theories abound. Zipf believed it all stems from the tendency of human beings to follow the path of least resistance, so that the inertia in the system tends to make the big bigger. There are various "winner takes all" theories which are commonly spouted in business circles. Economists espouse phenomena of "increasing returns" and "switching costs", which are appealing in certain contexts. It is all very fascinating and intellectually rich.

But what explains the degree of regret --- why is IT fat tailed? I wish I had an explanation, but all I have is more speculation: perhaps circumstances are responsible, i.e. things happen in a "fat tailed" manner so that we react to them that way. Or perhaps it is our reactions which are more responsible, i.e. after an accumulation of little things reaches some limit  (a camel-back breaking limit, so to speak) we react in an extreme fashion.

Finally, it would seem that other emotions, e.g. happiness work the same way. Repeat the experiment and see for yourself.

The mystery of it all! If only  I knew more. And yes, regret has struck again!