Survivor Bias

Moved to Blogger

2010-03-16T23:04:00.000-07:00

Apparently the site has been giving an error as I had not paid wordpress.com the domain mapping fee - so I moved it to blogger ;-) Hopefully I will get around to posting something one of these days...

Michael Lewis on the myth of prediction in the financial crisis

2009-06-28T20:22:00.000-07:00

[youtube=http://www.youtube.com/watch?v=E_XWuIeuN1c]

Another nail in the Black-Scholes coffin

2008-02-21T22:56:00.000-08:00

Michael Lewis has an article about how the markets have begun to realize that the Black-Scholes model for risk doesn't really work:

"No one believes the original assumptions anymore," says John Seo, who co-manages Fermat Capital, a $2 billion-plus hedge fund that invests in catastrophe bonds—essentially bonds with put options that are triggered by such natural catastrophes as hurricanes and earthquakes. "It's hard to believe that anyone—yes, including me—ever believed it. It's like trying to replicate a fire-insurance policy by dynamically increasing or decreasing your coverage as fire conditions wax and wane. One day, bam, your house is on fire, and you call for more coverage?"

Does this mean they have to give back the Nobel prize? I would have thought Black-Scholes was pretty universally questioned after the whole LTCM fiasco...

The subprime gray swan

2008-04-07T03:14:00.000-07:00

Nassim Taleb has been getting a lot of press in the wake of the subprime mortgage fiasco - the general theme is that Taleb ideas are harkening in a new era of financial rationality in the markets. But it's more likely this is the fear part of the usual greed-fear rollarcoaster. From Fortune:

Most people seem to have been caught off-guard by the subprime crisis, yet such an event was not only predictable but also inevitable. It was a Black Swan, yes?

The Black Swan is a matter of perspective. A turkey is fed for 1,000 days - every day lulling it more and more into the feeling that the human feeders are acting in its best interest. Except that on the 1,001st day, the butcher shows up and there is a surprise. The surprise is for the turkey, not the butcher. Anyone who knows anything about the history of banking (or remembers the 1982 Latin American debt crisis or the 1990s savings and loan collapse) will tell you that the subprime crisis was so bound to happen. Banks are exposed to such blowups. Bankers have been the turkey, historically.

So I call these crises "gray swans." I've been telling anyone willing to listen that banks have a tendency to sit on time bombs while convincing themselves that they are conservative and nonvolatile.

Good to not so great?

2008-07-29T04:13:00.000-07:00

Over at Freakonomics, Steven Levitt looks into the book Good to Great - it analyses 11 companies that transformed themselves and became "great" companies. Turns out the companies aren't all doing that well:

Ironically, I began reading the book on the very same day that one of the eleven “good to great” companies, Fannie Mae, made the headlines of the business pages. It looks like Fannie Mae is going to need to be bailed out by the federal government. If you had bought Fannie Mae stock around the time Good to Great was published, you would have lost over 80 percent of your initial investment.

Another one of the “good to great” companies is Circuit City. You would have lost your shirt investing in Circuit City as well, which is also down 80 percent or more. Best Buy has cleaned Circuit City’s clock for the last seven or eight years.

Hmmm. While it's true that you can learn from the mistakes and successes of those that came before us, what the salient lesson is (presuming there is one), the role of causality and luck or how to generalize to new situations make it a fool's errand. Good to Great and other successmanship manuals pick and choose winners to fit the evidence and ignore the losers.

The Apple Product Prediction Scam

2008-12-18T17:51:00.000-08:00

One of my least favorite parts of the tech business is the army of predictors of future Apple products. It's a great game because anyone can play from professional analyst firm, to financial news reporters or random bloggers and since there are so many people playing, the game is rife with survivor bias. So how do you play? Here's some tips for making "predictions" about future products:

The vaguer the better and don't forget to pepper your prognostications with weasel language like "sometime in the first half of 2009" or broad categories of products like "netbooks".

Don't have a source for this info? Make one up. Or don't - just hypothesize about how it "makes sense based on the market".

Use other wild speculation as a primary source - it's the wisdom of crowds!

Once Apple makes (or doesn't make) an announcement, determine if you won. With so many people playing, just by random chance someone will "predict" what's going on which will only make your future predictions that much more "valuable". But what if you weren't right?

It depends on what the meaning of is is: If you did things right you can fudge a vague prediction into the win column by talking about generalities or redefining your terms after the fact. Try squinting.

Blame the victim: They should have produced this product so they were wrong or it wasn't ready in time. This is a science man!

Better luck next time: Hey, nobody is right all the time but if you play for long enough, you can appear to be!

This really works for any kind of prediction and with the end of the year fast approaching, it's time to make those 2009 predictions and weasel around those 2008 ones!

For more on this topic, check out Nassim Taleb on the Scandal of Prediction.

UPDATE: Brad Feld seems to feel similarly:

This has been one of my pet peeves for 20+ years. For a while I managed to ignore them completely. At some point I started getting asked for my predictions and succumbed to my ego for a few years and participated in the prediction folly. At some point I realized that there was zero correlation between my predictions and reality and that by participating, I was merely helping perpetuate this silliness.

Scott Adams on survivor bias

2008-12-23T16:16:00.000-08:00

Still, there are plenty of civilian investors who have done well buying value stocks and holding for the long run. But wouldn't you expect a wide distribution of luck in any gambling arena? If every investor picked stocks entirely randomly, you would still produce a good number of Warren Buffetts entirely by chance. And our brains are wired to assume those winners had the secret formula for investing.

Mr Adams modest proposal for overhauling stock investing is here.

Misunderstanding Risk

2009-01-05T00:29:00.000-08:00

The New York Times published an article on the role of VaR (Value at Risk) financial models in the current fiscal crisis:

VaR isn’t one model but rather a group of related models that share a mathematical framework. In its most common form, it measures the boundaries of risk in a portfolio over short durations, assuming a “normal” market. For instance, if you have $50 million of weekly VaR, that means that over the course of the next week, there is a 99 percent chance that your portfolio won’t lose more than $50 million. That portfolio could consist of equities, bonds, derivatives or all of the above; one reason VaR became so popular is that it is the only commonly used risk measure that can be applied to just about any asset class. And it takes into account a head-spinning variety of variables, including diversification, leverage and volatility, that make up the kind of market risk that traders and firms face every day.

Another reason VaR is so appealing is that it can measure both individual risks — the amount of risk contained in a single trader’s portfolio, for instance — and firmwide risk, which it does by combining the VaRs of a given firm’s trading desks and coming up with a net number. Top executives usually know their firm’s daily VaR within minutes of the market’s close.

As you might expect of a discussion of complicated statistical modeling in the mainstream press, the story oversimplifies and comes up a little short. Naked Capitalism does a good job picking the article apart in "Woefully Misleading Piece on Value at Risk in New York Times" - essentially the article make the classic mistake of assuming everything is normally distributed (the ludic fallacy):

Now when I say it is well known that trading markets do not exhibit Gaussian distributions, I mean it is REALLY well known. At around the time when the ideas of financial economists were being developed and taking hold (and key to their work was the idea that security prices were normally distributed), mathematician Benoit Mandelbrot learned that cotton had an unusually long price history (100 years of daily prices). Mandelbrot cut the data, and no matter what time period one used, the results were NOT normally distributed. His findings were initially pooh-poohed, but they have been confirmed repeatedly. Yet the math on which risk management and portfolio construction rests assumes a normal distribution!

...

It similarly does not occur to Nocera to question the "one size fits all" approach to VaR. The same normal distribution is assumed for all asset types, when as we noted earlier, different types of investments exhibit different types of skewness. The fact that VaR allows for comparisons across investment types via force-fitting gets nary a mention.

He also fails to plumb the idea that reducing as complicated a matter as risk management of internationally-traded multii-assets to a single metric is just plain dopey. No single construct can be adequate. Accordingly, large firms rely on multiple tools, although Nocera never mentions them. However, the group that does rely unduly on VaR as a proxy for risk is financial regulators. I have been told that banks would rather make less use of VaR, but its popularity among central bankers and other overseers means that firms need to keep it as a central metric.

Similarly, false confidence in VaR has meant that it has become a crutch. Rather than attempting to develop sufficient competence to enable them to have a better understanding of the issues and techniques involved in risk management and measurement (which would clearly require some staffers to have high-level math skills), regulators instead take false comfort in a single number that greatly understates the risk they should be most worried about, that of a major blow-up.

More Literal Survivor Bias from Car Talk Puzzler

2009-02-28T23:09:00.000-08:00

Car Talk recently had a puzzler that involved another example of (literal) survivor bias:

It's World War II, an RAF airfield north of London. A dimly lit Quonset hut filled with air crews just returned from bombing runs over Germany.

The meeting opens with the chaplain leading the men in prayer for their lost comrades. He is followed by the flight operations chief, who begins the debriefing by asking the airmen, "From what direction were you attacked by the German fighter planes?"

Without hesitation or dissent, the reply was, "From above and behind."

As in our previous example of this (which also involved world war II pilots...), the problem is that they are only interviewing the survivors.

Spolsky on Survivor Bias

2009-02-28T23:24:00.000-08:00

Joel Spolsky (of Joel on Software fame) has a column "How Hard could it be? Startup Static" about survivor bias when trying to emulate successful companies:

The problem is that trying to copy one company's model is a fool's errand. It's hard to figure out which part of the Starbucks formula made the business a smash hit while so many of its rivals failed. Starbucks's success is the product of a combination of factors that came together in precisely the right way at precisely the right time. It's nearly impossible to isolate which one was the most important. You would probably have to look at the hundreds of small coffee chains that didn't make it big before you stood a chance of seeing what really distinguished Starbucks.

...

The survivorship bias in entrepreneurship was on my mind a few months ago. My company was putting together a conference in Boston, and I invited my friend Jessica Livingston to speak. Jessica is the co-founder of a small angel investment group called Y Combinator. Its model is to give a few thousand dollars to groups of two or three geeks to start tech companies. She has also written a book called Founders at Work, in which she interviews the founders of about 30 successful start-ups. When she asked me what she should speak about, I asked her to consider describing all the different ways a start-up can fail, rather than the usual stuff about lessons learned from people who succeeded.

"That would be boring," she told me. "They all fail for the same reason: People just stop working on their business." Um, yeah, well, sure, and most people die because their heart stops beating. But somehow dying in different ways is still interesting enough to support 40 hours a week of prime-time programming.

But the more I thought about it, the more I realized Jessica was onto something. Why do start-ups fail? As she pointed out, it's usually a collapse of motivation -- everyone wanders back to civilian life, and the start-up ends, not with a bang but a whimper.

I wrote on this topic earlier in "Beware advice from the successful" and "Good to Not so great". Startups fail for lots of reasons, they also succeed for lots of reasons and what worked for Starbucks (even if you could rigorously determine why) won't really help you much.

People have a strong tendency to attribute to skill everything they did right and luck to everything they did wrong. In reality, it’s best to think of the world as a very large roulette table (or Russian Roulette depending on your predicament) - skill will allow you to place more bets on the table but it’s not a guarantee of success. Couple this bias with the lack of data and an inability to reproduce the experiment - people generally don’t have the ability to rerun their lives to determine what would have happened if they had made some choice differently. This is where Survivor Bias appears - by asking the successful, your ignoring all the people who were unsuccessful.

Survivor Bias in Log Cabins

2009-02-28T23:41:00.000-08:00

Michael Graham Richard has a great post about survivor bias of frontier log cabins - essentially all the ones you see are well made because they were well made:

I have to chuckle whenever I read yet another description of American frontier log cabins as having been well crafted or sturdily or beautifully built. The much more likely truth is that 99% of frontier log cabins were horribly built—it’s just that all of those fell down. The few that have survived intact were the ones that were well made. That doesn’t mean all of them were.

He also makes the point that the classical music you hear today is good precisely because it is still around - history buries the mediocre.

Survivor Bias for retailers

2009-03-29T19:38:00.000-07:00

From Forbes, is appears the good news is, if you can stay in business things might start to be looking up - survivors in retail get are picking up business for

Rationalize your fears with Lethal

2009-04-24T05:35:00.000-07:00

One of the big criticisms after 9-11 was that terrorist attacks, while horrifying, were a lot less likely to kill you than mundane occurrences like car accidents and smoking. In general people fixate on the newsworthy rather than the everyday risks in life because your more likely to hear about the former and more likely to fixate on it. Bruce Schneier has articulated this quite well:

I tell people that if it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." It's when something isn't in the news, when it's so common that it's no longer news -- car crashes, domestic violence -- that you should start worrying.

Additionally, the very horrific nature of specific events makes us incorrectly assess the risk due to anchoring:

Anchoring and adjustment is a psychological heuristic that influences the way people intuitively assess probabilities. According to this heuristic, people start with an implicitly suggested reference point (the "anchor") and make adjustments to it to reach their estimate.

An audience is first asked to write the last 2 digits of their social security number, and, second, to submit mock bids on items such as wine and chocolate. The half of the audience with higher two-digit numbers would submit bids that were between 60 percent and 120 percent higher than those of the other half, far higher than a chance outcome; the simple act of thinking of the first number strongly influences the second, even though there is no logical connection between them.

Lethal is an iPhone app designed to combat this irrationality - based on your current location it gives information about the relative likelyhood of different types of lethal events:

Want to know everything in your area which poses a threat? LETHAL uses auto-location to deliver information you need to be on your guard. Find out more about the dangers which could surround you — the hostile animals, the likelihood of crimes, the prevalence of disease, and the potential accidents and disasters.

Drawing from a proprietary database compiling information from government and academic statistics and research, LETHAL offers information on 650 locations in the US and Canada.

Mitigated speech and disaster avoidance

2009-07-01T22:00:00.000-07:00

Interesting article on accidents caused in the aviation and medical industries where people notice deadly problems but don't convey the seriousness to other team members out of politeness or deferring to a senior team mate. For example:

Korean Air Flight 801, almost the same exact situation as Air Florida. In trying to warn the captain of severe weather problems that would eventually lead to the deaths of 228 of the 254 people on board, the first officer says, "Don't you think it rains more? In this area, here?" and "Captain, the weather radar has helped us a lot"…

Captain, the weather radar has helped us a lot?! What are these people doing? They're hinting at the impending problem, in hopes that the guy who's a little busy with the whole "flying an airplane" or "trying to bring 99 planes circling Kennedy airport in for a landing" thing is going to catch on, read their mind, and solve the problem for them self.

...

The official term for this is "mitigated speech" and Malcolm Gladwell provides a fascinating account of how it has effected the airline industry in his book Outliers. He defines it as "any attempt to downplay or sugarcoat the meaning of what is being said." and explains that "we mitigate when we're being polite, or when we're ashamed or embarrassed, or when we're being deferential to authority."

Why you won't live forever

2009-08-02T17:44:00.000-07:00

Interesting (and depressing!) post on the exponential mortality rate:

"Your probability of dying during a given year doubles every 8 years. For me, a 25-year-old American, the probability of dying during the next year is a fairly miniscule 0.03% — about 1 in 3,000. When I’m 33 it will be about 1 in 1,500, when I’m 42 it will be about 1 in 750, and so on. By the time I reach age 100 (and I do plan on it) the probability of living to 101 will only be about 50%. This is seriously fast growth — my mortality rate is increasing exponentially with age."