Unsustainable

The following question was given as a homework problem in a course I'm TAing:

CNBC had an interesting program on the current financial crisis. They located one investor who noticed that since the late 1990's housing prices have been growing 10 percent every year (that is, each year, the average home price is 1.1 times the average price in the previous year) while income was only increasing by 5 percent each year (that is, each year, the average income was only 1.05 times the average of the previous year).

Explain why it is "absolutely clear that this situation could not go on forever", in the words of the investor (who made over a billion dollars because of this observation).

This simple question goes right to the heart of the financial collapse. I would only add that, not only did this particular investor make billions off this observation, but our whole economy lost trillions, because the vast majority of financial decision makers were either unable or unwilling to make this same observation.

(Anyone who needs help with the mathematics of this problem can meet me in the comments.)

Human Cultural Transformation Triggered by Dense Populations

Biologically,modern humans first appeared 160,000 to 200,000 years ago. But the transition to complex human societies, with art, music, advanced tools, occurred a good deal more recently, and moreover, occured at different times in different parts of the world. An article in June's Science magazine (see a less technical write-up here) argues, based on historical evidence and computer simulations, that in each case the transition was triggered once the population density had reached a critical threshold. At this threshold, there is sufficient interaction to allow for complex ideas to be passed down through generations, enabling rapid cultural evolution.

This highlights an interesting evolutionary tension: as I've written before, evolutionary theory tells us that cooperative behaviors are more likely to evolve (biologically speaking) in populations that are dispersed over space rather than densely packed. But I'm beginning to think that cultural evolution may be different enough from biological evolution to require its own body of theory.

Inferring Social Security Numbers from Birth Data

An article in July's PNAS investigates the possibility of predicting a person's Social Security number from their birth date and place. Exploiting patterns in how SSN's are assigned, authors Alessandro Acquisti and Ralph Gross developed an algorithm which could correctly predict the first 5 digits of a social security number 44% of the time, for people born after 1988 (older SSNs are significantly harder to predict). The accuracy varied from state to state; for smaller states and recent birthdays, the algorithm could sometimes predict an entire SSN on the first try.

Think you're safe?

The Quandaries of Quantifying Complexity

My good friend and computer scientest Kyle Burke has recently started a highly interesting blog on his research field: combinatorial game theory. The idea of this field is to use games as a tool for studying issues of complexity. Though his blog is only a month old, some important foundational ideas have begun to rear their heads, one of which I'll explore in this post.

Understanding complexity is important for almost any human endeavor, but defining it in rigorous terms is notoriously difficult. For example, which is the more complicated game, chess or tic-tac-toe? Almost anyone would say chess, but suppose you had a computer that was designed only to play chess. In fact, this computer has no capacity for calculation; it simply has the best move for any given chess position hardwired into its architecture. To get this computer to play tic-tac-toe, you would have to program it to translate each tic-tac-toe position into an analagous chess position, so it could then find the best chess move and translate this move back into tic-tac-toe. This computer would certainly find chess an easier game to play.

Computer scientists have a way around this paradox: instead of looking at individual games or problems, they look at classes of problems. Each problem in the class has a certain size, and they look at how complexity increases in relation to size.

For example, you could easily imagine playing tic-tac-toe on boards of various sizes. Computer scientists can analyze how the complexity of tic-tac-toe varies with the size of the board. (Chess, on the other hand, doesn't generalize as easily to larger sizes, which makes it difficult to talk about its complexity.)

Unfortunately, if we are faced with a real-world issue (such as how to provide for the needs of a large population), we will want to know the complexity of the specific problem at hand, not how the complexity might theoretically scale with problem size. Part of the reason that complexity issues are so often ignored (to the detriment of many well-meaning policies and programs) is that defining and quantifying complexity is so unavoidably slippery.

Further reading

The Criminalization of Poverty

Barbara Ehrenreich had an excellent article in yesterday's New York Times on the many ways that being poor can land you in trouble with the law. One striking example:

In just the past few months, a growing number of cities have taken to ticketing and sometimes handcuffing teenagers found on the streets during school hours.

In Los Angeles, the fine for truancy is $250; in Dallas, it can be as much as $500 — crushing amounts for people living near the poverty level. According to the Los Angeles Bus Riders Union, an advocacy group, 12,000 students were ticketed for truancy in 2008.

Why does the Bus Riders Union care? Because it estimates that 80 percent of the “truants,” especially those who are black or Latino, are merely late for school, thanks to the way that over-filled buses whiz by them without stopping. I met people in Los Angeles who told me they keep their children home if there’s the slightest chance of their being late. It’s an ingenious anti-truancy policy that discourages parents from sending their youngsters to school.

The column was based on a report by the National Law Center on Homelessness and Poverty, which finds that the number of ordinances passed and tickets issued for crimes related to poverty has grown since 2006.

Hey, no one likes poverty, right? Let's pass a law!

The Evolution of Bad Ideas

It is by now common wisdom that our current financial crisis is due in large part to misplaced incentives in our financial system. Analysts and fund managers were rewarded for short-term thinking and risk-taking. If we can rework our financial system to reward long-term, careful planning, it is often argued, we can avoid collapses like this in the future.

While I agree that misplaced incentives were a fundamental problem, the question of how to change this is rather more deep and complex than I think many people realize.

Our economy is, of course, an evolutionary system. Successful businesses grow in size and their practices are imitated by others; unsuccessful businesses vanish. This process has led to many good business practices, even in the financial sector.

However, evolution does not always yield the best outcomes, in biology or in economics. Our recent crisis illustrates two key limitations of evolutionary systems, limitations which allow bad ideas to evolve over good ones.

The first problem has to do with time lags. Suppose Financial Company A comes up with an idea that will yield huge sums of money for five years and then drive the company to bankruptcy. They implement the idea, obfuscating the downside, and soon the company is rolling in cash. Investors line up to give them money, magazines laud them, and other companies begin imitating them.

Not so Company B. Company B believes in long-term thinking, and can see this idea for the sham it is. They persue a quiet, sound strategy, even when their investors begin pulling money out to invest in A.

We would like to think that in the end, Company B will be left standing and reap them benefits of their foresight. But there is a fundamental problem of time-scales here: by the time A folds, B may already be out of business, due to lack of interest from investors. In theoretical terms, there is a fundamental problem when the evolutionary process proceeds faster than the unfolding of negative consequences. In these situations, good ideas never have a chance to be rewarded, evolutionarily speaking.

One might argue that investors, not to mention government regulators and ratings agencies, should have forseen the flaw in A's plan. But this highlights a second limitation of the evolutionary process: it favors complexity. Simple bad ideas can be detected by intelligent agents, but complex ones have a chance to really stick. If Company A's idea was so complicated that no one aside from a few physicists could figure it out, investors and regulators could easily be fooled.

It's not clear to me how to patch these flaws in the evolutionary system. Increased transparency and oversight will help, but unless we can somehow cap the complexity of financial instruments (difficult) or slow down the evolutionary process (impossible), I'm not sure how we'll avoid similar crashes in the future.

Biodiversity and Entropy

On Tuesday, my Erdos number dropped from infinity to four. That's right: after four years of grad school, I am now officially published!

The article, “A New Phylogenetic Diversity Measure Generlizing the Shannon Index and Its Application to Phyllostomid Bats,” by Ben Allen, Mark Kon, Yaneer Bar-Yam, can be found on the American Naturalist website or, more accessibly, on my professional site.

So what is it about? Glad you asked!

Protecting biodiversity has become a central theme of conservation work over the past few decades. There has been something of a shift in focus from saving particular iconic endangered species, to preserving, as much as possible, the wealth and variety of life on the planet.

However, while biodiversity may seem like an intuitive concept, there is some disgreement about what it means in a formal sense and, in particular, how one might measure it. Given two ecological communities, or the same ecological community at two points in time, is there a way we can say which community is more diverse, or whether diversity has increased or decreased?

Certainly, a good starting point is to focus on species. As the writers of the Biblical flood narrative were in some sense aware, species are the basic unit of ecological reproduction. Thus the number of species (what biologists call the "species richness") is a good measure of the variety of life in a community.

But aren't genes the real unit of heredity, and hence diversity? Is the number of species more important than the variety of genes among those species? Should a forest containing many very closely related tree species be deemed more diverse than another whose species, though fewer, have unique genetic characteristics that make them valuable?

And while we're complicating matters, what about the number of organisms per species? Is a community that is dominated by one species (with numerous others in low proportion) less diverse than one containing an even mixture?


There is no obvious way to combine all this information into a single measure for use in monitoring and comparing ecological communities. Some previously proposed measures have undesirable properties; for example, they may increase, counterintuitively, when a rare species is eliminated.

In this paper we propose a new measure based on one of my favorite ideas in all of science: entropy. You may have heard of entropy from physics, where it measures the "disorderliness" of a physical system. But it is really a far more general concept, used also in mathematics, staticstics, and the theory of automated communication (information theory) in particular. At heart, entropy is a measure of unpredictability. The more entropy in a system, the less able you will be to accurately predict its future behavior.

The connection to diversity is not so much of a stretch: in a highly diverse community, you will be less able to predict what kinds of life you will come across next. Diversity creates unpredictability.

To be fair, we weren't the first to propose a connection between diversity and entropy. This connection is already well-known to conservation biologists. But we showed a new and mathematically elegant way of extending the entropy concept to include both species-level and gene-level diversity. It remains to be seen whether biologists will take up use of our measure, but whatever happens I am happy to have contributed to the conversation.