Information

Uif nbq jt opu uif ufssjupsz.

Were you able to tell what I was trying to communicate there? Let me express it differently:



Click here for the sentence I've been trying to convey, and its significance. The point is that the same information can be expressed in different ways, and understood (with some work) even if you don't know the the communication system (i.e. code) I'm using.

I started thinking recently about how one would define the concept of information. I don't have a definition yet, but I think one of its crucial properties is that information is independent of the system used to communicate it. The same information can be expressed in a variety of languages, codes, or pictures, and received through lights on a screen, ink arranged on paper, or compressed waves of air. To reach us, information might travel in the form of electrical pulses (as were used in telegraph machines), radio waves, light in fiber-optical channels, or smoke signals. This independence of physical form distinguishes information from other physical quantities. Energy, for example, can come in many forms; but you would react much differently from coming in contact with light energy versus heat energy or kinetic energy.

It takes a certain level of intelligence to conceive of information independently of its form. We humans understand that ink on paper can refer to the same thing as sound waves, whereas to a bacterium these are completely different physical phenomena. It would be interesting to investigate which animals can understand different physical processes as conveying the same message.

One might be tempted to conclude that information exists only in the minds of intelligent beings, with no independent physical meaning. But this is not true: information appears in the laws of physics. The second law of thermodynamics, for example, says that the closed systems become less predictable with time, and more information is therefore required to understand them.

So information is a physical quantity, but exists independently of its many possible forms. This falls far short of a definition, but it may help explain the uniqueness of information among the quantities considered in science.

How much can we know?

Scientific progress is often viewed as an inexorable march toward increasing knowledge. We'll never know everything about the universe, but we've gotten used to the idea that we keep knowing ever more, at an ever-increasing rate.

However, as we discussed some time ago, human beings are creatures of finite complexity. There is only a finite amount we can do, and, more relevant to the present discussion, there is only a finite amount we can know. It's very likely that the human brain holds less pure information than the average hard drive. So while we humans as a collective might be able to increase our knowledge indefinitely, our knowledge as individuals has a definite limit.

What does this limit mean for the study and practice of science? For one thing, it limits the knowledge that a single scientist can apply to a particular problem. A researcher studying a virus can't apply all of science, or all of molecular biology, or all of virology to his study. Even just the scientific knowledge about this particular virus might be too much to fit into this researcher's brain. As scientists, we attack our problems using whatever knowledge we've gained from coursework, reading, and conversations with others--a tiny fraction of the wealth of potentially relevant knowledge out there.

Worse, as the frontier of knowledge keeps expanding, the amount of background knowledge needed to comprehend a single patch of this frontier increases steadily. I started my math career in differential geometry/topology: a beautiful subject, but one that requires years of graduate coursework to understand current research questions even on a superficial level. Since we have finite brainpower, no individual can maintain this kind of expertise in more than a few subjects. So we become specialists, unable to discuss our research with anyone outside our narrowly defined field. Before I switched to complex systems, I was continually frustrated by the isolation that came with specialized research. And I hear this same frustration from many of the other math/science grad students I talk to.

The danger is that science will keep branching into smaller, more arcane, and more isolated subsubdisciplines. This would make interdisciplinary research increasingly difficult, and the prospect of a science career ever more daunting and unappealing for students. And it would not get us any closer to solving some of our biggest problems in science, which lie not at the fringes of some highly specialized discipline, but in the synthesis of results from all branches of science.

What is needed is a sustained push for big-picture thinking. Whereas small-picture science focuses on the complex and the narrowly defined, big-picture sceince seeks the broad and the simple. It combines the many complex discoveries made by small-picture scientists, and distills them into ideas that can fit in a single human's head.

Here's a useful example, stolen from the website eigenfactor.org and based on this paper:



The above is a diagram of a yeast protein interaction network. It represents the cumulative work of many scientists who investigated whether and how certain proteins interact with each other. A remarkable achievement, certainly.

But the sheer volume of information makes this diagram useless to anyone but a specialist, and probably not very helpful for the specialists either. Trying to draw conclusions from a diagram like this would be like trying to navigate cross country using a map that shows every side street and alley in the US. It's just too much information for one brain to handle.

The authors go on to describe an algorithm that can transform complex networks like this:



into simplified ones like this:



that represent simple, understandable relationships.

I don't mean to belittle the work done by small-picture scientists; without them the big picture thinkers would have nothing to talk about. But I think the scientific establishment is so structured around the small-picturists that big picture thinking often gets squeezed out, which only impedes our understanding of science in general.

Causality

My friend Seth asks, via gchat away message, "How is it that there are phenomena which are independant of all but a very small set of conditions?" In essence, he is asking about the concept of causality, and how it is that this concept can even make sense.

One of the first abstract ideas we are asked to understand as kids is the idea that some actions cause others. It fell because I dropped it. She's crying because I kicked her. But as we grow older, we see that causality is rarely so simple. Most events depend on a great many past events, none of which can be identified as a single cause. Yet we still use the language of causality ("Today's earnings report caused stocks to close lower") and, at least sometimes, this usage seems appropriate. So how can we tell when causality makes sense and when it doesn't?

In my conversations with Seth on this idea, I was reminded of a principle from special relativity: causality can't travel faster than light. For example, nothing you do today can affect events on Alpha Centauri tomorrow, because not even light can get from here to there that quickly.

This leads to the idea of a "causal cone" in spacetime. Consider the following picture:



The blue "cone" coming out of point A shows all the points in spacetime that light could possibly reach from point A. So points B and C can be affected by something that happens at point A, but Point D cannot because, like Alpha Centauri tomorrow, it is too far away in space and not far enough in the future.

In most everyday situations, causality travels at a speed much slower than light. The specific speed depends on the medium through which causality is travelling. For example, if an underwater earthquake causes a tidal wave, causality travels at the speed by which waves move through water. A rumor travels at the speed it takes people to hear the rumor and repeat it. The point is that, in all cases, causality moves at a finite speed. You can't affect something that is too close to the present (timewise) and too far away in a spatial sense or an information-sharing network sense (i.e. too many degrees removed from you.)

Now consider a situation where we have three potentially causal events:



Suppose we know that events D, E, and F could only have been caused by A, B, or C. Clearly D was caused by B, since A and C are too far away to have influenced D. E, on the other hand, could have been caused by A, B, or both, and F could have been caused by any combination of the three.

In real life, there are millions of events happening all the time, all of which have the potential to cause or influence other events. In the immediate aftermath of an event, the causal cone is undiluted by other cones (as in event B above). But as we get further away (spatially and temporally) from the event, other cones intersect and complicate the effects caused by the original event. This leads to our conclusion:

The statement "A causes B" is most likely to make sense if B happens immediately following and in close proximity to A. Otherwise, there are too many other effects that could dilute the influence of A.

Visually, this would probably look like a "causal flame" coming out of event A, representing the points in spacetime over which A the the most direct influence.



In short, you could reasonably say that dropping the urn caused it to break. But you'd have a much harder time arguing that this event caused your relationship to break up two years later.

The Prisoner's Dilemma

You and an acquaintance are charged (and rightfully so!) as co-conspirators in a train robbery. You are being interviewed separately by the police. You can either rat your buddy out or keep silent and do more time. Your acquaintance has the same choices.

If one of you rats and the other remains silent, the one who cooperated with the police gets off free and the other serves 10 years. If you both keep silent, they can only convict on a lesser charge (for lack of evidence), so you each do a year. If you both rat on each other, you each do five years.

Both you and your acquaintance know this information. Assuming neither of you cares what happens to the other, and there are no recriminations in the outside world (we'll revisit both of these assumptions later), what is likely to happen?

Under the assumptions we've made, neither of you has any incentive to help the other. No matter what the other guy does, you get a better result by ratting on him. You both come to the same conclusion, so you both do five years.

This game is one of the most famous examples in game theory. It presents something of a dilemma: By each choosing the option that serves them best, the two "players" in the game end up with a result (5 years each) that is worse than if they had each kept silent. Choosing the best option individually leaves them worse off as a whole.

The game is traditionally phrased in terms of prisoners, but it applies pretty well to any situation when people have an opportunity to screw someone else over for their own benefit. If it truly is better in each situation to screw the other person, then everyone will end up screwing everyone else (in a bad way), and everyone will be worse off.

I think of this game when I drive up my street after a snowstorm, looking for a parking spot. People on my block (and all over Boston, from what I've seen) have the perverse idea that if they dig their car out of the snow, they "own" the spot they parked it in until the snow melts. They mark their spots with chairs or traffic cones. I've thought about doing the same. On the one hand, I think it's ridiculous for people to "claim" spots, just because they happened to park there before the storm. On the other hand, if everyone else does it and I don't, I can't park anywhere. If everyone else has chosen the selfish option, why shouldn't I? Classic Prisoner's Dilemma.

So does this bode ill for humankind? Is this a game-theoretic "proof" that we're all going to stab each other in the back? To answer these questions, let's look back at the assumptions we made.

First, we assumed that you don't care what happens to the other person. If you do care, you'd be much more likely to keep silent, which would end with a better result for both of you. A little selflessness helps everyone.

We also assumed that there were no consequences to your actions beyond what was spelled out in the game. A lot of ways you can screw others for your own benefit, such as breaking into your neighbor's house, are illegal. Laws can't deal with all prisoner's dilemma situations, but they can eliminate some of the worst ones.

There is a third, hidden assumption that we made: we assumed the game would only be played once. If the game is played over and over many times, is it possible for more cooperative strategies to emerge successful? This question was addressed by David Axelrod in The Evolution of Cooperation who found that, while selfishness is the best short-term strategy, cooperative strategies will win in the long run if the game is played enough times. More specifically, he identified four hallmarks of successful strategies: (here I quote Wikipedia)

• Nice: The most important condition is that the strategy must be "nice", that is, it will not defect before its opponent does. Almost all of the top-scoring strategies were nice; therefore a purely selfish strategy will not "cheat" on its opponent, for purely utilitarian reasons first.


• Retaliating: However, Axelrod contended, the successful strategy must not be a blind optimist. It must sometimes retaliate. An example of a non-retaliating strategy is Always Cooperate. This is a very bad choice, as "nasty" strategies will ruthlessly exploit such softies.


• Forgiving: Another quality of successful strategies is that they must be forgiving. Though they will retaliate, they will once again fall back to cooperating if the opponent does not continue to play defects. This stops long runs of revenge and counter-revenge, maximizing points.


• Non-envious: The last quality is being non-envious, that is not striving to score more than the opponent (impossible for a ‘nice’ strategy, i.e., a 'nice' strategy can never score more than the opponent).

Are these principles to live by? Perhaps. Axelrod and others think the success of these kinds of strategies may help explain the evolution of altruistic behavior in animals. At any rate, it seems to suggest that nice guys can get ahead, if they're willing to be mean at the right moments.

The language of autism

An autistic woman demonstrates, then explains, her personal language and way of communicating. It blew my mind.



Here's a New York Times article on her. I don't have too much to add other than that nature didn't make any bad brains. Just way different ones.