Information, Part Deux

First, a note of personal triumph: I have a paper up on the arXiv! For those who don't know, the arXiv is a way for researchers to distribute their work in a way which is free for all users, but also official, so that no one can scoop you once you've posted to the site. In the paper, I argue that a new and more general mathematics of information is needed, and I present a axiomatic framework for this mathematics using the language of category theory.

For those unfamiliar with such highfalutin language, it's really not as complicated as it sounds. I'll probably do a post soon explaining the content of the paper in layperson's terms. But first, and based partly on the feedback to my last post, I think it's important to say more on what information is and why I, as a complex systems theorist, am interested in it.

I'm currently thinking that information comes in three flavors, or more specifically, three broad situations where the concept comes in handy.

• Statisitcal information: Some things in life appear to be random. Really, this means that there's information we don't have about what's going to happen. It turns out there's a formula to quantify the uncertainty of an event---how much we don't know. This enables us to make statements like "event A is twice as uncertain as event B", and, more powerfully, statements like "knowing the outcome of event C will give us half of the necessary information to predict event B." The second statement uses the concept of mutual information: the amount of information that something tells you about something else. Mutual information can be understood as quantifying the statistical relationship between two uncertain events, and forms the basis of a general theory of complex systems proposed by Bar-Yam.



• Physical Information: If the "uncertain event" you're interested in is the position and velocity of particles in a system, then calculating the statistical uncertainty will give you what physicists call the entropy of the system. Entropy has all the properties of statistical information, but also satisfies physical laws like the second law of thermodynamics (entropy of a closed system does not decrease.)



• Communication Information: Now suppose that the "uncertain event" is a message you'll receive from someone. In this case, quantifying the uncertainty results in communication information (which is also called entropy, and there's a funny reason* why.) Communication information differs from statistical information in that, for communication, the information comes in the form of a message, which is independent of the physical system used to convey it.

The neat thing about these flavors of information is that they are all described by the same mathematics. The first time I learned that the same formula could be used in all these situations, it blew my mind.

One might think that is concept is already amazingly broad; why is a "more general mathematics of information" needed? The answer is that people were so inspired by the concept of information that they've applied it to fields as diverse as linguistics, psychology, anthropology, art, and music. However, the traditional mathematics of information doesn't really support these nontraditional applications. To use the standard formula, you need to know the probability of each possible outcome of an event; but "probability" doesn't really make sense when talking about art, for example. So a big part of my research project is trying to understand the behavior of information when the basic formula does not apply.

*Having just invented the mathematical concept of communication information, Claude Shannon was unsure of what to call it. Von Neumann, the famous mathematician and physicist, told him "You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, no one really knows what entropy really is, so in a debate you will always have the advantage."

Information

Uif nbq jt opu uif ufssjupsz.

Were you able to tell what I was trying to communicate there? Let me express it differently:



Click here for the sentence I've been trying to convey, and its significance. The point is that the same information can be expressed in different ways, and understood (with some work) even if you don't know the the communication system (i.e. code) I'm using.

I started thinking recently about how one would define the concept of information. I don't have a definition yet, but I think one of its crucial properties is that information is independent of the system used to communicate it. The same information can be expressed in a variety of languages, codes, or pictures, and received through lights on a screen, ink arranged on paper, or compressed waves of air. To reach us, information might travel in the form of electrical pulses (as were used in telegraph machines), radio waves, light in fiber-optical channels, or smoke signals. This independence of physical form distinguishes information from other physical quantities. Energy, for example, can come in many forms; but you would react much differently from coming in contact with light energy versus heat energy or kinetic energy.

It takes a certain level of intelligence to conceive of information independently of its form. We humans understand that ink on paper can refer to the same thing as sound waves, whereas to a bacterium these are completely different physical phenomena. It would be interesting to investigate which animals can understand different physical processes as conveying the same message.

One might be tempted to conclude that information exists only in the minds of intelligent beings, with no independent physical meaning. But this is not true: information appears in the laws of physics. The second law of thermodynamics, for example, says that the closed systems become less predictable with time, and more information is therefore required to understand them.

So information is a physical quantity, but exists independently of its many possible forms. This falls far short of a definition, but it may help explain the uniqueness of information among the quantities considered in science.

How much can we know?

Scientific progress is often viewed as an inexorable march toward increasing knowledge. We'll never know everything about the universe, but we've gotten used to the idea that we keep knowing ever more, at an ever-increasing rate.

However, as we discussed some time ago, human beings are creatures of finite complexity. There is only a finite amount we can do, and, more relevant to the present discussion, there is only a finite amount we can know. It's very likely that the human brain holds less pure information than the average hard drive. So while we humans as a collective might be able to increase our knowledge indefinitely, our knowledge as individuals has a definite limit.

What does this limit mean for the study and practice of science? For one thing, it limits the knowledge that a single scientist can apply to a particular problem. A researcher studying a virus can't apply all of science, or all of molecular biology, or all of virology to his study. Even just the scientific knowledge about this particular virus might be too much to fit into this researcher's brain. As scientists, we attack our problems using whatever knowledge we've gained from coursework, reading, and conversations with others--a tiny fraction of the wealth of potentially relevant knowledge out there.

Worse, as the frontier of knowledge keeps expanding, the amount of background knowledge needed to comprehend a single patch of this frontier increases steadily. I started my math career in differential geometry/topology: a beautiful subject, but one that requires years of graduate coursework to understand current research questions even on a superficial level. Since we have finite brainpower, no individual can maintain this kind of expertise in more than a few subjects. So we become specialists, unable to discuss our research with anyone outside our narrowly defined field. Before I switched to complex systems, I was continually frustrated by the isolation that came with specialized research. And I hear this same frustration from many of the other math/science grad students I talk to.

The danger is that science will keep branching into smaller, more arcane, and more isolated subsubdisciplines. This would make interdisciplinary research increasingly difficult, and the prospect of a science career ever more daunting and unappealing for students. And it would not get us any closer to solving some of our biggest problems in science, which lie not at the fringes of some highly specialized discipline, but in the synthesis of results from all branches of science.

What is needed is a sustained push for big-picture thinking. Whereas small-picture science focuses on the complex and the narrowly defined, big-picture sceince seeks the broad and the simple. It combines the many complex discoveries made by small-picture scientists, and distills them into ideas that can fit in a single human's head.

Here's a useful example, stolen from the website eigenfactor.org and based on this paper:



The above is a diagram of a yeast protein interaction network. It represents the cumulative work of many scientists who investigated whether and how certain proteins interact with each other. A remarkable achievement, certainly.

But the sheer volume of information makes this diagram useless to anyone but a specialist, and probably not very helpful for the specialists either. Trying to draw conclusions from a diagram like this would be like trying to navigate cross country using a map that shows every side street and alley in the US. It's just too much information for one brain to handle.

The authors go on to describe an algorithm that can transform complex networks like this:



into simplified ones like this:



that represent simple, understandable relationships.

I don't mean to belittle the work done by small-picture scientists; without them the big picture thinkers would have nothing to talk about. But I think the scientific establishment is so structured around the small-picturists that big picture thinking often gets squeezed out, which only impedes our understanding of science in general.

Quantifying Complexity

Complexity matters. This will hopefully become evident through the course of our discussion, but for now let's accept the principle that, in a great many situations, the extent to which something is complicated can be hugely important.

For a mathematician or scientist, a natural step after identifying something important is to attempt to quantify it, in the hopes of determining some of its properties. Us humans actually have a decent intuitive sense of different quantities of complexity. For example, we could all agree that Mozart's 40th is more complex than Twinkle Twinkle Little Star, or that sovling a crossword puzzle is more complex than tying a shoe. Other comparisons are less clear: Is a horse a more complex animal than a lion? Is China's economy more complex than India's?

Complexity researchers have identified several different ways that complexity can be quantified. These measures roughly fall into three categories:

• Variety - the complexity of an object can be quantified in terms of the number of actions it can take or the number of states in which it can exist.
  
• Descriptive complexity - the complexity of an object can be quantified in terms of the length of the shortest complete description of that object
  
• Algorithmic complexity - the complexity of a process can be quantified in terms of the number of steps or amount of time required to complete that process.
  

These three categories can be linked mathematically, which supports the idea that they are three expressions of the same concept rather than three different concepts. However, none of these can be unambiguously applied to the real world. For example, the number of actions or states of an object can be difficult to quantify. How many different actions can a human take? Descriptive complexity notions are dependent on the language used to describe something, and on what counts as a "complete" description. Similarly, algorithmic complexity notions depend on how a process is broken into tasks. This is not to say that the above quantification schemes are useless; just that care should be used in applying them and the values they give should be seen as approximate.

This approximateness is a problem for many scientists, who are used to dealing with the exact. How can we apply our analytical tools to a quantity which can never be precisely measured?

The way forward, in my opinion, is as follows. We (complex systems researchers) will investigate abstract models in which complexity can be mathematically quantified. The goal of investigating such models will be to discover laws of complexity which may carry over to the real world. At the same time, these laws must be checked against real-world experiment and observation. Because of the semi-fuzzy nature of complexity, the laws we discover will likely not be quantitative (e.g. F=ma or e=mc^2), but qualitative (e.g. "energy is conserved.")

In the near future, we will investigate two examples of such laws: Occam's Razor (the simplest explanation is the likeliest) and Ashby's Law (an organism must be as complex as its environment.) In the meantime, can anyone think of other qualitative laws of complexity/complex systems?