digital vs physical businesses

In the first part, I will analyze how digital businesses and physical businesses are complementary to each other via the following dualities:

  1. Risk of Death vs Potential for Growth

  2. Controlling Demand vs Controlling Supply

  3. Network Effects vs Scale Effects

  4. Mind vs Body

  5. Borrowing Space vs Borrowing Time

In the second part, I will analyze how the rise of digital businesses against physical businesses is triggering the following trends:

  1. Culture is Shifting from Space to Time

  2. Progress is Accelerating

  3. Science is Becoming More Data-Driven

  4. Economy is Getting Lighter

  5. Power is Shifting from West to East

Duality 1: Risk of Death vs Potential for Growth

Since information is frictionless, every digital startup has a potential for fast growth. But since the same fact holds for every other startup as well, there is also a potential for a sudden downfall. That is why defensibility (i.e. ability to survive after reaching success) is often mentioned as the number one criterion by the investors of such companies.

Physical businesses face the inverse reality: They are harder to grow but easier to defend, due to factors like high barriers to entry, limited real estate space, hard-to-set-up distribution networks etc. That is why competitive landscape is the most scrutinized issue by the investors of such companies.

Duality 2: Controlling Supply vs Controlling Demand

In the physical world, limited by scarcity, economic power comes from controlling supply; in the digital world, overwhelmed by abundance, economic power comes from controlling demand.
- Ben Thompson - Ends, Means and Antitrust

Although Ben’s point is quite clear, it is worth expanding it a little bit.

In the physical world, supply is much more limited than demand and therefore whoever controls the supply wins.

  • Demand. Physical consumption is about hoarding in space which is for all practical purposes infinite. Since money is digital in its nature, I can buy any object in any part of the world at the speed of light and that object will immediately become mine.

  • Supply. Extracting new materials and nurturing new talents take a lot of time. In other words, in the short run, supply of physical goods is severely limited.

In the digital world, demand is much more limited than supply and therefore whoever controls the demand wins:

  • Demand. Digital consumption is information based and therefore cognitive in nature. Since one can pay attention to only so many things at once, it is restricted mainly to the time dimension. For instance, for visual information, daily screen time is the limiting factor on how much can be consumed.

  • Supply. Since information travels at the speed of light, every bit in the world is only a touch away from you. Hence, in the short run, supply is literally unlimited.

Duality 3: Scale Effects vs Network Effects

Physical economy is dominated by geometric dynamics since distances matter. (Keyword here is space.) Digital economy on the other hand is information based and information travels at the speed of light, which is for all practical purposes infinite. Hence distances do not matter, only connectivities do. In other words, the dynamics is topological, not geometric. (Keyword here is network.)

Side Note: Our memories too work topologically. We remember the order of events (i.e. temporal connectivity) easily but have hard time situating them in absolute time. (Often we just remember the dates of significant events and then try to date everything else relative to them.) But while we are living, we focus on the continuous duration (i.e. the temporal distance), not the discrete events themselves. That is why the greater the number of things we are pre-occupied with and the less we can feel the duration, the more quickly time seems to pass. In memory though, the reverse happens: Since the focus is on events (everything else is cleared out!), the greater the number of events, the less quickly time seems to have passed.

This nicely ties back to the previous discussion about defensibility. Physical businesses are harder to grow because that is precisely how they protect themselves. They reside in space and scale effects help them make better use of time through efficiency gains. Digital businesses on the other hand reside in time and network effects help them make better use of space through connectivity gains. Building protection is what is hard and also what is valuable in each case.

Side Note: Just as economic value continuously trickles down to the space owners (i.e. land owners) in the physical economy, it trickles down to “time owners” in the digital economy (i.e. companies who control your attention through out the day).

Scale does not correlate with defensible value in the digital world, just as connectivity does not correlate with defensible value in the physical world. Investors are perennially confused about this since scale is so easy to see and our reptilian brains are so susceptible to be impressed by it.

Of course, at the end of the day, all digital businesses thrive on physical infrastructures and all physical businesses thrive on digital infrastructures. This leads to an interesting mixture.

  • As a structure grows, it suffers from internal complexities which arise from increased interdependencies between increased number of parts.

  • Similarly, greater connectivity requires greater internal scale. In fact, scalability is a huge challenge for fast-growing digital businesses.

Hence, physical businesses thrive on scale effects but suffer from negative internal network effects (which are basically software problems), and digital businesses thrive on network effects but suffer from negative internal scale effects (which are basically hardware problems). In other words, these two types of businesses are dependent on each other to be able to generate more value.

  • As physical businesses get better at leveraging software solutions to manage their complexity issues, they will break scalability records.

  • As digital businesses get better at leveraging hardware solutions to manage their scalability issues, they will break connectivity records.

Note that we have now ventured beyond the world of economics and entered the much more general world of evolutionary dynamics. Time has two directional arrows:

  • Complexity. Correlates closely with size. Increases over time, as in plants being more complex than cells.

  • Connectivity. Manifests itself as “entropy” at the lowest complexity level (i.e. physics). Increases over time, as evolutionary entities become more interlinked.

Evolution always pushes for greater scale and connectivity.

Side Note: "The larger the brain, the larger the fraction of resources devoted to communications compared to computation." says Sejnowski. Many scientists think that evolution has already reached an efficiency limit for the size of the biological brain. A great example of a digital entity (i.e. the computing mind) whose growing size is limited by the accompanying growing internal complexity which manifests itself in the form of internal communication problems.

Duality 4: Mind vs Body

All governments desire to increase the value of their economies but also feel threatened by the evolutionary inclination of the economic units to push for greater scale and connectivity. Western governments (e.g. US) tend to be more sensitive about size. They monitor and explicitly break up physical businesses that cross a certain size threshold. Eastern governments (e.g. China) on the other hand tend to be more sensitive about connectivity. They monitor and implicitly take over digital businesses that cross a certain connectivity threshold. (Think of the strict control of social media in China versus the supreme freedom of all digital networks in US.)

Generally speaking, the Western world falls on the right-hand side of the mind-body duality, while the Eastern world falls on the left-hand side.

  • As mentioned above, Western governments care more about the physical aspects of reality (like size) while Eastern governments care more about the mental aspects of reality (like connectivity).

  • Western sciences equate the mind with the brain, and thereby treats software as hardware. Eastern philosophies are infused with panpsychic ideas, ascribing consciousness (i.e. mind-like properties) to the entirety of universe, and thereby treats hardware as software.

We can think of the duality between digital and physical businesses as the social version of the mind-body duality. When you die, your body gets recycled back into the ecosystem. (This is no different than the machinery inside a bankrupt factory getting recycled back into the economy.) Your mind on the other hand simply disappears. What survive are the impressions you made on other minds. Similarly, when digital businesses die, they leave behind only memories in the form of broken links and cached pages, and therefore need “tombstones” to be remembered. Physical businesses on the other hand leave behind items which continue to circulate in the second-hand markets and buildings which change hands to serve new purposes.

Duality 5: Borrowing Space vs Borrowing Time

Banking too is moving from space to time dimension, and this is happening in a very subtle way. Yes, banks are becoming increasingly more digital, but this is not what I am talking about at all. Digitalized banks are more efficient at delivering the same exact services, continuing to serve the old banking needs of the physical economy. What I am talking about is the unique banking needs of the new digital economy. What do I mean by this?

Remember, physical businesses reside in space and scale effects help them make better use of time through efficiency gains. Digital businesses on the other hand reside in time and network effects help them make better use of space through connectivity gains. Hence, their borrowing needs are polar opposite: Physical businesses need to borrow time to accelerate their defensibility in space, while digital businesses need to borrow space to accelerate their defensibility in time. (What matters in the long run is only defensibility!)

But what does it mean to borrow time or space?

  • Lending time is exactly what regular banks do. They give you money and charge you an interest rate, which can be viewed as the cost of moving (discounting) the money you will be making in the future to now. In other words, banks are in the business of creating contractions in the time dimension, not unlike creating wormholes through time.

  • Definition of space for a digital company depends on the network it resides in. This could be a specific network of people, businesses etc. A digital company does not defend itself by scale effects, it defends itself by network effects. Hence its primary goal is to increase the connectivity of its network. In other words, a digital company needs creation of wormholes through space, not through time. Whatever facilitates further stitching of its network satisfies its “banking needs”.

Bankers of the digital economy are the existing deeply-penetrated networks like Alibaba, WeChat, LinkedIn, Facebook, Amazon etc. What masquerades as a marketing expense for a digital company to rent the connectivity of these platforms is actually in part a “banking” expense, not unlike the interest payments made to a regular bank.

Trend 1: Culture is Shifting from Space to Time

Culturally we are moving from geometry to topology, more often deploying topological rather than geometric language while narrating our lives. We meet our friends in online networks rather than physical spaces.

Correlation between the rise of the digital economy and the rise of the experience economy (and its associated cultural offshoots like hipster movement and decluttering movement) is not a coincidence. Experiential goods (not just those that are information-based) exhibit the same dynamics as digital goods. They are completely mental and reside in time dimension.

Our sense of privacy too is shifting from space dimension to time dimension. We are growing less sensitive about sharing objects and more sensitive about sharing experiences. We are participating in a myriad of sharing economies, but also becoming more ruthless about time optimization. (What is interpreted as a general decline in attention span is actually a protective measure erected by the digital natives, forcing everyone to cut their narratives short.) Increasingly we are spending less time with people although we look more social from outside since we share so many objects with each other.

Our sense of aesthetics has started to incorporate time rather than banish it. We leave surfaces unfinished and prefer using raw and natural-looking rather than polished and new-looking materials. Everyone has become wabi-sabi fans, preferring to buy stuff that time has taken (or seems to have taken) its toll on them.

Even physics is caught in the Zeitgeist. Latest theories are all claiming that time is fundamental and space is emergent. Popular opinion among the physicists used to be the opposite. Einstein had put the final nail on the coffin by completely spatializing time into what is called spacetime, an unchanging four-dimensional block universe. He famously had said “the distinction between past, present, and future is only a stubbornly persistent illusion.”

Trend 2: Progress is Accelerating

As economies and consumption patterns shift to time dimension, we feel more overwhelmed by the demands on our time, and life seems to progress at a faster rate.

Let us dig deeper into this seemingly trivial observation. First recall the following two facts:

  1. In a previous blog post, I had talked about the effect of aging on perception of time. As you accumulate more experience and your library of cognitive models grows, you become more adept at chunking experience and shifting into an automatic mode. What was used to be processed consciously now starts getting processed unconsciously. (This is no different than stable software patterns eventually trickling down and hardening to become hardware patterns.)

  2. In a previous blog post, I had talked about how the goal of education is to learn how not to think, not how to think. In other words, “chunking” is the essence of learning.

Combining these two facts we deduce the following:

  • Learning accelerates perception of time.

This observation in turn is intimately related to the following fact:

What exactly is this relation?

Remember, at micro-level, both learning and progress suffer from the diminishing returns of S-curves. However, at the macro-level, both overcome these limits via sheer creativity and manage to stack S-curves on top of each other to form a (composite) exponential curve that literally shoots to infinity.

This structural similarity is not a coincidence: Progress is simply the social version of learning. However, progress happens out in the open, while learning takes place internally within each of our minds and therefore can not be seen. That is why we can not see learning in time, but nevertheless can feel its acceleration by reflecting it off time.

Side Note: For those of you who know about Ken Wilber’s Integral Theory, what we found here is that “learning” belongs to the upper-left quadrant while “progress” belongs to the lower-right quadrant. The infinitary limiting point is often called Nirvana in personal learning and Singularity in social progress.

Recall how we framed the duality between digital and physical businesses as the social version of the mind-body duality. True, from the individual’s perspective, progress seems to happen out in the open. However, from the perspective of the mind of the society (represented by the aggregation of all things digital), progress “feels” like learning.

Hence, going back to the beginning of this discussion, your perception of time accelerates for two dual reasons:

  1. Your data processing efficiency increases as you learn more.

  2. Data you need to process increases as society learns more.

Time is about change. Perception of time is about processed change, and how much change your mind can process is a function of both your data processing efficiency (which defines your bandwidth) and the speed of data flow. (You can visualize bandwidth as the diameter of a pipe.) As society learns more (i.e. progresses further), you become bombarded with more change. Thankfully, as you learn more, you also become more capable of keeping up with change.

There is an important caveat here though.

  1. Your mind loses its plasticity over time.

  2. The type of change you need to process changes over time.

The combination of these two facts is very problematic. Data processing efficiency is sustained by the cognitive models you develop through experience, based on past data sets. Hence, their continued efficiency is guaranteed only if the future is similar to the past, which of course is increasingly not the case.

As mentioned previously, the exponential character of progress stems from the stacking of S-curves on top of each other. Each new S-curve represents a discontinuous creative jump, a paradigm shift that requires a significant revision of existing cognitive models. As progress becomes faster and life expectancy increases, individuals encounter a greater number of such challenges within their lifetimes. This means that they are increasingly at risk of being left behind due to the plasticity of their minds decreasing over time.

This is exactly why the elderly enjoy nostalgia and wrap themselves inside time capsules like retirement villages. Their desire to stop time creates a demographic tension that will become increasingly more palpable in the future, as the elderly become increasingly more irrelevant while still clinging onto their positions of power and keeping the young at bay.

Trend 3: Science is Becoming More Data-Driven

Rise of the digital economy can be thought of as the maturation of the social mind. The society as a whole is aging, not just us. You can tell this also from how science is shifting from being hypothesis-driven to being data-driven, thanks to digital technologies. (Take a look at the blog post I have written on this subject.) Social mind is moving from conscious thinking to unconscious thinking, becoming more intuitive and getting wiser in the process.

Trend 4: Economy is Getting Lighter

As software is taking over the world, information is being infused into everything and our use of matter is getting smarter.

Automobiles weigh less than they once did and yet perform better. Industrial materials have been replaced by nearly weightless high-tech know-how in the form of plastics and composite fiber materials. Stationary objects are gaining information and losing mass, too. Because of improved materials, high-tech construction methods, and smarter office equipment, new buildings today weigh less than comparable ones from the 1950s. So it isn’t only your radio that is shrinking, the entire economy is losing weight too.

Kevin Kelly - New Rules for the New Economy (Pages 73-74)

Energy use in US has stayed flat despite enormous growth. We now make less use of atoms, and the share of tangibles in total equity value is continuously decreasing. As R. Buckminster Fuller said, our economies are being ephemeralized thanks to the technological advances which are allowing us to do "more and more with less and less until eventually [we] can do everything with nothing."

This trend will probably, in a rather unexpected way, ease the global warming problem. (Remember, it is the sheer mass of what is being excavated and moved around, that is responsible for the generation of greenhouse gases.)

Trend 5: Power is Shifting from West to East

Now I will venture far further and bring religion into the picture. There are some amazing historical dynamics at work that can be recognized only by elevating ourselves and looking at the big picture.

First, let us take a look at the Western world.

  • Becoming. West chose a pragmatic, action-oriented attitude towards Becoming and did not directly philosophize about it.

  • Being. Western religions are built on the notion of Being. Time is deemed to be an illusion and God is thought of as a static all-encompassing Being, not too different from the entirety of Mathematics. There is believed to be an order behind the messy unfolding of Becoming, an order that is waiting to be discovered by us. It is with this deep conviction that Newton managed to discover the first mathematical formalism to predict natural phenomena. There is nothing in the history of science that is comparable to this achievement. Only a religious zeal could have generated the sort of tenacity that is needed to tackle a challenge of this magnitude.

This combination of applying intuition to Becoming and reason to Being eventually led to a meteoric rise in technology and economy.

Side Note: Although an Abrahamic religion itself, Islam did not fuel a similar meteoric rise, because it was practiced more dogmatically. Christianity on the other hand self-reformed itself into a myriad of sub-religions. Although not too great, there was enough intellectual freedom to allow people to seek unchanging patterns in reality, signs of Being within Becoming. Islam on the other hand persecuted any such aspirations. Even allegorical paintings about Being was not allowed.

East did the opposite and applied reason to Becoming and intuition to Being.

  • Becoming. East based its religion in Becoming and this instilled a fundamental suspicion against any attempts to mathematically model the unfolding reality or seek absolute knowledge. Of course, reasoning about Becoming without an implicit belief in unchanging absolutes is not an easy task. In fact, it is so hard that one has no choice but to be imprecise and poetic, and of course that is exactly what Eastern religions did. (Think of Taoism.)

  • Being. How about applying intuition to Being? How can you go about experiencing Being directly, through the “heart” so to speak? Well, through non-verbal silent meditation of course! That is exactly what Eastern religions did. (Think of Buddhism.)

Why could not East reason directly about Becoming in a formal fashion, like West reasoned directly about Being using mathematics? Remember Galileo saying "Mathematics is the language in which God has written the universe." What would have been the corresponding statement for the East? In other words, what is the formal language of Becoming? It is computer science of course, which was born out of Mathematics in the West around 1930s.

Now you understand why West was so lucky. Even if East had managed to discover computer science first, it would have been useless in understanding Becoming, because without the actual hardware to run simulations, you can not create computational models. A model needs to be run on something. It is not like a math theory in a book, waiting for you to play with it. Historically speaking, mathematics had to come first, because it is the cheaper, more basic technology. All you need is literally a pen, a paper and a trash bin.

Side Note: Here is a nerdy joke for you… The dean asks the head of the physics department to see him. “Why are you using so many resources? All those labs and experiments and whatnot; this is getting expensive! Why can’t you be more like mathematicians – they only need pens, paper, and a trash bin. Or philosophers – they only need pens and paper!”

But now is different. We have tremendous amounts of cheap computation and storage at our disposal, allowing us to finally crack the language of Becoming. Our entire economy is shifting from physical to digital, and our entire culture is shifting from space to time. An extraordinary period indeed!

It was never a coincidence that Chinese mathematicians chose to work in (and subsequently dominated) statistics, the most practical fields within mathematics. (They are culturally oriented toward Becoming.) Now all these statisticians are turning into artificial intelligence experts while West is still being paranoid about the oncoming Singularity, the exponential rise of AI.

Why have the Japanese always loved robots while the West has always been afraid of them? Why is the adoption of digital technologies happening faster in the East? Why are the kids and their parents in the East less worried about being locked into digital screens? As we elaborated above, the answer is metaphysical. Differences in metaphysical frameworks (often inherited from religions) are akin to the hard-to-notice (but exceptionally consequential) differences in the low-level code sitting right above the hardware.

Now guess who will dominate the new digital era? Think of the big picture. Do not extrapolate from recent past, think of the vast historical patterns.

I believe that people are made equal everywhere and in the long-run whoever is more zealous wins. East is more zealous about Becoming than the West, and therefore will sooner or later dominate the digital era. Our kids will learn their languages and find their religious practices more attractive. (Meditation is already spreading like wildfire.) What is “cool” will change and all these things will happen effortlessly in a mindless fashion, due to the fundamental shift in Zeitgeist and the strong structural forces of economics.

Side Note: Remember, in Duality 4, we had said that the East has an intrinsic tendency to regulate digital businesses rather than physical businesses. And here we just claimed that the East has an intrinsic passion for building digital businesses rather than physical businesses. Combining these two observations, we can predict that the East will unleash both greater energy and greater restrain in the digital domain. This actually makes a lot of sense, and is in line with the famous marketing slogan of the tyre manufacturing company Pirelli: “Power is Nothing Without Control”

Will the pendulum eventually swing back? Will the cover pages again feature physical businesses as they used to do a decade ago? The answer is no. Virtualization is one of the main trends in evolution. Units of evolution are getting smarter and becoming increasingly more governed by information dynamics rather than energy dynamics. (Information is substrate independent. Hence the term “virtualization”.) Nothing can stop this trend, barring some temporary setbacks here and there.

It seems like West has only two choices in the long run:

  1. It can go through a major religious overhaul and adopt a Becoming-oriented interpretation of Christianity, like that of Teilhard de Chardin.

  2. It can continue as is, and be remembered as the civilization that dominated the short intermediary period which begun with the birth of mathematical modeling and ended with the birth of computational modeling. (Equivalently, one could say that West dominated the industrial revolution and East will dominate the digital revolution.)


If you liked this post, you will probably enjoy the older post Innovative vs Classical Businesses as well. (Note that digital does not mean innovative and physical does not mean classical. You can have a classical digital or an innovative physical business.)

hypothesis vs data driven science

Science progresses in a dualistic fashion. You can either generate a new hypothesis out of existing data and conduct science in a data-driven way, or generate new data for an existing hypothesis and conduct science in a hypothesis-driven way. For instance, when Kepler was looking at the astronomical data sets to come up with his laws of planetary motion, he was doing data-driven science. When Einstein came up with his theory of General Relativity and asked experimenters to verify the theory’s prediction for the anomalous rate of precession of the perihelion of Mercury's orbit, he was doing hypothesis-driven science.

Similarly, technology can be problem-driven (the counterpart of “hypothesis-driven” in science) or tool-driven (the counterpart of “data-driven” in science). When you start with a problem, you look for what kind of (existing or not-yet-existing) tools you can throw at the problem, in what kind of a combination. (This is similar to thinking about what kind of experiments you can do to generate relevant data to support a hypothesis.) Conversely, when you start with a tool, you try to find a use case which you can deploy it at. (This is similar to starting off with a data set and digging around to see what kind of hypotheses you can extract out of it.) Tool-driven technology development is much more risky and stochastic. It is a taboo for most technology companies, since investors do not like random tinkering and prefer funding problems with high potential economic value and entrepreneurs who “know” what they are doing.

Of course, new tools allow you to ask new kind of questions to the existing data sets. Hence, problem-driven technology (by developing new tools) leads to more data-driven science. And this is exactly what is happening now, at a massive scale. With the development of cheap cloud computing (and storage) and deep learning algorithms, scientists are equipped with some very powerful tools to attack old data sets, especially in complex domains like biology.


Higher Levels of Serendipity

One great advantage of data-driven science is that it involves tinkering and “not really knowing what you are doing”. This leads to less biases and more serendipitous connections, and thereby to the discovery of more transformative ideas and hitherto unknown interesting patterns.

Hypothesis-driven science has a direction from the beginning. Hence surprises are hard to come by, unless you have exceptionally creative intuition capabilities. For instance, the theory of General Relativity was based on one such intuition leap by Einstein. (There has not been such a great leap since then. So it is extremely rare.) Quantum Mechanics on the other hand was literally forced by experimental data. It was so counter intuitive that people refused to believe it. All they could do is turn their intuition off and listen to the data.

Previously data sets were not huge, so scientists could literally eye ball them. Today this is no longer possible. That is why now scientists need computers, algorithms and statistical tools to help them decipher new patterns.

Governments do not give money to scientists so that they can tinker around and do whatever they want. So a scientist applying for a grant needs to know what he is doing. This forces everyone to be in a hypothesis-driven mode from the beginning and thereby leads to less transformative ideas in the long run. (Hat tip to Mehmet Toner for this point.)

Science and technology are polar opposite endeavors. Governments funding science like investors fund technology is a major mistake, and also an important reason why today some of the most exciting science is being done inside closed private companies rather than open academic communities.


Less Democratic Landscape

There is another good reason why the best scientists are leaving the academia. You need good quality data to do science within the data-driven paradigm, and since data is so easily monetizable the largest data sets are being generated by the private companies. So it is not surprising that the most cutting edge research in fields like AI is being done inside companies like Google and Facebook, which also provide the necessary compute power to play around with these data sets.

While hypotheses generation gets better when it is conducted in a decentralized open manner, the natural tendency of data is to be centralized under one roof where it can be harmonized and maintained consistently at a high quality. As they say, “data has gravity”. Once you pass certain critical thresholds, data starts generating strong positive feedback effects and thereby attracts even more data. That is why investors love it. Using smart data strategies, technology companies can build a moat around themselves and render their business models a lot more defensible.

In a typical private company, what data scientists do is to throw thousands of different neural networks at some massive internal data sets and simply observe which one gets the job done better. This of course is empiricism in its purest form, not any different than blindly screening millions of compounds during a drug development process. As they say, just throw it against a wall and see if it sticks.

This brings us to a major problem about big-data-driven science.


Lack of Deep Understanding

There is now a better way. Petabytes allow us to say: "Correlation is enough." We can stop looking for models. We can analyze the data without hypotheses about what it might show. We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot.

Chris Anderson - The End of Theory

We can not understand the complex machine learning models we are building. In fact, we train them the same way one trains a dog. That is why they are called black-box models. For instance, when the stock market experiences a flash crash we blame the algorithms for getting into a stupid loop, but we never really understand why they do so.

Is there any problem with this state of affairs if these models get the job done, make good predictions and (even better) earn us money? Can not scientists adopt the same pragmatic attitude of technologists and focus on results only, and suffice with successful manipulation of nature and leave true understanding aside? Are not the data sizes already too huge for human comprehension anyway? Why do we expect machines to be able to explain their thought processes to us? Perhaps they are the beginnings of the formation of a higher level life form, and we should learn to trust them about the activities they are better at than us?

Perhaps we have been under an illusion all along and our analytical models have never really penetrated that deep in to the nature anyway?

Closed analytic solutions are nice, but they are applicable only for simple configurations of reality. At best, they are toy models of simple systems. Physicists have known for centuries that the three-body problem or three dimensional Navier Stokes do not afford a closed form analytic solutions. This is why all calculations about the movement of planets in our solar system or turbulence in a fluid are all performed by numerical methods using computers.

Carlos E. Perez - The Delusion of Infinite Precision Numbers

Is it a surprise that as our understanding gets more complete, our equations become harder to solve?

To illustrate this point of view, we can recall that as the equations of physics become more fundamental, they become more difficult to solve. Thus the two-body problem of gravity (that of the motion of a binary star) is simple in Newtonian theory, but unsolvable in an exact manner in Einstein’s Theory. One might imagine that if one day the equations of a totally unified field are written, even the one-body problem will no longer have an exact solution!

Laurent Nottale - The Relativity of All Things (Page 305)

It seems like the entire history of science is a progressive approximation to an immense computational complexity via increasingly sophisticated (but nevertheless quiet simplistic) analytical models. This trend obviously is not sustainable. At some point we should perhaps just stop theorizing and let the machines figure out the rest:

In new research accepted for publication in Chaos, they showed that improved predictions of chaotic systems like the Kuramoto-Sivashinsky equation become possible by hybridizing the data-driven, machine-learning approach and traditional model-based prediction. Ott sees this as a more likely avenue for improving weather prediction and similar efforts, since we don’t always have complete high-resolution data or perfect physical models. “What we should do is use the good knowledge that we have where we have it,” he said, “and if we have ignorance we should use the machine learning to fill in the gaps where the ignorance resides.”

Natalie Wolchover - Machine Learning’s ‘Amazing’ Ability to Predict Chaos

Statistical approaches like machine learning have often been criticized for being dumb. Noam Chomsky has been especially vocal about this:

You can also collect butterflies and make many observations. If you like butterflies, that's fine; but such work must not be confounded with research, which is concerned to discover explanatory principles.

- Noam Chomsky as quoted in Colorless Green Ideas Learn Furiously

But these criticisms are akin to calling reality itself dumb since what we feed into the statistical models are basically virtualized fragments of reality. Analytical models conjure up abstract epi-phenomena to explain phenomena, while statistical models use phenomena to explain phenomena and turn reality directly onto itself. (The reason why deep learning is so much more effective than its peers among machine learning models is because it is hierarchical, just like the reality is.)

This brings us to the old dichotomy between facts and theories.


Facts vs Theories

Long before the computer scientists came into the scene, there were prominent humanists (and historians) fiercely defending fact against theory.

The ultimate goal would be to grasp that everything in the realm of fact is already theory... Let us not seek for something beyond the phenomena - they themselves are the theory.

- Johann Wolfgang von Goethe

Reality possesses a pyramid-like hierarchical structure. It is governed from the top by a few deep high-level laws, and manifested in its utmost complexity at the lowest phenomenological level. This means that there are two strategies you can employ to model phenomena.

  • Seek the simple. Blow your brains out, discover some deep laws and run simulations that can be mapped back to phenomena.

  • Bend the complexity back onto itself. Labor hard to accumulate enough phenomenological data and let the machines do the rote work.

One approach is not inherently superior to the other, and both are hard in their own ways. Deep theories are hard to find, and good quality facts (data) are hard to collect and curate in large quantities. Similarly, a theory-driven (mathematical) simulation is cheap to set up but expensive to run, while a data-driven (computational) simulation (of the same phenomena) is cheap to run but expensive to set up. In other words, while a data-driven simulation is parsimonious in time, a theory-driven simulation is parsimonious in space. (Good computational models satisfy a dual version of Occam’s Razor. They are heavy in size, with millions of parameters, but light to run.)

Some people try mix the two philosophies, inject our causal models into the machines and enjoy the best of both worlds. I believe that this approach is fundamentally mistaken, even if it proves to be fruitful in the short-run. Rather than biasing the machines with our theories, we should just ask them to economize their own thought processes and thereby come up with their own internal causal models and theories. After all, abstraction is just a form of compression, and when we talk about causality we (in practice) mean causality as it fits into the human brain. In the actual universe, everything is completely interlinked with everything else, and causality diagrams are unfathomably complicated. Hence, we should be wary of pre-imposing our theories on machines whose intuitive powers will soon surpass ours.

Remember that, in biological evolution, the development of unconscious (intuitive) thought processes came before the development of conscious (rational) thought processes. It should be no different for the digital evolution.

Side Note: We suffered an AI winter for mistakenly trying to flip this order and asking machines to develop rational capabilities before developing intuitional capabilities. When a scientist comes up with hypothesis, it is a simple effable distillation of an unconscious intuition which is of ineffable, complex statistical form. In other words, it is always “statistics first”. Sometimes the progression from the statistical to the causal takes place out in the open among a community of scientists (as happened in the smoking-causes-cancer research), but more often it just takes place inside the mind of a single scientist.


Continuing Role of the Scientist

Mohammed AlQuraishi, a researcher who studies protein folding, wrote an essay exploring a recent development in his field: the creation of a machine-learning model that can predict protein folds far more accurately than human researchers. AlQuiraishi found himself lamenting the loss of theory over data, even as he sought to reconcile himself to it. “There’s far less prestige associated with conceptual papers or papers that provide some new analytical insight,” he said, in an interview. As machines make discovery faster, people may come to see theoreticians as extraneous, superfluous, and hopelessly behind the times. Knowledge about a particular area will be less treasured than expertise in the creation of machine-learning models that produce answers on that subject.

Jonathan Zittrain - The Hidden Costs of Automated Thinking

The role of scientists in the data-driven paradigm will obviously be different but not trivial. Today’s world-champions in chess are computer-human hybrids. We should expect the situation for science to be no different. AI is complementary to human intelligence and in some sense only amplifies the already existing IQ differences. After all, a machine-learning model is only as good as the intelligence of its creator.

He who loves practice without theory is like the sailor who boards ship without a rudder and compass and never knows where he may cast.

- Leonardo da Vinci

Artificial intelligence (at least in its today’s form) is like a baby. Either it can be spoon-fed data or it gorges on everything. But, as we know, what makes great minds great is what they choose not to consume. This is where the scientists come in.

Deciding what experiments to conduct, what data sets to use are no trivial tasks. Choosing which portion of reality to “virtualize” is an important judgment call. Hence all data efforts are inevitably hypothesis-laden and therefore non-trivially involve the scientist.

For 30 years quantitative investing started with a hypothesis, says a quant investor. Investors would test it against historical data and make a judgment as to whether it would continue to be useful. Now the order has been reversed. “We start with the data and look for a hypothesis,” he says.

Humans are not out of the picture entirely. Their role is to pick and choose which data to feed into the machine. “You have to tell the algorithm what data to look at,” says the same investor. “If you apply a machine-learning algorithm to too large a dataset often it tends to revert to a very simple strategy, like momentum.”

The Economist - March of the Machines

True, each data generation effort is hypothesis-laden and each scientist comes with a unique set of biases generating a unique set of judgment calls, but at the level of the society, these biases get eventually washed out through (structured) randomization via sociological mechanisms and historical contingencies. In other words, unlike the individual, the society as a whole operates in a non-hypothesis-laden fashion, and eventually figures out the right angle. The role (and the responsibility) of the scientist (and the scientific institutions) is to cut the length of this search period as short as possible by simply being smart about it, in a fashion that is not too different from how enzymes speed up chemical reactions by lowering activation energy costs. (A scientist’s biases are actually his strengths since they implicitly contain lessons from eons of evolutionary learning. See the side note below.)

Side Note: There is this huge misunderstanding that evolution progresses via chance alone. Pure randomization is a sign of zero learning. Evolution on the other hand learns over time and embeds this knowledge in all complexity levels, ranging all the way from genetic to cultural forms. As the evolutionary entities become more complex, the search becomes smarter and the progress becomes faster. (This is how protein synthesis and folding happen incredibly fast within cells.) Only at the very beginning, in its most simplest form, does evolution try out everything blindly. (Physics is so successful because its entities are so stupid and comparatively much easier to model.) In other words, the commonly raised argument against the possibility of evolution achieving so much based on pure chance alone is correct. As mathematician Gregory Chaitin points out, “real evolution is not at all ergodic, since the space of all possible designs is much too immense for exhaustive search”.

Another venue where the scientists keep playing an important role is in transferring knowledge from one domain to another. Remember that there are two ways of solving hard problems: Diving into the vertical (technical) depths and venturing across horizontal (analogical) spaces. Machines are horrible at venturing horizontally precisely because they do not get to the gist of things. (This was the criticism of Noam Chomsky quoted above.)

Deep learning is kind of a turbocharged version of memorization. If you can memorize all that you need to know, that’s fine. But if you need to generalize to unusual circumstances, it’s not very good. Our view is that a lot of the field is selling a single hammer as if everything around it is a nail. People are trying to take deep learning, which is a perfectly fine tool, and use it for everything, which is perfectly inappropriate.

- Gary Marcus as quoted in Warning of an AI Winter


Trends Come and Go

Generally speaking, there is always a greater appetite for digging deeper for data when there is a dearth of ideas. (Extraction becomes more expensive as you dig deeper, as in mining operations.) Hence, the current trend of data-driven science is partially due to the fact that scientists themselves have ran out of sensible falsifiable hypotheses. Once the hypothesis space becomes rich again, the pendulum will inevitably swing back. (Of course, who will be doing the exploration is another question. Perhaps it will be the machines, and we will be doing the dirty work of data collection for them.)

As mentioned before, data-driven science operates stochastically in a serendipitous fashion and hypothesis-driven science operates deterministically in a directed fashion. Nature on the other hand loves to use both stochasticity and determinism together, since optimal dynamics reside - as usual - somewhere in the middle. (That is why there are tons of natural examples of structured randomnesses such as Levy Flights etc.) Hence we should learn to appreciate the complementarity between data-drivenness and hypothesis-drivenness, and embrace the duality as a whole rather than trying to break it.


If you liked this post, you will also enjoy the older post Genius vs Wisdom where genius and wisdom are framed respectively as hypothesis-driven and data-driven concepts.

future of pharmaceutical industry

What will the future of pharmaceutical industry look like?

It is clear that we are reaching one end of a paradigm, but what most people still do not get is how big the oncoming changes will be. We are on the cusp of a great intellectual revolution, on par with the revolution in 20th century physics. Computer science is unlocking biology, just like mathematics unlocked physics, and the consequences will be huge. (Read this older post for a deeper look at this interesting analogy between analogies.)

For the first time in history, we are engineering solutions from scratch rather than stumbling into them or stealing them from nature. Western medicine is only now truly taking off.

Not only will this transformation be breathtaking, but it will also be unfolding at a speed much faster than we expect. As biology becomes more information theoretical, pharmaceutical industry will become more software driven and will start displaying more of the typical dynamics of the software industry, like faster scaling and deeper centralization and modularization.

Of course, predicting the magnitude of change is not the same thing as predicting how things will actually unfold. (Sometimes I wonder which one is harder. Remember Paul Saffo: “We tend to mistake a clear view of the future for a short distance.”) Let us give a try anyway.


1. Splitting and Centralization of the Quantitative Brain

Just like the risk analytics layer is slowly being peeled out of big insurance companies as it is becoming more quantitative (small companies could not harbor such analytics departments anyway), the quantitative layer of the drug development process will split out of the massive pharmaceutical companies. (Similarly, in the autonomous driving space, companies like Waymo are licensing out self-driving technologies to big car manufacturers.)

Two main drivers of this movement:

  • Soft Reason. Culturally speaking, traditional (both manufacturing and service) companies can not nurture software development within themselves. Big ones often think that they can, but without exception they all end up wasting massive resources to realize that it is not a matter of resources. Similarly, they always end up suffocating the technology companies they acquire.

  • Hard Reason. Unlike services and manufacturing, software scales perfectly. In other words, the cost of reproduction of software is close to nil. This leads to centralization and winner-takes-all effects. (Even within big pharmas bioinformatics and IT departments are centralized.) Software developed in-house can never compete with software developed outside, which serves many customers, takes as input more diverse use cases and improves faster.

Study of complex systems (which biology is an example of) is conducted from either a state centric or process centric perspective, using either statistical (AI driven) or deterministic (algorithm driven) methods. (Read this older post for a deeper look at the divide between state and process centric perspectives.)

In other words, the quantitative brain in biology will be centralized around four different themes:

  1. Algorithm Driven + State Centric

  2. AI Driven + State Centric

  3. Algorithm Driven + Process Centric

  4. AI Driven + Process Centric

Xtalpi is a good example for the 4th category. Seven Bridges in its current form belongs to the 1st category. There are other examples out there that fit neatly into one of these categories or cut across a few. (It is tough to cut across both state centric and process centric perspectives since latter is mostly chemistry and physics driven and tap into a very different talent pool.)


2. Democratization and Commodification of Computation

Big pharma companies could afford to buy their own HPCs to run complex computations and manage data. Most are still holding onto these powerful clusters, but they are all realizing that this is not sustainable for two main reasons:

  • They either can not accommodate bursty computations or can not keep the machines busy all time. So it is best for the machines to be aggregated in shared spaces where they are maintained centrally.

  • Since data size is exploding doubly exponentially, it is becoming harder to move and more expensive to store. (Compute needs to go where data is generated.)

Cloud computing took off for reasons entirely unrelated to biomedical data analysis, which will soon be the biggest beneficiary of this revolution as biomedical data sizes and computation needs surpass everything else. (It is not surprising that the centralized disembodied brain is developing in the same way as our decentralized embodied brains did. It got enlarged for social reasons and deployed later for scientific purposes.) Small biotechs can now run complex computations on massive data repositories and pay for computation just like they pay for electricity, only for the amounts they use. Big pharmas too are migrating to the cloud, finally coming to terms with the fact that cloud is both safer and cheaper. They are no longer uncomfortable departing with their critical data and no longer ignorant about the hidden costs of maintaining local hardware.

Long story short, democratization of computation is complete (aside from some big players with sunk cost investments) and the industry has already moved on to its next phase. Today we are witnessing a large scale commoditization of cloud services, driven by the following two factors:

  • Supply Side. Strong rivals arriving and catching up with Amazon Web Services.

  • Demand Side. Big players preferring to be cloud agnostic and supporting multi-cloud.


3. Democratization, Uniformization and Centralization of Data

Democratization. Big pharmas are hoarding data. They are entering into pre-competitive consortiums and forming partnerships with or buying diagnostics companies straight out. Little pharmas (startup biotechs) are left out of this game, just as they were left out of the HPC game. But just like Amazon democratized computing, National Institutes of Health (NIH) is now trying to democratize data. (Amazon and NIH are playing parallel roles in this grand story. Interesting.) Sooner or later public data will outstrip private data simply because health is way too important from a societal point of view.

Uniformization. NIH is also trying to uniformize data structures and harmonize compliance and security standards across the board, so that data can flow around at a higher speed.

Centralization. NIH not only wants to democratize and uniformize data, but it also wants to break data silos. Data is a lot more useful when it all comes together. (Fragmentation problem is especially acute in US.) Similarly, imagine if everyone could hold all of their health data on a blockchain that they can share with any pharma in return for a compensation. This is another form of centralization, radically bringing together everything at an individual level. All pharma companies need to do is to take a cross section across the cohorts they are interested in.

With its top-down centralized policy making and absence of incumbent (novel drug developing) big pharmas, China will skip all of the above steps just as Africa skipped grid-based centralized electricity distribution and is jumping straight into off-grid decentralized solar power technologies.


4. Streamlining and Cheapening of Clinical Trials

It is extremely time consuming and expensive to get a drug approved. In 2000s, only 11 percent of drugs entering phase 1 clinical trials ended up being approved by FDA. Biotech startups that can make it to phase 3 usually end up selling themselves completely (or partially on a milestone basis) to big pharma companies simply because they can not afford the process. In other words, the final bottleneck for these startups in getting to the market on their own is clinical trials.

This problem is much more multi dimensional and thorny, but there is still hope:

  • Time. Regulations are being more streamlined and thereby making the processes faster.

  • Cost. Genomics and real world data are enabling better targeting (or - in the case of already approved drugs - retargeting) of patients and resulting in better responding cohorts and thereby driving costs down.

  • Risk. As we get better at simulating human biology on hardware and software, parallelizability of experimentation will increase and thereby the number of unnecessary (sure to fail) experiments on human beings will decrease. In other words, just as in the software world, experiments will fail faster.


5. Democratization and Decentralization of Drug Development

As some of the largest companies in the world, big pharmas are intimidating, but from an evolutionary point of view, they are actually quite primitive. The existing fatness is not due to some incredible prowess or sustained success, it is entirely structural in the sense that the industry itself has not fully matured and modularized yet. (In fact, there is little hope that they can execute the necessary internal changes and evolve a contemporary data-driven approach to drug development. That is why they seek acquisitions, outside partnerships etc.)

If you split open a big pharma today, you will see a centralized quantitative brain (consisting of bioinformatics and IT departments) and a constellation of independent R&D centers around this brain. This is exactly what the whole pharma industry will look like in the future.

Once quantitative brain is split off and centralized, computation is democratized and commoditized, data is democratized, uniformized and centralized, and clinical trials is streamlined and cheaper, there will be no need for biotech startups to merge themselves into the resource-rich environments of big pharma companies. Drugs will be developed in collaboration with the brain and be co-owned. (Currently we have already started seeing partnerships between the brain and the big pharma. Such partnerships will democratize and become common place.)

Biology will start off in independent labs and stay independent, and the startups will not have to sell themselves to the big guys if they do not want to, just as in the software world.

Biology is way too complex to allow repeat successes. Best ideas will always come from outsiders. In this sense, pharma industry will look more like the B2C software world rather than the B2B software world. Stochastic and experimental.

We have already started to see more dispersed value creation in the industry:

“Until well into the 1990s, a single drug company, Merck, was more valuable than all biotech companies combined. It probably seemed as if biotech would never arrive—until it did. Of the 10 best-selling drugs in the US during 2017, seven (including the top seller, the arthritis drug Humira) are biotech drugs based on antibodies.”

- MIT Tech Review - Look How Far Precision Medicine Has Come

(I did not say anything about the manufacturing and distribution steps since the vast majority of these physical processes is already being outsourced by pharma companies. In other words, these aspects of the industry have already been modularized.)

Future of Pharma.png

data as mass

The strange thing about data is that they are an inexhaustible resource: the more you have, the more you get. More information lets firms develop better services, which attracts more users, which in turn generate more data. Having a lot of data helps those firms expand into new areas, as Facebook is now trying to do with online dating. Online platforms can use their wealth of data to spot potential rivals early and take pre-emptive action or buy them up. So big piles of data can become a barrier to competitors entering the market, says Maurice Stucke of the University of Tennessee.

The Economist - A New School in Chicago

Greater centralization of internet and growing importance of data are basically two sides of the same coin. Data is like mass and therefore is subject to gravitation-like dynamics. Huge data-driven companies like Google and Facebook can be thought of as mature galaxy formations.

Light rays travel through fiber optic cables to transfer data and cruise across vast intergalactic voids to stitch together a causally integrated universe.

biology as computation

If the 20th century was the century of physics, the 21st century will be the century of biology. While combustion, electricity and nuclear power defined scientific advance in the last century, the new biology of genome research - which will provide the complete genetic blueprint of a species, including the human species - will define the next.

Craig Venter & Daniel Cohen - The Century of Biology

It took 15 years for technology to catch up with this audacious vision that was articulated in 2004. Investors who followed the pioneers got severely burned by the first hype cycle, just like those who got wiped out by the dot-com bubble.

But now the real cycle is kicking in. Cost of sequencing, storing and analyzing genomes dropped dramatically. Nations are finally initiating population wide genetics studies to jump-start their local genomic research programs. Regulatory bodies are embracing the new paradigm, changing their standards, approving new gene therapies, curating large public datasets and breaking data silos. Pharmaceutical companies and new biotech startups are flocking in droves to grab a piece of the action. Terminal patients are finding new hope in precision medicine. Consumers are getting accustomed to clinical genomic diagnostics. Popular culture is picking up as well. Our imagination is being rekindled. Skepticism from the first bust is wearing off as more and more success stories pile up.

There is something much deeper going on too. It is difficult to articulate but let me give a try.

Mathematics did a tremendous job at explaining physical phenomena. It did so well that all other academic disciplines are still burning with physics envy. As the dust settled and our understanding of physics got increasingly more abstract, we realized something more, something that is downright crazy: Physics seems to be just mathematics and nothing else. (This merits further elaboration of course, but I will refrain from doing so.)

What about biology? Mathematics could not even scratch its surface. Computer science on the other hand proved to be wondrously useful, especially after our data storage and analytics capabilities passed a certain threshold.

Although currently a gigantic subject on its own, at its foundations, computer science is nothing but constructive mathematics with space and time constraints. Note that one can not even formulate a well-defined notion of complexity without such constraints. For physics, complexity is a bug, not a feature, but for biology it is the most fundamental feature. Hence it is not a surprise that mathematics is so useless at explaining biological phenomena. 

The fact that analogies between computer science and biology are piling up gives me the feeling that we will soon (within this century) realize that biology and computer science are really just the same subject.

This may sound outrageous today but that is primarily because computer science is still such a young subject. Just like physics converged to mathematics overtime, computer science will converge to biology. (Younger subject converges to the older subject. That is why you should always pay attention when a master of the older subject has something to say about the younger converging subject.)

The breakthrough moment will happen when computer scientists become capable of exploiting the physicality of information itself, just like biology does. After all hardware is just frozen software and information itself is something physical that can change shape and exhibit structural functionalities. Today we freeze because we do not have any other means of control. In the future, we will learn how to exert geometric control and thereby push evolution into a new phase that exhibits even more teleological tendencies.

A visualization of the AlexNet deep neural network by Graphcore

A visualization of the AlexNet deep neural network by Graphcore


If physics is mathematics and biology is computer science, what is chemistry then?

Chemistry seems to be an ugly chimera. It can be thought of as the study of either complicated physical states or failed biological states. (Hat tip to Deniz Kural for the latter suggestion.) In other words, it is the collection of all the degenerate in-between phenomena. Perhaps this is the reason why it does not offer any deep insights, while physics and biology are philosophically so rich.

philosophy of dockerization

To persist you can either be inflexible and freeze your local environment into constancy or be flexible and continuously morph along with your environment. Former is the direction digital entities pursue and latter is the direction biological entities pursue. (Either way, at the extreme end, complete correlation with the environment results in complete diffusion of identity.)

Non-adaptive entities like pieces of code can only survive via dockerization. Adaptive entities persist in a weaker sense but they can do so by themselves. Non-adaptive entities on the other hand can only persist with the help of adaptive entities whom they need for the execution of the dockerization processes.


Going back to our childhood neighbourhoods and seeing them completely changed is so sad and destabilising. I wish we could dockerize our moments so that we can visit them later.

Dockerization in this sense is the ultimate form of nostalgia.

machine-to-human communication

Black box models make us feel uneasy. We want to have an intuitive grasp of how a computer reaches a certain conclusion. (For legal considerations, this is actually a must-have feature, not just a nice-to-have one.)

However, to exhibit such a capacity, a computer needs to be able to

  1. model its thought processes and,
  2. communicate the resulting model in a human understandable way.

Let us recall the correspondence between physical phenomena and cognitive models from the previous blog post on domains of cognition:

  • Environment <-> Perceptions
  • Body <-> Emotions
  • Brain <-> Consciousness

Hence, the first step of a model being able to model itself is akin to it having some sort of consciousness. Tough problem indeed!

The second step of turning the (quantitative) model of a model into something (qualitatively) communicable amounts to formation or adoption of a language which chunks the world into equivalence classes. (We call these equivalence classes "words".)

Qualitative communication of fundamentally quantitative phenomena is bound to be lossy because at each successive modelling information gets lost. 

  • That is essentially why writing good poetry is so hard. Words are like primitive modelling tools.
  • Good visual artists bypass this problem by directly constructing perceptions to convey perceptions. That is why conceptual art can feel so tasteless and backward. Art that needs explanation is not art. It is something else. 
  • Similarly, good companions can peer into each others' consciousnesses without speaking a word. 

Instead of expecting machines to make a discontinuous jump to language formation, we should first endow them with bodies that allow them to sense the world which they can then chunk into equivalence classes.

genericity and artificiality

Now that we proved faces are generic with respect to genes, life feels even more like a computer game

Left Real, Right Predicted

Left Real, Right Predicted

Finite variations within genomes explain most of the differences between our faces. The rest of the differences seem to be due to wear and tear.

There is a correlation between the extent of observable variation and the feeling of naturalness. An object feels natural if the variation among the relevant population looks infinite. Otherwise it feels artificial.


Despite all the apparent complexity and drama, variations among personalities too seem to be quiet contained. Big Five personality traits explain most of the variance. The output structure of IBM Watson's semantic take on personality analysis does not look too rich neither.

Watson Personality Insights takes your social media feed as an input and spits out a graph like above as an output.

Watson Personality Insights takes your social media feed as an input and spits out a graph like above as an output.

Of course, personality is a relational concept. How one behaves changes with respect to who one is interacting with. But focusing solely on one's relationship with a common reference point should be good enough for comparative purposes.

This approach is similar to extracting a variant from a genome by comparing it to a reference genome constructed out of the set of all genomes of the relevant population. Everyone's social media feed reveals how they interact with "the public", which acts sort of like a "reference personality", an average entity representing one's social network.

On a related note, dialogues with humanoid robots feel unnatural today partly due to the non-relational aspects of their personalities. Someone behaving in exactly the same manner regardless of context is deemed to be abnormal.

Consistency shows character, but too much of it is inhuman, as so eloquently pointed out by Walt Whitman in his famous quote: "Do I contradict myself? Very well. Then I contradict myself. I am large. I contain multitudes."

hierarchy and testosterone

Hierarchies select for testosterone heavy traits. In a world where only the high testosterone people can rise to the top, decisions will be testosterone driven. Hence, if you want to make the world a little less aggressive place, you should start by making the internal structure of the decision making entities less hierarchical. But how do you proceed?

  • Keeping the size of the entities small is one option. But competition and scale effects favour consolidation. Hence this will not work out in any sensible economic regime.
  • Trying out non-hierarchical organisational structures like holacracy is another option. But these flat fantasies never last too long. None of the large entities can even hope to give them a try.
  • Waiting for artificial intelligence to mature seems to be the most feasible option at the moment. AI will dramatically decrease the need for human decision making so that even the largest entities can be run like a small entity.

machine learning revolution

Take agriculture, for example. In agriculture you plant seeds, you fertilize them, you spray them, and then you wait six months and hope for the best. Once upon a time, people really understood the land, they understood the plants, and they had this intuitive feel for what would happen. Then farms got bigger and there were fewer people on them. We didn't have the ability to walk the land, and feel it, and close the loop. In the absence of information, we had to resort to monoculture. We closed the loop on an industrial scale. We went to predictability: If I use this Monsanto seed, spray it with this chemical, in this very well understood soil, I'm likely to get a pretty good outcome. In the absence of information, we went with predictability, simplicity, and a lot of chemicals.

- Closing the Loop (Chris Anderson)

We literally rationalised the shit out of the environment after we successfully formalised the inner workings of the consciousness. (We basically got super excited once we found out how to exert scalable control over the nature.)

Now we are living through another massive revolution. This time we have discovered the inner workings of the unconsciousness and it is a lot better at dealing with the sort of complexities and non-linearities exhibited by ecosystems. Intuitive control has finally become scalable.

The worst is behind us. Our relationship with the environment will slowly recover. Machine learning will allow control structures to stay beautiful and organic even as they scale.


Artificial intelligence (AI) is a misnomer. Intelligence is a notion we associate with consciousness (which works rationally / analytically), not unconsciousness (which works intuitively / statistically.)

This is not surprising, since the actual historical evolution of the idea of AI started off as an analytical project and ended up as a statistical one. (Analytical approach to automate human-level intelligence failed.)

This too is not surprising, since nature itself too invented unconsciousness first. (In other words, the straight-forward looking consciousness has to be deeper and more complex than unconsciousness.)

Side Note: Here, by unconsciousness, I am referring to the intuitional part of the reasoning mind, not to the lower level older brain parts that are responsible for things like the automation of bodily functions. Hence the remarks are not in contradiction with Moravec’s Paradox.

Notice that there is a strange alteration going on here. Human reason was built on top of human intuition. Now machine intuition is being built on top of (advanced) human reason. In the future, machine reason will be built on top of machine intuition.


The reason why our unconscious mind copes better with complexity has to do with its greater capacity to simultaneously deal with many variables. Rational mind on the other hand extracts heuristics and stories out of patterns. This process results in a drastic reduction of number of variables which in turn results in a loss of accuracy.

Take, for example, the problem of predicting the distribution of trees in a forest using only map data, such as elevation, slope, sunlight, and shade. Jock Blackard and Denis Dean did the original calculations in 1999, and left behind an enormous public database for other mathematicians to use. According to Vapnik, when computers were trained using 15,000 examples, their predictions were 85 to 87 percent accurate. Pretty good. But when they were fed more than 500,000 examples, they developed more complex rules for tree distribution and boosted the accuracy of their predictions to more than 98 percent.

“This means that a good decision rule is not a simple one, it cannot be described by a very few parameters,” Vapnik said. In fact, he argues that using many weak predictors will always be more accurate than using a few strong ones.

- Teaching Me Softly (Alan S. Brown)


What distinguishes excellent service (and maintenance) from very good service (and maintenance) is anticipatory power. Machine learning techniques are increasing our predictive capacity. We will soon be able to fix problems before they even occur. All problems, including those related to customer unhappiness and mechanical breakdowns, exhibit early signs which can be picked up by machine learning frameworks.

Life is about to become significantly happier and safer for our children.