Bringing magic back to technology

Back in 2011, I was one of the discoverers of "Bytebeat", a type of very short computer programs that generate music. These programs received quite a lot of attention because they seem to be far too short for the complex musical structures they output. I wrote several technical articles about Bytebeat (arxiv, countercomplex 1, countercomplex 2) as well as a Finnish-language academic article about the social dynamics of the phenomenon. Those who just need a quick glance may want to check out one of the Youtube videos.

The popularity of Bytebeat can be partially explained with the concept of "hack value", especially in the context of Hakmem-style hacks -- very short programs that seem to outgrow their size. The Jargon File gives the following formal definition for "hack value" in the context of very short visual programs, display hacks:

"The hack value of a display hack is proportional to the esthetic value of the images times the cleverness of the algorithm divided by the size of the code."

Bytebeat programs apparently have a high hack value in this sense. The demoscene, being distinct from the MIT hacker lineage, does not really use the term "hack value". Still, its own ultra-compact artifacts (executables of 4096 bytes and less) are judged in a very similar manner. I might just replace "cleverness of the algorithm" with something like "freshness of the output compared to earlier work".
Another related hacker concept is "magic", which the Jargon File defines as follows:

1. adj. As yet unexplained, or too complicated to explain; compare automagically and (Arthur C.) Clarke's Third Law: "Any sufficiently advanced technology is indistinguishable from magic." "TTY echoing is controlled by a large number of magic bits." "This routine magically computes the parity of an 8-bit byte in three instructions." 

2. adj. Characteristic of something that works although no one really understands why (this is especially called black magic). 

3. n. [Stanford] A feature not generally publicized that allows something otherwise impossible, or a feature formerly in that category but now unveiled. 

4. n. The ultimate goal of all engineering & development, elegance in the extreme; from the first corollary to Clarke's Third Law: "Any technology distinguishable from magic is insufficiently advanced".

Short programs with a high hack value are magical especially in the first two senses. How and why Bytebeat programs work was often a mystery even to their discoverers. Even when some theory about them was devised, it was often quite difficult to understand or apply. Especially bitwise arithmetic tends to have very esoteric uses in Bytebeat.

The hacker definition of magic indirectly suggests that highly advanced and elegant engineering should be difficult to understand. Indecipherable program code has even been celebrated in contests such as IOCCC. This idea is highly countercultural. In mainstream software industry, clever hacks are despised: all code should be as easy as possible to understand and maintain. The mystical aspects of hacker subcultures are there to compensate for the dumb, odorless and dehumanizing qualities of the industrial chores.

Magic appears in the Jargon File in two ways. Terms such as "black magic", "voodoo programming" and "cargo cult programming" represent cases where the user doesn't know what they are doing or may not even strive to. Another aspect is exemplified by terms such as "deep magic" and "heavy wizardry": there, the technology may be difficult to understand or chaotic to control, but at least there are some talented individuals who have managed to. These aspects could be called "wild" and "domesticated", respectively, or alternatively "superstition" and "esoterica".

Most technology used to be magical in the wild/superstitious way. Cultural evolution does not require individual innovators to understand how their innovations work. Fermentation, for example, had been used for thousands of years without anyone having seen a micro-organism. Despite this, cultural evolution can find very good solutions if enough time is given: traditional craft designs often have a kind of optimality that is very difficult to attain from scratch even with the help of modern science. (See e.g. Robert Boyd et al.'s articles about cultural evolution of technology)

Science and technology have countless examples of "wild magic" getting "domesticated". An example from computer music is the Karplus-Strong string model. Earlier models of acoustic simulation had been constructed via rational analysis alone, so they were prohibitively expensive for real-time synthesis. Then, Karplus and Strong accidentally discovered a very resource-efficient model due to a software bug, and nowadays it is pretty standard textbook material without much magical glamor at all.

Magic and rationality support each other. In good technology, they would coexist in symbiosis. Industrialization, however, brought a cult of obsolescence that prevented this kind of relationship. Traditions, time-proven designs, intuitive understanding and irreducible wisdom started to get obsoleted by one-dimensional reductive analysis. Nowadays, "magic" is only tolerated as bursts of inspiration that must be captured within reductivist frameworks before they break something.

In the 20th century, utilitarian industrial engineering started to get obsoleted by its bastard offspring, tumorous engineering. This is what I discussed in my earlier essay "The resource leak bug of our civilization". Accumulation of bloat and complexity for their own sake is making technology increasingly difficult to rationally understand and control. In computing, where tumourous engineering dominates, designers are already longing back to utilitarian industry where simplicity, controllability, resource-efficiency and expertise were still valued.

When advocating the reintroduction of magic, one must be careful not to endorse the kind of superstitious thinking that already has a good hold on how people relate to technology. Devices that hide their internal logic and instead base their interfaces on guessing what the user wants are kind of Aladdin's lamps to most. You don't really understand how they work, but at least their spirits fulfill your wishes as long as you don't make them angry.

The way how magic manifests itself in traditional technology is diagonally opposite to this. The basic functional principles of a bow, a canoe or a violin can be learned via simple observation and experimentation. The mystery lies elsewhere: in the evolutionary design details that are difficult to rationally explain, in the outworldish talent and wisdom of the master crafter, in the superhuman excellence of the skilled user. If the design has been improved over generations, even minor improvements are difficult to do anymore, which gives it an aura of perfection.

The magic we need more in today's technological world is of the latter kind. We should strive to increase deepness rather than outward complexity, human virtuosity rather than consumerism, flexibility rather than effortlessness. The mysteries should invite attempts at understanding and exploitation rather than blind reliance or worship; this is also the key difference between esoterica and superstition.

One definition of magic, compatible with that in the Jargon File, is that it breaks people's preconceptions of what is possible. In order to challenge and ridicule today's technological bloat, we should particularly aim at discoveries that are "far too simple and random to work but still do". New ways to use and combine the available grassroots-level elements, for instance.

A Bytebeat formula is a simple arrangement of digital-arithmetic operations that have been elementary to computers since the very beginning. It is apparently something that should have been discovered decades ago, but it wasn't. Hakmem contains a few "sound hacks" that could have evolved into Bytebeat if a wide enough counter had been introduced into them, but there are no indications that this ever took place. It is mind-boggling to think about that the space of very short programs remains so uncharted that random excursions there can churn out new interesting structures even after seventy years.

Now consider that we are surrounded by millions of different natural "building blocks" such as plants, micro-organisms and geological materials. I honestly believe that, despite hundreds of thousands of years of cultural evolution, their combinatory space is nowhere near fully charted. For instance, it could be possible to find a rather simple and rudimentary technique that would make micro-organisms transform sand into a building material superior to everything we know today. A favorite fantasy scenario of mine is a small self-sufficient town that builds advanced spacecraft from scratch with "grassroots-level" techniques that seem magical to our eyes.

How to develop this kind of magic? Rational analysis and deterministic engineering will help us to some extent, but we are dealing with systems so chaotic and multidimensional that decades of random experimentation would be needed for many crucial leaps-forward. And we don't really have those decades if we want to beat our technological cancer.

Fortunately, the same Moore's law that empowers tumorous engineering also provides a way out. Computers make it possible to manage chaotic systems in ways other than neurotic modularization. Today's vast computational capacities can be used to simulate the technological trial-and-error of cultural evolution with various level of accuracy. Of course, simulations often fail, but at least they can give us a compass for real-world experimentation. Another important compass is "hack value" or "scientific intuition" -- the modern manifestations of the good old human sense of wonder that has been providing fitness estimations for cultural evolution since time immemorial.