Discovering the Unconscious Through Data Analysis
- Thilo Weber
- 3. März
- 17 Min. Lesezeit
Relationships in the Emergence of the Major Ideas in Art, Science, Philosophy, and Spirituality
In this article, I explore how major ideas in art, science, philosophy, and spirituality have emerged and influenced each other throughout history. Inspired by Carl Jung’s theories of the collective unconscious, I wanted to see if patterns in intellectual history could reveal something about the hidden structures shaping human thought. By analyzing historical events and their relationships over time, I discovered some interesting correlations between these four fields. The analysis identifies the Axial Age (around 500 BCE) — a time when profound cultural shifts happened simultaneously across different civilizations, sparking an explosion of ideas in all four domains — as well as a similar rise in cultural development since 1500 CE. Beyond that, the results suggest that art is primarily an inspirational force, often preceding breakthroughs in other fields. Science and spirituality, on the other hand, seem to be both inspirational and analytical, sometimes leading, sometimes following other developments. Meanwhile, philosophy appears to be the most retrospective and analytical, typically absorbing and structuring ideas rather than generating them first. These insights resonate with Jung’s idea that the unconscious mind influences culture in hidden yet meaningful ways. While this analysis has its limitations, it offers a first glimpse into how human creativity and rationality interact over time, shaping the evolution of thought in ways we may not always consciously realize. The code is published on GitHub.
Introduction to the Unconscious
Reading several of Carl Jung’s books over the past year has opened up a whole new set of questions for me about our world and our minds. For example, what is this mind we walk around with every day really like? And how does it interact with the world?
As I understand it, Jung suggests that mind and matter are actually both part of one great dynamic process, and both follow a trajectory of evolution. Of this process, what we call matter is mostly perceived as something common to everyone, while mind is mostly perceived as something personal. Somewhere in between, Jung suggests the process of a “collective mind,” a mind that gradually extends the personal mind first to a smaller group of people, then to an entire nation, to all of humanity, to all living beings, and finally to what we call the material world. This collective mind, so to speak the whole spectrum of transmission processes between the two poles of the world and our own personal mind, is often unconscious to us, which is why Jung refers to it as the “collective unconscious”.
The unconscious, consisting of both collective and personal parts, is probably more than just the transmission medium between individual minds and the common world. As part of the whole body-mind continuum, it has its own dynamics and exerts its own influence on the world and individual minds. Bringing more knowledge, awareness, and understanding to the dynamics of the unconscious was therefore one of Jung’s greatest missions. For ignorance of the underlying processes of life leads to uncertainty and poses a danger to everyone involved.
Jung discovered various gateways to explore the unconscious mind, particularly through the analysis of dreams, myths, religious symbols, and works of art. Learning about his work opened up a whole new understanding of the creative process. Dreams and myths can be seen as unconscious processes that emerge into consciousness in the abstract forms of images and language. Art, on the other hand, can be seen as a way for the unconscious to manifest directly into the material world, in other words, to appear into consciousness through material manifestation.
Despite their different ways of appearing in consciousness, dreams, myths, and works of art all contain an element of primary inspiration, that is, direct messages from the unconscious. Analysis and rational thought, on the other hand, are always secondary processes that use the symbols produced by various forms of primary inspiration and attempt to discover some sort of structure within them or relationships between them. In particular, the main tools of analysis and rational thought, namely the grammar and semantics of language, logic, mathematics, geometry, algebra, and so on, are themselves symbols that emerged from the unconscious through direct inspiration at some point.
Unconscious processes emerge as symbols either in an individual mind — in which case we tend to speak of the person associated with the mind as a genius — or in the collective mind of a group, organization or society, or through material expressions in art or labor, or even through changes in the natural world.
In this light, we can think, for example, of Charles Darwin’s discovery of the evolution of species as a structure in the unconscious mind, which he discovered through a careful and comprehensive analysis of the relationships between living organisms. Jung discovered similar evolutionary processes by analyzing the myths and stories of different cultures. One of the most prominent archetypal patterns Jung identified is the hero’s journey, which represents the universal process of transformation, growth, and self-discovery, and appears in various mythologies of many cultures, as well as in modern stories such as Star Wars and The Lord of the Rings (see my previous blog for my personal interpretation of Frodo’s quest to destroy the ring).
As a trained data scientist, these discoveries by Darwin and Jung make me wonder if there might be a great deal of unconscious structure yet to be discovered in the vast amount of data that humanity has created throughout history, the amount of which has only begun to explode in recent years. So for this article, my idea is to do a little data analysis to explore the relationships in the major ideas in the history of the four fields of art, science, philosophy, and spirituality. My hope is to find some interesting patterns that might reveal some structure among these fields and inspire ideas for further analysis.
Data Analysis
My idea for a data analysis that will hopefully reveal something about the structure of the unconscious is to compare the relationships in the emergence of major ideas in history. My assumption is that the collective unconscious reveals itself primarily through the four major fields of art, science, philosophy, and spirituality, and that revelations in each field correlate with revelations in the other fields.
My hypothesis is that there is a general order in which ideas occur within these fields, a chain of inspiration so to speak. In particular, I imagine that this chain might look like this:
In this graph, art is the main field of inspiration, capturing new ideas and expressing them in a first intuitive physical form. Science takes these ideas and brings them into a well-structured and rational form. Philosophy abstracts the ideas of science and brings them into a unified theory of reality. Spirituality connects the understanding of philosophy back to the unconscious and serves as a source of inspiration for art.
To test this hypothesis, I designed a data analysis with the following four steps:
Creating Dataset: Generate four datasets of all major ideas in the four fields, art, science, philosophy, and spirituality, using ChatGPT.
Years Transformation: Fitting an exponential function to the cumulative sum of all events that is used to transform the years in order to get a more uniform distributions of events over years.
Smoothed Event Signals: Convert the events streams to four time series spike trains and smooth the trains with a Gaussian window.
Calculate Time Shifts With Cross-correlation: Calculate the time shift for each pair of smoothed spike trains by calculating the cross-correlation between each two signals.
According to my hypothesis shown in the graph above, I would expect positive time shifts from art to science, from science to philosophy, from philosophy to spirituality, and again from spirituality back to art.
1. Creating Dataset
I asked ChatGPT to create four datasets for each of the four fields with the following prompt:
Create a table of all major new ideas and influential individuals along with a year column and a weight column that weights the impact between 1 and 10, and the continent on which this idea emerged, in the field of <FIELD> between the years -5000 to 2000. The year column should be a positive or negative integer, where years BCE are negative numbers. Output only the table, no additional text.The full dataset is shown at the end of this article.
2. Years Transformation
The following plot shows the cumulative sum over all weighted events of major new ideas, as returned by ChatGPT, along with an exponential function fitted to the cumulative sum:
The increasing slope of the cumulative sum indicates that the rate of important events increases approximately exponentially over time. The exponential increase can be explained in part by the fact that we have much more detailed information about more recent history and tend to make more detailed categories of recent events, while we necessarily have to lump ancient history into much larger categories. But the increase also shows an acceleration in human history, where new ideas propagate faster and faster through time and space, fostering again the inspiration of new ideas at an ever-increasing rate.
It is interesting to note the steep rise of many new ideas around 500 BCE with the advent of Taoism and Confucianism in ancient China, Upanishadic thought, Jainism and Buddhism in ancient India, and ancient Greek culture in Europe. Karl Jaspers introduced the term Axial Age to describe this span of time in which cultural changes occurred in several different cultures and places around the world, resulting in a simultaneous increase of new ideas in all four fields of art, science, spirituality, and philosophy. The Axial Age is followed by a long period of rather little innovation, with the only advances occurring mostly in the Middle East due to early Christianity, Islam, and Arabic mathematics. Since the 15th century there has been another steep rise, beginning with the Scientific Revolution, Baroque and Rationalism in Europe, and in the 20th century more and more in North America.
For further analysis, I transform the years using the fitted exponential function. This transforms, for example, the year 0 CE into a transformed year of 30 and the year 1000 CE into a transformed year of about 50. With this transformation, the events are spread fairly evenly throughout the transformed domain. This technique for transforming years is inspired by histogram equalization used in image processing.
3. Smoothed Event Signals
The following plot shows the event signals for the four fields of art, science, philosophy, and spirituality, along with text annotations indicating some of the events along the timeline:
The x-axis is equidistant in the transformed time domain. Hence, recent years are spread out while earlier years are compressed. I smoothed the event spikes with a Gaussian kernel, which visually groups close spikes into larger pulses and helps to produce smooth cross-correlations, which are used to quantify the temporal relationships between the four signals, as described in the next section.
4. Calculate Time Shifts With Cross-correlation
To examine potential temporal dependencies between art, science, philosophy, and spirituality, I calculated the time shifts between each pair of smoothed event signals using cross-correlation. Cross-correlation measures the similarity between two time series as one is shifted relative to the other, allowing us to identify potential lead-lag relationships.For each pair of fields (e.g., art vs. science, science vs. philosophy), I computed the cross-correlation function and identified the time shift at which the correlation reaches its maximum. A positive time shift indicates that the first field tends to precede the second, while a negative time shift suggests the opposite.
The following plot shows all pairs of cross-correlations for the full history of (in blue) and for the history since 1500 CE (in orange). The number in parentheses in each subplot title indicates the time shift in the transformed year domain between the first and second field, averaged over the 20 percent largest values of both cross-correlations (represented by the transparent fields in the plots):
In general, we can see that there is some correlation between all pairs of signals from the fact that all cross-correlations have a maximum peak close zero and are not just flat. However, the maximum peaks are not always centered exactly at zero — which would imply a general co-occurrence of events — but are often shifted slightly to the right — indicating that there seems to be a tendency for events to occur in a particular order. For example, events in philosophy have a clear tendency to occur about 2.8 transformed years after an event in art.
The transformed time shifts correspond to different yearly shifts in different years. As the following graph shows, a time shift of 2.8 transformed years corresponds to 66 years around 2000 CE, 180 years around 0 CE, and 452 years around 2000 BCE:
The resulting time shifts obtained from the various cross-correlations don’t support my hypothesis of a “circular chain of inspiration” as depicted in the graph at the beginning of this article. Rather, they suggest the following structure in the subsequent events between the different fields:
New events in art and spirituality show a strong correlation in synchronous occurrence with almost no time shifts. Events in science show a correlation peak that is synchronous with all three other fields-spirituality, art, and philosophy—but then another peak that suggests that some science events tend to precede the other fields, resulting in an average lead of science over all other fields.
Probably the most significant finding is that philosophy tends to lag considerably behind the other three fields — although it also has a synchronous peak without time shift with spirituality and science, respectively.
So the overall — admittedly rather speculative — interpretation of the correlation plots could be as follows: Philosophy is a purely analytical field, as shown by the fact that it does not precede any of the other fields. It only analyzes the results of the other fields after the fact. Science and spirituality seem to have an inspirational component — correlation spikes that precede philosophy — as well as an analytical component — correlation spikes that occur synchronously with philosophy. Art, on the other hand, seems to be a purely inspirational field that doesn’t show any synchronous correlation with philosophy. However, there is also a small “pre-peak” from science to art, suggesting that art may be partly inspired by new ideas in science.
Limitations and Conclusion
The presented analysis includes a number of heuristic steps, all of which affect the final result — such as the exponential transformation of the years, the shape and size of the smoothing kernel, and the calculation of the average time shift. Furthermore, the selection and weighing of important new ideas is entirely delegated to ChatGPT, and there may be a bias in the selection.
And the greatest speculation, of course, is whether these cross-correlations reveal anything at all about the mutual influence between these fields, as well as their inspirational connection to the unconscious.
In particular, this analysis ignores the aspect of space and correlates events that happened in different regions of the world. In our everyday understanding, we think of such events as uncorrelated, especially for earlier stages of history where different cultures are often thought of as independent entities. However, my assumption here is that such seemingly independent cultures may still have had correlations with each other that originated in the collective unconscious. Examples of fundamental new ideas emerging simultaneously in different cultures around the world — such as the new ideas of the Axial Age, or early alchemy emerging “independently” in ancient China and ancient Egypt — seem to at least support the possibility of some kind of connection across cultures.
Despite these limitations, the presented analysis provides a first idea of how data analysis can help to discover hidden structures and patterns in the collective unconscious that shape human thought across time and space. Frankly, I quite like the findings that art is primarily inspirational, science and spirituality are partially inspirational and partially analytical, and philosophy is primarily retrospective and analytical.
May this analysis be a source of inspiration for more fascinating thoughts to come.
Full Dataset
+--------+---------+----------------------------------------+-------------------------------------------------------+----------------------+----------+| Year | Field | Idea/Movement | Influential Individuals | Continent | Weight ||--------+---------+----------------------------------------+-------------------------------------------------------+----------------------+----------|| -4000 | philo | Proto-Religious Thought | Early Mesopotamian and Egyptian thinkers | Middle East | 7 || -4000 | science | Early Metallurgy | Unnamed Sumerian and Egyptian metalworkers | Middle East | 7 || -3500 | science | Writing Systems | Sumerians, Ancient Egyptians | Middle East | 8 || -3000 | science | Early Astronomy | Mesopotamian and Egyptian astronomers | Middle East | 8 || -3000 | philo | Ancient Chinese Philosophy | Unnamed early Chinese sages | Asia | 7 || -2600 | philo | Egyptian Moral Philosophy | Ptahhotep | Africa | 7 || -2600 | science | Medicine in Egypt | Imhotep | Africa | 8 || -2000 | philo | Vedic Philosophy | Vedic Rishis | Asia | 8 || -2000 | science | Babylonian Mathematics | Unnamed Babylonian scholars | Middle East | 8 || -2000 | spirit | Vedic Religion | Vedic Rishis | Asia | 9 || -2000 | art | Ancient Egyptian Art | Imhotep, Thutmose, Senenmut | Africa | 9 || -1800 | spirit | Zoroastrianism | Zoroaster | Asia | 9 || -1600 | art | Minoan Art | Unnamed fresco painters of Knossos | Europe | 8 || -1600 | philo | Babylonian Ethics | Hammurabi | Middle East | 8 || -1600 | science | Egyptian Medicine | Unnamed Egyptian physicians | Africa | 7 || -1500 | science | Early Chemistry (Alchemy) | Chinese and Egyptian alchemists | Asia, Africa | 7 || -1500 | spirit | Ancient Egyptian Religion | High Priests of Amun | Africa | 8 || -1200 | philo | Hebrew Monotheistic Ethics | Moses | Middle East | 9 || -1200 | science | Iron Smelting | Hittite Metallurgists | Middle East | 8 || -1200 | spirit | Hebrew Monotheism | Moses | Middle East | 10 || -1100 | art | Ancient Chinese Art | Unnamed artisans of the Shang Dynasty | Asia | 9 || -800 | science | Greek Natural Philosophy | Thales, Anaximander, Heraclitus | Europe | 9 || -800 | art | Ancient Greek Art | Phidias, Polykleitos, Praxiteles | Europe | 9 || -800 | spirit | Taoism | Laozi | Asia | 9 || -800 | philo | Upanishadic Thought | Early Hindu sages | Asia | 9 || -600 | philo | Taoism | Laozi | Asia | 9 || -600 | philo | Jain Philosophy | Mahavira | Asia | 9 || -600 | science | Early Atomic Theory | Leucippus, Democritus | Europe | 8 || -600 | spirit | Jainism | Mahavira | Asia | 9 || -563 | spirit | Buddhism | Siddhartha Gautama (Buddha) | Asia | 10 || -550 | spirit | Confucianism | Confucius | Asia | 9 || -500 | science | Hippocratic Medicine | Hippocrates | Europe | 9 || -500 | art | Ancient Indian Art | Unnamed sculptors of the Maurya Empire | Asia | 9 || -500 | philo | Greek Philosophy | Socrates, Plato, Aristotle | Europe | 10 || -500 | spirit | Greek Philosophy & Spirituality | Pythagoras, Socrates, Plato, Aristotle | Europe | 9 || -500 | philo | Confucianism | Confucius | Asia | 9 || -400 | philo | Buddhist Philosophy | Siddhartha Gautama (Buddha) | Asia | 10 || -300 | science | Euclidean Geometry | Euclid | Europe | 10 || -300 | philo | Stoicism | Zeno of Citium | Europe | 9 || -250 | science | Archimedean Mechanics | Archimedes | Europe | 10 || -200 | art | Ancient Roman Art | Apollodorus of Damascus, Unnamed mosaic artists | Europe | 8 || -100 | philo | Roman Philosophy | Cicero, Seneca, Marcus Aurelius | Europe | 8 || 30 | spirit | Christianity | Jesus Christ | Middle East | 10 || 150 | science | Ptolemaic Astronomy | Claudius Ptolemy | Europe | 9 || 200 | philo | Neoplatonism | Plotinus | Europe | 8 || 250 | spirit | Mahayana Buddhism | Nagarjuna, Ashvaghosha | Asia | 9 || 300 | art | Byzantine Art | Unnamed icon painters, mosaicists | Europe/Asia | 9 || 500 | philo | Christian Philosophy | Augustine of Hippo | Europe | 9 || 570 | spirit | Islam | Muhammad | Middle East | 10 || 700 | art | Islamic Art | Calligraphers and architects of the Umayyad Caliph... | Middle East | 9 || 800 | philo | Islamic Philosophy | Al-Farabi, Avicenna, Al-Ghazali | Middle East | 9 || 800 | spirit | Vajrayana Buddhism | Padmasambhava | Asia | 8 || 800 | science | Algebra | Al-Khwarizmi | Middle East | 9 || 1000 | art | Romanesque Art | Unnamed cathedral sculptors and architects | Europe | 8 || 1000 | science | Optics | Ibn al-Haytham | Middle East | 9 || 1100 | spirit | Sufism | Rumi, Al-Ghazali | Middle East | 9 || 1100 | science | Early Medicine | Avicenna | Middle East | 9 || 1150 | art | Gothic Art | Giotto di Bondone, Abbot Suger | Europe | 9 || 1200 | philo | Scholasticism | Thomas Aquinas | Europe | 9 || 1200 | science | Scientific Method (Islamic) | Alhazen, Al-Biruni | Middle East | 9 || 1300 | art | Renaissance | Leonardo da Vinci, Michelangelo Buonarroti, Raphae... | Europe | 10 || 1400 | philo | Renaissance Humanism | Erasmus, Pico della Mirandola | Europe | 9 || 1469 | spirit | Sikhism | Guru Nanak | Asia | 9 || 1543 | science | Heliocentric Theory | Nicolaus Copernicus | Europe | 10 || 1600 | science | Modern Scientific Method | Francis Bacon, Galileo Galilei | Europe | 10 || 1600 | spirit | Christian Mysticism | St. John of the Cross, Teresa of Ávila | Europe | 8 || 1600 | art | Baroque | Caravaggio, Rembrandt van Rijn, Peter Paul Rubens | Europe | 9 || 1600 | philo | Rationalism | René Descartes, Baruch Spinoza, Leibniz | Europe | 10 || 1687 | science | Classical Mechanics | Isaac Newton | Europe | 10 || 1700 | art | Rococo | Jean-Antoine Watteau, François Boucher, Jean-Honor... | Europe | 8 || 1700 | philo | Empiricism | John Locke, David Hume | Europe | 10 || 1750 | art | Neoclassicism | Jacques-Louis David, Antonio Canova, Jean-Auguste-... | Europe | 8 || 1750 | science | Electricity Studies | Benjamin Franklin | North America | 9 || 1780 | spirit | Enlightenment Spirituality | Immanuel Kant, Voltaire | Europe | 9 || 1780 | philo | German Idealism | Immanuel Kant, Georg Hegel | Europe | 10 || 1800 | philo | Utilitarianism | Jeremy Bentham, John Stuart Mill | Europe | 9 || 1800 | art | Romanticism | Francisco Goya, J.M.W. Turner, Caspar David Friedr... | Europe | 9 || 1820 | science | Electromagnetism | André-Marie Ampère, Michael Faraday | Europe | 10 || 1840 | philo | Existentialism | Søren Kierkegaard | Europe | 9 || 1850 | philo | Marxism | Karl Marx, Friedrich Engels | Europe | 10 || 1850 | art | Realism | Gustave Courbet, Jean-François Millet, Honoré Daum... | Europe | 8 || 1859 | science | Evolution by Natural Selection | Charles Darwin | Europe | 10 || 1870 | art | Impressionism | Claude Monet, Édouard Manet, Pierre-Auguste Renoir | Europe | 10 || 1875 | spirit | Theosophy | Helena Blavatsky, Henry Steel Olcott | Europe/N. America | 8 || 1880 | art | Post-Impressionism | Vincent van Gogh, Paul Cézanne, Georges Seurat | Europe | 10 || 1893 | spirit | Modern Hindu Revival | Swami Vivekananda | Asia | 9 || 1900 | art | Expressionism | Edvard Munch, Wassily Kandinsky, Egon Schiele | Europe | 9 || 1900 | philo | Analytic Philosophy | Bertrand Russell, Ludwig Wittgenstein | Europe | 10 || 1905 | science | Special Relativity | Albert Einstein | Europe | 10 || 1910 | art | Cubism | Pablo Picasso, Georges Braque, Juan Gris | Europe | 10 || 1920 | spirit | New Thought Movement | Ernest Holmes, Emmet Fox | N. America | 8 || 1920 | philo | Phenomenology | Edmund Husserl, Martin Heidegger | Europe | 9 || 1920 | art | Surrealism | Salvador Dalí, André Breton, Max Ernst | Europe | 9 || 1927 | science | Quantum Mechanics | Werner Heisenberg, Erwin Schrödinger | Europe | 10 || 1940 | art | Abstract Expressionism | Jackson Pollock, Willem de Kooning, Mark Rothko | North America | 10 || 1950 | philo | Postmodernism | Michel Foucault, Jacques Derrida | Europe | 9 || 1950 | spirit | Zen & Eastern Spirituality in the West | D.T. Suzuki, Alan Watts | N. America/Europe | 9 || 1950 | art | Pop Art | Andy Warhol, Roy Lichtenstein, Richard Hamilton | North America | 10 || 1953 | science | DNA Structure | James Watson, Francis Crick, Rosalind Franklin | Europe/N. America | 10 || 1960 | spirit | New Age Movement | Various spiritual leaders | N. America | 8 || 1960 | art | Minimalism | Donald Judd, Agnes Martin, Frank Stella | North America | 8 || 1969 | science | Moon Landing | NASA Apollo 11 team | North America | 10 || 1970 | art | Conceptual Art | Sol LeWitt, Joseph Kosuth, Lawrence Weiner | North America | 8 || 1980 | science | Personal Computing | Steve Jobs, Bill Gates | North America | 9 || 1980 | art | Street Art | Jean-Michel Basquiat, Keith Haring, Banksy | North America/Europe | 9 || 1990 | art | Contemporary Art | Damien Hirst, Jeff Koons, Yayoi Kusama | Europe/Asia | 9 || 1990 | spirit | Mindfulness Movement | Jon Kabat-Zinn | N. America | 9 || 2000 | science | Human Genome Project | International Team | Global | 10 |+--------+---------+----------------------------------------+-------------------------------------------------------+----------------------+----------+
Kommentare