From the Artificial Neuron to ChatGPT: Eighty Years of a Journey That Changed the World

The article traces the eighty-year history of artificial neural networks, from the McCulloch–Pitts mathematical model of the neuron (1943) through the perceptron, the AI winters, backpropagation, and convolutional and recurrent networks, all the way to the transformer architecture and the emergence of ChatGPT. It follows key milestones including AlexNet, Word2Vec, the attention mechanism, and RLHF, and shows how each breakthrough solved a specific technical limitation of the previous generation.

In the autumn of 2024, the Royal Swedish Academy of Sciences awarded the Nobel Prize in Physics to two men who had spent their careers working on something seemingly unrelated to physics — artificial neural networks. John Hopfield and Geoffrey Hinton received the award for "foundational discoveries that enable machine learning." Many in the community asked: why them? And why now?

The answer requires a journey back — eighty years — to the moment when a neurophysiologist and a logician first described mathematically how a machine might think.

In 1943, Warren McCulloch and Walter Pitts proposed a mathematical model of a neuron. A simple threshold unit: if the weighted sum of the inputs exceeds a boundary, the neuron fires. No learning, no adaptation — the weights were set by hand. Yet it was a fundamental idea: computation and thought can operate on the same principle.

Fifteen years later, Frank Rosenblatt added what the McCulloch–Pitts model lacked. His perceptron could learn from data. Using a simple rule, it updated its weights based on the errors it made. The U.S. Navy funded the research, and in 1958 The New York Times wrote of an "electronic brain." Rosenblatt predicted machines that would walk, talk, and be aware of their own existence.

The enthusiasm did not last. In 1969, Marvin Minsky and Seymour Papert published the book Perceptrons, in which they proved mathematically that a single-layer perceptron cannot handle even trivial tasks. It cannot learn XOR — a logical function any schoolchild understands. Their book was not the only reason, but it became a symbol of the turn: the British Lighthill report of 1973 and the subsequent cuts in American and British funding did the rest. The first "AI winter" set in.

While neural networks stagnated, attention shifted to expert systems — programs made of hand-coded rules of the type "if the patient has a fever and a headache, then consider meningitis." Stanford's MYCIN, work on which began in 1972, diagnosed bacterial infections on the basis of roughly five to six hundred such rules. In tests it achieved accuracy comparable to specialists in infectious diseases.

Expert systems looked like the future. Companies invested billions. But the rules were written by humans, and humans cannot capture the complexity of the real world. The systems were brittle — they worked in a narrow domain and failed outside it. By the late 1980s the market for expert systems had collapsed. The second AI winter.

The era of hand-coded knowledge was closed by the second AI winter. But even later, more sophisticated approaches ran into limits. IBM Watson demonstrated this most dramatically.

Watson was not an expert system — its DeepQA architecture combined natural language processing, information retrieval, and over fifty machine learning models. In 2011 it defeated human champions on the television quiz show Jeopardy!, and IBM tried to deploy it in oncology. Watson for Oncology was meant to analyze the medical literature and recommend cancer treatments. The result was unsatisfactory — the system could not reliably understand unstructured clinical documentation, required enormous manual curation for each hospital separately, and could not learn from local patient data. In 2022, IBM sold off the Watson Health division.

Watson's story illustrates a deeper truth: even a massive engineering approach with elements of machine learning is not enough if the system cannot itself extract patterns from data. That required a different path — a path back to neural networks.

In parallel with the collapse of expert systems, a revolution was quietly unfolding. The backpropagation algorithm — the back-propagation of error — had roots in Seppo Linnainmaa's automatic differentiation of 1970 and in the work of Paul Werbos of 1974. But it was the 1986 paper by David Rumelhart, Geoffrey Hinton, and Ronald Williams that showed backpropagation works in practice for training multilayer networks.

The idea is elegant. The network makes a prediction, compares it with the correct answer, computes the error, and propagates that error back through the layers — from output to input. In each layer it adjusts the weights so that next time it errs less. By repeating this over millions of examples, the network learns to recognize patterns that no human explicitly described to it.

This overturned Minsky's limitation. Multilayer networks could handle XOR and far more complex tasks. At the end of the 1980s, Yann LeCun designed convolutional neural networks for recognizing handwritten digits — the first architecture the U.S. Postal Service actually deployed in practice.

Despite partial successes, neural networks remained on the margins of the mainstream for two decades. This was not due to the community's prejudices alone — the hardware and the data were missing.

The breakthrough came in September 2012. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton entered a convolutional network called AlexNet in the ImageNet competition. The task was to recognize objects in millions of photographs. AlexNet crushed every other approach — its error rate was more than ten percentage points lower than that of the second-best solution. The key was two things that had not existed in the 1980s: NVIDIA graphics cards capable of massive parallel computation, and the ImageNet dataset with more than a million annotated images.

The consequences came at once. Large technology companies began investing massively in deep learning. Google bought startups focused on neural networks; Facebook founded an AI research lab led by Yann LeCun. Within two years, deep networks dominated image, speech, and text recognition. Hinton's student Sutskever became one of the key architects of the entire revolution to come.

Recognizing an image is one thing. Understanding a sentence is another — words arrive in sequence and meaning depends on context. For processing sequences, recurrent neural networks emerged. Their hidden state acts as memory: at each step the network takes a new input and combines it with what it remembers from previous steps.

The problem showed up when training on longer sequences. The gradients that drive learning either shrink exponentially or, conversely, explode as they pass through dozens of time steps. The network "forgets" distant dependencies, or the training becomes unstable.

The architectural solution came in 1997 from Sepp Hochreiter and Jürgen Schmidhuber — Long Short-Term Memory. LSTM introduced control "gates" that learn which information to keep and which to forget. For two decades it became the standard for machine translation, speech recognition, and text generation.

Around 2010, Tomáš Mikolov, a Czech researcher from the Brno University of Technology, showed that even simple recurrent networks without LSTM can be excellent language models — provided the problem of exploding gradients is solved. His solution was surprisingly straightforward: clipping gradients that exceed a threshold value. The technique known as gradient clipping stabilized training, and Mikolov's RNNLM outperformed, on a number of benchmarks, both classical statistical models and more complex architectures.

Three years later, by then at Google, Mikolov introduced Word2Vec — a method that assigns each word a dense vector in a mathematical space. Similar words have similar vectors, and what is more — the geometric relationships between the vectors reflect the semantic relationships between words. The example that became iconic: subtract the vector of the word "man" from the vector of the word "king," add the vector of the word "woman" — and you get a vector close to the word "queen."

Distributed representations of words had existed earlier. Yoshua Bengio proposed a neural probabilistic language model as early as 2003. Mikolov's innovation lay in efficiency — clever simplifications of the architecture and training tricks such as negative sampling made it possible to process billions of words on ordinary hardware. Word2Vec became the de facto standard in natural language processing and inspired a number of follow-up works — GloVe from Stanford, FastText from Facebook, on which Mikolov again collaborated.

Machine translation is among the oldest problems in artificial intelligence. And it was precisely this problem that became the catalyst for the breakthroughs that led to today's language models.

The team around Ilya Sutskever at Google proposed in 2014 an architecture in which one module encodes the input sentence into a vector and another generates the translation from it. It worked, but for longer sentences the quality dropped — the entire meaning had to pass through the bottleneck of a single vector.

The solution came that same year. Dzmitry Bahdanau proposed an attention mechanism — attention. When generating each word of the translation, the model "looks back" at the entire input sentence and learns which words to pay attention to. When translating the word "cat," it gives high weight to the English "cat." When generating a Czech case ending, it looks at the whole phrase. Simple, but transformative.

A team at Google Brain published a paper in 2017 whose title became a mantra of the entire field. They proposed an architecture called the transformer, based solely on the attention mechanism — without any recurrence.

How does the transformer work? Each token in the input sequence is transformed into three vectors: Query (what I am asking about), Key (what I identify myself by), and Value (what information I carry). The dot product of one token's Query with all the Keys determines which tokens to pay attention to. The result — a weighted sum of Values — combines information from the relevant parts of the text.

The key advantage: all tokens are processed at once, in parallel. Recurrent networks had to read word by word, like a person reading a book. The transformer sees the whole page at once. On graphics cards, optimized for parallel computation, this meant an order-of-magnitude speedup in training.

The original transformer was tested on machine translation — English–German, English–French. That it would become the universal architecture for all of artificial intelligence was something no one expected in 2017.

In 2018, OpenAI and Google showed almost simultaneously that pre-training a huge transformer on unlabeled text creates a model that understands language. GPT (Generative Pre-trained Transformer) predicted the next word in a text. BERT (Bidirectional Encoder Representations from Transformers) learned to fill in masked words from the rest of the sentence. Both approaches revolutionarily improved results across language-processing tasks.

Then came scaling. GPT-2 had 1.5 billion parameters. GPT-3, in 2020, already had 175 billion. What is a parameter? An adjustable numerical value — the weight of a connection between neurons or the bias of a neuron. The more parameters, the more patterns and relationships the model can capture. A model with 175 billion parameters needs hundreds of gigabytes of memory just to store the weights — and that does not yet count the memory needed for the text generation itself.

With size came capabilities that no one had planned. GPT-3 could solve tasks it had not been trained on — it was enough to give it a few examples in the prompt. It generated code, wrote essays, translated between languages. Researchers began calling these emergent capabilities — although the term is a matter of debate. Some scientists argue that the apparent emergence may be an artifact of the way performance is measured, rather than a genuine phase transition.

Predicting the next word is not in itself enough to make a model useful as an assistant. GPT-3 could continue a text, but it would just as willingly continue a toxic text as a useful one.

The solution became fine-tuning based on human feedback — RLHF (Reinforcement Learning from Human Feedback). People rated pairs of the model's responses — which is better, more accurate, safer. From these ratings a reward model was trained, against which the language model itself was then optimized. The key step was InstructGPT, a 2022 paper in which OpenAI showed that even a relatively small model fine-tuned with RLHF can, in users' eyes, outperform a hundred-times-larger un-tuned model. The result: a model that better follows instructions and responds in the way people prefer.

In November 2022, OpenAI made ChatGPT available — GPT-3.5 fine-tuned by this procedure, with a simple conversational interface. A hundred million users in two months. The moment when neural networks moved from academic papers into everyday life.

The year 2023 brought an avalanche. GPT-4 added image processing alongside text. Meta released LLaMA — open-source models that democratized access to the technology. Anthropic introduced Claude, with an emphasis on safety. Google answered with Gemini.

Today, at the beginning of 2026, attention is shifting from merely enlarging models to new capabilities. Reasoning models explicitly "think" before answering — they break complex problems down into steps. Agentic systems connect language models with tools — the model can search the web, run code, work with files. Quantization and distillation allow models to run on ordinary computers.

McCulloch and Pitts showed that a neuron can be described with mathematics. Rosenblatt taught it to learn. Minsky and Papert showed the limits and froze the research for a decade. Rumelhart, Hinton, and Williams broke through those limits with the back-propagation of error. Hochreiter and Schmidhuber taught networks to remember. Mikolov stabilized their training and gave words geometry. Krizhevsky, Sutskever, and Hinton proved on ImageNet that deep networks and GPUs change the rules of the game. Bahdanau taught models to pay attention to what matters. Vaswani and colleagues built the transformer out of it. OpenAI scaled it. And billions of people discovered that you can talk to a machine.

Each of these steps solved a specific technical problem of the previous generation. None of them was inevitable. And an open question remains: is the transformer the architecture that will lead us to artificial general intelligence, or just another stop on a road whose end we cannot yet see?

Methodological note

The conception, structure, and editorial line of the article are the work of the author, who prepared the content sketch, set out the key theses, and directed the entire creative process. Generative AI (Claude, Anthropic) was used as a technical tool for research, fact-checking, and fleshing out the author's outline.

The author edited the outputs on an ongoing basis, verified the key findings, and approved the final wording. No part of the text was published without human review. All factual data were verified against the publicly available sources cited in the text.

The procedure complies with the requirements of Article 50 of EU Regulation 2024/1689 (the AI Act) on transparency of AI-generated content. #poweredByAI