ChatGPT is made from 100 million of these [The Perceptron]

Frank Rosenblatt, a psychologist, invented the perceptron in 1957 and unveiled it at a press conference on July 7th, 1958.

Frank Rosenblatt discovered the perceptron learning rule in 1957, providing a simple procedure guaranteed to find a solution when one exists. This breakthrough enabled machines to learn automatically from examples not being explicitly programmed.

The perceptron architecture uses weighted inputs combined with a bias term, a mathematical structure that enables flexible linear classification. This design translates biological neural concepts into computable mathematical operations.

Albert Novikov proved mathematically in 1962 that the perceptron learning rule is guaranteed to find solutions, but only when patterns are linearly separable. This formalized the perceptron’s fundamental geometric limitation.

The exclusive OR (XOR) problem became a major criticism of Rosenblatt’s perceptron and early neural networks, representing the simplest pattern that single-layer perceptrons fundamentally cannot learn. This limitation nearly halted neural network research.

Frank Rosenblatt himself recognized that multi-layer architectures could solve the linear separability problem. Before his death in 1971, he worked on multi-layer networks closely resembling modern neural networks. However, finding a training algorithm remained elusive for decades.

Marvin Minsky and Seymour Papert’s critical 1969 book “Perceptrons” contributed to neural network research decline. The field revived dramatically with Rumelhart, Hinton, and Williams’ 1986 backpropagation paper, ultimately leading to today’s AI boom.

Bernard Widrow and graduate student Ted Hoff discovered Least Mean Squares (LMS) algorithm Friday afternoon, fall 1959, at Stanford. They built circuit over weekend, successfully testing by Sunday evening.

Gradient descent emerged as general optimization method, applied to neural network training by Widrow and Hoff in 1959, using calculus to compute exact gradients.

Walter Pitts and Warren McCulloch developed the first mathematical model of an artificial neuron in the 1940s, translating biological neural behavior into computational logic. Their model became the foundation for all subsequent artificial neural networks, including the perceptron.

Walter Pitts and Warren McCulloch’s 1940s artificial neuron model used binary step activation functions—neurons either fire (output 1) or don’t (output 0), creating critical mathematical problem for multi-layer learning.

David Rumelhart, Geoffrey Hinton, and Ronald Williams published the modern backpropagation algorithm in 1986—27 years after Widrow and Hoff discovered the LMS algorithm. Their paper explicitly cited LMS as the “Delta Rule” and derived backpropagation as its generalization using the chain rule with sigmoid activations.

OpenAI trained GPT-3 in 2020 using backpropagation to optimize 175 billion learnable weights across 96 layers. GPT-4, reportedly 10 times larger, contains approximately 100 million artificial neurons—staggering scale compared to Rosenblatt’s original single perceptron.