But what is a neural network? | Deep learning chapter 1

The MNIST dataset contains handwritten digits rendered at 28x28 pixel resolution, serving as a classic introductory problem for machine learning. Researchers and students worldwide use this dataset to learn neural network concepts and test algorithms.

Artificial neurons serve as the fundamental computational units in neural networks, each holding a single numerical value called its activation. In the MNIST example, 784 input neurons each represent one pixel’s grayscale value.

Neural networks organize computational units called neurons into distinct layers that transform input data into meaningful outputs. The MNIST digit recognition network exemplifies this structure with 784 input neurons, two hidden layers of 16 neurons each, and 10 output neurons.

The feed-forward process defines how neural networks transform input data into output predictions by propagating activations sequentially through layers. A trained network performs this process to classify images, recognize speech, or generate predictions.

Layered neural networks build progressively abstract representations by combining simple features into complex patterns across successive layers. The MNIST network exemplifies this with layers potentially learning edges, then shapes, then complete digits.

Neurons in early hidden layers can learn to detect simple visual features like edges through appropriate weight configurations. The second layer of the MNIST network potentially contains neurons specialized for recognizing edges at various positions and orientations.

Each connection between neurons in adjacent layers carries a weight parameter that determines how strongly one neuron’s activation influences another. A single neuron in the second layer connects to all 784 first-layer neurons, each connection having its own weight.

The sigmoid function, also called the logistic curve, serves as a mathematical transformation applied to weighted sums in neural networks. Early neural networks predominantly used sigmoid activation, though modern architectures favor alternatives.

Each neuron in hidden and output layers possesses a bias parameter that shifts the activation threshold. The bias acts as an additional learnable parameter independent of incoming connections.

Neural networks contain thousands or millions of learnable parameters—weights and biases—that training algorithms adjust to solve specific tasks. The MNIST example network has approximately 13,000 total parameters requiring optimization.

Neural network computations express layer transitions through matrix-vector multiplication combined with activation functions. Machine learning libraries optimize these linear algebra operations for computational efficiency.

The Rectified Linear Unit (ReLU) functions as the dominant activation function in modern deep neural networks, replacing sigmoid in most architectures. Researchers adopted ReLU after discovering it makes training deep networks significantly easier.