Perceptron and Fully-connected Neural Networks

What it is

A perceptron is the simplest artificial neuron. It multiplies each input by a weight, adds a bias, and passes the sum through an activation function to give an output.

A fully-connected neural network (also called a multilayer perceptron, MLP) stacks layers of neurons, where every neuron in one layer connects to every neuron in the next. It is trained by repeating a forward pass, a loss calculation, backpropagation and a weight update.

How it works

The perceptron

The output is y = f(w1x1 + w2x2 + ... + b). Rosenblatt's original perceptron used a step function: output 1 if the sum is above 0, else 0.

A single perceptron draws one straight line (a hyperplane) between classes, so it solves only linearly separable problems. It can do AND and OR, but not XOR, which Minsky and Papert pointed out in 1969.

Adding a hidden layer with nonlinear activations solves XOR. That is why multilayer networks replaced the single perceptron.

Network structure

A layer with n inputs and m outputs has n x m weights plus m biases. For 784 inputs and 128 neurons: 784 x 128 + 128 = 100,480 parameters.

Activation functions

Without a nonlinear activation, stacked layers collapse into one linear function, so the network cannot learn complex patterns.

Softmax of logits [2, 1, 0]: e squared = 7.389, e = 2.718, 1, sum 11.107. The probabilities are 0.665, 0.245 and 0.090.

Loss functions

Training process

Backpropagation applies the chain rule from the output layer back to the input, giving the gradient of the loss for every weight. The optimizer then updates the weights. One pass over the whole training set is an epoch.

Optimizers

Vanishing and exploding gradients

In deep networks, gradients are multiplied layer by layer. With sigmoid, factors of at most 0.25 shrink gradients toward zero, so early layers barely learn (vanishing gradient). Large weights can make gradients grow out of control (exploding gradient).

Remedies include ReLU, careful weight initialization, batch normalization, residual connections and gradient clipping.

Regularization in deep learning

Dropout is active only in training. Calling model.eval() turns it off for inference.

Training in PyTorch

import torch
from torch import nn

model = nn.Sequential(nn.Linear(784, 128), nn.ReLU(), nn.Linear(128, 10))
loss_fn = nn.CrossEntropyLoss()          # takes raw scores (logits)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

for x, y in loader:                      # loader is a DataLoader
    optimizer.zero_grad()                # clear old gradients
    loss = loss_fn(model(x), y)          # forward pass and loss
    loss.backward()                      # backpropagation
    optimizer.step()                     # update weights

A worked example

Build AND with one perceptron: w1 = 1, w2 = 1, b = -1.5 and a step function. Input (1, 1) gives 1 + 1 - 1.5 = 0.5, above 0, so output 1.

Input (1, 0) gives -0.5 and input (0, 0) gives -1.5, both output 0. No single choice of w1, w2 and b can do the same for XOR, because its positive points (0, 1) and (1, 0) cannot be split from (0, 0) and (1, 1) by one line.

How the exam asks about this

• "Why can a single-layer perceptron not solve XOR?" → XOR is not linearly separable, and a perceptron draws only one linear boundary.
• "Which activation function is usually used in the output layer for multi-class classification?" → Softmax.
• "Which technique randomly deactivates neurons during training to reduce overfitting?" → Dropout.