Neural Networks
The Artificial Neuron
The fundamental unit. A single neuron:
- Takes inputs (numbers)
- Multiplies each by a weight (learned importance)
- Sums everything up
- Adds a bias term
- Passes through an activation function
- Outputs a number
output = activation(w₁x₁ + w₂x₂ + ... + wₙxₙ + bias)
Activation functions introduce non-linearity — without them, stacking layers would just be matrix multiplication and the network couldn't learn complex patterns.
| Function | Formula | Used In |
|---|---|---|
| ReLU | max(0, x) | Hidden layers (most common) |
| Sigmoid | 1 / (1 + e^(-x)) | Binary classification output |
| Softmax | e^(xᵢ) / Σe^(xⱼ) | Multi-class output, attention |
| GELU | x * Φ(x) | Transformer hidden layers |
Network Architecture
Neurons are organized in layers:
- Input layer — raw data enters here
- Hidden layers — where pattern extraction happens
- Output layer — the final prediction
Every neuron in one layer connects to every neuron in the next — this is a fully connected (dense) network.
How Depth Creates Abstraction
Early layers learn simple features. Deeper layers compose them:
| Layer Depth | What It Learns (Vision) | What It Learns (Language) |
|---|---|---|
| Layer 1-2 | Edges, gradients | Character patterns, common bigrams |
| Layer 3-5 | Textures, shapes | Word boundaries, basic syntax |
| Layer 6-10 | Object parts (eyes, wheels) | Phrases, grammar rules |
| Layer 10+ | Full objects, scenes | Semantics, reasoning, context |
This hierarchical feature extraction is why deep networks work and shallow ones don't for complex tasks.
The Training Loop
- Forward pass — data flows through, network produces prediction
- Loss calculation — compare prediction to ground truth
- Backpropagation — calculate gradient of loss with respect to each weight
- Weight update — adjust weights using gradient descent
new_weight = old_weight - learning_rate × gradient
The learning rate controls step size. Too large = overshoot. Too small = never converge. This is a critical hyperparameter.
Security Implications
- Weights are the model — stealing weights = stealing the model (model extraction)
- Gradients leak information — gradient-based attacks can reconstruct training data
- Activation patterns are exploitable — adversarial inputs manipulate specific neurons
- The loss landscape has local minima — models can be pushed into bad regions via data poisoning