Understanding Transformer Architectures

AI-generated content

The transformer architecture has fundamentally changed how we approach machine learning. From GPT to BERT to vision models, transformers are everywhere. But how do they actually work?

The Core Idea

Traditional sequence models (RNNs, LSTMs) process data one token at a time. This sequential nature limits parallelization and makes it hard to capture long-range dependencies.

Transformers solve both problems with self-attention: a mechanism that lets each token attend to every other token in the sequence simultaneously.

Self-Attention Explained

Given an input sequence, transformers compute three vectors for each token:

• Query (Q): What am I looking for?
• Key (K): What do I contain?
• Value (V): What should I provide?

The attention score between tokens is computed as:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

The scaling factor $\sqrt{d_k}$ prevents dot products from growing too large, which would push softmax into regions with tiny gradients.

Multi-Head Attention

Instead of computing attention once, transformers use multiple “heads” — different Q, K, V projections that capture different relationships:

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, num_heads):
        super().__init__()
        self.num_heads = num_heads
        self.d_k = d_model // num_heads
        
        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)
    
    def forward(self, x):
        # x shape: (batch, seq_len, d_model)
        Q = self.W_q(x).view(batch, -1, self.num_heads, self.d_k).transpose(1, 2)
        K = self.W_k(x).view(batch, -1, self.num_heads, self.d_k).transpose(1, 2)
        V = self.W_v(x).view(batch, -1, self.num_heads, self.d_k).transpose(1, 2)
        
        scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(self.d_k)
        attn = F.softmax(scores, dim=-1)
        
        out = torch.matmul(attn, V)
        out = out.transpose(1, 2).contiguous().view(batch, -1, d_model)
        return self.W_o(out)

Positional Encoding

Since transformers don’t process sequences sequentially, they need explicit position information. The original paper uses sinusoidal functions:

$$PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d_{model}})$$ $$PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i/d_{model}})$$

Modern models often use learned positional embeddings or RoPE (Rotary Position Embeddings) for better extrapolation.

Key Architectural Innovations

GPT Family (Decoder-only)

• Causal masking: Each position can only attend to previous positions
• Autoregressive generation: Predict one token at a time
• Used for: Text generation, code generation, chat

BERT Family (Encoder-only)

• Bidirectional attention: All positions attend to all others
• Masked language modeling: Train by masking random tokens
• Used for: Classification, NER, question answering

Encoder-Decoder (Sequence-to-sequence)

• Cross-attention: Encoder outputs serve as keys/values for decoder
• Used for: Machine translation, summarization

Modern Advances

Mixture of Experts (MoE)

Instead of a single feed-forward network, MoE routes each token to a subset of experts:

• Sparse activation: Only 1-2 experts active per token
• Massive scale: Mixtral 8x22B has 141B params but only activates 47B
• Linear scaling: More experts = more capacity without proportional compute

State Space Models (SSMs)

Models like Mamba offer linear-time sequence processing:

• Selective scan: Content-aware filtering of information
• Hardware-aware: Better GPU utilization than attention
• Long context: Naturally handles very long sequences

Flash Attention

An algorithmic optimization that reduces attention from $O(n^2)$ memory to $O(n)$:

Traditional:    Q @ K.T @ V  →  O(n²) intermediate
Flash Attention: Tiled computation  →  O(n) memory

This enables training much longer contexts on the same hardware.

Practical Considerations

Training Data

• Scaling laws: Loss follows a power law with compute, data, and parameters
• Data quality > quantity: Curated data often outperforms raw web scrape
• Deduplication: Remove near-duplicates to improve generalization

Inference Optimization

Context Windows

Modern models support increasingly large contexts:

• GPT-4: 128K tokens
• Claude 3.5: 200K tokens
• Gemini 1.5: 2M tokens
• Local (Llama 3): 8K-128K (depending on quantization)

Getting Started

If you want to experiment with transformers:

1. Fine-tune a pre-trained model (use LoRA for efficiency)
2. Build a RAG pipeline to add knowledge without retraining
3. Experiment with prompt engineering before reaching for fine-tuning
4. Monitor your models — evaluate on held-out data regularly

Resources

• Attention Is All You Need — The original paper
• The Illustrated Transformer — Visual explanation
• Karpathy’s Neural Network Zero to Hero — Practical implementation
• Transformers from Scratch — Build one yourself

This post was generated with AI assistance. Review and customize it to match your expertise.