Phase 3: Architecture~8 minadvanced

šŸ’”Modern Innovations

State-of-the-Art Techniques

Flash Attention, RoPE, GQA, Mixture of Experts, and other cutting-edge improvements.

Flash AttentionRoPEGQAMixture of Experts

State-of-the-Art Improvements

The original 2017 Transformer has been significantly improved. Modern LLMs incorporate numerous optimizations that enable longer context, faster training, and more efficient inference.

Tuning a Race Car

The original Transformer is like a solid car that wins races. But engineers keep finding ways to make it faster ā€” better aerodynamics (Flash Attention), lighter materials (GQA), more efficient engines (MoE). Each improvement compounds, and today's models are dramatically more capable.

Key Innovations

Flash Attention2022

IO-aware attention that reduces memory reads/writes

Enables longer context, faster training2-4x
RoPE2021

Rotary Position Embedding for relative positions

Better extrapolation to longer sequencesContext extension
GQA2023

Grouped-Query Attention shares KV heads

Faster inference with minimal quality loss1.5-2x inference
MoE2022

Mixture of Experts activates subset of weights

Scale parameters without scaling compute8x parameters/FLOP
SwiGLU2020

Gated activation function combining Swish + GLU

Improved performance over ReLU/GELUBetter quality
RMSNorm2019

Simplified normalization without mean centering

Faster than LayerNorm, similar quality~10% faster

Flash Attention Deep Dive

Flash Attention is arguably the most impactful recent optimization. Standard attention is memory-bound ā€” it reads/writes the full attention matrix to GPU memory. Flash Attention uses tiling and recomputation to keep everything in fast SRAM.

2-4x
Speedup
5-20x
Memory Savings
O(N)
Memory (vs O(NĀ²))
Exact
No Approximation

Mixture of Experts (MoE)

MoE models like Mixtral have 8x the parameters but only activate a fraction per token. A "router" network decides which expert(s) process each token.

šŸ§  ā†’ šŸŽÆ ā†’ šŸ”§šŸ”§
Token ā†’ Router ā†’ Top-2 Experts (out of 8)
46.7B
Total Params
12.9B
Active Params
3.6x
Efficiency Gain
āœ…
Key Takeaways
  • Flash Attention enables 2-4x speedup and longer context
  • RoPE allows context extension beyond training length
  • GQA reduces inference costs with minimal quality impact
  • MoE scales parameters without proportional compute increase