The Evolution of AI & Machine Learning

Alan Turing's "Computing Machinery and Intelligence" proposed the imitation game — framing intelligence as behavioral indistinguishability. Set the philosophical frame for AI for 70 years.

Good Old-Fashioned AI: intelligence as symbol manipulation. Logic rules, search trees, knowledge bases. Programs like LISP, PROLOG. Works when problem space is fully enumerable — fails at perception and ambiguity.

Binary threshold neuron modeled on biology. The conceptual atom of all neural networks. No learning — weights were hand-set. The spark, not the fire.

First algorithm that could learn weights from data. Single-layer, linear classifier. Proved convergence theorem — if data is linearly separable, it will find the boundary. Minsky & Papert's 1969 critique (XOR problem) triggered the first AI Winter.

DARPA funding collapsed. Symbolic AI hit intractable search spaces. The "combinatorial explosion" — brute-force search doesn't scale. Real AI needs representation, not just computation.

Rule-based systems encoding domain expert knowledge. MYCIN diagnosed bacterial infections with IF-THEN rules — matched physician accuracy on narrow cases. XCON configured VAX computers at DEC, saving $40M/year. First real industrial AI deployment.

Probabilistic graphical models for reasoning under uncertainty. Judea Pearl's 1988 framework. Directed acyclic graphs encoding conditional independence. Foundation of modern probabilistic ML and causal inference.

Efficient computation of gradients via chain rule through layered networks. Multi-layer perceptrons could now learn non-linear functions. Trained on XOR — killed Minsky's critique. The engine inside every modern neural network today.

Convolutional Neural Networks for image recognition. Local receptive fields, weight sharing, pooling. LeNet-5 read handwritten digits on bank checks at scale. The blueprint for all vision models — ResNet, EfficientNet, ViT all descend from this.

Long Short-Term Memory solved the vanishing gradient problem for sequential data. Gates control what to remember, forget, output. First architecture capable of long-range sequence modeling — powered speech recognition and early machine translation until 2017.

Maximum-margin classifier with kernel trick — maps to high-dimensional spaces without explicit computation. Vapnik-Chervonenkis theory provided generalization bounds. State-of-the-art for classification until ~2012. Still used in production today for tabular data.

Breiman's Random Forest (2001): bagging over decision trees with random feature selection. XGBoost / LightGBM / CatBoost: gradient boosting on residuals. Still the dominant approach for tabular ML in production — Kaggle winners, fraud detection, risk scoring.

"You shall know a word by the company it keeps" (Firth). Latent Semantic Analysis (LSA), pLSA, LDA for topic modeling. The philosophical basis for all modern embeddings — meaning from co-occurrence.

Stack of Restricted Boltzmann Machines trained layer-by-layer. Showed deep networks could learn meaningful representations. Revived deep learning research after the second AI winter. Pre-training → fine-tuning paradigm that GPT still uses today.

NVIDIA's CUDA (2007) made general-purpose GPU computing accessible. Matrix multiplication — the core operation of neural networks — maps perfectly to GPU parallelism. This hardware shift enabled every modern deep learning breakthrough.

Won ImageNet LSVRC with 15.3% top-5 error vs 26.2% second place. Deep CNN with ReLU activations, dropout regularization, GPU training. Every major CV breakthrough since is a direct descendant. The shot heard round the ML world.

He et al.'s Residual Networks: skip connections solve vanishing gradients in 100+ layer networks. Batch Normalization stabilizes training. ResNet-50/101 still widely deployed in production CV systems at scale today.

ImageNet pre-trained weights transfer to any vision task. Fine-tune final layers on domain data. 10x less labeled data needed. The pattern used in every industrial CV deployment — wildlife recognition, medical imaging, autonomous vehicles.

Generative Adversarial Networks: generator vs discriminator in minimax game. Learned to generate photorealistic images. Mode collapse, training instability, Wasserstein GAN improvements. Direct ancestor of Stable Diffusion and image generation models.

Dense word embeddings from co-occurrence. Skip-gram and CBOW architectures. Semantic arithmetic: king - man + woman ≈ queen. Foundation for all modern text embeddings. Replaced TF-IDF in NLP overnight — and the conceptual basis for pgvector today.

Encoder-decoder LSTMs for machine translation. Attention mechanism (Bahdanau, 2014): instead of single context vector, attend to all encoder states. Solved the bottleneck problem. The "attention" in "Attention is All You Need" was born here.

Replaced RNNs entirely. Multi-head self-attention: every token attends to every other token in parallel. Positional encodings instead of recurrence. Scales with data and compute in a way LSTMs never could. The backbone of GPT, BERT, T5, PaLM, Claude, and every frontier model.

Attention(Q,K,V) = softmax(QKᵀ/√dₖ)V — queries, keys, values from the same sequence. The √d scaling prevents softmax saturation. KV cache (reuse keys/values at inference) is why vLLM exists and why GPU memory is the bottleneck.

Bidirectional Encoder Representations from Transformers. Masked Language Modeling: predict [MASK] tokens using both left AND right context. Fine-tune a single model on any NLP task. SOTA on 11 benchmarks simultaneously.

Decoder-only Transformer. Causal Language Modeling: predict next token. GPT-2 (1.5B params) was so good OpenAI delayed release. Demonstrated emergent capabilities — arithmetic, translation, summarization — from next-token prediction alone. The architecture of every modern LLM.

Hugging Face's transformers library democratized BERT/GPT access. spaCy for production NLP pipelines. Model Hub: thousands of pre-trained models one line of code away.

175 billion parameters. In-context learning: no fine-tuning needed — examples in the prompt teach the task. Emergent capabilities appear suddenly at scale thresholds. Sparked the "prompting" paradigm — prompt engineering became a real skill.

Loss scales as a power law with compute, data, and parameters. Optimal model size for a compute budget is predictable. Chinchilla (2022) refined: Llama-sized models trained on more data beat GPT-3-sized models — leading to Llama 3.1, Mistral, Phi architectures.

CLIP: contrastive learning across image-text pairs. Encodes images and text in a shared embedding space — enables zero-shot image classification and cross-modal retrieval. DALL-E: text-to-image via autoregressive Transformer on image tokens. Foundation of Stable Diffusion and midjourney.

Three phases: SFT on demonstrations → reward model trained on human preference rankings → PPO fine-tuning against reward model. InstructGPT (2022) showed RLHF made 1.3B model more helpful than raw 175B GPT-3. ChatGPT is RLHF on GPT-3.5. The technique behind Claude.

Low-Rank Adaptation: freeze base model, add trainable rank-decomposition matrices (ΔW = BA where rank r ≪ d). Trains 0.1–1% of parameters. QLoRA (2023): 4-bit quantized base + LoRA adapters — 70B model on single A100.

Reformulates RLHF as supervised learning on preference pairs. No explicit reward model, no PPO instability. Loss directly optimizes: preferred response probability up, rejected response probability down. Simpler, stabler, now dominant in fine-tuning pipelines.

Chunk documents → embed → store in vector DB → at query time: embed query, ANN search, retrieve top-k, inject into prompt, generate grounded response. Solves hallucination and knowledge cutoff. Hybrid search (BM25 + dense) + cross-encoder reranking is the 2024+ production standard.

KV cache grows O(sequence_length × layers × heads × d_model) per request. vLLM's PagedAttention manages KV cache in non-contiguous memory blocks — like OS virtual memory. Continuous batching: new requests slot into vacated positions. 24x throughput over naive serving.

LangChain: chains of LLM calls, tool integrations, memory. LangGraph: stateful agent graphs with explicit state machine — nodes (agents/tools), edges (transitions), state shared across nodes. Enables retry logic, human-in-the-loop, parallel branches, conditional routing.

Reason + Act interleaved: Thought → Action → Observation → Thought → … Agents call tools (search, code execution, APIs) based on LLM reasoning. Tool calling APIs (OpenAI function calling, Anthropic tool use) standardized the interface. Failure modes: infinite loops, hallucinated tool calls, context bloat.

Speculative decoding: small draft model generates k tokens speculatively — large verifier accepts/rejects in parallel. ~2–3x speedup on output-heavy tasks. Quantization: INT8/INT4 (GPTQ, AWQ, GGUF) — 4x memory reduction with 1–3% accuracy loss. Production standard now.

Vision-Language Models: image tokens projected into the LLM's embedding space. Unified architecture handles text + images + (increasingly) audio, video, code. Enables document understanding, chart reading, screenshot analysis. Production use: contract review, medical imaging, e-commerce.

128K → 1M token context windows. Gemini 1.5 Pro: 1M tokens. RoPE (Rotary Positional Encoding) replaces absolute positions — enables extrapolation. Flash Attention 2/3: IO-aware attention — 2–4x faster by minimizing GPU memory transfers. Sliding window attention for efficiency.