Microsoft CTO Kevin Scott on How Far Scaling Laws Will Extend

The current LLM era is the result of scaling the size of models in successive waves (and the compute to train them). It is also the result of better-than-Moore’s-Law price vs performance ratios in each new generation of Nvidia GPUs. The largest platform companies are continuing to invest in scaling as the prime driver of AI innovation. Are they right, or will marginal returns level off soon, leaving hyperscalers with too much hardware and too few customer use cases? To find out, we talk to Microsoft CTO Kevin Scott who has led their AI strategy for the past seven years. Scott describes himself as a “short-term pessimist, long-term optimist” and he sees the scaling trend as durable for the industry and critical for the establishment of Microsoft’s AI platform. Scott believes there will be a shift across the compute ecosystem from training to inference as the frontier models continue to improve, serving wider and more reliable use cases. He also discusses the coming business models for training data, and even what ad units might look like for autonomous agents. Hosted by: Pat Grady and Bill Coughran, Sequoia Capital Mentioned: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, the 2018 Google paper that convinced Kevin that Microsoft wasn’t moving fast enough on AI.  Dennard scaling: The scaling law that describes the proportional relationship between transistor size and power use; has not held since 2012 and is often confused with Moore’s Law. Textbooks Are All You Need: Microsoft paper that introduces a new large language model for code, phi-1, that achieves smaller size by using higher quality “textbook” data. GPQA and MMLU: Benchmarks for reasoning Copilot: Microsoft product line of GPT consumer assistants from general productivity to design, vacation planning, cooking and fitness. Devin: Autonomous AI code agent from Cognition Labs that Microsoft recently announced a partnership with. Ray Solomonoff: Participant in the 1956 Dartmouth Summer Research Project on Artificial Intelligence that named the field; Kevin admires his prescience about the importance of probabilistic methods decades before anyone else. 00:00 - Introduction 01:20 - Kevin’s backstory 06:56 - The role of PhDs in AI engineering 09:56 - Microsoft’s AI strategy 12:40 - Highlights and lowlights 16:28 - Accelerating investments 18:38 - The OpenAI partnership 22:46 - Soon inference will dwarf training 27:56 - Will the demand/supply balance change? 30:51 - Business models for data 36:54 - The value function 39:58 - Copilots 44:47 - The 98/2 rule 49:34 - Solving zero-sum games 57:13 - Lightning round