236: We Now Measures the Largest Chips Used to Generate an LLM – or a 21st century #$%& Measuring Contest

Welcome to episode 236 of the Cloud Pod Podcast, where the forecast is always cloudy! Are you wandering around every day wondering just who has the biggest one? Chips, we mean. Of course. Get your mind out of the gutter. Did you know Azure was winning that battle for like 8 whole minutes? Join us for episode 236 where we talk about chip size, LLM’s, updates to Bedrock, and Toxicity Detection – something you will never find applied to the podcast. Not on purpose, anyway. 

Happy Thanksgiving!

Titles we almost went with this week:

• You Can Solve All Your AI Problems by Paying the Cloud Pod 10 million Dollars. 
• Cloud Pods Interest in AI Like Enterprises is Also Shockingly Low
• Llama Lambda Llama Llama Lambda Lambda… or How I Went Crazy
• Comprehends Detects Toxicity with the Cloud Pod
• You Didn’t Need Comprehend for Me to Tell You I’m Toxic
• The Cloud is Toxic, Run!

A big thanks to this week’s sponsor:

Foghorn Consulting provides top-notch cloud and DevOps engineers to the world’s most innovative companies. Initiatives stalled because you have trouble hiring?  Foghorn can be burning down your DevOps and Cloud backlogs as soon as next week.

AI is Going Great! 

00:39  OpenAI’s New Weapon in Talent War With Google: $10 Million Pay Packages for Researchers (listeners note: paywall article)

• The battle for AI talent is heating up between open AI and Google. With compensation packages but also promises of access to more hardware, better chips and more. 
• Open AI depends on Microsoft for its cloud resources, whereas Google owns its cloud and is manufacturing their own AI chips. 
• Salaries are crazy with stock compensation with Open AI saying their stock compensation could be worth as much as 5-10m.  
• Of course assuming that recruits start before the company goes public or gets completely acquired by MS. 
• So, bottom line? Money. Are you shocked? We’re shocked. 

01:30 Jonathan – “I guess it’s quite a concern actually that since Google bought DeepMind they have pretty much two-thirds of the entire global AI talent at their own disposal. So I guess this is a desperate needs, call for desperate measures kind of thing.”

01:49 Nvidia Unveils New AI Chip, Upping Ante with AMD (listeners note: paywall article)

• Nvidia on Monday announced a new graphics processing unit, the H200, which next year could become the most advanced chip on the market for developing AI. 
• The chip’s memory capacity has been significantly upgraded compared to the H100, which has been in high demand and boosting NVIDIA stock 240% since Jan 1.  The increased memory allows LLM models powered by H200 chips to generate results nearly twice as fast as those running on H100s
• Cloud companies should have the new chips available in 2nd quarter 2024 and will put these in tight competition with AMD’s MI300X gpu’s slated for release later this year.  

02:29 Matthew – “ I feel like we’re seeing the speed curve of processors and now we’re just watching the same things that happened in the 90s and 2000s happen with GPUs. It’s like, it will double every 18 months. That’s