What is Q*? | FULL DEEP DIVE | Tina Huang

0:00A few days ago, I asked what do you think about Q-star in relation to OpenAI's quest to AGI? AGI being artificial general intelligence, in which an AI has the same intelligence and capacity were greater intelligence and capacity than a human would. The answers were pretty divided. While in a recent interview by Diverge, it seemed like Sam did confirm that Q-star is a thing. He said, no particular comment on that unfortunate leak. That implies that there was something, whereas he would have said something like the unfortunate rumors. Elon Musk also tweeted, ilia had a good moral compass and does not seek power. He would not take such drastic actions unless he fell was absolutely necessary in reference to drastically firing Sam Altman. So it does seem like the evidence points to the fact that Q-star must have spooked ilia to some degree, which contributed towards everything that happened at OpenAI. Although this is of course still speculation since nobody at OpenAI has given any concrete answers.

1:02But in any case, in this video, we're going to be focusing on Q-star. Would it could be? Why the fact that it being able to do great school math is such a big deal? Into the common person, what does this all mean and why should you care? This video is sponsored by Zendask, who is leading AI safety and customer experience. Check out the links in the video description to learn how big companies are considering AI's impact on their customers. After Sam got fired and then reinstated four days later, there was a lot of speculation about why he got fired in the first place. Maybe he was his issue with Helen Toner, who published this research paper that was slightly critical of ChatGbT. Maybe it was because OpenAI agreed to buy $50 million of AI trips from a startup that was personally backed by Sam Altman. Maybe it was because he was also fundraising in the Middle East. But what really caught everybody's attention was when this Reuters article initially came out, saying that OpenAI researchers warned board of AI breakthrough ahead of CEO Auster. The researchers let it raise concerns that the discovery could

2:10threaten humanity. Yeah, pretty dramatic. There's honestly quite little details surrounding this. Reuters said they were unable to actually review a copy of the letter. And after being contacted by Reuters, OpenAI, which declined to comment, actually declined to comment, acknowledged in an internal message to staffers about a project called QStar and a letter to the board before the weekends events. I was pretty confusing this message sent by long-term executive Mir Morati, alerted staff to certain media stories without commenting on their accuracy. So basically, it's like one huge question mark. There was a couple more details. Given vast computing resources, the new model was able to solve certain mathematical problems, though only performing math in the level of grade school students, async such tests made researchers very optimistic about QStar's future success. You might be wondering why is performing math at a grade school level something to be so excited about? Well, the reason is because generative AI is not pretty good at language-based stuff like writing and language translations, but there is a lot of room for error because there isn't one single correct answer. But, conquering the ability to do math where there's only one

3:15right answer implies AI would have greater reasoning capabilities resembling human intelligence. This would essentially open up a whole other branch of capabilities, especially in feels like scientific research where logic is very, very important. So even though we can only do math right now at a grade school level, now that that capacity is there, actually optimizing it and refining it, is just the matter of time and money. It would be a huge step forward towards AI. Obviously, this is not super solid confirmation of anything. Of course, there are prominent figures in AI field, like Young Le Koon, he says, please ignore the deluge of complete nonsense about QStar. It's not anything particularly new. François Chalet, who also does the learning at Google, and is the creative Kira's, and every single month from here on, there'll be rumors of AI having been achieved eternally, just rumors never any actual papers, product releases or anything of the sort. So yeah, take everything with a huge, huge, huge grain of salt. But I did personally do a lot of digging into all of the theories that people had about QStar. And I thought I was sure with you guys a couple

4:20of the most believable theories. At the very minimum, give you some context and some insights into what some people are working on in the AI field, how they're being applied now, and what implications they can have for our future. This Twitter post by Sylas Auburti is a pretty good summary of the two major theories that people have that seem like the most possible. The first one is some optimization related to QLearning, and the second one is some combination of the A-Star algorithm and QLearning, given its name. We are going to unpack both of these now. Starting with QLearning. I asked Chat2pt, what is QLearning? Can you give an analogy to explain it? And it came up with a pretty good analogy, the maze and the treasure. It says, imagine a mouse and a maze. The mouse's goal is to find a piece of cheese hidden somewhere in the maze. Each time the mouse moves from one point in the maze to another, it learns something new about the maze. The mouse doesn't know the layout of the maze initially, nor does it know where the cheese is. But as it explores, it starts to learn which paths lead to the cheese and which don't. Valley came out with these cute illustrations,

5:22and the first one here is the mouse taking a step into the maze. As it takes a step into the maze, it then starts learning a Q value, which is a score that tells the mouse how good each step is in relation to getting closer to the cheese. The mouse keeps track of the Q values in a table, called the Q table, and over time, it fills out this table with values based upon experiences after learning which steps are most likely to lead to the cheese. As the mouse keeps exploring the values more and more, the Q values in the Q table keep getting updated and they became more accurate, representing being able to find the cheese more efficiently. Throughout the whole process, the mouse always chooses the step with the highest associated Q value, which is the step that is most likely to lead to the cheese based upon its prior experiences exploring the maze. So yeah, this is how Q learning works. The mouse learns to make decisions or where to step next in the maze in order to maximize its reward of finding the cheese based upon the Q values that represent its experience. Kind of extrapolating from this example, the key idea here is that the agent,

6:23like the mouse in this case, learns from its own experiences by exploring the environment, and it updates its decisions based upon past rewards and actions. Q learning is used in a lot of different fields. Like for example, in robotics, it's used for pathfinding and avoiding different obstacles. In trading, Q learning algos can optimize stock trading strategies and it's used to learn market data and maximize different returns. It's also used for autonomous vehicles like self-driving cars, supply chain and return management, healthcare, personalized medicine, let's go on. Okay, so coming back to this tweet, it sounds like it's related to Q learning. For example, Q start denotes the optimum solution of the Bellman equation. What he's referring to here as the Bellman's equation, the simplest explanation is that it's an equation that helps a model guide towards an optimal solution, which is called the Q star. So the whole point of Q learning is get towards this optimal solution of Q star, which is why people are speculating that this AI called Q star is somehow improving Q learning in some way to get closer to the optimal Q star.

7:25So I asked Chad to be to speculate a little bit and it says that it could be convergence towards Q star in terms of accuracy and optimality, because one of the main issues with Q learning now is that it does try to approximate towards the optimal, but it doesn't always get to the optimal. It could also be speed of convergence, which is efficiency. Q star right now can be very slow to converge depending on certain environments, especially with very complex environments. It could be better at learning things, it can be more stable, etc, etc. So a lot of different ways that you can improve Q learning that Q star may be. Second, plausible explanation is a combination of A star algorithm and Q learning. Heads the name Q star. There's this really good article on LinkedIn that explains how it could be combined together. So I'm going to walk you through the example that the article gave. Full credit here to Siva. Although I did ask Chad to be to draw some images, just ignore the fact that it's not very good at spelling, but anyways. So in this example, we'll think about Q learning in the context of learning a new recipe. Imagine Q learning as learning to cook a complex dish without having a recipe. So each ingredient and cooking step is a

8:28decision that you have to make. First thing that you do is trial and error. You try a lot of different combinations of ingredients and cooking techniques. Some of it is going to taste yummy, which is a reward, and some of it is going to taste not very good, which is going to be a penalty. As you experiment, you keep notes at which combinations and techniques work the best, and you put that all together into a notebook, which is your Q table. With each cooking attempt, you keep refining your notes, learning about successful and unsuccessful outcomes, aiming to perfect this dish. Until you get the recipe for the perfect dish that has the best ingredients and the best techniques. Now let's talk about A-Star search, which is an algorithm that goes all the way back to the 60s. It's usually used to find the shortest path between like two different things based upon some information, some experiences that it has. So here's the analogy. You can think about A-Star as the process of planning a complex meal where you have to prepare multiple dishes efficiently. In the beginning, you're evaluating different steps of the cooking process.

9:29You're looking at different options like which dishes to start with, which ingredients to prepare next, which cooking method to use. For each option, A-Star then evaluates how choosing it will affect the overall meal preparation time. So it's thinking like, hmm, if I roast the vegetables now while I simmer the sauce, then will everything come together more quickly or not. So it keeps going through this process, choosing which seems to optimize the cooking process, which doesn't, trying to make sure that all the dishes are prepared in the shortest combined time. With the ultimate goal to find a sequence of all the cooking steps that lead to the most efficient preparation of this entire meal. So it does need to have information about what the recipes are composed of. Can you roast vegetables while you're simmering the sauce? How long does steaming take? I would that information is able to optimize for the shortest amount of time and at least the amount of effort in order to make sure that everything is done correctly. So how does Q-learning and A-Star Search come together? Well, imagine that you're organizing like a really big dinner party where you need to cook multiple complex dishes, each with its own set of ingredients and its own cooking steps. A-Star

10:33Search comes into play by planning the sequence of cooking steps for the entire meal. It would consider factors like how long things take to cook, what are the preparation steps in order to figure out the most efficient way of preparing the dishes while also using Q-learning to refine each dish's preparations based upon its learnings about successes and failures. And all of this coming together, this dual approach makes sure that each individual dish is as tasty and yummy as possible, but the entire meal is also coming together in the most efficient way possible. That's an example of how Q-learning plus A-Star Search could come together and make a very powerful algorithm called Q-Star. Okay, so there's one more piece of the puzzle that we know. It was reported that Setskyver's breakthrough within OpenAI allowed OpenAI to overcome limitations on obtaining enough high-quality data to train models, a major obstacle for developing next-generation models. The research involved using computer-generated rather than real-world data like text-reimages pulled from the internet to train new models. This is what is called synthetic data. So synthetic data is data that's generated

11:34from computers as opposed to real-world data that most of these large language models are trained on. By being able to use synthetic data efficiently, this can be really powerful. For example, there are certain topics in certain industries in which there's not that much high-quality human-generated data in order to train algorithms on. Synthetic data would be able to solve that problem. Also, human data is messy, it's dirty, it's biased. So by being able to use synthetic data, we're able to train the models minus the things that humans do that we wouldn't want AI to have. For example, be prejudice, we're aggressive. The theory is that Q-star is either partially or completely trained on synthetic data, which could be why it's a breakthrough. In any case, I'm sure more information about this is probably going to come out, and I just want to make a note that this really is just a lot of speculation over here. Even if these components are true, there's probably other piece of the puzzle that we don't know about. Most advanced AI systems use a combination of different techniques. So I'm sure there's other bits and pieces that form this algorithm called Q-star.

12:37Whether Q-star is real, not real, impactful, not impactful, the one thing that is undeniable is the speed of AI development and how much effect it's already having in many, many different industries right now. That's why I want to show you some examples of super, super, fast progress in different industries. I keep in mind that the faster AI develops, the faster these innovations will be happening. And all this is within the past few weeks to few months. First off, material side. Google deep minds, new AI tool will help create more than 700 new materials. Newly discovered material can be used to make better solar cells, batteries, computerships, and more. And these are not just hypothetical materials. Google AI and robots are joining forces to build these new materials. We're all of healthcare. AI scours documentation for cancer studies. And AI helps diagnose and manage kidney diseases. There's predictions that healthcare is one of the industries that is going to be most disrupted going to 2024. In the world of customer service, customer service investments in ARS, accelerating, after seeing what recent AI events are capable of, future ready support leaders

13:38have already started thinking about how to integrate AI power technology into their tech staff. With 69% planning to invest more in AI in the next year, customer service predicted to be one of the most impacted industries, which is why I really appreciate Zendask, who is a leader in this field for opening up conversations on how AI is being integrated into a customer services. Zendask is also the sponsor of the video. They asked me to spread the word about an economist impact roundtable discussion that they sponsored. Discussing the impact of AI and customer services such as operations for efficiencies, but also the risks. Roundtable features top leaders in the customer experience and customer service field. Adrian starts off by explaining the way that Zendask is thinking about AI, which is to enhance customer experiences rather than automation were replacing. The expert found really interesting about how it's impacting customer service professionals and how it's shifting their work, a subject that I know a lot of people are interested in. So what we see is that a lot of the road repetitive work that an agent has to do will be taken over

14:43by AI by automation and operations will become a lot more efficient because you'll be able to, for example, detect the intent of a customer inquiry and then route it to the right agent with a right skill. So the consensus of the panelists is that they do agree that agents would remain the point of escalation for the next decade at least. That will be the full and a gen A.I. will provide it will supercharge many of these roles, but you know, human touch will be needed for a long time. As Adrian explains, humans ultimately like to talk to humans and they intend to always keep humans in the loop. They were all really excited about this technology, but they were also very open and expressing that we needed to exercise caution. Because even though technology is good at making predictions, it can also be very confidently wrong. It lacks human reasoning and is prone to errors, which has severe regulatory and legal implications. Everybody agree that gen A.I. proved an opportunity that can greatly improve customer

15:45services, but it needs to be thoughtfully implemented. I think that this is a thought that should be echoed across all different industries and all different companies. You can watch the economist impact webinar that send us sponsored or check out the event summary. Both are linked below in descriptions. Alright, so before I end this video, I want to tie everything together that we discussed today. I think that every single person should be keeping an eye on A.I. development, especially in relation to the field that they're currently in. We already saw how much A.I. is already changing the world around us and how fast innovations are happening. If there's one thing I want you to take away from this video is realizing that A.I. is here to stay and is very, very important to start embracing it because the change is exponential and it's only going to get faster and faster from here. So it's much better to be in the loop, taking advantage of the opportunities that A.I. presents and being the one shaping how A.I. is going to be influencing the things that you care about. If you're interested in learning more about A.I. where maybe even getting your hands dirty and integrating A.I. into different products, I do have a weekly, completely free lunch and learn

16:49series where we talk about A.I. and also do workshops and tutorials. Now is definitely a time to start learning about these things. There are a lot of opportunities in the A.I. space, especially if you know how to code where you're interested in learning how to code. All right, thank you guys also much for watching and I will see you guys in the next video where live stream.