Speech Recognition Is Not a Solved Problem — Pavan Kumar Reddy

0:00College football is back. So, Hilton called to me the superstition concierge to make your fan rituals a reality. Need a room to match your lucky number? We got you. Want to make sure our team doesn't wash your lucky jersey? Oho, that smells lucky. Hilton's unmatched hospitality can keep up with any superstition. Even a marching bandwink up call it 555 and 55 seconds. Hit it! When you need a team that will do whatever it takes on game day, it matters where you stay. Hilton, for this day. Good things come in force. Like ukulele strings, boy band members. Oh yeah! And PayPal payin' for. No fees, no interest, no impact on your credit score. Just the flexibility to pay the way that works for you. Strum it, hum it, pay it in for. Download the PayPal app to get started. Subject to approval. Learn more at paypal.com slash payin' for. Paypalling. NMLS 910457. I think a lot of people don't know how audio generation works. Voice is one of the primary ways humans communicate.

1:03Definitely came much before text was a thing. The world looked very different, I think, or with neural codecs and autoregressive generation, which was really cool that the field progressed so far. And when we made it work, it was pretty cool. It's an auto-regressive architecture. So what you predicted already is in a sense committed. So it becomes context and you can't edit it out. It's like a voice, an end-to-end speech model, but primarily to serve as an interface to other systems. Yeah, I'm a research scientist at Mistral. I work on audio research. At the moment, I lead the audio research team here at Mistral. I've been broadly focused on multi-model models at Mistral since I joined like two years ago. This episode was produced in partnership with Mistral AI. So Mistral is a full-stack AI company.

2:04And you're in the thick of it. You're doing the actual research. What does that actually mean deep down? Yeah, definitely. The full-stack description of Mistral applies to the whole company as in it's serving needs at different layers of the AI stack, I suppose. I think the company is most well known for the model layer. Since the original Mistral 7B, we've been releasing open models, both text models and over time, multi-model models and reasoning models. That's one layer that we continue to focus on and one of the most important layer and the place I work on primarily. But on top of it, we have offerings at the application layer. We have Mistral Wipe, which is a coding offering, a chat app, and also a command line based interface.

3:08So there is an on top of it, we also have like Wipe work, which is similar agent, tech working experience. So that's at the product layer and we also have AI studio, which has our API offerings and workflows where you can build agents, AI agents easily. And we also have a forge, which is a fine tuning platform and also other kinds of model adaptations, which where you can take any of our open models, some of our non-open models as well and just adapt them to your specific needs. Again, applies to both text models and multi-model models. And over the last year or so, we also have Mistral Compute, which is the AI Cloud offering, which is bare bones compute on top of which many of the other work happens, including the research work.

4:12It's a full stack AI company in that we do have some presence and offering across various layers of the AI stack because most of the enterprises, which tries to incorporate AI usually require integration at several points in their stack. And we try to meet them where they are and give them the efficiency gains or productivity boost that they're looking for. Very cool. And what's it like, though, just being a research scientist at a frontier AI company? I mean, you must be doing, you know, you must be looking at lots of interesting research that other folks are doing going to conferences and so on. I mean, how is all of that work kind of being integrated into your workflow? What is your day in a life look like? Yeah, that's a great question. It's a really fun to work at a frontier company, which is also a general frontier company, I suppose to a task based company. Again, I think there are trade offsets, not it's not a strict one is better than the other, but for me personally speaking, it's always nice to have visibility into the whole work happening across the board.

5:25Even though I focus on audio, I closely model our work on text pre-training, post-training, reasoning, vision and more recently, we have a venture into robotic foundation models as well. And I think that's why I don't work across the board at any given point in time. It gives me good exposure to what's going on and it also I think helps me incorporate the latest techniques because I think at this point, not just for audio, but across modalities and tasks, the approaches are getting converging. The paradigm of pre-training, post-training, there are subtle differences and specific nuances to specific niches, but at the same time, the broad techniques are super general. I mean, that's the success of the latest paradigm, I suppose. So an average day involves doing a lot of my own research work, training models, running uplations, investigating e-walls and stuff, but also following the work that's happening across the company.

6:40Yeah, I suppose it's cool as well because you folks get to publish a lot of your research so you can kind of speak openly to the community and a lot of your models are also open weights as well. Yeah, definitely. I think from the get-go, we have the open weight ethos, our very first set of model, as Mr. 7B was the one which got us prominence and we continue to open weight. Most of our models, especially the largest models, and the hope is basically we operate as one open ecosystem. We contribute to the ecosystem and take from the open ecosystem as well. And it's a, the whole open frontier keeps moving. So it's also, as a researcher, it's nice to be able to share our research work. It's personally speaking, it's always great to share the work and also borrow developments outside of our company.

7:45There's a lot of the ecosystem is quite big. So it's, it would be a missed opportunity to not leverage the developments and it's also always nice to contribute your own in your own small way. Traditionally in machine learning, all of the different modalities were quite fractured into different expertise. So, you know, the language people were one ecosystem and the audio people were another ecosystem. And you're working on, on so many modalities and isn't it interesting that many of the same types of approaches work across those modalities? Yeah, it's actually, I mean, I've been working in machine learning and AI for, I think at this point, close to 10 years, I suppose. And it's actually fascinating. It converged to a point where the same, firstly, it's fascinating that architecture says so. All of them, most of them are transformer based. Not some variance across modalities as modality specific variance and now there are hybrid models with non attention based layers going on, but broadly speaking, the architectures have converged.

8:58And even in audio, which is a modality, I started working on since two years ago, I suppose at this point, the techniques are converging more and more to a unified approach. I think it just comes, the forcing function, at least comes from the place that the way you use them is you want to use them as a single entity, like in many of these cases, at the end of the day, from a user standpoint, from an interface standpoint, it's a, it's a model that you would like to give input through text communicate through vision, inputs and talk to it and hope it writes back or speaks back and in some cases generate images. So I guess it's a natural forcing function for people to explore how to make it all work in a single model because it's quite elegant way to do it. It's just I was at least surprised when it started not just as a nice to have, but it also turned out to be the way that that seems to be working really, really well.

10:08Well, we should start talking about some of the audio work that you folks have been doing. Can you tell me about that? Yeah, I have to share our work. So we started working on audio last year and the very first model we released was Waxroll Chat. It's a audio input text to text LLM model. So you give it audio input and a textual instruction or it doesn't need a textual instruction because your question can be in the audio and then the model produces a text response. So the idea behind that is to have a general interface for audio understanding. So the model can do transcription, speaker segmentation or summarization or any questions like you can have a big audio document essentially, which is like an earnings call or a meeting recording and then ask it questions like what you would against a text document.

11:11So you can ask it about what was the topic of the conversation, who spoke when did this thing happen? It's specifically useful when analyzing for instance podcasts like if you want to at what time point did the speakers talk about this topic, things like that. So that was the first model that we worked on and then since then our focus was to build foundational building blocks for audio agents to that and we released a transcription model a real time variant of that and then a TTS model earlier this year and we are continuing to work on improving those models and the hope eventually is to build a end to end speech to speech model in the space. Yeah, really cool and that first model that you gave an example of I guess you would call that an audio understanding model and people might not be familiar with that because you know that there are omnimodels that have audio capability and there are transcription models and because this model has all of those capabilities built into it you gave the example you can actually say when did Tim ask the question about the audio model and it actually knows when certain events are and what time those events are.

12:35So that's actually a new type of AI that I don't think many people have seen before yeah, definitely people might already do this in a cascaded way you basically take an audio document get a transcription of it and feed it to an LLM chat bot and you can ask it all all kinds of questions that you just mentioned. This model basically does the same thing but in an audio native way so you don't need the intermediate transcription so there are few advantages to handling it natively I think the general principle of ML is the more you do end to end the more you are removing the error propagation through a cascade like approach and in this case you can ask it about things like emotion which is not necessarily transferred captured in a transcription. Also if you want to query about some information which is not just the transcript you need to have that information be captured in the intermediate format in the transcription case like for instance for the time stamp question you are asking when was this topic when when it mention this in the podcast.

13:41You are basically assuming the transcript has time stamps are the granularity you care about so you need to capture very rich information in the intermediate representation in this case it's a transcript I suppose and it's always better when the quality matches up to the expectations to have it be done and to end purely because you don't need to capture things that you can do. So I think it's a head of time it's the power of attention I guess you just focus on the things you care about by directly attending to the relevant aspects of the audio input in this case. Very cool now as I understand it you folks use one of your text models as a trunk and you know like the promise of deep learning was always this beautiful idea that it's like Lego right we can compose all these different models together so you can have a frozen trunk over here and then and then you can build you know an audio head on the top but I suppose the question is do you lose anything by doing that you know like what what are the trade offs by building deep learning models in this compositional way yeah definitely it is composition from the

14:54architectural composition standpoint so just to take a step back and describe the architecture of this model that you alluded to so we have the trunk which is a 3B text model that we train that we call the role series of models and audio input is provided to the model through a audio encoder but unlike say whisper or models like that where the audio input goes through an encoder which is then fed into the decoder through cross here audio encoder produces tokens in this case embed continuous representations through embeddings and then they're fed into the main decoder model just as a direct token input similar to how you would feed text input in the text case it's rather simple encoding scheme you

15:54send it through a tokenizer and you get token IDs and then you just have a embedding table in this case the encoder is a little bit more sophisticated at least in the voxel chat the encoder is very close to whisper encoder although for the later models we optimized it and adapted it we just try to reduce the number of layers to the minimum number required for getting the performance but in the first model it was very close to the whisper encoder and the encoder is itself a transformer model so it starts with extracting the mel spectrogram and having a couple of convolutional layers at the very beginning and later we have a transformer stack and a down sampling layers so you basically get an audio token for every 80 milliseconds so we are getting 12.5 tokens per second with this it's a fixed frame rate approach and the the compositional aspect of it is you could train this encoder independently which is very common in vision for instance vision encoder so train with the large captioning data set and then added into a decoder transformer like this and train with the transformer

17:20and then we have a free training task with vision text align documents but in the audio case rather at least in this model we don't quite train this encoder in a separate phase with a different different laws or different approach it's a fairly simple recipe and we always try to have simple recipes because it's I think stands the test of time and also easier to scale benefits of keeping the recipe as simple as it can possibly be so in this case we just pre train the model end to end with the encoder and the decoder on actually two basic tasks that we describe in the paper a continuation task where we have interleaved audio text sequence so there is an image in a podcast like this where it's let's say it segmented based on speaker so if my speeches are present in audio your speech would be represented in text and it's an interleaved training on the text 12 blocks and also we train it on the transcription like pattern for basic alignment between audio and text modalities.

18:34So since it's trained and to end it's almost like a singular unit although it's a compositional from how the audio is fed into the model so I personally feel the line is pretty blurry that's not it's not that the audio encoder is solely responsible for audio understanding it's clearly the joint optimization which leads to the audio understanding being quite spread out in the model although audio encoder is the first component which at least presents the information in a way that the decoder can ingested similar to text and how in the architecture do you combine the two modalities I think you said you summed them up and what what with the alternatives big yeah that's a good question so in the walk through chat it's an interleaved audio text model so at given position you either have audio or you have text more specifically if you have audio document you basically will have audio audio

19:35occupying the prefix and then comes the text part which is first a text question and then the model generates a text answer I think the summing part that you're mentioning is in the real time model in this case that's a slightly different architecture where we the encoder part is the same but when the information is fed into the decoder the model is processing audio and generating text at the same time that's the same time that's the reason why it's a dual stream model and the specific task here is the model is expected to produce a transcript with an expected delay and the model is consuming the tokens it generated itself that's a autoregressive component and the audio stream is a read only stream since it's not generating audio in this case it's just ingesting the audio and the way you can imagine more complex fusion but again in the spirit of keeping things as simple as they can get we went with summation and it works just fine compared to other more fancy fusion techniques.

20:59So what I'm discussing today is the trade off between having like a big omnimodel and building specialized models and using cascades and this is a great example because when whisper came out it was an amazing model when there was loads of innovation in the open source community and some people even hacked it to be a streaming model right so they were they were doing this kind of sliding window type thing because you know the obvious question is well why do we need to have like a streaming model architecture and a batch model architecture I mean maybe could you just explain the difference between the and what the trade offs are you know for both of them definitely and I'll prefer this by saying audio is interesting to me also because the space is still evolving there is no architecture which is like the end form I suppose in text to that is true to certain extent but in audio it is I would say it's even more so so whenever I want to I can't trust anything with something else it's with the caveat that it would probably look a little different next year or two years from now but at least as it stands today the motivation behind streaming is to make it natively streaming so if the model can natively so the the property we want is for the model to continuously be listening

22:21not listening chunks and we wanted to see how far we can push there's also this approach this approach is also inspired from the delayed stream modeling that q type publish and the key difference is the encoder is trained from scratch in an end to end fashion in this case and the idea here is to have the audio fed to the model at the same frame rate that the encoder emits the tokens in this case. The encoder emits the token every 80 milliseconds so the model is natively fed the audio every 80 milliseconds essentially and one interesting aspect I mentioned the model is expected to produce the transcript it's listening but there is a subtle trade off here the model can produce the transcript after listening to the all of the audio which would be the case with walks of chat by its design and whisper originally as well it takes the audio 30 seconds of it and then produces the text it this model can do that to adjust what we so the tokens it's producing when it's listening is like a a filler token like a streaming listen kind of token and when it's ready to produce a transcript it produces the text tokens corresponding to the transcript that it is just heard and one extreme I just mentioned is listening all of the audio and producing the other extreme is producing the text tokens it hurts

23:48text tokens it hurt almost immediately if it hears new york and then it produces the text token corresponding to that word right after it heard new york but you can you can see how the this extreme cases actually ambiguous because in many cases a full word could involve prefixes which are themselves stand alone words so the more you wait the more the less ambiguity there would be and the higher even theoretically like this is assuming oracle information the model which does perfectly well even in that case there is ambiguity and hence producing errors and the more you wait the less ambiguity and the lower the chances of errors and in this case the expected delay that the model is supposed to have is actually a parameter to the model itself we are conditioned the model with the target delay at hand and the target delays are target number of frames it has to wait before it produces the word it just heard so the model gives you a nice quality latency trade off that you can control based on the application that you're building or whether you're feeding it to a downstream LM or whether it's a subtitle use case where you're showing it

25:08are the bottom of the video in a conversation like this so that's flexible yeah it's so interesting because there must be levels of latency where you get different types of degeneracy so I'm you know I'm imagining if you make it respond too quickly at some point it'll just fall off a cliff but there's also this more sophisticated form of of you know text production where as you say it takes the context into account to give you an example when I do some ASR transcription and I'll get Claude to do a transcription refinement and it can take the entire document into consideration and maybe even other stuff as well and it will know oh it's mistranscribed on Dracopathy's name and it's actually talking about this paper and there's a technical figure you know there's a technical term in that paper and it's absolutely wonderful and this is the kind of thing that practitioners can do right so you could put a custom vocabulary in there or you could have varying levels of transcription refinement so you could you could have a fast path maybe that refines within a five second window it may be like a 20 second path and you see what I mean you can just kind of lay

26:11around with engineering various forms of refinement to make it work better yeah definitely I think at the end of the day it's a system that you're building voice agencies the specific system that is of most interest to us but having said that the models are built to be applied in all kinds of contexts including the completely offline case where you only care about performance at the end of the quality don't care about latency at all and there are cases where you don't even care about the cost you're happy to call many models exactly like the case that you mentioned where you're post like post protection task where you're taking a podcast recording like this or a earnings call and trying to do analysis over it where this is not the biggest cost consideration and you care about being as right as you possibly can in that case you can do all kinds of refinement like you mentioned where you can bring a reasoning LLM into the mix and actually the walks will chat it was last year but if you were to do again this year that would be a

27:20reasoning model so it would listen to the audio natively and then would not produce the transcript or analysis that you asked immediately it couldn't in its own reasoning chain transcribe the necessary parts do a web search fetch appropriate context it needs to disambiguate certain things that it is a little finds a little inconsistent based on what was said and then retranscribe in its own thinking chain so this can this can be done in a single model pass I suppose it's not even a complex system but you can also augment it with other like you can give tools to this model that can provide you context you can condition it with the customer vocabulary like you mentioned but customer cabbary is applicable where the vocabulary is like finite and fixed but in some medical domains it could be not infinite but quite large and in which case it can be a tool which the model can call and fetch appropriate context bites on

28:23a wall is shown and produce a higher quality transcribe that's definitely that's definitely true yeah it's a so interesting how we have these cascades now so I'm a top 1% user of whisper flow and I use it I talk to my agents more than I talk to real human beings apparently and it's so interesting so whisper flow will do some refinement of what I say and then it would go into my coding agent and it just has this almost clear void ability to understand what I mean so even if something was mistranscribed it will still understand what I mean and you see what I'm saying that we will have voice agents and they won't you know yeah they'll have some intelligence and now have some reasoning maybe even adaptive reasoning but then it'll probably go to another agent and the other agent will have more context and it's almost as if coherence will just manifest in in the cascade yeah definitely I think because now we have cascaded cascaded systems are source of errors usually like in the case of ASR LLM TTS but in the cascades that are emerging now with the use case that you're mentioning for instance the coding agents but also the co work agents where you are trying to

29:35leverage agentic systems to get not just coding but other kinds of workflows done and voice is a primary interface each of the components is also a smart system which tries to be robust against the mistakes made from the previous system so it definitely like overall works quite well but it's not a purely a when when I suppose there are cases where if you if you are trying to like in the context of coding if you're trying to be very precise about a function name or a method like method or some other aspect of your code base that you're trying to fix and it mistranscribes for whatever reasons because it doesn't have context and if there are multiple entities which could match what you describe you can imagine how it can it can go wrong so it's there still value in trying to improve the understanding layer whatever it transcription being the simplest case but in an audio understanding model it would be the holistic native audio input LLM or native audio input agent I suppose I think there is still a lot of value in trying to still bring down the

30:54the basic audio understanding quality and that's what we're trying to do with both our transcription models by making them more contextual so it can give all kinds of context that just as a transcription model and you then use this transcription to feed it to whatever downstream system in whatever way you want or in a more into and model care about the specific cases where the failures are costly and try to address them. Let me read this out so it's a decoder only transformer trunk predicting audio as auto regressive continuous latents reconstructed by a neural codec so I think we should talk about the continuous latents and the neural codec because they're both very interesting so first of all why the continuous latents. Yeah that's a good question I think it came after several iterations so we were trying to take a architecture for generation and that was so the walks to TTS model which we released in March the goal is dual fold I suppose want to provide a offering from mistral for voice generation voice synthesis for mostly voice agents but also other use cases.

32:10And at the same time for us to also do some research and come up with a architecture that we're happy with and continue to it right on both to improve the TTS model itself but also to incorporate in a future speech to speech system and the lay of the land is essentially neural codec based auto regressive generation seems to be a dominant paradigm when we started and it continues to be the case but the key difference is a lot of models then were based on discrete token auto regressive decoding and the attractiveness of discrete tokens is it fits well into the existing text token based auto regressive frameworks. So if you so the neural codec itself would have a quantized bottleneck which is producing discrete tokens and examples of this is encodex sound stream me me and it's not just discrete it's not just one token so similar to how I said the audio encoder for the understanding model produced a embedding every 80 milliseconds it's similar here but instead of an embedding it would be a

33:27set of tokens usually corresponding to a residual vector quantizer and you also have a audio decoder here or work order in some cases where which basically reconstructs based on these discrete tokens and the model this whole auto encoder is trained end to end to minimize the reconstruction losses and a bunch of other losses and the discrete tokens are then used in auto regressive decoder model which is the let's say the TTS model which takes text conditioning and produces the discrete tokens one by one. So once the two your question why continuous so at each time step these discrete tokens what is not just a single discrete token like in text it's a set of discrete tokens usually an ordered list I suppose in the residual vector quantizer case and the way many models approached is they have a architectural affordance just for handling predicting this multi token set.

34:34There are many ways to do it there is a delay pattern way to do it but I think one of the more popular ways was to have a depth transformer so it's also auto regressive encodes where at so there's auto regression in time and there is auto regression in codes so we are trying to see if we can reduce this complexity around this auto regression in codes because the number of steps you need to do in the auto regression codes is the number of code books that you have. And in our case it was 30 plus code books so it's it's at every time step you're doing at 30 step water regression again although it's a smaller model and given this consideration we wanted to see explore approaches which are more controllable which provide you a more delicate trade off between the number of steps and the quality and so we started exploring diffusion based approaches and the model that we are going to do is the same. The end up with was the flow matching head so instead of a residual vector quantizer which is a discrete set of tokens we have a continuous latent embedding in this case and the model takes the as this flow head takes as input the final hidden state from the decoder transformer and does the flow matching.

36:01Inferences I suppose where it's doing a velocity prediction it's very similar to denoising task but slightly different in that it's predicting a velocity on a velocity field and the inference is to do a integration over this velocity field which comes down to just a simple summing of velocity with the time step over a fixed number of steps and this provides a neat trade off. You don't the number of steps is controllable and you can also there are also techniques to reduce the number of steps with a few step techniques so that's one of the reasons we picked it but the continuous embedding is also attractive in that the discrete codes in some sense also have this connotation of communication where you're trying to reduce the bits of information that you're sending over a channel but in our case that is not a consider we are not.

37:01Using this neural code for compression or communication over a band limited channel or something like that it's purely for generation purpose so that bottleneck doesn't make sense in our context and that's one of the reason we also thought this would be the approach which allows us to expand the bottleneck and explore a larger design space. It's so interesting I remember the the encodic paper I think it was met her. I think it was about three or four years ago that that came out and it is fascinating at the time pick up a can you just explain a little bit more detail about what that paper introduced and what the lineage was it's mainly residual vector quantities of approach so you have. It's it's an auto encoder architecture where you take the waveform on the input side and generate the waveform on the decoder side and you have a vector quantizer bottleneck specifically this residual vector quantizer bottleneck and then came Mimi which is an enhancement on top of it and one of the.

38:17One of the aspects of this family of models is this semantic code book and acoustic code books and the key difference between these is the semantic code book gets a distillation supervision the motivation behind this is to keep the this code book closer to the text space and the order in which you predict the codes in the decoder. TTS model or generation model is you generate the semantic code books first and the idea is that it's close to the text space so it's easier to predict and then condition on this predicted semantic code book you predict the acoustic code books which are assumed to carry more of the acoustic information which is not so you can have multiple ways to pronounce the same word and there is this variance and this acoustic code books are assumed to carry more of this information and. First getting the word broadly right and then predicting all these it's not as interpretable as I'm trying to say here but that is a motivation behind it but in our model the key differences we still maintain the semantic code book in the first release we did as a separate vector quantized code book but the acoustic once we made it a fsq based quantization.

39:41And it is a numerical quantization so instead of a residual vector quantization here we have levels so in this case we had 21 levels so it's a 36 dimensional embedding with 21 levels and since these are discrete levels you can treat them as a continuous vector without the quantization or with quantization you can actually treat them as discrete codes so that's one of the reasons we picked fsq we are. Actually were exploring discrete code prediction based approaches to begin with and then later relax them to be more continuous as we build confidence that the quality of generation was either similar or improving with this relaxation there's a bit of a theme here that we're going to talk about which is that when you build machine learning models if you want them to be more sample efficient and work it's more scale there's a whole bunch of tricks like engineering tricks that you can put into them and I think I believe this is one of them but.

40:41Another one for example is you could in principle in an ASR model just learn from the waveform data directly or what you could do is something akin to a short time for a transform with the so the the male spectrogram I believe is it's like a logarithmic Lee and it's like a curve which is calibrated to human hearing with some discretization and that's often used in machine learning as perceptual audio features for human dialogue and that's the way to do it. That is a form of kind of dimensionality reduction I suppose so you folks have used this so I suppose it's an interesting trade off like presumably if you had a huge amount of compute in a huge model and lots of data you could just use the waveform data but you decided not to can you explain the rationale. Yeah actually I think you covered most of the rational pretty well I think the view the viewpoint here being the model especially when you're operating in small models the encoder offers us to be a little bit more sample efficient I suppose and the on the input side there is no need to model the aspects of the waveform which are not necessary for.

42:00Human perception understanding and the male spectrogram based feature is one such feature is which is which is still pretty broad to be honest because it is not a I think in old school machine learning you have very handcrafted features which are known to help with certain aspect of the task at hand in this case this is still the male spectrogram is quite general it doesn't it doesn't just capture the speech it can capture a lot of different auditory features I suppose and background sounds. It at least empirically shown to capture rich information but it's still a believer on getting getting a little bit more sample efficiency and you're totally right I think as a models get larger and as you train these over large and large data sets some of these become unnecessary and you can feed.

43:06The way from directly into the model and even without an encoder which is becoming more common more recently and the belief also is that you're doing two accomplishing two things by removing these things one I you're letting the general purpose model find I guess a better point without your inductive bias which is generally shown to work as you. So you can go to more end to end approaches and secondly the idea here is that as you if you can feed in a more general purpose way the scaling becomes more predictable as you go to large and large models because you have one less component to worry about. Let's talk a little bit about noise so for example you know ASR models they they have varying levels of performance based on how much background noise there is and I suppose there's a whole bunch of things you could do you could do noise augmentation you could do some kind of data pre processing or voice isolation before the data goes in.

44:19I don't think you've you've done that it just works reasonably well out of the box but can you can kind of tell me about how you can make these models robust to noise. Yeah definitely I think noise adding a lot of approaches involves augmenting which is true with vision models as well augmentation is a way to expand the amount of data to cases and making the model in some way invariant to transforms of the data which shouldn't matter for the task. In the object recognition case for instance if you have a flipped image ideally it shouldn't matter and scaling of the image shouldn't matter in the audio case noise in some way is a similar transform in the sense that if you have speech that is being spoken up to a point noise shouldn't change what was spoken from a perceptual standpoint and hence the prediction task should stay the same.

45:24And we want to also make the model robust to various acoustic scenarios which we might not have coverage for in the data and this is a more automatic way to get these acoustic conditions simulated in the limited data we have and yet again work with the data limitations but for the at least for the initial versions of the model. We didn't have to do a lot of noise augmentation precisely because one of the things we try to cover in our data sets is to cover various acoustic conditions but having said that we do do some noise augmentation especially to cover the tail cases and it does add like robustness to these models and it's quite important like I think when people report ASS and the performance and stuff it's usually presented in on data sets which are like more academic data sets and the it's not like they're very clean system there are values like e valves like

46:37modular common voice and things like that where the acoustic conditions are a bit more noisy bit more uncontrolled because it's people recording on their laptop in all kinds of background settings but still when you deploy it in the wild the diversity of acoustic conditions is like lot more and usually models which are specifically tuned for a specific set of acoustic conditions sometimes struggle to generalize to these other scenarios and you get models which sharp drop off in performance and that's that's where we're trying to avoid. Yeah because I suppose even the data set selection and augmentation that is an inductive bias at the end of the day and it's one of those things that as you say if you have sufficient diversity then it will just learn to generalize so you know at the small scale it might overfit to certain acoustic environments which means on this particular microphone it works very well on this particular microphone it doesn't work very well and I just wonder how much work have you have you folks done around that is this the kind of thing where you identify failure modes and you lean into them so you say OK we need we need to do more adaptation and augmentation here or have you just reached a sufficient level of data and diversity where they just kind of washes away.

47:51Yeah that's a great question I think the model is generally performant in wide variety of acoustic condition and like you said when we identify areas where it doesn't work well from as a function of feedback from customers who deployed it in the wild we try to gather data which is covering that distribution and try to improve the model but yet again I think this is one area where the model adaptation is quite. Important and that's also used to the popularity of fine tuning whisper in the community because usually when you deploy this kind of ASR model by virtue of deploying you have the data distribution the precise data distribution that you care about it's it's the one that you where you're using this model so you're very well equipped to adapt the model time through especially even if you have. A small reasonably small amount of competitive the large corpus which is usually used to train these models the small amount of exactly in distribution data and having some of it annotated to a high quality transcription having an internal workflow where you can transcribe these.

49:12The audio and that's one of the reasons why we have forge the model adaptation platform where you can take any of our existing transcription models and with your own data you can adapt them and usually one of the concerns people have have is. Some of this data might be very sensitive especially if it's like meeting recordings in the context of a company or something like that so you would like to keep it yourself I guess and find in the model and have a variant which is very very much applicability you and performs really well in your situations you don't care about how it does in other acoustic situation because you deployed in this very specific city you don't want don't need a general purpose model I suppose in that way and if you're happy with the performance you can. And deploy it in the context of your company or wherever you deployed it. Yeah and can you tell me more about fine tuning so is it one of those things that with great power comes responsibility and you need to be extremely careful about you know still maintaining diversity and your fine tuning data or is this the kind of thing that you folks can do as a platform so you can still inject more diverse data just to stop the model becoming degenerate I mean talk me through that.

50:28It's it's more so for the text models but even for audio models it's important to have a diversity even when you're fine tuning precisely like you mentioned to avoid the catastrophic forgetting of the general general purpose training the model has received in the earlier stage. But at the same time I think for a task like transcription it's a little bit easier to find you in the recipes fairly straight forward and people have been doing it for a while in the community especially you see a lot of fine tune transcription models on hugging face for instance because the transcription task I think is fairly well understood at this point and the the thing that I'm going to do is I'm going to do it. The thing that you want to end up with a good fine tune artifact is having high quality data with precise acoustic conditions you care about a noise conditions and recordings and etc along with like high quality transcriptions which is very important because if you have mistakes there.

51:38It is especially since you're fine tuning it's not part of a large large stage training it's important to have high quality data and then starting from a model which is already fairly good at it and that's one of the reasons we also pick text based trunks for our transcription models and in future it would be a holistic audio capability as part of a large language model in general because transcription while a lot of it is very localized. It also benefits from just world knowledge I think for specific it's very obvious in the case of like getting entity names right. If the model is already aware of these entities it and based on the context in which you are speaking it can easily guess so starting from a good starting point and having a high quality data set are the most important ingredients but some of the general principle. General principles about having some diversity in the fine tuning stage to avoid forgetting the catastrophic for getting is also applicable in our case for text models it lot more applicable you do need to have a good mix otherwise there is a getting the details right is quite important fine tuning and that's where the platform part is also important because we would offer recipes which are well tested in various contexts.

53:06And usually should work reasonably well out of the box very cool now let's talk about speaker diarization. Now it's possible to do this in a specialized model but you folks have built this into the model so the model is actually outputting speaker change tokens as part of their output and maybe you should explain whether it's in your streaming model and your batch model how does that work and it must be quite a difficult problem what what kind of issues do you have with the model. Yeah that's a good point like you were alluding to the way we do speaker diarization is as a auto regressive task it's not a special head it's not a separate stream in our offline or async model the way to usually done is you have the audio and then for a plane transcript the model would just produce the text transcript. And in the context of timestamps we just produce segmented transcript where you would have the start time of a segment and then the transcript and then the end time unless some special tokens to represent the time delimit is I suppose and the model would just predict the timestamp for this speaker it's basically one additional piece of information on the segmented timestamp in addition to the start time on the end time the model also predicts the same.

54:32The model also predicts the speaker ID it's a fairly simple convention it goes from speaker zero to speaker I assuming there are I plus one speakers in the conversation since the model is trained the model is trained with the same auto regressive loss just with the data with the speaker representation incorporated in the case of the diarization training. I use a lot of ASR tech myself so speaker diarization is a great example of something where prime a face when you look at it looks all great this is done an amazing job and then when you actually look at the speaker changes you see the failure modes and what I what I tend to see is that sometimes it just takes a few seconds to get the speaker change so it will get it but it will just mix it up by a few seconds and what I also see is that there are significantly more failure modes on the streaming models than the batch models. And what I tend to see is that they will create additional speakers so the batch models will be quite good at sort of correctly identifying let's say four speakers but the streaming models might kind of like pick up a fifth speaker and a sixth speaker when they were actually previous speakers.

55:40Yeah it's streaming models are fundamentally more challenging just because they're operating with less operating with less context and they have to make do with whatever context they have like for in our API and also in as a recipe for the streaming model that we have the model provides flexible target latencies like I was mentioning earlier in our conversation so you can give it a target latency of say one six to me a second. And you can give it and you can have another stream open where you give it like a second or two second delay and the idea behind this is for anything you would want to display to the user right away like in the case of subtitle you want to showcase the transcript with as less delay as possible just to keep it relevant for what's going on. And then if this transcription is going into being logged somewhere for for future reference in that case you can use this slow stream to correct the mistakes on the fast stream more context it has the better it gets and with this model it's as simple as just opening a second stream because the delay is a parameter that is fed to a particular stream that is open until point on derisation yeah it's actually quite challenging I think it's going to be a good thing to do.

57:08It's far from salt in my opinion, especially multi speaker more than two speaker in the context of a meeting for instance even day to day meetings that happen in the company where they're like four or five people speaking over each other it's very hard to get that right the challenges are quite diverse because the audio model is like as a human you also have a visual input you have a kind of you know the audio source where it's coming from and you are present in one part of the room and you know where all the other people are located so you have a lot of additional auxiliary auxiliary signals which you can leverage to disambiguate who is speaking when and when you actually listen to these meeting recordings I don't know if you ever try to annotate very painful exercise and it is I am not going to be able to do that.

58:08Actually like you can empathize with the models there it's a really hard task to recognize who is who without you have no video just just have to listen and and you're not listening to the people that you know who you already know how they sound these are people from a random meeting who you never met and some of the people do sound somewhat similar and you don't know if they're the same person speaking and then you have to take a step back and read the context and see oh okay it doesn't make sense for the same person to be interrupting themselves so it has to be a different person or something like that. So it's a very challenging task especially since the way we cast the task is a audio only single stream collapsed. There is definitely a ceiling that you can achieve with this constraint that you have with this but having said that the current performance is well below that ceiling in my opinion I think especially in a more than two speaker noisy environment and also regular speech even in a two person conversation and especially in more than two person conversation has a lot of overlap which is which is just a natural property of speech it's not a problem.

59:21Not taking that's not I speak and then I abruptly stop or you only start after I completely finish my thought that's not usually how regular dictate conversations go and that adds to the challenge because you'll have many many segments of audio where more than one person is speaking and not only do you need to recognize that more than one person is speaking you recognize who all are overlapping and what each one of them is speaking independent. Very cool. Yeah so you know that the cocktail party problem has been a huge problem in speech tech for many many years which is this famous setting where you're in a cocktail party and there's many many different people talking there's John talking behind me and we can focus our attention right we have two ears and we we are are is are actually doing this kind of time based recognition so to put you know when when the sound waves hits both of our ears we can actually distinguish our brain can distinguish. Different sounds and and as you said you know like these models they they don't have all of that additional awareness so they they they can't do the kind of conversational understanding of the dynamics that that we can do and maybe in the future that'll change right because we're you know we're going to have models that have duplex and and have much more information but on that note though I did notice working with speech models about two years ago some of the front is speech models could do what I can only describe intuitively as active as we can do that.

1:00:50And I think that's a very important thing to do is to talk about the active speaker locking which meant if there was cross talk it would lock on to what it thought was the active speaker and it would continue to transcribe that voice and ignore other voices and maybe that was an emerging property of that particular frontier ASR model but have you noticed stuff like that. It also comes from the closed caption data I suppose it's usually transcribing the active speaker and the models which are trained from data model at the end of the day I think a lot of properties of the model also can be attributed to the annotation quirks and it usually explain with it not everything is explained by but a lot of it is usually explained by that I think if you take a second look at the data you'll see a lot of. Properties of the model is exhibiting can be attributed to the properties of the dataset that you're trying it on but having said that it is a challenging problem but at the same time the models don't need to be the model the whole point I suppose is that they're at least in principle can be better than they don't need adhere to the constraints of humans they can be super human that way and unlike unlike us.

1:02:07We can only focus at one speaker at a time and we have this nice ability to decide who to focus on which is pretty cool but. Models at least as long as they have enough information like in this case if they have instead of a single channel or instead of a single mic let's say they can take input from several mics across the room and they have additional information where these mics are things like that they can. Follow all the speakers all the time it doesn't need to focus on a single speaker in in a more simple way in our model we are trying to tackle this problem when there is cross talk or overlap speech the expectation is the model transcripts both the speakers and identifies when the first speaker stopped and the second speaker started and even when the start of the second speaker is before the first speaker stopped speaking. So this is trying to transcribe multiple speakers at the same time but it's a limited setting in the sense that the input is still a single stream collapsed audio channel I suppose but it is at least the goal is to make it super human and it is actually does better than humans in quite a few settings but there's still always room to go go for the great stuff let's talk about hallucinations so you remember back in the day.

1:03:35So it's almost a day of GPT 3 when it first came out and it would get stuck in these degenerate loops because it's almost surprising isn't it that we can use an auto regressive model and it's coherently not only transcribing things but it's putting you know like a time index is in there and speaker change labels and so on it's amazing but there are there is still occasionally hallucinations and you folks are using DPO to minimize those hallucinations. Can you explain that whole process as you rightly pointed out the hallucination problem at least the degenerate generation problem can be somewhat attributed to the architecture itself owing to the limitations of the existing auto regressive architecture and once the model makes a few mistakes. It tends to commit to those mistakes and especially when those mistakes take the model out of its training distribution that's usually the scenario where it goes into a degenerate mode of infinite generations or looping the same prediction or skipping a whole segment of transcription because it decided to skip the model.

1:04:55One part of the transcription now it continues to skip the whole part because it doesn't want to predict halfway again owing to the out of distribution nature of its initial mistake I suppose and the way the many ways to tackle this and DPO is a simple way which which is what at the same time quite powerful to fix issues like this which are. They fall into a small set of categories and the recipe here is fairly simple and you collect the generations from the model which have been degenerate and you generate a correct version of them and that would become a winner loser pair and the DPO training process usually involves training on pairs of data like this directly as opposed to a. Explicit reward model that is also a equally valid and effective way that also works DPO is just a single stage process and for fixes like this which which you can consider alignment of sorts I suppose it is quite effective so you would have a winner which is the correct transcript and a loser which is degenerate generation from the model and usually you would look at the model and you would see the same thing as the first one.

1:06:24I like to limit to the distribution where the losers are actually a generation from the model so it's more in policy of the precise model you're trying to fix like you're not trying to generate infinite generation for scenarios the model never actually does which is off policy and for all the category of errors you generate pairs like this and it it is quite quite effective in fixing most of the problem same thing works with hallucinations or skip transcriptions or. Some family of transcription mistakes it's not it's a fine tool it's not a way to support a new language or something like that it's a nice post training tool to fix small but important problems in the model and can you give me your intuition on what's actually going on there with DPO above and beyond let's just say using supervised fine tuning because my intuition is as you just described your health is not going to be a good thing to do.

1:07:24So helping the model develop a kind of conceptual space understanding so this is good this is bad and and it's kind of exploring around that space reflexively and you're kind you're kind of like helping it conceptualize in a way that perhaps wouldn't be possible in the base model or with sft I mean is that a reasonable intuition. Yeah that's a reasonable intuition and I at least from my point of view one of the primary tools that the DPO gives you is the way to give negative supervision with both retraining and sft the supervision is purely positive you're just reinforcing what is the right thing to do in pre training it's just general text corpus and in sft the specific behavior you want to get out of the model whether it's a chat model or a transcription model but they don't offer a mechanism for the model. For the DPO to say this is wrong like penalize this like reduce the probability of this sequence and DPO is not the only way even many of the are all are techniques are all adjacent techniques offer a similar mechanism and DPO is one of the simpler variance which provides you a mechanism where the winner gets

1:08:45reinforce and the loser or the one which is erroneous gets penalized by the objective function. Yeah that makes sense so you're saying the the most load bearing thing sorry to use a Claude is in their load bearing my god I've been using Claude too much but you're saying it's mostly pushing the energy landscape down on the negative so we have the base distribution we push the energy landscape down but is there another thing going on as well there is it is it pulling up the energy landscape and it's not going to be a good idea. So the energy landscape and is it actually learning and exploring more in the positives is is that part of the story as well. Yeah it's it's so the the loss if you see the last term of the DPO it it looks like a weighted positive loss on the winner and then it's a negative loss on the loser it's not there is a weighted nature to it and nice formalism in the original DPO paper as well and there is lot more lot more fall of work trying to build in Tishon what the algorithm actually accomplishes but it it is a margin up to me it looks like a margin loss which pushes the positives on the more positive side on the negatives to be

1:10:03predicted less by the model after the stage along with a component which tries to keep the model not too far away from the starting point so you don't want to move too far from the initialization because you want the DPO model to fix the subtle mistakes while at the same time keeping most of the properties that you got from the sft stage. And how do you detect degeneracy and presumably you use that as a lens to synthesize or select these positives and negatives for DPO so how does that process look like it usually comes from many different places the e valves is one of the primary ways to detect this. You can define metrics for each of the degenerate subclasses in finite generation being one repetitive generation or based on looking at the error patterns that you yourself notice or reported by the users of the model you can and there's some error patterns which are not captured by your e valves which is a good sign that you need to build a new evil to capture these cases

1:11:19because it's likely to happen in future models as well. And once you capture the error patterns then you would have a data generation phase which would involve reproducing these kind of bad patterns and having pairing up with a ground truth equivalent in those cases and it totally depends on how you source this data set. There are many many ways to do it. You can have a ground truth transcript annotated data set and you can run your model and you can take those subset of cases where it produced any of the triggered any of the detected that you had for degeneracy and generate a paired sample out of that very cool very cool. Let's move back to TTS. So the technology is getting incredibly good you folks have a TTS model. I mean what fascinates me is like where does the emotion come from.

1:12:22You know like where does how does it even do that? Yeah the model is it the way the model is trained is like the exact opposite of the ASR tasks then put would be text and the output would be audio. In our case the loss loss is a flow loss which is an L2 loss for velocity estimation. And the model is trying to model distribution of human speech from whatever data set you have and any reasonable data set that you have where there is enough variability of human speech which you should we should see from any any audio recording I suppose and the model makes associations like any other deep learning model the between the actual semantic content. So the first level of mapping it needs to do is map the words to appropriate phoneme implicitly older TTS models explicitly did it like phonemes were either generated or it's an intermediate stage of the model.

1:13:34But in the more recent versions of TTS architectures and the one that we actually released it's completely into and and this kind of association happens somewhat late in the model. And a first level of mapping is to just get the what has to be spoken the high level thing and then the next level would be a function of the semantic content of the text. You can have text based on the semantic content you should be able to not always true because you can have neutral text which depending on the emotion of the person can be spoken in different ways but there's some text which is only spoken in certain way like for instance you can have sports commentary so if you read a sports commentary it would have lots of ups and downs depending on what's going on the field. You'll have very excited intonation ups when there is a exciting event going on like a goal being scored or a home run in the context of baseball you can have all kinds of ups and downs in a game and just based on the text you should be able to tell reasonably and ambiguously where the emphasis should be.

1:14:47And for all the other places where there is multiple options I think one of the aspect of the TTS learning is this one too many learning which is the somewhat in in one way distinguishes itself from the ASA task where it's like mostly a many to one mapping in TTS it reverses itself from one to many. So I think it's a multi model distribution I suppose and that's one of the reason why techniques like flow or diffusion based techniques work really well because it models the multi model distribution quite well. Is there a bit of a trade off between kind of the naturalness and the variability because I mean I'm sure many folks are home you know you go on 11 labs and you generate a bunch of TTS utterances and they vary quite a lot and you know humans are incredibly good at detecting change in tonality and also there's a lot of meaning in tonality and what we don't want is to have TTS systems or voice systems that are disingenuous right so what we tend to do is we just

1:16:01do many, many generations and we select the one that sounds most natural but obviously like in real time production systems you just got to take the one that's generated because you can't go back and regenerate many, many times so is it is something that you know do you take a position on this I mean do we do we want to have something that's quite stable or you know very human but run the risk of it being disingenuous. Yeah in the case of answer the start with the question around what if our stance on it our stance is making it customizable so as a user you get to choose which mode you want to operate the model in and some of the customization the there are few knobs to control the behavior of the TTS model and the biggest one is the voice you pick from the system and the model does support the model. So you can give a recording of a person the persona you want to give the voice is the is derived from the reference so for short references where there is quite a bit of ambiguity there is opportunity for more variability but even there you can control the behavior of the TTS system by having voice reference which is quite neutral and quite safe and less.

1:17:23Variance and on the other hand if you're using it for a creative use case which is not real time which is also a perfectly good use case for both RTTS model and many other TTS models then you would actually want to pick a more variable somewhat even wild variances in some cases because you're looking for different candidates in your creative process to pick from and that's actually a feature not not a bug and so depending on the use case you can pick the property as a research scientist and as a research team when we are training the model one of the goals that we want is to train a model which is general enough that it can represent both behaviors in the same model and control that behavior through the knobs voice being the primary one but will also plan to expose instruction for the model. So I'm going to go ahead and try to get the model into the model and you can see that the TTS model is not a very similar to ASR you can say the word that is not there or skip the word that is there or keep repeating the word and going to infinite loop or similar problems exists although the goal is to be much more robust to these problems because

1:18:51this model will be used in real time context and it's much more jarring to see these bad generations in an audio output as opposed to a text output because you have to listen to them. So we aim to make them much less prevalent in this but based on your use case if your tolerance is extremely low because you're putting it in a real time context in a customer service agent or something which interfaces with consumers at scale where you have hundreds of thousands or millions of consumers then picking a safe voice seems like the more conservative option to go but that also depends I suppose because you can you can have a there are all kinds of audio agents now not just for professional consumer service that personal use cases where people do uses might expect more variance from their audio agents in which case it becomes a trade off based on your use case.

1:19:58Yeah and what kind of use cases are you expecting your customers to you know to derive from these models I think if I understand correctly you are not currently going for the long form generation it's still quite short form. Yeah it is a short form generation model it can generate long for more you it's just that long for more you has properties which are a little different from short for more you like imagine generating a podcast on an audio book you like like it to stay consistent but at the same time you would want inflections are long form consistency and variability in the long form so in some cases so that it's not more monotonous so the considerations are little different and you can put together a system which can leverage the short form TTS to produce long form again the TTS model is also contextual similar to the ASR and the hope is you give it more context you can give it previous or the conditions it can continue the generation so that can help you keep consistency across a long form generation but the primary use case is short form generation because the deployments in areas or voice agents and primarily enterprise focus voice agents whether it be customer service or productivity assistance which are customized for enterprise use case like I was mentioning one of the things that I was going to do is to get a good example of the use case.

1:21:20So I was mentioning one of the things in why work and why code is ability for people to also make custom AI agents for their workflows and the hope is you have a similar lever with audio agents as well where you can customize an audio agent for all kinds of one of use cases in your enterprise context. You can deploy them like for all kinds of use cases in an enterprise so in these cases there are all productivity assistance in some way but they are aimed at a slightly different thing. So those are the main use cases that we are targeting with the system which is productivity assistance and something which interacts with your external customers through customer support being the top one but that itself is a big category with lot of subcategories. So when there are adjacent ones as well similar to that very cool now if I understand correctly so you folks are about sovereign AI and allowing engineers to build systems from their component building blocks.

1:22:33So engineers would build cascades from your models they would integrate into their systems and they would wire them all together. Obviously that gives incredible flexibility because it is running on premises and it is small models and they can fine tune it they can control their data and all of that kind of stuff but is it also quite difficult as well do you provide recipes I mean how do you make it easy for folks to compose all these things together. Yeah so one of the things we also have is a applied science applied engineering team which is also like a forward deployed engineering equivalent and the main idea behind this is we do provide the building blocks for you to make workflows or proof of concept exploratory product products slash applications in your enterprise context or any any other. AI related tool in the context of your enterprise and usually this is the cases where you already have a.

1:23:37Three set opinion and experience on what you want to build and you're just looking for building blocks to just use in what you're building in which case you it's kind of like a combination of a self serve plus interaction with our company to get that going or in the other case like model adaptation or customization or where it's a. Slightly open exploratory project in that case the usual workflow here would be to engage with our applied team and work together and our job would be to make sure your use case the success of your use case and finding the best way to get it done with the tools that we already. Yeah I mean just a sketch out an example if I wanted to make a customer support agent using this tooling so I guess I would have some kind of database with intense in I would be using a language model agents and then I would be using TTS to like you know to generate some kind of conversation flow and then I would be using ASR to listen to the customer when they said something but some of the complexities here I guess are when the TTS is talking the ASR needs to be turned off.

1:25:05Because otherwise there would be a feedback loop and you see what I'm saying so so we would kind of construct this cascade but there's a little bit of complexity there just just to stop the system from going crazy use cases precisely this kind of stuff so you want to get started right away you don't want to. Spend a lot of time building this scaffolding which has already been somewhat solved the cascaded system itself will have limitations owing to it being a cascade but. If you want a state of the art cascaded system today with the state of art components and the components we are building hopefully are pretty state of art especially for you use case at the end of the day you care about whether it's. Frontier for your use case night frontier on some external benchmark and you can adapt either of the components in the case like we were talking about how to adapt a ASR model for your acoustic settings and the same with the TTS yeah because I think that this is the thing that I want to get across that when you can do adaptation and second strengths you can always strictly win because a lot of people at home they might just be saying well you know work on I can I can.

1:26:18I just use a nominee model or G. B. T. live or something like that and there's almost too much flexibility right I'm I'm in large enterprise and I want to have an observability layer I want to have hard constraints I want to say when people are talking about this it needs to be logged in the database or this needs to respect this security boundary it needs to be running on my server because we're fine shooting it with my data and you see we can we can do a lot of customization and when we do a few iterations of that we can actually make sure that we can. That we can actually make something which is strictly better for my particular use case exactly I think the current voice act. Owing to it being a cascade also gives you a lot of observability into the system big uninterpretability because each of the components the interface is natural language which is quite. Interpreter which makes it quite interpretable and on top of it these audio agents. The LLM component of it has a text counterpart in many of the cases like if you're deploying customer service agent you probably also have a text version of this customer service agent where people can chat with it over it takes based interface to get you a problem.

1:27:34It's all like if you have a food ordering service and you probably already deployed a text based interface for dealing with customer complaints on delivery problems or getting the wrong stuff and you probably already have a safety system on top scaffold observability traceability and all other enterprise compliance layers added on the case for instance you just want to add a audio interface to this system that you already have and this allows for you to build it fairly easily because in the audio agent your goal is to not. You could be but in many cases your goal is to just give a more pragmatic practical interface to the system through audio so you can make a phone call and interact with it and the goal is to not have the most personal emotional conversation through this system but rather just get the task task done in which case it is a fairly powerful system to replace with a more.

1:28:50More end to end approach eventually sure I suppose but today these off the level of flexibility and control you get out of these systems is quite nice and what does the front here or voice tech look like today. Yeah a lot of places I think there's a lot of claims about even the basic ASR task being solved but when we talk to customers we spoke to a few customers just a couple of weeks back and the primary complaint is it's far from sold in the precise scenarios that deploying it and it makes a ton of mistakes even for the most prominent customer service case they feel it's not sold they have to add so much. So much scaffold just to deal with all the conic cases that you encounter when you deploy an audio agent over millions of sessions so there are their components around which is not necessarily model related problems but there are very much model related problems as well including things basic things like the model not being good in languages it has a sharp drop off outside of the top languages again owing to the data there's a lot of English.

1:30:06Audio data in the wild similar to it's a very similar story on the tech side as well but in the audio case it presents itself as the transcription quality going down significantly when you go off the top languages when you go off the acoustic like take an example where I audio system is deployed on a factory floor which is fairly common use case I suppose because you want to provide a interface to. Control and automated system not necessarily an agent but I'm sure it would be agents going forward because they can help you solve hands free when you're already doing certain other job or managing having mentioned or something like that and in those cases you can have all kinds of background sounds noise a lot of people speaking with each other's there's a.

1:31:06Crowd noise going on so in that setting for instance getting especially for an interface to a command system you need to get it very much right and in many many cases the current systems for for short and that's one of the reasons why the customization is quite interesting it's a while we always aspire to incorporate more and more variability into a. General purpose main model there's always a data or distribution which is not captured by the which is probably so unique that it is only in your context that you see that kind of audio and the model is reasonably good but you can always make it better and since it's not unlike text models where I would believe it's also true with even any general purpose reason.

1:32:06So you can always make them better for use kids but the cost of doing so is so high and the general purpose model keeps improving year over year where the the math changes and you'd rather use the general purpose model and hope for the best but in audio that you don't need to be constrained by that the models are fairly small customisable and you do have data to customize and the cost of experimentation is easy you're not trying to. You're not trying to make one T model here so I think it's easy both logistically where you don't need that much GPU compute to do it and also easy from the data side it's a fairly simple task it's hard to get the quality control right to get the annotation quality high but at the same time the task is fairly easy to describe just write down what you hear. So I think in the wild performance is the main thing that is not not solid for a wide variety of settings and language coverage is one of the primary things I always hear when it comes to audio models.

1:33:22Yeah and I think that voice tech is going to become quite ubiquitous over the next five years I think voice just seems to be almost a special case I don't know where you agree with that is something that people will increasingly kind of train and it will be embedded into their ways of work and and their life in a way that perhaps a large language model might not be yeah it's a voices. One of the primary ways humans communicate definitely came much before text was a thing and even today there are a lot of places where voices the main way people communicate and even in text heavy. The environment I think voice is still maybe not the primary way people talk like in work context but it's still like we have meetings all the time you don't need to have meetings you can just work over slack but it's a adds a level of depth which cannot be conveyed through just purely textual interfaces so it's always been the case that voice would be a big.

1:34:29Interface for interacting with machines and even other humans as well and it's always envisioned in like sci fi and that's one of their goals I worked on Google assistant for a while and that was actually the goal there as well for instance but I think the previous generation of technology was not not quite there with a. Even today's cascades are much more powerful than the previous generation of cascaded systems and but I think today's technology that a point where there's very seamless on the present voice based assistance is becoming more of a thing and the cascaded approach itself can solve a lot of the problems but I think that's where more and when a duplex approaches are also coming up because the technology is more. It's not making it possible to build more robust duplex approaches it's not sufficient to have like a prototype demo it needs to actually work in a real time context and all kinds of weird corner cases that you would encounter on a day to day basis and whether it be a real time translator to make make it more accessible to talk to someone who doesn't speak your language or as you are all you think to as a language learning tool.

1:35:54I mean there is a primary thing where people try to learn language by reading it never happens that you have to talk to someone for regularly to learn a new language and with a voice agent which can actually capture the nuances of the target language that you're trying to learn and also captured the subtle mistakes you make in pronunciation or aspect of you speaking that language you can improve much more significantly. So it's a lot of opportunity language learning and also just as an interface to AI agents for getting as a orchestrated to getting work done. It probably the interface to this would look like how a human human interface would look like in today's organizations where you're delegating work to other humans or collaborating with other humans to get your work done and you speak to them. Many times through a meeting and I can imagine it might look very similar with interfacing with AI agents which are also getting your work done and you can describe what needs to be done in the in an audio interface and they can communicate back in an audio it's not the exclusive way of communication they can communicate through text or whatever other channels but audio is definitely going to be.

1:37:21One of the key pillars of interfaces I suppose I don't know whether you would agree that in some ways it's it's actually cognitive debt promoting. So I love listening to audio podcasts and it's it's a non interactive stream of information and you know audio books and it goes into my mind and for close ended applications it's wonderful so like we have a let's say I build an application with a very clear interface and I'm talking to you. I'm talking to the voice agent and if I already understand the interface and I understand the information this is this is very efficient but if I use a voice agent for doing coding or my email triage or researching a podcast that's an absolute nightmare and I'm trying to understand why it is I think it's because when I have the text in front of me using an agent I can very rapidly kind of select interesting branches of inquiry and I could say go there go there and it might just be because the voice agents aren't very intelligent and sometimes I don't trust what they're doing. So when I when I tell it to do something in codex I don't know whether it's using my existing codex model I don't know whether it's giving me an answer or whether it's rooting the answer to the codex model I don't know whether it's making a new codex session or using an existing one and and I feel a little bit blind.

1:38:35So you see what I'm saying it's kind of like magical but there's also this weird new form of cognitive debt at the same time. That's a great point. I think that one of the things there was one experiment that we did a while back not not here but basically if you try to get people to order make make a food order people try to make it through pure audio interface and it's much harder to do it because when you have a menu in your hand you see all that. So you are talking to the person who is taking the order but you're doing two tasks at the same time you're absorbing the information making a decision going back and forth and also communicating with the person and if you do it imagine not having the menu so the person will read out the person serving you would read out the full menu and now you have to like to like load it all in memory and you'll forget like because the new items are coming in and a rapid fire succession. So it's just not the optimal way to do it but it's at the same time people still do at least a subset of people still do enjoy interacting with the waiter and making the order while having the menu.

1:39:49I think the same philosophy applies to these voice interfaces too. It's still very nascent so I'm sure it will evolve over time but basically it would look like a mix where you have your favorite ID or interface basically a visual medium also and voice is a augmentation or auxiliary thing on top. So you don't need to worry about the visual medium provides you two things it provides you a way to ingest information rapidly and go back and forth and also it provides you confidence and proof of what it what it says it's doing is actually what is being done so you have like two ways to validate what is going on. So yeah I think it like there are places where pure audio stuff makes sense like if you're wearing headphones or glasses or all these new new gadgets that people are exploring now.

1:40:51They are audio only makes sense because there's no screen and you would probably not do certain things there like probably wouldn't make it's like the favorite example of making flight bookings because it's a costly and a way to get it wrong you need to do a bunch of stuff to undo it so. That might not be the right task for that specific thing but you can still do it in an audio interface when there is a screen at hand for instance so it would probably be a auxiliary communication medium on top of what you already have and which works best people will experiment and figure it out and your other question on the audio models being less intelligent than the text counterparts today I think that's also just as a. Artifact of the way they are developed today and there's no fundamental reason for them to be any less intelligent than text model especially when it's a single model and even even if that is the case with I don't know how some of the other models are developed I can only speak to the models that we are developing and.

1:42:03It could be the case initially but there is no fundamental reason for it to be any any less intelligent or any different from it takes base model yeah I suppose I mean it's all a trade off at the end of the day because there's always a bigger model yeah so and it's not just the bigger model that this bigger model might have more context and might have access to my memory system it might be doing Google searches and whatnot so that they're you know there exists some perfect cascade yeah and we're kind of trade we're trading off intelligence and latency and a bunch of different things but no it's fascinating I think we're going to have a lot of fun over the next five years just kind of engineering all of these systems and figuring out how can we it's quite interesting as well that you know maybe the reason our memory is so bad is because we've become so conditioned to having these visual user interfaces and maybe when we use voice agents again our memory will improve and it might be less less of an issue wonderful stuff Paven it's been such an honor having on the show thank you so much for joining us today thanks thanks a lot for having me on the show it was a really fun chatting about audio and all other research efforts around Mistral yeah hope to see you again

1:43:17Ryan Reynolds here for Mint Mobile with your summer price forecast now unfortunately we're seeing rising costs across the country with possibility that big wireless hates you 100% now over here at Mint Mobile we're seeing sunny skies and dropping prices every plan to just $15 a month give it a try at mint mobile dot com slash switch up from payment of $45 for three months $90 for six months or $180 for 12 month plan required $15 per month equivalent taxes and fees extra new customer offer for initial plan term only greater than 50 gigabytes me slow and work is busy see terms