Traceloop OpenLLMetry — Open-source OpenTelemetry tracing for LLMs and GenAI across providers

0:00You know, when you board a commercial flight, you trust the pilot, but there's always that comfort in knowing there's a flight data recorder on board. Right, the black box. Exactly. It doesn't steer the plane, obviously, but if something goes wrong, it tells you exactly what happened. And today, as always, we were checking out the AI agent store.AI website page, and we found something that does basically exactly that but for AI. Yeah, it's an open source project called Trace Loop Open Elementary. And looking at the site, they highlight this really specific design choice, which is a 1% autonomy level. About 1% autonomy level. Right. Meaning, it makes absolutely zero decisions for your AI system. You know, it's whole purpose is just to observe and record. Which is exactly what we want to figure out today. Our goal is to understand how this tool actually monitors generative AI behind the scenes. Because traditional software monitoring is pretty straightforward, right? Yeah, you just track server response times or, you know, simple error codes. Right. But AI is just incredibly non-deterministic.

1:02You feed it a prompt and you could get like a completely different answer or resource strain every single time. It's total chaos compared to traditional software. It really is. So how does this tool actually track that chaos? Well, that non-deterministic nature is exactly why standard tracking fails here. With AI, you aren't just logging a crash server. You need a lot more detail. Exactly. You need to know the exact phrasing of the prompt, the hidden context injected into it, and how many computational tokens were burned. Oh, right. Token usage is huge. It is. So open-elmetry tackles this by hooking directly into the underlying code frameworks developers use to build the AI. So it just sort of sits in the middle and watches the traffic? Yeah, pretty much. It relies on the open telemetry standard. Which acts as this universal language for logging software performance. Right. And it automatically traces the whole life cycle of an AI request. It captures the input, tracks the processing time, and logs the final output. It gives developers a complete step-by-step reconstruction.

2:02Okay, I get why that data is super valuable. But you know, I have a practical question here. Realistically, most professional engineering teams already pay a fortune for massive observability bashboards. Yeah, they definitely do. So I just can't see a senior developer agreeing to adopt and monitor a totally separate platform just for their AI features. And they shouldn't have to, honestly. The primary value here is that open-elmetry doesn't force a new dashboard on you. Oh, it does it. No, it acts purely as a router. It grabs all that highly specific, complex AI telemetry and seamlessly pipes it directly into the established observability back ends the team is already using. Okay. That's a big sense. It adapts to their existing workflow instead of fighting it. Exactly. But deploying this does have some friction, right? The documentation shows it's installed as an SDK. Right. A software development kit embedded right inside the application. Yeah. But the catch on the website page is that they only really highlight setup examples for Python and TypeScript.

3:03That's true. So if you're engineering team codes in another language, you know, you really have to verify compatibility before jumping in. Plus, this is strictly open source. Like there's no paid tier listed and absolutely no service level agreement mentioned. Which means if it breaks, you're the one fixing it. Exactly. So it's a bit of a calculated trade off. It is, but think about developers building complex features like R.E. systems. Right. Retrieval augmented generation where the AI pulls specific facts from a private database. Yeah. If that system hallucinates or grabs the wrong data, the developer is flying completely blind without this kind of telemetry. You just can't fix a logic error if you can't see the exact sequence of events that caused it. Exactly. You need that objective truth because relying solely on the AI providers themselves to tell you how well their models perform isn't exactly objective. No, it's really not. Which is the big takeaway for anyone managing AI right now. You need a vendor neutral observer. And honestly, it brings up a really fascinating question about the future.

4:05How so? Well, as AI models become more integrated into critical infrastructure and frankly, more opaque, will independent vendor neutral black boxes like this eventually become a mandatory legal requirement for anyone deploying AI? Oh, wow. Yeah. That's a serious regulatory question. Accountability requires visibility. Definitely something for you to consider the next time a chatbot gives you a surprisingly specific answer. If you want to check out the tool for yourself, head over to aayagentstore.ai. Thank you so much for rating the podcast. It genuinely helps us out. Until next time.