LLM Wiki, Content Intake, and Defuddle

Yo, welcome back to the slightly caffeinated podcast. I'm T.J. Miller and I'm Chris Camille.

So Chris, what's new in your world, man?

Too much this week.

We've got a lot of stuff going on, but we are going camping with the Scouts tomorrow.

So we're recording on Thursday.

So I have the day off tomorrow and we're going camping from Friday afternoon to come back on Sunday.

And then it's a quick turnaround for me because on Monday, I head out to Chicago for an engineering team offsite.

Ooh, until Thursday, then fly back on Thursday.

My mom is also coming in on Thursday because it's my daughter's birthday the following week.

But that weekend, she's next weekend that she's coming in for.

We're going to the beach.

So that's what my daughter wanted for her birthday gift, activity, whatever for this year.

So just going to be bouncing around and doing the beach for the weekend, coming back the following Sunday.

So it's just a whole lot of travel, a whole lot of stuff prepping for things that are still a little week away just less time at work for a condensed week.

So just trying to keep everything straight.

I have like a million lists going and things to tick off and just trying to try to do my best.

Yeah, I very much understand that.

Yeah, it's a lot going on, man.

Camping sounds pretty fun.

Yeah, I want to get out and do a camping trip soon before like summer heat heads.

But it's April and Michigan is such a gamble.

Our my wife and I's wedding 17th wedding anniversary was yesterday.

Oh, nice. Congrats.

Yeah, thank you.

And we were just we were at dinner and we were talking about just reminiscing about like our wedding and everything.

And we remember it was so cold when we got married.

Like we were bundled up.

It was absolutely freezing and raining.

And yeah, I just remember being like we just remember being like so cold.

And it's like late April, you know?

Yeah.

And so we've had like a big mix of like 70 degree days and then like back into the 30s and just been all over the place.

But I forgot where I was going with that.

But you want to go camping?

Yeah, we want to go camping.

So yeah, I want to go camping.

But I don't want to like have that happen where it's like 70 degrees.

And then we call the sudden we wake up and it's 30, you know?

So I want to get out and camping before the summer heat really sets into.

I know that's a big goal for this year is to like get out and do a little bit of like camping with the kiddo.

He's been wanting to go really bad.

But yeah, so like I knew in my world that we had a wedding anniversary.

That was great.

I think our best wedding anniversary dinners.

Like we've ever had like we try to go out to like a nice spot for our anniversary and everything.

Like I was bouncing around around a couple of places and like settle on this Italian Mediterranean place.

And do this was out of this world.

My gosh, I cannot believe it.

So that was great.

And then over the weekend.

My son's really into rollerblading right now.

Like I got a new roller blades a couple of weeks ago.

He comes home from school immediately puts his blades on and then just like leaves like goes up.

Skates in the neighborhood goes over to his elementary school.

He went to and like skates around there.

I love it because technology is great.

He's got an iPhone.

So I can like track him and like figure out where he's at and make sure he's safe and everything.

But in a world where like it seems less and less common.

It reminds me so much of like my feral childhood, right?

Where we like it was like be back when the street lights turn on and like out of the house.

Go get out.

Go leave.

Go have fun.

Do things.

Be a kid.

So it reminds me a lot of that.

And so that's been a lot of fun seeing him get like really into it.

But when he first got his skates, he wanted to go to the skate park.

So it took him to a skate park.

It was packed.

And he was just so overwhelmed, really intimidated.

Honestly, I was really blunt with him while we were there.

I'm like, but I think this is over your head right now.

And he was really discouraged and like we ended up leaving without him really doing any skating because it was just like he tried.

He tried to roll in.

It's.

It feels like a lot like a pump track, but it's all like bowls and and like hips and stuff.

So it's like you can just kind of like pump and vibe through the whole place, which is cool.

So he like dropped in like a little roller and like eight shit.

And like it was just a big mess.

So he left.

But a few weeks go by.

We went back.

And he is now just like jamming through the bowls and like rolling in everywhere.

And like it was so fun.

And I got out and busted my skateboard out and like pumped around a little bit, which was fun.

But it was it was funny, you know, just just skating around.

And I haven't been on my skateboard in a minute and some kids were there on their scooters too.

And they were like pumping my kid my son up.

So like he was doing like bigger stuff.

And he kind of like one of the kids like made some comment to me about something about me skating.

And I'm like, Hey, man, for being almost 40 years old and pumping around a skate park like this is pretty cool.

Man, I'll take it.

I haven't done this in a while.

Give me a break.

It's not only like not only have I not like really skated at a park and like probably eight or nine years.

My knees are not up to the task right now.

Like, Hey, I am to the point in my life where ice knees and through my back out.

So I'm pretty lucky to be on a skateboard right now.

So y'all.

Yeah.

It was so fun.

So I see us going back and doing more of that, which is, which is great.

But yeah, that's really it.

That's all I've been doing.

I've been hacking a lot on Iris and we can talk about that too.

But yeah, yeah, it's what we've been jamming on.

That's awesome.

Yeah.

I don't have anything too exciting for a coffee basically.

Just enjoying the same thing that I've been doing with the cool brew in the afternoon or cold espresso.

And yeah, pretty boring over here.

Any changes for you?

No, not really.

I am cutting back on monster.

You know, I have a coffee in the morning and then somehow managed to get through like two or three monsters a day after that, which is ridiculous.

Yeah.

That's a lot.

I know that part of it is like using that caffeine to self-medicate my 80.

But it's so excessive.

That's so many monsters.

Like, that's just it's not good.

So I've been working on cutting back.

So I've really kind of gotten strict about like having my coffee in the morning.

Maybe a second cup of coffee and then basically like gating myself of like between like two and three 30.

You can have a monster.

That's your monster window.

You can have it then and then that's it.

So.

Yep.

Totally.

It was funny though.

Because last week when we were talking about monster and gas prices and stuff like that.

It was like an hour later, I think.

I got a text message from one of the gas stations about like, hey, say 50 cents when you buy like three monsters.

I'm like, they're listening.

Yep.

They're listening.

It's got me locked in.

My gosh, it's so funny man.

That's funny.

So yeah, I know we're on a little bit of a time crunch today, both of us are.

But I really wanted to dive into some talking about like what I've been doing with Irish.

Because I've been doing some like pretty interesting things and taken under taken a big, a big challenge there.

But it's all been really centered around working with this like wiki stuff that you've been working on with Obsidian.

And like I've, I've taken your blog posts and I've been starting to like contextualize that into context for Iris.

And then I've got corpathies on wiki description.

I've got that in there.

And then I've been kind of, I guess to jump ahead and we'll kind of go back.

I've been kind of taking the approach like lessons learned from you of just kind of letting Iris figure out schemas and templates like as we go.

And just kind of like letting it all evolve in like I was very upfront when I was talking with Iris about like what we're the mission of what we're doing of this is like.

Like look, I'm going to give you content.

We're going to work together to like synthesize and curate this content.

And then like your job is to like link it all together.

So like it should be built in a way that like I can open Obsidian and find content and explore and reference things.

But for the most part like outside of maybe some journaling like I'm not going to be writing these notes.

It's it's on Iris to like do all of that really.

And that's been a pretty good approach like so far.

I haven't ingested a lot, but I've been starting to like throw links at Iris and get some content in there like I had her pulling on my blog posts and a bunch of different things.

But like kind of taking a step back to the roots of all of it.

So we talked about this before.

Iris part of the Iris experiment that I was running initially was how much can we make this like a direct message experience of like it's me talking with my AI companion.

And it's like it would just be like you and I texting back and forth right wanting to use it more in a like utilitarian sense of like, hey, go do this.

You know, I didn't want these like one off requests or very specific conversations about things like polluting the context of like maybe my main thread of like my main conversation with Iris.

So I introduced threats.

And so that like the concept of threads is it's everywhere like Claude has them chat GPT has them when people first cut their hands on Iris.

They're like, where's threads like what is this?

And so I built threads that's been really good, but on top of threads this PR just keeps growing like it's crazy.

But for me to make sure threads worked in the way that I wanted them to I just kind of had to keep building up to the point where I can enable the workflow that I'm looking for.

And then I can kind of go back through the system and say yes, threads works well then like layered on top of that is this other thing.

So it's kind of been like, well, I know I need to do threads for like context management, but I don't know if I've like got the abstractions right until I keep building more.

And so this has gotten pretty crazy.

So like I built threads, then it was like, all right, well, I want now that we have clean context per thread.

You know, each thread, we've talked about Iris has like summarization systems. Now those are scopes to each thread.

So like the only things that bleed between threads are truths and memories.

And then each thread has its own summarization like process that takes place.

So I wanted a way to like have a thread have specific contacts and this kind of came out of your work with your open claw bot in discord where you have different channels for different topics right.

So thinking about that is you can really kind of scoping that channel or thread to a like contextual prompt, right.

Like this is what this thread is for. This is what we're doing in this thread and like that way anytime you talk in that thread, it's just always known, right.

Yep. So I wanted to mirror some of that. So on top of threads, I built custom prompts where it's there's a whole crowd interface for it.

We go in there, you give it a title, a title, a description and then the like prompt body.

You save those it's an eloquent record. It's in the database. But then in a thread, you can pin a prompt or a number of prompts.

So for example, in a thread, if I want to talk with Iris about the LM wiki, right. Like I have a custom prompt saved that has all of Carpathi's LM wiki.

It's got excerpts from your blog posts in there. And then at the bottom of it, it says here's the local path to the obsidian wiki directory.

And so whenever I want to talk about the wiki, I just go into the thread. I go to thread settings and then I pin that prompt and that prompt is now always going to be part of the system prompt.

So that context is permanent, which is like really nice. Then you can have that conversation and like it's not going to like lose that prompting because it's part of the system prompt now instead of just part of like a messaging prompt.

And then I did the same thing with skills skills are pinnable per per thread and you can pin any number of skills and that just like permanently adds that context to the system prompting as well.

So we're we're very skill focused. I just didn't want to like rely on the LM to like pick a skill where I know if I want that skill and I know we're going to use it.

I'm just going to pin it to the context. It's always there. So those things worked really well, but I wanted more power user features out of it. Like I don't want to have to let go click on the settings menu and then go to the thing that I want.

So in a two slash commands.

And they're not.

They're not where I want them, but for a first iteration on slash commands. It's not bad. There's two slash commands. There's slash skill and slash prompt and those auto complete to skills or prompts and then that just triggers pinning them to the thread context.

So instead of like having to go through the settings money, you can just do it through some slash commands.

It's pretty well. And it's nice for like toggling context on the fly of like, hey, I need I need this prompt in.

I want to activate the skill and then do something with it and then like toggle it back off like, you know.

So that works really well. But then I'm like, all right, let's get into this. I started feeding content to iris.

And then asking iris to like ingest or like curate, you know, this like go grab this blog post and then ingest it into the system.

And that led to like another request of like, all right, now just organize everything, right? Like we've dumped everything in here.

We've evolved. We've had like a lot of churn in directory structure and templates and like, just go through the whole damn thing and tighten it all up.

Like get everything in order figure out new templates. Just and I pose it to her is like, all right, in the concept of all this, you're going to go garden these notes.

That resulted in like 56 tool calls or something.

It was, it was nuts. Oh, and that brings me to this, right? So if we're going to be in the wiki, iris only had shell tools.

That's like that's really the only tool she had to interact with like the file system.

So it was like constantly having to come up with like crazy escape string patterns and hear docs to like cat into files.

And then it was like trying to rip grip and said and do all these things to like try and manipulate files.

And it was just like wildly inefficient. And it would take maybe three or four attempts to like get the escaping right.

And it's just like, this is not cool. So I sat down with Claude and I'm like, hey, why don't you just build iris or set of file system tools that replicates your file system tools of like file reads, create files, edit files, glob delete files.

And then like the create the read file, for example, like has like offset and max. So like it can narrowly scope in on like where insert stuff the create net a file tools get like much, much smarter.

So now that we're working with like files, like I got these like file specific tools. So anyhow runs through the tools and it does like 56 tool calls moving stuff around updating content, linking things.

And it's doing a great job. But all the sudden there are questions like everything blows up like the whole thing just like the system completely collapses on itself.

And what I find out happens was that we just blew out the context window like massively. So we're using sonnet four or five for iris. I'm going to upgrade to four or six soon. But it's only got a 200K context window.

So we were like just blowing it out because like a file read or if iris needed to read a file, it was going to like cat the file. Now all of that file content is in the context window because we send tool results as part of the messaging chain.

Because it needs to be aware of like what tools that called with tool results were because that's context feeding into the next message and so on. So I'm like great. Now I have like a massive context window management problem.

I take a step back and I'm like, all right, I knew this day would come. I knew this day from like day one was going to come where I was being really simplistic about context window management. I basically was just using message count.

Right. So we're just going to include the last 50 messages in a rolling context window no matter what. And we just like hope you don't send a ton of long messages.

All right. If someone was doing that, like sending tons of really long messages, they would hit the context window before it compacted and went to like summarization.

So now I had to implement token management, like token based context window management. But it gets so much more complicated than that because let's say we're in a loop of tool calls that's like 50 tool calls deep.

You know, the tool calls and results, the arguments and results alone are going to blow out the context window, let alone the message that you send.

So I had to go a step deeper and get really smart about truncating tool call and tool results. And so now we've built this like crazy system that estimates like how big your system prompt is.

Estimates the incoming message and like tries to do a prediction on like your token budget for your incoming message.

And then as it's streaming that message out, it's on the fly like truncating older tool calls and older messages and all of that to like make sure you've at the very least like fit in the context window.

And I'm still in the throws of testing that change and I'm pretty scared about that. I just don't want like I don't want missing context, but like we have to manage the context window and then trying to like expose and visualize that in the front end.

What I've done is sort of what cloud code does and it's not as reliable as I'd like it yet, but we're showing in the UI at the message of like this is this is like we show like a horizontal rule across the chat is like this little like

pretty big like pill that goes across the chat window that's like earlier messages have been summarized. So at least you know looking at the conversation history.

When that compaction hit and turned into a summary and where you're like new context begins. Don't try to expose that to the user.

But man, it's just it's it's spiraled out of control.

Yeah, it seems like lots of big problems to solve and I was thinking when you were talking about the tool calls, I do think those are valuable for the system as it is, especially with multiple people using it on whatever system that they have.

But for obsidian that you're using a four or two, I would take a look at the obsidian skills that are available from the creator of obsidian. I don't know if you've seen that repo.

I'll put it in the heavens and send it to you.

But it has certain skills to utilize obsidian CLI to how about like right to the files.

It has a whole handful of different skills in there. I think there's like three or four of them, but that might be a good one to try on your side to because installing the obsidian CLI via that application on the server.

That would give you some additional benefits to that and might find and be able to crack those files faster.

Cool. Yeah, I'll definitely check that out. One of the like I had, I had iris set up obsidian sink and everything like the headless sink on the server that went pretty smooth.

But yeah, I'll add some of those obsidian skills. The other thing I sent you and I was looking at a little bit was graphy is that what it was, but it's this thing that it's been going around.

It's a skill that's like got a set of skills and commands that you use that like look over your content and generate like grab a crawlable graphs that are meant for LLM's to be able to like crawl through your information and like find stuff.

And it's kind of oriented towards code bases, but there is like there is direct support for it for obsidian stuff.

And I started trying to play around with it and I was asking iris if it made any sense to use for the LLM wiki and it's like, but you're not really at that scale yet.

So you might be able to make take advantage of it because I know you have a ton of content in your system. So it might be useful for you.

We're not quite there yet. I think I tried running it on the iris code base and I just didn't know what to do with the data. So I just kind of moved on.

But it looks really interesting. We will include it in the show notes, but I wonder how much like you could make use of that since you've got so much more content.

Yeah, I've definitely seen this on like Instagram and other socials of people promoting it.

Most a little hesitant to install some of these things, but this one because it keeps on coming up and it seems pretty well supported.

And it has a lot of options in it. I might just do a eval on it and see where it goes.

I think this could be interesting, not only for the obsidian vault for code bases as well, because there's additionally notice like more chunking for doing that like investigative work when doing a new feature or big code bases and things like that.

So this could potentially help with that too.

Yeah, I was thinking about it in regard like in relation to my PRD generation skill is like part of the process that I lay out in that skill is like, let me tell you what I'm looking to build.

We have a conversation about it. Then there's like a research phase and then we have another conversation about it.

So like we're kind of like it's kind of scoping what we're looking to do.

Then go and researching. I think that research phase could like definitely benefit from those types of like reports and graphs and stuff of like.

Here's where I need to actually target my investigation before we go back and like continue to interview process on generating this PRD.

Like I think it makes it so much smarter than trying to just like wrap around and like find stuff.

You actually have this like mapped out tree that I can crawl through.

Yeah, I definitely want to experiment with it some more on like a code base side of things, but definitely seemed interesting on the wiki front too.

Yeah, for sure.

Yeah, I want to dig into that a little bit more and you know by free time.

Yeah.

There's so many things that I've like kept or like bookmarked or whatever.

I'm like, oh yeah, this seems cool to look into and this seems cool to look into and I just add it into my to do us in box.

Also getting things done process of like, yeah, I'll just throw this in here. I'll organize it or like look at it later.

But there's been just so much stuff over, you know, the last three to six months that like I never have a have had a chance to get back to it.

And I feel like a couple of weeks ago, I'm like, this is crazy.

I got like almost a hundred things like in my inbox. I don't like it.

There's like, I'll get in there. There's probably a bunch of things that are now like old and outdated, you know, because AI stuff is just changing so often.

So I hooked up the to do us MCP to Claude and said like, hey, go through all the things in the inbox, see if you can grab the content or use some like other tools to see if you can grab the content and

and break them all and see like what's actually valuable valuable or what things might have like fallen off and either like popularity or usage or because the models and Claude code have changed significantly.

So it did like a pretty good job of like ranking all of them of like, oh, no, you don't have to worry about this anymore because it's handled, you know, out of the box or like, hey, this would be really good to use for like AI work or here's like a really good like obsidian one.

So yeah, it helped me just kind of organize it really quickly and just kind of clear up that list. So now it's building up again, but yeah, yeah, we'll practice to get into, I guess.

That's what I'm really excited about is I've got links in like a couple places and I really need to like centralize all of it.

That's what's been kind of cool about getting this system bootstrapped is I threw like I saw on.

I think your latest dev notes newsletter, you had linked to a article about context management with Claude code and like the 1 million context, it was unfortunately it was a Twitter article.

And those are kind of like a pain to get so I had iris install some like tooling to make it easier for her to get it, but had iris added to.

The uninjusted so the way I have the obsidian set up with that like inbox raw and wiki like it's three directories so the inbox is for like uninjusted things right it's just like like a markdown document with a link in it or a blog post like a full content blog post or maybe like a link to a blog post whatever.

And then when it comes to ingestion and curation time like I'll deal with that later, but then that gets after it's been like ingested then it goes into the raw directory and then it's referenced in the wiki directory.

So it was I had iris scrape that article down and immediately iris was like this is directly applicable to what you're currently working on an iris with context management and I'm like oh that's so cool you're already making the connections and like making this relevant so it's like oh great I like I actually need to go and read this article because.

Like it is directly related to the context management that I'm trying to do with iris so I thought that was that was really cool that like the systems already kind of exposing like links and relevant things because like I didn't even read the article I was just like here's the URL like go get it and.

And through that process iris was like no this is like something you should read like it's applicable to what you're doing which I thought was.

Super cool like this is exactly why i'm building it like the whole purpose is i'm like I want to two things I want a second brain reckon like ask you questions about things and like you can go and like find relevant stuff but also like and that's why i'm.

A little on the fence about automated curation is like I kind of want to participate in that curating process to like make sure things are relevant that I do want to hold onto that stuff.

So I'd like iris to drive that process but I still kind of want a little bit of a human in the loop right the very least be able to like surface the stuff like hey this is like further investigation for you to do.

But man I was yeah that was just like such a cool moment of like yeah no this is like related to the thing you're actively having conversations with me about you know.

So awesome yeah that was rad man.

I send it to you on the side it's also included in the obsidian skills repo that will have in the show notes but they have a default.

Skill in there and you install it via brew and that actually grabs Twitter and reddit and a couple of other social like kind of scraper avoid links so that has worked for me for a bunch of.

Intake for different URLs and articles and things like that that I want to grab that club can't do like on its own boss so that's a nice little work around.

So yeah quick little install via brew and then use the skill and I think I have a I guess that up a memory for like always use the default skill for Twitter and reddit and maybe a couple other.

Newer else so it'll always just use that and not try to you know to a bunch of tries and it's out of the box options yeah found a project called like camo fox or something like that it's a Firefox distribution the whole purpose of.

That project is to avoid that stuff right like avoid the browser sniffing and like bypass all of that so that it looks like it is like a real browser traffic rather than like playwright which is going to set like playwright headers and like.

Abide by like robot text files and stuff you know this this kind of like cuts through all of that it has a ton more features than I've used it for but I use that to like get through to.

Twitter I actually think it even got through one time using curl and like header manipulation which I thought was really neat like yeah do it here's that's great nice.

Yeah and the funnel is actually made by the same obsidian maker so it's a service to grab the content like it tries to grab the main content of the page and converts it to mark down so it'll try and get like the headers out of there the side bar content the footer stuff like that to actually get the main content of the web page I think that's what the obsidian like web clipper uses behind the scenes so yeah definitely check it out I'll have that.

The direct link in the show notes for there's a website and then you install the CLI via brew so lots of options there and just converts it to mark down automatically.

This whole default thing is like a whole I as default things like a whole holding CLI project this is pretty neat yeah so it's been working great for me for the links that I need to grab from some of the social sites.

Yeah no totally use this is awesome it's great I just set up a thread in iris for intake right I just I said I built a safe prompt added that iris and so like it's just a thread for me to dump things to get filed on the inbox so I just just linked to this I'm like put it in the inbox we'll check it out.

Nice yeah yeah I think I need to move my intake from to do this and just throw it into like discord or the obsidian vault and have it like do a quick pass and summarize and organize it or see if it's still relevant or not.

Because I know I wouldn't have to organize it myself and do double duty on that so yeah yeah thanks for a tip I'll check it out all right on that note you want to wrap things up.

Yeah it's about it.

All right man thank you all so much for listening to the slightly caffeinated podcast show notes and all the links from things that we talked about in social channels will be down below as well as available at slightly caffeinated.fm.

If you have questions for us or a content suggestion go to the ask a question page on our site and we'll feature it on an upcoming episode.

Thank you all so much for listening we'll catch you soon.