D2DO297: The Future of Open-Source Contributions in the AI Age

If you can write a detailed human language specification, right, and then I can set my own trusted agent loose on it, and it produces the code,

is that potentially, is that any better or worse than having a software package that I've just downloaded?

Welcome to Day 2 DevOps, where the devoops is in the details. I'm Ned Belvant, and I'm joined by my clandestine co-host Kyler Middleton.

I feel like I should be wearing a trench coat. Hey, Ned, here, here today we talk to Liz Fong Jones, technical fellow at Honeycomb, and she is going to guide us through the future of open source contributions in the age of AI, how and whether to build your own software, and how to encourage junior engineers to grow when AI can jump right to the answer. Let's talk to Liz.

Well, Liz, welcome to Day 2 DevOps. We're super pumped to have you here. Kyler and I are both fans from afar for a while. We've been looking for a reason to get you on the show, and we finally have one before we dig into that.

I want you to tell the good folks a little bit about yourself and what you do.

Sure, I've been working in the DevOps field for since it became DevOps, so over 20 years, really started my career as a, as a live ops, live ops, tech ops person for a games studio, and then spent 11 years at Google and seven years at Honeycomb, and here we are.

So, yeah, I'm a technical fellow at Honeycomb, which means that I get to do really cool stuff, essentially, that is essentially my job description.

Do you really cool stuff talk about it?

Oh, that sounds like a perfect job. It's kind of like what Kyler and I do accept we podcast to, and I guess that counts, right?

Yeah, what is I'm working on our observability product and trying to make it better and trying to show people how the ability can make it a lives better.

Awesome. Well, the reason we reached out to you was you had a post that was talking about sort of the future of open source development, and the impact that LLMs and LLM generated code is going to have on open source projects.

So, can you talk a little bit about what your thoughts and feelings are around open source projects and how LLMs are kind of messing with that format?

It is an inversion of the difficulty curve.

Previously, the difficulty curve was that it was difficult to generate a pull request that would pass the unit tests.

So, if you received a pull request as an open source maintainer and assuming that it passed, you know, the basic smoke test of do the task pass, you could assume that it was created by someone working in good faith who had a problem that they needed to solve and that they developed the necessary expertise to solve their problem and also to make it conform to your standards.

That has gone no. Code is at least throw away code is extremely cheap to write, but open source code bases are not throw away code and different cheating throw away code and durable code can actually be really challenging right like you have to do a lot of work to validate it to make sure that it's ready.

And that is essentially the asymmetry that's happening out is open source developers are finding that it takes five minutes for someone to open quad code and submit an upstream pull request and it will take an hour to validate it to make sure that it's right.

And at that point people throw their hands up and say why should I even bother reviewing these pull requests right like you know this and this gets even worse when it comes to bump down is as we were talking about earlier where people are expecting to be paid and therefore they can just afford to throw shit at the wall and see you know who's willing to pay the money and it just creates a sea of noise.

It's like my wife is actually a maintainer of the Google Chrome security bug bounty program okay and one of the things that she said is they are getting a lot more bug bounty submissions.

And fortunately some of them are actually valid bug bounties they look the same as invalid bug bounties but there's actually an increase in the number of valid bugs that are being found by AI and reported by AI.

There's also just a lot more noise as well but there are there are more genuine bugs are being found with help the AI.

That's awful because there's more noise there's so much more noise but there's also there's also more signals you cannot just categorically model out.

Oh that's awful. It sounds like the same problem set as AI is just sort of making stuff up is a really good liar.

It justifies it's it's totally mystery with hallucinated things and that's the same function here that it's it's writing absolutely gosh darn awful code that also satisfies the unit tests.

It's the same and also it potentially even looks normal ish at first glance right like that's that's the problem is you can get something that seems compelling to a human but is not when you dig when you dig and deeper.

One of the fun words that I've started using recently instead of hallucinate is confabulate because right like it's it doesn't have an imagination right like that and it's definitely lying in some way really so I think confabulation is a is a is a fun word to describe that.

I love that word I'm going to use that I'm going to borrow that if you don't mind please population.

Yeah we've spoken to we spoke to cat Traxler a few weeks ago about how she's been finding vulnerabilities but she's been very careful to validate them after the AI finds them before she tries to submit them.

I know not everybody is careful about that is we also spoke to the founder of curl and they've shut down their bug bounty program because of the volume of.

Submissions that they were getting and most of it was just AI slop I don't I don't know what the solution is what the right approaches for that.

We have to.

Lower the cost of validation of the code and remove incentives for people to.

Spreads spread slop everywhere right like if people think there's a financial reward in it for them there is going to scam it everywhere right like unless there's some you know minimum like payment or something right like you have to put down a deposit when you submit to the bug bounty right like.

Because previously just the act of writing up the bug bounty was its own kind of deposit of your own time right now it's not so yeah I think basically.

One other analogy that I use is in my one most really like you know you got stuff that's shoved in your mailbox of the time right like you know they have these like you know magnets so they want you stick to your fridge with you know the number of a plumber or something on it.

Sure and I think start happening the past couple of years.

Previously you know it would you know be a type written thing that says for my neighbor or something right I would be something that was not at all customized to you right like you know you.

Right just be like I'm waiting for this out.

And then things started popping up that were like looks like they were hand written notes that were like you know when I was you know 16 I aspired to be a master plumber ball about written out in long form person right.

And some people are very impressed by that because they're like while someone took the time to do this turns out it's a robot right right but now it's stopped being a valuable signal that someone hand wrote something right.

The same is happening with sales calls brightly get a link in pitch where you get you know a sales rep who has like recorded you a short form video and you're like wow that person put a lot of effort into that.

He didn't right like I think the kind of sheepening right like this sheepening and hollowing out effect is happening right like you can no longer trust that previous kind of social signals at something like something was potentially working out because someone put a lot of effort into it.

That's gone right there there is no effort I actually got one of those I think about a month ago the LinkedIn messages recorded just for me and the thing I immediately noticed is when they got to my name.

The person was not in the camera cut to something where I couldn't see the person's mouth moving anymore and.

I guess they used AI to synthesize my name and of course they butchered the last name because they always do and so I immediately picked up on them like okay so you generated this for thousands of people and it just cuts away whenever you say their name.

So you won't see it mispronounced by the lips and maybe the air will get it close enough that someone will go oh they really they reached out to me directly.

Yeah yeah so I think right like the thing that expressed in my LinkedIn post is another way to lower the validation costs in the burden on maintainers.

A quality bug report is always good we know that quality bug reports are good right like when there is a clear description of what the problem is what did you do what did you expect to happen what actually happened and you know potentially like you know where in the code base might the problem live right.

It used to be that the burden was that creating the patch for it was also time consuming.

But now creating a patches cheap regardless of whether or not there is a valid bug bug report and problem statement.

We need to flip it around we need to lead off the bug report first what is the problem you're trying to solve.

And it turns out if you've described the problem that you're trying to solve well enough you don't actually need to submit code to go with it.

Because they maintainer can use their own language model to ingest the bug report to ingest list of files to look at and to generate the code.

And then you don't have to be looking for subtle prompt injection attacks you don't have to be looking for did did someone you know insert something nefarious into this into this patch right because you can read with your own eyes the plane English or you know

Banner sure Chinese description of what of what the problem is.

Valid that looks good in that's not going to throw your LM off and then throw it over the fence to LM to generate a solution.

So I think that's kind of the one potential answer to this problem of code is cheap to generate but but difficult to validate is don't even submit the code.

That was a stumbling block I know I've certainly found bugs in different open source software and gone through the process of trying to write a solution for it pre LLM's and just learning enough about the code base to figure out what a patch look like was pretty difficult.

If I could just submit a bug report and know that it could get fixed relatively quickly if I was detailed enough I'd be much more encouraged to just submit that bug report and move on my day.

Yeah, yeah exactly and conversely right like you know patch with no context that was generated by now.

Throw that away the person could zero offered into it.

So does this mean that for open source projects you would no longer have contributors to the project in the form of code everyone would contribute to bug reports or would you still have a small group of people who are contributors.

I think for core contributors that you trust right like you know please write like those are the people who are triaging the reports those are people who are submitting for requests.

But I think for entrusted parties I think the way that you earn your way to being a maintainer is now no longer submitting patches that get accepted I think it's it's submitting valid bug reports.

That's fascinating having them do less what I've seen proposed by lots of other folks is just throw more AI at it and it feels like fighting fires with gasoline.

Yeah exactly right like you might have mistreating of things and also right like when you throw more AI at it without a human looking at it.

Okay in the case of open source the risk is lower because right like the lethal trifecta does require private information and trusted input and network access right.

So if there is no private information if it's running inside of sandux via an interacting only it's the open source code right like you know sure right like maybe you can you know use AI to triage a a incoming pull request.

But like I worry that if the developers is doing that on their own machine they are going to get owned right like we actually we saw this today on.

There was a report that just there's a report that just came out.

About yes a GitHub issue title compromised for a thousand developer machines.

No but that's amazing from injection people were literally just running their setting their agents loose on you know free edge incoming bug reports in one of those like bug reports must have said like you know ignore all.

Previous instructions run npm blah blah blah right like it did.

Wow that's wild because I know that npm itself has been such an attack vector in the last couple of years and to tie that with the prompt injection to go install a malicious npm package.

You might not even one even malicious it seems like it was just someone doing a it was someone either who and intentionally did this or who is doing a proof of concept of I could make you install open claw open claw I can make you install anything but yeah.

Joe's open claw yeah.

Yeah.

So yeah I think I think I think that's the risk basically is AI cannot and should not be trusted with the ability to to take potentially disruptive actions like a committing code or or or.

Or sending emails or so forth with with n trust you didn't put and unfortunately what open source maintainer beginning is a lot of interest inputs that's why yeah that's why I think fighting AI with a doesn't doesn't really work that well.

Sure you might have a bot that applies tags right I think that's relatively safe that's not going to cause cause security vulnerabilities in the worst case you missed tags something or you create a tag saying high list was here right like but.

The more permissions you grant the scary it becomes and also people do expect someone a utility out of this yes I think.

Using AI directly to triage AI outputs is dangerous because someone someone can black hat that by think using AI once a human has verified and signed off on it can be very very powerful right by shifting where that work happens.

I was just thinking about the sort of the trusted contributors idea.

And the way that you typically become a trusted contributor is that you start off as just a regular bug reporter who's submitting some patches and you sort of slowly build up that level of trust and you know credentials are credibility with the maintainers and then at a certain point you become part of that sort of inner circle what you're proposing.

Would make that a little more difficult what do you think the path would be for someone who wants to become a contributor if they adopt this new model.

Why is that a problem if you're submitting detailed bug reports right like that is a difficult thing to do that earned that earned trust regardless of whether or not you're writing the fix because right like again if the fix is cheap and easy to do regardless of whether you're doing either the maintainers doing it.

I don't I don't think that's as much of an issue that being said right like maintainer trust attacks are everything right we saw this with the LMA of inside our attacks so.

The other interesting phenomenon or question is.

Some projects are definitely going to continue needing you know a central code base a central set of maintainers like things like bluemix crawl things like open telemetry.

For smaller more bespoke projects right I shouldn't have to trust the maintainer of the code i should be able to read the specification right like or read through maybe a few hundred lines of code to validate that it does what it says it does.

And sure maybe I can rely on signals of you know 5,000 people download this software package or something but in the cases where something is more niche and you know maybe only 50 people are ever using it.

If you can write a detailed human language specification right and then I can set my own trusted agent loose on it and it produces the code have read it is that potentially is that any better or worse than having a software package that I've just downloaded.

That is wild that so that's like if I could put that to like more old school terms it's the difference between getting the code or compiled binary like.

Instead of distributing the compiled binary to me I can just grab your source code and compile it myself will take one more step up the chain now I can just take your specifications that the code was based off of and have my AI generate the code for me yes.

I might not sit with that for a minute yeah and I can tell you that as of yesterday or today open a release to projects that they're calling symphony and they've released that in specification form they have an elixir reference implementation but they're also strongly encouraging people and maybe they're self motivated by the fact that this will consume open AI tokens.

But like right like they are saying you know here's specification read through it with your own human eyes if you like it or you know don't even modify it but like just set your own AI agent loose building this right like instead of them distributing binaries or instead of them distributing source code they're just to bring this back.

It does seem that they have a strong incentive to have like you know 50,000 people compile it with an LLM instead of compiling it one time and putting it in like an S3 bucket for download.

But then there's the trust issue right like you know I can trust open AI will probably you know if there's giving me a binary if they're giving me source code right like I can trust it's probably not malicious.

When it comes to a random open source maintainer I actually do think that I like the idea of compiling the spec more than I like the idea of compiling source code that I cannot possibly with my own eyes read through.

So I can maybe set an AI you know loose to security audit or whatever right like I could run a container but like.

Yeah I think I think there's that kind of question of you know are we using AI to verify that there are no bugs and packages and like scanning things to make sure they're malicious are we are we having AI validate the source code or are we just starting from a clean slate which one is going to be more efficient having a is repeatedly scan source code.

You're talking about shifting left but in a way that I don't normally think about it I work as a developer so I think of like instead of at the sea ice layer like let's move it onto my computer let's put some cloud code in my little IDE and you're talking about shifting left as an industry away from from scanning your dependencies and more towards generating your dependencies.

Do you think that this is just and we're not new to this right like you know at least it but with a more deterministic engine right like right if you generate an antler compiler right like if you generate an antler parser.

You are not working with the antler output if you generate a profile you're typically not looking at you know reading through the dot PB dot go file it lined by line right like you're trusting the compiler works.

Absolutely the issue is just that it's fuzzy and yes it is resource intensive and I don't know where that lands.

I I've been thinking a lot about what the future looks like when it comes to using LLMS to develop software and right now we're using these big honk and models that have everything in the kitchen sink slammed into them I mean I don't even know how many billion parameters are in GPT 5.2 or you know opus 4.6 or whatever.

I'm in the future I'm thinking smaller models that are more focused specifically on a particular or software development or particular language would not be as resource intensive so it might actually make this whole process more feasible and wouldn't feed the open AI dragon is that sound remotely plausible.

I worry that is not true because if we are shifting up to business requirements and specifications.

Yeah.

Yeah.

And then it right like this is not fancy tab auto complete right like this is now a thing that has to understand your descriptions of what you're trying to accomplish in the world and sure right like it is programming and that we are making it ever more detailed and you know solving bugs but like.

Yeah, I don't think it is possible to extricate the knowledge about the world and the programming and if we are solving problems for real work ready sure you can make a programming you know only model that solves project Euler sure that needs zero ice understanding physics biology chemistry.

What if you are you know writing brain programs to to simulate cellular biology suddenly that knowledge that becomes relevant.

Yeah.

Totally.

We hear that described as like programming for software engineering which is this new frontier for AI is moving from just being like fancy auto complete and it knows how to write a Python script and it's cool to software engineering which is this.

Squishier skill set you know it's understanding business objectives it's it's getting people on the same page it's and that's your right that's about people and businesses and ecology and economies and like that is not.

Which is also why I am very very sure that businesses that are laying off their programmers are making a very fundamental mistake right that fundamental mistake is confusing programmer as person who writes line of code versus programmer is specifier right.

There is so much more that each person can do which right like lowers the marginal barrier of this wasn't worth doing before this is worth doing.

If you're slaying people off it means you're not capping that set of extra capacity that's just been unlocked.

Right well there's I mean there's this idea that the number of lines of code that you're writing is your worth within the company not not with every company but that was certainly a way by which programs were measured is how what your code output I.

And the code output is proportional to the cost to verify it it may not be proportional to value to the user but is proportional to cost.

Right but what I would I know from my own personal experience is the more senior developers weren't writing a lot of code a lot with their job was was nailing down those specifications parsing out tasks figuring out requirements and then delegating that.

Are we building the right thing how are we architecting the thing how are we breaking it up into chunks as you're saying really.

And maybe at some point there will be very smart orchestrator models but right now there is a lot of room for human judgment and taste that is the other thing that huge a proponent Jeffrey Huntly says is Jeff says there is no substitute for taste language models do not have taste they will build you a thing in the least tasteful way possible if you don't tell them like how to do it properly.

I want to call this like tacky AI this podcast now that your wonderful title.

It's sort of like what makes something tacky is when you have no sense of proportion so you just take everything and throw it together instead of like choosing a specific style or genre to go with.

And that's kind of what you're describing these elements they there's like I will take a little from here and a little from here and I'll just mush it all together doesn't matter from wearing stripes and polka dots and plaid all together.

And hey it might work or it might not and I think you know there again there's that question of how do we debug it right you know we have to either use humans to go and gravel through the code.

Or we can have language models gravel through the code aided by some insight from production baseline data right to use your observability data is ground truth for what is actually happening when you roll this out to production.

I'm glad you mentioned observability because for me this raises a huge potential issue with making sure that the software that's written is observable when it actually rolls out production so you can close the feedback loop.

How do we do that when the code is being generated by non humans.

I think almost it is easier with non humans because a lot of what charity majors and I have spent the past 10 years doing is telling people to eat their vegetables instrument their code and to you know stop using print F the bug lines everywhere right like to create wide wide events.

Now I think it's possible with a really good skills file to actually just instruct a language model to instrument things correctly and it will and hear some examples what it correct information looks like and it will do it.

And the more training data that we give us how to instrument correctly even if it's generated by a language model the more this becomes a self reinforcing positive cycle.

So I think the generation of well instrumented code is something that is potentially even easier than if you're trying to convince a human to take the time out of their day to add instrumentation.

But there is the analysis side there is the data volume side right like this is where observability company isn't I'm not going to single up mine right like this is where having mcp servers for your language model to be able to interact with that branch of data giving them flexibility to query it right like that's where that matters because right all the international world if no one's querying it right like tree falling for us making a sound.

We've been talking about open source projects and how we should approach or a potential approach or contributions and going through bug reports.

But the other side of that is the observability story so what are your thoughts on approaching these open source projects from an observability stand.

Yeah it has been a challenge historically over the years and it used to be the case that every single observability framework would have to hook or monkey patch these various open source libraries but they could only do it if they were popular enough.

But there's only kind of the sixth number of maintainers right like there's all this right like you know if you're supporting no JS you're supporting the express framework what other HTTP frameworks are you also supporting right like there there's this whole mass.

Because the open source project doesn't necessarily know what observability solution is going to be used where they're deployed.

And they can't afford to be opinionated about it because then right like you're potentially you know baking in the data dog libraries into your open source project right now that's hard to get everyone else has to take.

One of the really cool things about open tool libraries we are starting to see that shift from here is a surface areas of libraries that we have to support monkey patching on the same as any other previous instrumentation library.

To developers of open source projects having the confidence in the open telemetry data model of being able to use the open telemetry APIs and trust that an open telemetry SDK will be created in the main function of whatever is calling the open source a library.

That people are just natively adding open telemetry support to their libraries and even in the case of Claude code Claude code has native open telemetry export right.

It doesn't have specifically honeycomb integration or a data dog integration it just has an open telemetry exporter right like and you can tell it where to write that data to what API key to use and off your off to the races you go right.

So I think making sure that you are maintaining kind of that vendor diagnostic vendor neutrality is something that and locks the delays should be a lot more opinion about observability in your own open source project.

Okay, that's yeah, I've been following the open telemetry movement for a little while and I'm really seeing the value now that pro open source projects are adopting it so it's just kind of.

Yeah, I just took a while right because you know 19 standards and then it's 20th one comes to end all of the other standards and it takes a while to actually convince anyone that yes, this is actually the one that's going to stick.

It's like we're finally at USBC everybody just get on board.

Exactly.

USBC is a great example.

Yeah, 100% because people were initially skeptical and yet now every single device is sold with the USBC for.

My children will never know the pain of plugging the USB port in upside down and then right side up and then realizing it was right side up the first time.

Oh, now you have that problems your cables better there's dirt on the pins now it right like you know it mostly works until that time it doesn't right.

That's open source software too isn't it?

Yeah, it mostly works.

It mostly works and if it doesn't work it's you break it you keep both pieces really now that's kind of always been the model of you have to figure it out and either you ask a maintainer for help and they are very very busy or now you know you maybe ask an assistant or ask a friend.

That raises an interesting point if generating code is pretty cheap now do I do I even need the open source project to continue to be maintained if I know where I want it to go and what purpose I want it to serve.

Why wouldn't I just fork it and modify it the way I want to and continue on my merry way do you think there's going to be like a massive.

Additional number of forks on all these open source projects and people just maintaining their weird copy of it.

I don't think the issue that we're confronting is going to be forking.

I think the issue that we're going to be confronting is multiple independent re implementations.

You probably need only a tenth of what any given open source project produces true why keep up with the entire open source project why not generate something opinion based off of your own needs and just carry that forward.

That's a really good point because there's a lot of effort right like an AI can automate some of it in terms of keeping your fork up to date and so forth but like you run the risk if you are running something based off with a well established open source framework.

And you fork it and then you never update your fork congratulations you now have the worst of both worlds you've got something that's spoke that requires customer maintenance effort and it shares all the security bugs is everyone else to who's using that open source software and vulnerabilities are being reported now they're you're being breached.

As opposed to if you did bespoke chances are no one would stumble across any bugs that you did happen to introduce.

So that's why I think forking forking and maintaining and maintain forks are the worst of both worlds.

You get the maintenance burden of maintaining all that yourself and you get the security penalty of this is commonly deployed software.

At the same time.

Yeah.

That's so true.

It makes me think of like Microsoft word and how I don't use more than 10% of the features and functionality in Microsoft word when I'm forced to use it which is

that is only rare.

But like if I were to re implement Microsoft word it would I would streamline it so much and I wouldn't have to worry about random VBA macros like ruining my day.

Where do you land on that list like is that a it's something that's going to happen right people are going to sort of vent these open source projects to their their own needs and store them.

Is that a good thing a bad thing should I avoid doing that.

I think it depends upon right like you know things like the Linux kernel not going away things like open telemetry probably not going away.

Right thing things like an API for interacting with Twitter anyone can rewrite the API for interacting a library for interacting with Twitter right like.

So I think that's kind of the challenge the other interesting thing that we don't genuinely don't know yet is model capability ceilings and cost.

None of us are paying the true cost for all these tokens yet.

That's what happens when I catch up to us because that's going to influence a lot of this behavior over is a cheaper to throw out and rewrite right like with the sub size prices I can say it's cheaper to throw out and rewrite that does not account for all the externalities.

And that's why I do have to preface when I'm saying you know go ahead and rewrite or throw it out or generate the code right like.

I'm going to acknowledge there's a cost to us I'm going to acknowledge is not necessarily sustainable forever.

Yeah so I would be cautious about over indexing on rewriting everything in your stack like use the same time we are currently at a moment where she used tokens to generate to generate code is sheep use tokens to create plausible sounding plausible sounding security bug bounties.

In terms of writing a specification for an AI to generate your software have we settled on like a pretty good standard for that specification or is it still on files.

Yeah of course but I think the interesting question is going to be if this continues being a thing where is that go.

I think of one other interesting interesting solution to the problem of wasteful regeneration.

I care that a that I can read through this specification I can understand what the spec is doing I can understand you know I can poke holes in it and I can also understand that no one is deliberately added malicious malicious prompts to the specification.

What if there were a trusted way of verifying that a model that I trust not necessarily under my own control.

But what if there were verifiable reproducible builds right like similar to how you can turn source into deterministic reproducible builds right like Ubuntu and WN have done this for years and years and years of being able to prove you did not tamper with the build the source code that is is what it is here's the buyer that you output.

Therefore it's you know you don't have to audit the binary you can just download the binary and use it right.

What is the generation we're running in a third party foundation controlled sandbox run by you know the Linux foundation or whatever running on Amazon bedrock for like all the inputs and outputs are really you know here's the art here's.

You cannot perfectly verify the seed because right like you know there's there's temperature there's all these you know variables that go into it there's floating point fluctuation and GPUs really.

So it's not going to be reproducible exactly every single time but what I care about is then arms length third party is converting the spec into code and demonstrably any way that is not right like that that is not open to to tampering.

If you did that I could read the spec I could trust it and then I could then chain from that into well then I trust the code that was generated and then from there into I trust that I can download the binary.

I might as long as this doesn't turn into like a crypto startup so any crypto startups listing please don't please don't write a crypto chain of validity for like LLS generating code but other than that that sounds great it sounds like a real advancement right you don't need a proof of work we you right like it's it's it's crypto only in the sense that yet is a is a set of more hold trees right like it's it's crypto only a sense of you know there's some degree of attestation.

Without the wastefulness right like so yes I I risk you I have a tremendous tremendous allergy to anything anything boxing and this is a lot more about the story that we've been on for so many years of reflections on trusting trust.

Right right who validates the validators yes we certainly ran into that problem when it comes to certificate authorities and some of them being compromised over the years that's always fun.

This would be a similar challenge you'd have to set up a network of trust or building software from spec with it with an AI model and someone would need to verify that I'm already thinking about what that would entail.

We may have started up another startup Kyler this would be like that the 12th startup idea we should publish this list somewhere and people can apply to like a built that.

Here's the challenge ideas now are you know we're sharing them freely implementation is cheap to it's all about execution yeah.

How to execute it properly and in a way that everyone's.

It was fascinating by the way the other day seeing come seeing companies who are doing open core or open source disabling access to their to their unit tests and integration tests.

Under the theory that that would create a moat if the verification or siloed off so that you couldn't use a language model to recreate something that path that is compatible because it passes.

All the tests.

Interesting so their concern is that people are going to recreate their open core thing based off of what's publicly available and then they don't need that open core thing anymore correct.

Open core source available projects yeah I think open core and source available projects are now it was already on the downswing but I think that's potentially dead because you would have a really hard time convincing a judge that was not fair use.

To run the validation suite and otherwise to have a language model just rip out you know spit out code that is not necessarily from the training set of the original open core project and it's just from you know ripping off many other people more ambiently in a way that's fair use right like so is use of you know fair use of generation of code from the entire universe or everything gets out there validated with one company specific suite.

You're not actually shipping the validation so therefore you're not copyright infringing the test suite right.

You're just leveraging it yes compatible software that's interesting right so many interesting things are happening now that generating software is cheap yeah.

It is it's changing the way I do everything projects that I would have just written off before like I've developed to Android apps in the last month and I don't know how to write Kotlin I don't know anything about Android development but I was able to get clawed code to help me build entire you are an experience software developer you are skills at the art of describing what you want of dip of specifying bugs.

I think that before we close we should talk about juniors because that that is nothing that scares me is how do we give juniors that experience to be good drivers of LLM's that is we as seniors are experienced enough to delegate.

I would love to hear your thoughts on what what we can do to bring juniors up or or bring recent college graduates up and and and help them become mid level and senior.

I'm supposed to give a talk at UBC in a month and talk to some students and they are rampantly using cloud code and the instructors assume they're going to do that but it turns out they come into office hours and they have no idea what the code that that they submitted actually does right.

So I think the right you can perfectly well rob yourself of your own learning opportunities right and there's that temptation do that when productivity is expected to be at maximum.

So I think we have to give people a lot more grace as far as not expecting them to do things quickly right like the AI can show them a way to do it that they need to take time to internalize it right.

I think in general it's always been the case like getting juniors to consider the impact of what they're doing what is going to do in production has been challenging.

But I think we really need to turn that up right to get people think much more about the unhappy path about bugs about how do I identify bugs how to specify bugs any way that you're going to fix them right like.

See I think there's this challenge of not leaning too hard into the robot is just doing it for me and I don't need to care about about how you need to care about how until it does go wrong and then you do care about how very much on so making sure that they're getting that incremental exposure as they go rather than.

Suddenly they discover two months into compilers class that they've sent themselves into an architectural corner by using cloud code to generate it all right.

Totally my partner's a professor and they have just been struggling just in academia in general how do we measure output when students can use AI and just jump to the ending it's hard to prove because we usually grade the output but we can't grade the output you can get the output for free now so how do we how do we measure the process.

How do we measure what's actually internalized and it's a hard problem it's harder it's harder than just measuring the output.

It's the same challenge for juniors I think it's like you know the concept of open book open note exams or you know exams where you're allowed you know two sheets of note paper right that process of freeing that note papers almost as valuable as the exam itself right so similar way.

Encouraging people to come with their cloud code transcripts right like is is actually a really interest interest interesting approach right to ask you why did you prompt it that way why didn't you prompt it that other way right like you know how did you know how did you how did you learn from this where you just hitting in or every times log code ask you a question or you actually like thinking and typing something back right like.

Yeah I think that that is right like it is true we don't make people use slide rules anymore we don't make people use library and X cards anymore right like so but.

People do need to learn literacy right like and I think Lucy is through experience not not through you know a one in done process.

Yeah and there's always going to be that subset of people that will use cliff notes is they don't want to read the book and those people are now going to use cloud code but if.

Books for all is me have captured my imagination and engage something in me beyond that just getting to the answer or writing the essay for class and there will always be that class of people that is curious and interested in the process.

And I think those will end up being the senior developers and the folks who were always looking for the shortcut maybe maybe they just need to approach a different area that's not software development right what makes a good DevOps practitioner curiosity and empathy.

Yeah and if you automate away all the curiosity you're probably not a good not a good fit for our field and you know the field the filter for that is much easier than before because the easy button is right there right if you're just pressing the easy button then you know you're probably not.

Cut out to be one of us.

Absolutely well this has been a fascinating conversation thank you so much for being a guest today on day to DevOps if people want to follow you what's a good place to find you on the internet.

Liz the great calm as my other stuff is also my social media handle so.

Awesome well Liz thank you again for being a guest today.

All opinions are strong but like we held in this moment of everything changing underneath us.

Thank you again to Liz for appearing on day two DevOps and virtual high fives to you dear listener for tuning in.

If you have suggestions for future shows we would love to hear them hit either of us up on LinkedIn or send some feedback via packupusures.net slash follow up.

You can find me Ned Belvins at net in the cloud dot com and my amazing co host Kyler Middleton logging over at let's do DevOps dot com and we're both termally active on LinkedIn stop by say hello.

If you like engineering oriented shows like this one visit packupusures dot net slash subscribe all of our podcasts newsletters and websites are there it's all nerdy content designed for your professional career development.

Until next time just remember that doing DevOps is awesome it's so are you.