#236: AI Answers - No Time for AI, AI Budgets, Vendor Terms & Data Risk, AI Disclosure & Vanishing Entry Level Roles

0:00Where does management come from? What do managers do in the future if we don't train venture-level employees? So if we stop hiring as many entry-level staff, then we decimate the future managers and leaders of the company. Welcome to AI Answers, a special Q&A series from the Artificial Intelligence Show. I'm Paul Rateser, founder and CEO of SmartRex and Marketing AI Institute. Every time we host our live virtual events and online classes, we get dozens of great questions from business leaders and practitioners who are navigating this fast moving world of AI. But we never have enough time to get to all of them. So we created the AI Answers series to address more of these questions and share real-time insights into the topics and challenges professionals like you are facing. Whether you're just starting your AI journey or already putting it to work in your organization, these are the practical insights, use cases and strategies you need to grow smarter. Let's explore AI together. So, let's explore AI together.

1:01Welcome to Episode 236 of the Artificial Intelligence Show. I'm your host, Paul Rateser, along with my co-host, Kathy McPhillips. Welcome, Kathy. Thank you. Kathy is our Chief Marketing Officer at SmartRex. If you are a regular listener to the podcast, you have heard Kathy many times before. So Kathy is not taking Mike's job. Mike and I are still doing the weekly, and Mike is the host of the AI Transformation series that we've started. We've done three of those now. And then Kathy and I do these special editions of what we call the AI Answers episodes. So these episodes are based on questions that we receive during our monthly intro to AI and scaling AI classes, as well as some questions that we get from virtual butts. So these are all community-driven questions, and we just kind of go through. There's usually, I don't know, Kathy, what do we have? 15, 12 to 15 of these things? Yes we do. Kathy prepares a brief that I look at as we start recording.

2:03It's a wonderful brief, but I just got to dive in and we go. So yeah, so today's 15 questions. They are based on the last two intro to AI and scaling AI classes that we did. The intro to AI was July 29th, and scaling AI was July 9th. So these are both from late July. And yeah, so the interesting thing, and we'll kind of get into this as we go through these questions, is the intro class I have been teaching since fall of 2021. So we've had north of 60,000 people register for that series. And back in 2021, pre-chatchypt, you could imagine, we were getting some very foundational questions, just very basic, trying to understand what the technology was and where it was going and how to figure out use cases. And the intro questions have evolved. They have become much more advanced. So even the people that are attending the intro class generally have really, really good questions.

3:05And then the scaling AI, we have now done, probably going on two years, Kathy, I would say, we've been teaching that one. Just about. Yeah, we've done 18 of those. And so that's probably had 10 to 12,000 people I would guess go through with that series. And that one teaches our five step framework for scaling AI in an organization. So those questions tend to be more organizationally driven, team department driven. They're thinking bigger picture about how to move beyond just pilot projects and actually scale it in the organization. So if you've never had a chance to attend either of those, we do continue to teach them. You can look in the show notes and there's links to sign up for the next ones. I think we have one coming up at the end of September, maybe. Is that sound right, Kathy? We have September 10th is our next scaling AI. And September 30th is our next intro to AI. OK, so apparently I'm teaching a class next week. All right, so September, you can join us for both of these if you'd like. All right, actually, no, it's on Friday, actually. It's on the 11th because you're not going to be in town on the 10th. So we just shift that date.

4:06Yes, yes. OK. All right. So OK, so today's episode is brought to us by Marketing AI Month. This is a brand new thing. And I am going to let Kathy kind of tee this up for us. And then we will get into the Q&A. Also, do you think you ever need to say that people are worried I'm going to take Mike's job? I do because I think that these episodes with you, Kathy, are so popular. And people might dip in and just want to hear you and I talk. And they don't realize there's this broader show. So I don't know. I think I never know when someone enters into the artificial intelligence show world. And it might be their first experience with it is. And AI answers on like a YouTube clip. I don't know. So yeah, I always try and set it up that way. I'm not taking Mike's job. I don't want Mike's job. We are all good. I don't want Mike's job either. So thank you for being here. This episode is brought to you by us by Marketing AI Month. We launched that Tuesday of this week.

5:06So this month we are doing something really fun. We've got this amazing AI for Marketing Course series as part of AI Academy. Almost long, you can get it free. It's valued at $499. So you get the series. It's five expert led sessions by Mike Practical Tools, frameworks that we actually use as well and a professional certificate when you are done. So that's free to September 30th. You do not need to complete it by September 30th. You just need to enroll by that date. And to close out the month, which we're actually doing on October 1st, we're holding an AMA where similar to this format, I will be peppering Mike and Paul with all of your questions. Anyone that has registered for that or enrolled in the course is eligible to attend. So if you only want the AMA, read and enroll anyways, but the course itself is amazing and I highly recommend it. I took it myself over the holidays and kind of kicked off my year really strongly. So I was really excited about that. So if you visit smarderex.ai slash marketing, you can enroll right on that page. Anything to add on that, Paul?

6:07No, I take advantage of it. It's our most popular course series. AI Academy has I think 22 professional certificate course series right now. So the idea here was like, let's just accelerate marketing, understanding and adoption because our experience with a lot of our AI Academy business account customers is that marketing is often the tip of the spear. It's the department that's leading on AI adoption, but then also an essential part of communication and strategy to bring other departments in and drive AI adoption and transformation through other departments being communicated through the marketing team. So marketing to us is a compounding value for the industry, the more marketers we can get to understand this stuff and be adept at using it in a responsible way, the faster we can spread that throughout organization. So if you're a listener and you're not a marketer, make sure to send that link to your marketing peers within your organization because it again is open

7:07to everyone and it's a great value for everybody. For sure, which takes us to our first questions, let's jump right in. Number one, what are the best ways to use AI in marketing? I might turn this one back to you and say, how are you using a Kathy as our chief marketing officer? I'll keep it simple. And then if you have something to add, Kathy, I'd love to hear your perspective on this one. I just start with what do you do? Like I would look at the tasks and the workflows that you are expected to perform every day. I would look at the goals that you're expected to achieve. And then I would go in and just ask your favorite AI assistant. You could use our jobs GPT tool if you want, but you could just give this prompt to any of your favorite assistants and say, I am a director of marketing. My job is this. Here's the KPIs I'm responsible for. Help me figure out like the top three to five ways that I could be using AI today. These are the tools I've access to. Here's the data I can connect to. Just give it the information and let it help you figure that out. But the answer is it's personal to you

8:07and your role in the company is how I kind of simplify that. Agreed. I think I would add, I'm using it a lot for strategy. We are doing, we are rolling things out on a regular basis. So me getting a head start on getting all of the strategy out of my head on the paper. It's repurposing strategies that have worked in the past. It's building this knowledge base of all the things that we're doing. It frequently helps me write, because I do enjoy writing, but it does give me, you got check on some things. I'm using some of our paid advertising to help me figure out new ways to message. Looking at data, looking at trends that I might not be able to surface with my non-analyst eyes. So one thing we've done on the marketing team is we have these documented processes for virtually everything that we're doing. So we're trying to on a quarterly basis or whatever cadence we decide running it through saying, can we do something more? Can we do something more? So having the processes, having the use cases documented, and just because things are changing so rapidly, updating that regularly has been really helpful for our team.

9:10One example, I'm going to have to get into great detail on this, but the marketing AI month that we just talked about, we decided last Wednesday that we were doing this. It was a very spur of the moment, hey, let's just give AI for marketing away. Let's just try and accelerate awareness and understanding in the marketing industry, leading up to our make on event, to our big conference in October, which is for marketers and business leaders. And so Kathy, I don't even know how she did it all, but we launched that campaign three business days later. Like it was an entire thing, and I'm sure there was ways Kathy leveraged AI to help in the planning, the production of assets, things like that, but to launch a major campaign in matter of three business days is wild. Like you think, what could we have done historically? No, I'm sure Kathy worked really long hours, and it wasn't like she just hit a button and boom, here's your campaign, but even that, there's just no way you do that. We have a small team, like we don't have 20 marketers,

10:12where Kathy is just like, all right, everybody go do your things. It's like, no, we get our hands dirty. We're actually in this doing this stuff. So that's an example of like a very real relevant thing that I'm sure we couldn't have pulled off without the support of AI. And people, I mean, Jeremy Zimmer, our director of marketing for Academy, he was obviously integral in all of this, Mace Air, so it in social media. Everyone, and then I built this brief, and like here's all of the things we could be doing with my knowledge, with AI's knowledge, and I told the team, I'm like, I know I'm missing something. So please, as humans go in and help me figure this out, what are all the pieces parts that I'm missing? But yes, AI definitely gave me a very good roadmap to get it all done. Okay, number two, I have a team that is mostly made up of AI beginners. They are curious and optimistic, but also time-stharved. The most common pushback I hear is, I believe that if I invest time in AI, it will eventually generate efficiencies in my day-to-day work. But how do I carve out the time for now? Sales, targets, and client deadlines won't pause while I do this.

11:13Are there any good tips to respond? I would do a forcing function of a workshop. It could be a one hour workshop, it could be two hours, it could be half day, however you want to do it. But I would do an AI workshop where you teach them a framework. So we have a use case model. Before you can go, I mean, it's in our book, we teach it online. Like there's plenty of ways to go learn it and you can apply that framework to your own company. But basically, let's say it's, let's just pick 90 minutes. We are all gonna leave here in 90 minutes with one to two AI use cases that we can implement starting tomorrow morning that's gonna save us at least three hours a week. Like just set some parameters because what it does is it forces you to look through your tasks and workflows that you do over any given week and say which are the ones that take more than three hours. Okay, now you've narrowed that down to like, let's say there's five things on that list. Which ones have a repetitive process that we could automate elements of? Which ones require creation of strategy documents or analysis of data?

12:14Like you go look for the use cases that are a good fit and you just pick one and you build a project, you build an agent, like you just build something. So make the first like, you know, 20 minutes setting the stage, make the next 25, 30 minutes brain storming where people are prioritizing lists, make the last 30 minutes or so sharing what you've done or actually building the thing. So maybe you extend it to two hours and now you can actually have a build session with it. So you have to force it, like you have to have something that says we aren't leaving here today until everyone has built a thing that's going to save them time. And then if that works, do it every other week or do it once a month and like, then you just stack those things over time and all of a sudden everybody's got five agents or apps that are, you know, saving them 10, 15 hours a week. Yeah, and I think showing them how, you know, I think the jam sessions we're doing where people on our team are opening up their computer, sharing it on the TV, we're all in the same room, we're online, we're watching them, click the buttons and doing the things because that is daunting.

13:15People are like, okay, I think I can do that. I probably have the rain power and the capacity to do it. I just don't even know how. And I don't have time to really figure it out. So having someone show you how to do it is I think so valuable. Yeah, now I don't know. Again, I haven't really looked at these questions. So I don't know if what I'm gonna get into this and I'm not even actually 100% sure I want to share this yet. But like I started a project this morning where I'm basically going through and trying to look big picture at where agents can be infused into my role. And as I was doing, I was like documenting journaling it. Like I normally would. Like, okay, I'm going in here, I'm creating this, I'm gonna test this agent. And so I would go through in like this whole dialogue with myself, but then I realized, wait a second, maybe I'll just turn on screen recording and I'll just like record this in Zoom or Google wherever and then eventually take that and turn it into like demos for the team, but then I'm like documenting all the steps I'm going through, all the visualizations I'm seeing as well. So all of that is helpful. But I think you're right, Kathy. Like just you need to infuse whether they're workshops,

14:16build sessions, jam sessions, whatever you want to call them. Those have to just become part of the routine. And then it gives people the time and the permission to do these kinds of things. Right. The build sessions that make on this year, not to plug make on, but I will. I'm really excited about that because you bring your computer, you sit down and 45 minutes you leave with something tangible and it's just so helpful. Even if it needs to be, you know, sussed out after you get home, but just being there and watching someone else do it is pretty cool. Okay. Number three, we are a small company. How should we think about using external consultants to help the team is still busy doing day-to-day work that we are not making time to create agents and take us to the next level? We have lots of ideas, but need the time skills, what thoughts do you have on that? Yeah. I mean, there's a decent chance you can find consultants who come in and help you analyze your workflows, look for ways to drive innovation, look for ways to drive, you know, automate things, build some agents.

15:19I think that that's a short-term fix, honestly. I just feel like this is gonna be so critical to every organization's operational structure and the way they manage the company that you have to own it. Like, so I explain this a little bit on the podcast, but we're building smarter X labs within our organization. We're a small company, we have 19 employees. I think of labs as like the R&D unit and their job is going to be to go to each team or department or business unit and work with them to say, where are the barriers right now to achieving your goals? What are the workflows that take the longest or what are the major things that we should be doing as a company that we just don't have someone in that seat right now, like we can't do it? And so they're gonna basically function as four-deployed engineers, so that's the concept we talk in the podcast where you basically have people who go in, look for things to build problems to solve and then they build them for them.

16:20But what I believe the reason we have to internalize this is because we have to maintain that IP. Like, we have to maintain that know-how within the organization because if we go in and we solve something with Kathy and her team on the marketing side, it may translate over to the sales side or the op side, like an agent we build here might just need a couple of tweaks and all the sudden it's a sales agent too. And so because AI being infused into all these areas is so critical to what we're trying to build, I just see it as like absolutely fundamental skill we have to build internally. So if you're gonna rely on consultants in the early going, I think that's fine, but I would be training or developing people who can function in that role moving forward. And it could be a marketer who's just, also pretty technical, like very comfortable in the data and the systems that's trained to become like an AI ops person. And it doesn't, you don't have to go higher software development people, you don't have to go higher AI engineers.

17:21They can be business people, you know, knowledge workers with domain expertise who are just the most advanced users of the AI platforms you use. So that's how I would think about it. And you think about subject matter expertise, you know, being like you said, domain expertise, institutional knowledge like a consultant can help you with most of that stuff. So, are just so do you start building no books and knowledge bases now if you're going to start using someone on the outside? I mean, if you're gonna bring in a consultant, they're just gonna come in initially and like document processes, audit workflows, they're gonna look at what the AI's going. And again, all could really be valuable. Like if you truly don't have the time to do this and like the alternative is no one does it? Then yeah, like hire somebody to at least get this moving. But I don't know, it's one of those things I think is so critical to every company that I would be very aggressively looking at. What is that, like defining what that role is? And seeing do we have someone on our team who could, this is a career opportunity for them? Like they would work really, what we've seen this

18:22in companies, Kathy, you and I both know people who are in AI ops roles today who, you know, 12 months ago, 18 months ago, were marketers? Are they project managers or like whatever? So this to me is that whole creating, seeing opportunity for a career path for yourself, defining it and then going and building it. And when we start this internally, I would like to be tribute and I would like you to tear apart my department and tell me all the things that I will swallow my pride. And I can't wait to see what comes out of it. I think it is just like a totally collaborative thing. I think it's more of like you have a wish list, like you have roles you'd love to fill. I think this division is like, okay, Kathy, what are the five roles you would put on your team today if you had the approval to hire them? And they might look and say, I think we can get you like 50% of the way there on three of these in like two weeks. It's not like you're not gonna be your FTE, but like we could probably build something that'll at least give you that functionality. That's how I'm thinking about it. And then you kind of like go down from there. I'm ready. Okay. Number four, who should be the point person for AI transformation in the enterprise?

19:23And some organizations, it's the CFO, with project manager who understands the systems and knows where the data sets are hidden. Has this been your experience? I don't know that there's a right answer for this one yet. I do think it depends so much on the skill set of this person. So I've seen one where it was like a chief legal officer who became like the head of transformation because that person was just also extremely qualified on the AI side and understood the business deeply. I've seen it being driven. Our first AI transformation spotlight that Mike did with good karma sports, that is a CEO driven transformation. I mean, Ty, who did the interview, was telling he's a key driver of the whole thing, but that is being very much led from the top by Craig at that organization. And so this is a CEO who got into it, said this is the future of the company, I'm all in. And he is like right there every week and every meeting about AI strategy. So in that case, I don't know that they have a chief AI officer,

20:26but it's probably Craig like he's the driver. I've definitely seen where the CIO does it if that CIO has deep business knowledge. Like if they're not just thinking about it from a technical perspective, we've definitely seen instances where like Dan Slegan who's speaking at Mekon this year, marketer, CMO, and Dan just sort of emerged into being an AI transformation leader at a major company because he just became the most adept at the technology. So I don't think that there is a right answer. I really do think it is where is your organization at? Where is the CEO in the organization at? Because I am a believer that if the CEO is extremely AI forward, that is probably the person who should be at the top of the chain when it comes to spearheading the transformation. Obviously doesn't have the ability to be in every detail, but the CEO should be the driver. That's probably gonna be the most successful organizations is when it is a top three priority for the CIO and they are actively involved in everything.

21:28Yeah, it just opens up that permission and enable it when the CIO is in that role. Yeah, like I actually, I, this is totally behind the curtain stuff. I actually messaged our team today because I was working on something pretty significant for our organization. And so I just messaged, I was like, Hey, can you guys send me the excerpt from our AI policy regarding connectors because Tracy and Elizabeth are gonna laugh when they hear this now? Because I may be instituting an R&D exemption that's a term I just made up. I literally put in parentheses to them. And like approving things myself. So I'm not bypassing our approval channels in our AI policy, but I am. I mean, I'm excited. I'm excited. I'm accelerating them. And so within like 10 minutes I had, I was copied on emails to our IT team and our lawyers saying that basically our CEO is planning to accelerate approvals. We'll call it. All right, I'll see.

22:29So let's think about companies. Think about companies we know where the CEO has gone rogue, Paul. That's not it. Oh no, Tracy did her job. She was like, here's the 15 things that I'm basically like wanna check before I like, go back to Paul, but like, and they're all really smart things. It's like, yeah, I know. I'm not gonna just like hit the buttons. But yes, when the CEO has an urgency like, hey, I'm doing a presentation on this in seven days and I need these connections in place right now, then things happen. And you do it within the safest way possible. But you have to also like, and again, I don't wanna like spend a whole time on this. But like Tracy's thing is, okay, cool. You turn that on, does everyone have access? Do we have to like communicate with everybody, not to use the access you just granted? So while I wanna move fast, I, that is why I reached out to her first and said, I'm going to do this within our guardrails. We have to set that example. But we also have to find a balance here because I have no choice.

23:30Like I have to do this in the next seven days. So how do we do this in the most responsible way possible related to our guidelines and policies? So you may have, you may have some access in your chat, you can come later today. They listen, look familiar. Okay, number five, are there generic agents everyone can use? Or for agents to be beneficial for us, we need to build them on our own. It depends on what platform you're in. So this is actually the initiative I'm referring to. I'm working deeply on integration of agents into smart directs. And I had this, I even talked about this internally. So again, I was, it's not even joke. Like people think I'm like making this up maybe on the podcast. So I'm like, I know on our team knows, literally no one on our team knows about some of the things I'm working on. And they hear about it on the podcast. So a couple of weeks ago, I started a personal initiative to deeply infuse agents by the end of this calendar year. And in essence, figure out what that looks like

24:31moving forward, all the connections, how we're going to structure them through the smart directs labs, division, things like that. And so I have a forcing function that I have a presentation next week related to an agentic enterprises. And then I'm teaching an AI executive workshop at Maycom on October 13th, where I need to actually present a model for this. So I'm deep in the agent world at the moment. And what I'll say is like an example here would be if you're in chat GPT, they have standard agent templates that you could go in and just start with those, like when the pop some eyes, like a marketing strategy agent, they've prebuilt the skills for it, they've built the system instructions for it. All you do is go in and give it a knowledge base or connect it to like HubSpot or whatever it is you want to connect it to if you're allowed to. So I guess the short answer is yes, like there are generic agents if you're in co-pilot or if you're in Google, chat GPT, Claude,

25:31they've prebuilt some agents that are more universal. But most likely you're going to be building things that are specific to your role or to your organization because for them to be really valuable, they're going to need a knowledge base that is personalized to you or your company and they're going to need connections to data sources that your administrators are going to have to approve. Got it. Okay, number six, I love this question. Can you set up checks within an agent that require it to pause for human approval before taking specific actions? I wouldn't build an agent that you couldn't do that with. So yeah, it just depends on how technical you are. Like where you're building these agents, but again, if you're working, I'm going to assume most people that listen to our podcast are probably not like advanced codex users, cursor users, like they're probably not the, now I might be wrong.

26:33Again, we don't really, it's hard to know your podcast audience. It's just all these people, it's quite opaque. So I'm assuming that most of our listeners are more marketers, sales professionals, business leaders that are not deeply technical coders, software developers. So we, I'm just going to use the collective we hear, are likely we're sales force users and we're going to be trying to figure out how to use the sales force agents that are available to us. We're going to be building co-pilots in Microsoft. We're going to be building chat GPT agents. There's instructions within there that where you're going to say, this is the checkpoint where you're going to do it, or you're going to give it read only access where it can't actually go and send something or do something. So the vast majority of us are likely only working within building agents where humans are in the looper, in the lead. And yes, you should know when an agent would be taking an action and you should be the one that giving it permission to do that in low risks of scenarios you may give it permission

27:36to do the actions without approval. Like, example, they're send Kathy a summary report every Sunday night of marketing performance data the week prior and it goes to HubSpot, it gets the data, pulls it, it summarizes it's sensitive. So it's going to take the action because it's just between me and Kathy. There's no risk of it sending to our customer base. That kind of thing. OK, cool. Number seven, for organizations with many data sets containing personal and sensitive information, a complex enterprise architecture legacy systems shared services across very different departments and pockets of AI forward teams, how can agents be enabled for those teams without creating risk for the rest of the organization? With great care. That is like when you're talking about these agents that are accessing sensitive information, let's say like a healthcare system, law firm, a bank, wealth management company, whatever, like there's just, we all have sensitive information. We can even say smart racks, we have customer data

28:37that we always obviously want to be extremely sensitive to. So anonymizing data before you put it into these systems, things like that, you have to have IT and legal involved. Like, it's just, you can design what you want to have the outcome be, you can build prototypes of agents, you can do all of these things, but when it's going to do things that present risk to an organization, especially if you're in a heavily regulated industry, you have to know and advance the approval systems you have to go through, you have to deeply understand your AI policies. And that's the example I was giving earlier with Tracy. Like, Tracy just taught an entire course and we'll get to some AI policy stuff I see it highlighted below. She just taught a whole AI Academy live course on our AI policy, which is a very comprehensive document. And if you're going to be building agents, you have to be familiar with that. You have to know what you're allowed to do and not allowed to do what data you're allowed to put into it, what connections you're allowed to make. And so what we're trying to do as an organization is these sort of like, I'm just going to make up a term here,

29:37like horizontal approvals, like, okay, if we want to connect HubSpot to ChatGPT, let's approve it once so that if Kathy or Tamara or Dia or anybody wants to do something between ChatGPT and HubSpot, they don't have to come and ask for the connection permission every time. It's already been connected and then it's approved for read-only access for like all these different uses. So you do something once and then it permeates across the whole organization with permissions. And so I think you'd be in a situation like that. Like if you're in this kind of enterprise where you're going to be accessing personal sense of information, doing these complex architectures, try and solve for as many of those with a one time permission and training process as you can so that like everybody's not coming and trying to get approvals are basically the same thing. Right. Number eight, how should a company fund its AI investments over the next few years? And where should I expect the new spending to come from? Little boy.

30:41It's got to be dynamic. I don't know, like this is a really hard question to answer. Like I'm just thinking out loud here. People is my first thing. So like when I'm thinking about smarter X, I'll just take it to make a personal. So for our company over the next three years, where am I going to put AI investment dollars? My first investment is the people to help me figure out the answer to that question. So the people who think 24-7 about our AI investments, that could be human capital, it could be token capital. It's like where's our spend going to go? What platforms are we going to invest in? But first you have to have the right people to make those decisions. Second is going to be my platforms. So what are our token budgets? Where how are we going to optimize those token budgets? How do you even budget for token budgets? Yeah, like teach people what that even means and how do you manage them. And then third would be probably more of an infrastructure thing.

31:41Do I need to be buying Mac minis? Do we need to be buying GPUs? Like are there business cases from a hardware perspective where we actually need to be thinking differently about the company to where we have access? And do we need to be building? I guess the fourth would be, is there going to be an increasing argument that we should be training our own model at some point? I think that a lot of companies are going to head in that direction where they're going to get these open weight more efficient models. And they're just going to do reinforcement learning, training on them to be adept at their specific industry and business case. So yeah, I guess that's four. And again, I've never actually thought deeply about this question at all. So that's, I reserve the right to change my answer at some point. But it is hard for I met with the group this morning and they're on a three year cycle where they're trying to plan out the next three years. Like how do you even do that? And as leadership, are you able to give your team permission

32:42to just as intelligently as you can make those decisions knowing that they're going to change? Yeah, I think three years is hard, but every company's doing it. Every enterprise obviously thinks in three to five year roadmaps, like they're all looking in that direction. I think you need to make decisions that are as flexible as possible and as like evergreen as possible. So you're not like locking, I don't know, like one of the ways I think about this one is, you know, if we go really, really deep with chat GPT as an example and everything starts running through that and all the connections are for chat GPT and then something happens to open AI or like their models or whatever. And it's down for two hours, two days, two weeks. What happens to the company? There's no redundancies, there's no backup plan. And so that's a, you know, betting on a single provider of intelligence is in a super scalable thing probably.

33:45And so when you're looking at three years, I strategy be like we need intelligence redundancy. And that scales, that like becomes a strategy that lives over three years and actually starts to affect the decisions you make because we've set a priority around redundancies. So that would be good. Well, that thought because I do have another question later that I don't want you to answer right now. Okay, all right. Let's end it right. Let's end that right there. That's the end. Okay, number nine, kind of going into this a little bit. Beyond subscription costs, how are companies actually budgeting and forecasting total AI spend including training and education time, set up an integration, internal token usage and token usage driven by external user or customers? Is anyone doing this well enough to have a real forecasting model or is everyone still finding out when the invoice arrives? Are there any resources you would suggest? The best, I've heard, I don't even know if best is the right term here. A model I have heard is literally just sending or setting a cap each month. Like here's our budget for the month.

34:47They're definitely more and more organizations trying to figure out the token optimization and management. There was just something from Google, I think it was the Monday I saw this, where they're starting to offer that kind of capability within their systems to help you manage it. There's startups emerging that do token optimizations and basically route you to the most efficient models. I saw something I think it was this week where OpenAI is now actually experimenting with outcome-based pricing. So part of this is we're developing systems for the way that AI is priced right now, but most of these labs and providers of the intelligence know that this is not a viable scalable system. Per token utility-based pricing works sort of for developers, but once you start getting into departments like marketing sales, operations HR, that's a really, really bizarre way to try and price stuff.

35:48If it's not seat-based with unlimited usage or it's not outcome-based, if the outcome is very predictable and able to be attributed specifically to the model's work, then it gets kind of tricky. So yeah, I don't know, like an awareness of how the tokens are being used, what the different models charge per token. It's very early. I don't have a great answer for like, go look at these three companies because they're doing it perfect. I don't know anybody who's actually solved this yet, including some of the big tech companies that are responsible for building the models. I don't know that other than providing unlimited uses of tokens to their employees, but even meta just changed. Like they've made changes to the models that they're using for this reason because they couldn't manage the token budgets and they're a builder of their own models. Like, so. And if you do know somebody, listener, reach out. We have a lot of transformations.

36:49Yeah, totally. Okay, the next three questions are from the Academy Live. We had on Tuesday this week with Tracy or COO and then Samantha, early, your council. So these were, I thought these just fit in well with some of these discussions. So the first one is, what does a right sized AI vendor approval process look like? How long should a review take? What happens to employees while they're waiting and how do you catch it when a vendor update introduces new integrations or changes how an approved tool handles data? And I know that you are not doing, you know, a trait that ops is doing this, but as the leader, what thoughts do you have on this? I would start early. I mean, we've, we've been in situations with some customers of ours where they got excited about instituting, you know, different AI technologies into the organization. They made the business case for it. And then they go to procurement and they would be told there was like a multi-month wait time before procurement would even look at anything. So it starts with really just understanding

37:49your organization's procurement process and what kind of risk and security considerations need to be made as we start getting into AI technology that might require access to certain data sets. And yeah, I think it really is about an understanding of the process. And then from there, it's going through and building that in. So if you know that it's going to be three months before you can get the system you want or the approval for the use case you want or the connection you're looking for, you need a plan B about what that workflow looks like without those tools. So yeah, I think again, this one is very subjective to the organization and what their approval processes look like and how they're defined within the AI policy. So I know in our case, we have to submit now through form requests to the operations team when you want a new vendor, if you want to make a connection to a data source. And then I think the team says like within 48 hours or something that at least give a preliminary,

38:51hey, we're going to have to look deeper into this. This is going to be quick. Kind of the example I gave a trace earlier, like I put in a request, I will call it. And then there's a, you know, within a few minutes, there's an email to the legal team, there's an email to the IT team because for the ops team to make their decision, they need the expert perspective from those two bodies to make the informed decision around the vendor or the connection that's being requested. So yeah, it's going to vary by company to company, but the best thing you can do is deeply understand that process before you get started. So you don't run into that obstacle when you're ready to go. For sure. And in one instance that we had internally, the request was put in, it took longer than normal because it was brand new tech, we were trying to figure out, well, ops was trying to figure all of it out with legal. And there were some redundancies because things were being done twice, one the old way and one trying to do it with a new tool. So there was a minute of double work, but the long run's going to be so much better.

39:51Now that everything got approved the right way. Yep. And I will just say being on the other end of this, we have our AI Academy, which is a technology platform. And so we have to go through procurement, especially for these bigger enterprises, and you have to meet some very stringent standards that are only getting stricter as we go. And so Tracy can attest, like I mean, she spends a fair amount of her time getting through procurement systems. And she could probably sit here all day telling you about all the new requirements over the last 18 months as a result of AI. Yeah, there was one she did last week or two weeks ago that she was just like, this is the wildest form if ever filled out. Yeah, that's very. Yeah. Okay, number 11. When reviewing an AI vendor's terms and data policies, what do companies look for regarding what happens to their data if the vendor is acquired, goes bankrupt, or sells its assets? How can organizations manage or mitigate that risk? Or sells the data for training purposes?

40:52I'm like this one I just honestly feel like my best answer is you have to rely on your legal team. Like everybody's legal team's gonna have different appetites for risk. They're gonna look at it through different lenses. It is just simply not a decision that a, like a traditional business leader, department leader. You don't have the expertise and likely you don't have the permit in your company to make those decisions. You have to involve legal and whatever other channels are required. And hopefully based on a previous question, that's all clearly documented in the AI policy. And if it's not, it needs to get documented so that you can follow these procedures. And then you know when it's a non-starter. So you're getting really excited about this new AI tool. You go in and pull up their terms of use. You throw it into chat GPT and say, tell me how they handle our data. And immediately it pops up with like three red flags for you. Another way you could do it is take your AI policy

41:54related to vendor data usage. Drop that into chat GPT thread or Geminar wherever. And say, hey, here's our AI policy or build a project that has that AI policy baked into it. I'm going to give you a contract for a new vendor. Can you tell me how this relates to our terms of use requirements and flag for me what I should highlight for our attorneys? I do that kind of stuff all the time. Where I'm just like, here's this doc, here's this doc, compare them and tell me what I should ask the attorney. For sure. And then you're just thinking about when they change the terms and conditions or when something, I mean, there's so many, it's not like approve at once and it's done. It's like set up some normal cadence of just reviewing everything to make sure it's still. Yeah, like when Anthropic came out with Fable 5, they changed their data retention policy where they kept everything for 30 days. And a lot of people were up in arms about that. Right. Number 12, when should a company disclose externally that AI was used to create content or complete work?

42:54Is there a meaningful threshold or does it depend more on the nature of the work and its potential consequences? This gets in again to AI policies. Everybody needs to make these decisions for themselves. I feel like it's a bit of a sliding scale where more and more, it's just going to be kind of assumed and expected that AI played a role. And I think tagging every single thing with AI helped here, AI did this. I don't think most people are going to care most of the time. So I think that there's some basic rules that at least I tried to live by and I hope our company tries to live by. It's like when authenticity matters, you should be doing the work. It should be human in the lead. Like it's okay if you're using AI to edit or verify ideas and things like that, but it should be you. And you should stand behind that work. This came up on episode 234 or 35 of the podcast. We were talking about that one author who used it and got called out on it.

43:55He's like, of course I use it all the time. Like what are you talking about? I generally like to default to, if you're using it, it can be something as simple as like co-authored with Clawed or Clawed supported in the editing and revisions of this document or Clawed helped me with brainstorming, whatever. Like just disclose it because it doesn't do any harm but I don't think worth the point we need to make like a huge deal of it anymore. Right. So yeah, again, it's, you gotta make these decisions in your app policies and then provide that guidance to your team because everybody's wondering. It's like, I don't know if I'm supposed to tell somebody I didn't actually look at their email like my agent is now answering my email swore me. In that case, I would just say, yes, you should disclose it if your agent is answering your emails for you. But yeah, I think it's a very personal decision and it needs to be decided in the policies. I was working on something last week and I came to recall what it was but I kind of went into it blind, not really knowing what I was gonna be working, like what I was gonna be doing and I used AI a lot.

44:58And I put that in the, when I distributed it to the team, I was like, this is Clawed written with Kathy supporting it. Like I honestly, I don't know. So this is my first pass I'm learning and just some people looking at this like this doesn't sound like her. It's so obvious when it's not, you know? But I just want people to know that where I was on that process. Like one example and you could have tested this, Kathy, like sometimes we'll be working on a problem and I'll go into chat to people like, here's the exact problem. Here's the context you need. Here's what I'm trying to think through, build a brief for me. Like I just need to like think about this and it'll build a brief and I will take 10 minutes. I'll read the brief, I'll give it some thought and like this is really good. I will send it to the team internally and say, this is completely unedited, but I have reviewed this and this is a really good starting point and then I turn it over to the team. So they know I didn't do it, but they know I've signed off on like, this is a good direction. Come back to me when you all have made progress on this. So yeah, I just, I don't know. I don't think there's like rigid rules around this.

45:59I feel like it's just evolving as our use of the models changes. Okay, number 13. Entry level staff have historically learned their trade by implementing task-based work assigned by their managers. If that work is not going to be assigned to AI, how do entry level staff learn their trade and where do future managers come from? My theory is apprentice programs. I talked a little bit about this on the podcast this week. It was 235, I think it came up because there was that research that came out about apprenticeships were actually like the number one thing and it just, that happens to be what I'm talking about for Macon. Opening keynote is the architect, the orchestrator and the apprentice. It's a theory and that it's just a theory on what the future of work looks like and what the different roles we're going to need within organizations are. My belief is that we can actually accelerate learning and expertise through apprenticeships with the support of like guided learning from AI

46:59and personalized learning. But I'm not sure yet. I have six weeks to figure out what that exactly that is. I've been thinking about this model since March of this year. I don't know, this has been my great debate where does management come from? What do managers do in the future if we don't train entry level employees? So if we stop hiring as many entry level staff, then we decimate the future managers and leaders of the company. And I don't know that again, kind of like the earlier question. I don't know of an organization that has this figured out. Like it's, I've talked to lots of companies and there are lots of AI transformation leaders and HR leaders at big companies that are just starting to ask these questions themselves and nobody seems to really know where to go with it yet. Yeah. 41 days, in case you're wondering, tell me, Con. No pressure. 41. I only have three talks to create at that time period. Number 14, you described the actual mechanics

48:01of an LLM practice predicting the next word as a bit of a black box. Is it really, and if so, isn't that a little ominous? It is really a black box. There's a field of study called mechanistic interpretability where they try and understand why the models do what they do. The LLM's itself is making predictions based on proximity of like words to each other within a graph. So this is like a very oversimplified way to try and help people visualize this. Imagine all of human language as like this three-dimensional cube. And within that cube is like every word in the human language. And how close those words are to each other sort of shows a probability of them occurring next to each other. And so when chat GPT starts writing something, it's a essence going through this dimensional graph and it's like knowing that Kathy sat on the chair and it knows that chair is a likely word connected

49:02to where Kathy would sit, because people sit on chairs, could be a bench, could be the table, could be all these things. But in essence, like imagine in that cube, those words all appear near each other, sat, chair, desk, whatever. That's kind of like how it works. The way to think about the language model, the closest proximity to it is the human brain. And that would be like saying, well, do we really not know why Paul picked the words? He just picked when he was explaining this. Like, no, we actually don't. Like you could put it me under an MRI machine and it might see what neurons in my brain are firing when I'm talking, but it can't drill in and say, why did he choose the word he chose? It's likely just on my training and in my experience as a human of the words that appear near each other in the English language. So that's kind of it. Like it, yeah, it's ominous. Yeah, it's weird. Like we don't really understand this intelligence that we've created. All we know is that we give it more and better data

50:03and we give it more chips to train on that it seems to just get smarter and more capable of simulating human behavior and language. And that's kind of it. Like that's the basis for the first scaling law of AI that has now stayed true for the last decade, which is more data, more computing power, bigger the model, the better it seems to get. There was this sort of ominous thing that Ilya Setskov has said back in the day at OpenAI, which was they just want to learn. Like these models just want more information and they just get smarter. Okay, number 15, the nice thing about these questions is that I get to decide the question. So I actually dropped in my own question. Okay. My husband's been traveling for a week and I've had a lot of free time. So these are the things I think about. Okay. When I worked with Joe Pilatesy at the Content Marketing Institute for years, he always emphasized the importance of building on land you own, your blog, your email list, your audience. We've seen what can happen

51:04when people go all in on rented land, like LinkedIn, Twitter or another platform. The algorithm changes products audiences, audience becomes harder to reach the platform disappears. Now people are building products audiences and sometimes entire businesses on custom GPs and other AI platforms. Are we watching the same story repeat itself with LLMs and how should we think about the risk of building on rented land? That's a deep thought to have. I know, a lot of free time. Not really free time actually, alone time. Yeah, I think it kind of relates back to the thing I mentioned earlier about, if you build on a single platform, you're sort of a prisoner to what happens to that platform. I look at this a lot now with how transferable skills are between agents and platforms. The system instructions that are used to build a GPT can very easily be copied and pasted into Clawed and you can build the same capability over in Clawed. So I think that there's an inherent risk here.

52:07It's probably part of the reason why some organizations are now choosing to take open-weight models and train their own models. It's just a more reliable system. The model lives on internal servers. You're not captive to a Clawed service going down and you no longer have access to it. So I do think that more organizations will start to build redundancies and backup plans in. At a very surface level, you can start by making sure everything is documented. We've done this again internally at Smart or X, where we're now, at least from a governance perspective, documenting when apps and agents are created by different people. Ideally, you include what the system instructions are, what the knowledge base they have access to is, so we can go back and audit it. And so let's say like worst case scenario, you build everything around chat GPT. You've got 25 agents that are basically running the company. You've got 75 skills that those agents use to do what they do. And then chat GPT goes down for five days. You could worst case go grab the skills,

53:10recreate them in Clawed or you just mirror everything in Clawed. Like someone's job literally might just be maintaining a mirrored image of the chat GPT instance in a second platform. So that if something goes down, just flip a switch, everyone moves over to Clawed and continues their work. I would imagine there are organizations that are actively doing what I just described. I haven't talked to them. It's just one of those things you can kind of make an assumption that someone is ahead of the game doing this because we are talking about entire companies running on this. Now you have companies laying off thousands of people under the assumption that agents can do a lot of this stuff. Well, if those agents aren't available, work doesn't happen. So there needs to be redundancies built. Yeah, and I was talking to Claire, who is our producer of the podcast and works on our content. We were talking to you about how Mike had actually talked about this in one of his courses for Academy about how to extract some of that from one tool to be able to upload it and put it in another tool,

54:10or how to document it and what that process is. So it really is a critical step of all of this. All right, that is 15 questions. That's a good questions. I feel like there's all different questions too. I don't know if I've answered any of those. Maybe the marketing use cases has probably been answered at different times, but it changes, but the rest of these are relatively original, really good. You don't look at this beforehand, but Claire and I do. Yeah, so she does a lot of, she does a lot of, has this been asked 20 times already? That's not after that again. So yeah. Yeah, good stuff. So thank you, Paul. Well, you'll see Mike and Xtusay back on the podcast. If you are interested in learning more about marketing AI month or Mayconn, our big event coming up in October, please let us know. I'd love to help you get you to both of those things. They're coming to Cleveland. We've got, oh, do the pod, because we have the pod 100, which gets you a VIP lunch with me and Mike's. We're doing on that the final day of the October 15th of VIP lunch for anybody who uses the pod 100 when they register. So yeah, Mayconn.ai, get registered.

55:12We'd love to see there. pod 100, thanks Paul. Thank you. Thanks for listening to AI Answers. To keep learning, visit smarterx.ai, where you'll find on-demand courses, upcoming classes, and practical resources to guide your AI journey. And if you've got a question for a future episode, we'd love to hear it. That's it for now. Continue exploring and keep asking great questions about AI.