AI in Scientific Writing: Opportunity, Risk, and Responsibility

We love dogs. We tolerate their breath. But we know bad breath isn't just a nuisance.

It can be a sign of canine dental disease. Sure. Daily brushing to keep teeth clean is ideal,

but getting a dog to cooperate can be a real challenge for many pet owners.

Greenies. Dental treats are vet-recommended and clinically proven to promote oral care

when used daily between professional cleanings. Plus, they're irresistibly tasty,

and VOHC accepted for plaque and tartar control, with a texture designed to clean teeth effectively

with every chew. So, if you're looking for a routine they'll actually stick with.

Greenies dental treats are the perfect place to start.

Welcome to Veterinary Vertex, the AVMA journal's podcast,

where we delve into behind-the-scenes look with manuscript authors.

I'm editor-in-chief Lisa48, joined by associate editor Sarah Wright.

Today, we welcome a couple members of our journal team, Mordekonway, and Vic Shultz,

to discuss AI and scientific publishing.

Thanks for joining us, Mordekon. Thanks for having me, Sarah and Lisa.

Thanks for having us. So, Mordek, why are verifiable citations essential

for scientific credibility? Okay, so verified citations establish that the current research,

the manuscript that you're publishing, that the findings in it are firmly based on prior research.

So, it's a whole notion of the building blocks of science,

that you don't just plaque something completely new out of the air without it having a solid

pathway from earlier research. And that's basically the whole fundamental notion that

science progresses building block by building block to advance knowledge.

And so, if you can't verify the citations, the references in a work,

it means that that work is questionable at best.

Yeah, you know, it's starting to happen in grant review as well.

I was just speaking to someone on a review panel that read my latest editorial and they were

saying that on grant review panel, they found these hallucinated references as well and did not

score the grants. It was controversial for some people, but they drew a hard line like we did.

Yeah, more than how do these AI tools end up generating citations that don't exist?

So, these AI tools like ChatGPT and large language models are basically

they're not databases of factual information. They're not the same as PubMed or Google scholar.

They are the databases that the LLMs are trained on and what they're supposed to do is prioritize

and predict the next likely word, the next likely concept. And so, what happens is that I mean,

basically, they are not working from verified sources, the general GPTs or LLMs.

So, it's the difference between information processing and language processing systems,

and AI is based on, in this instance, on language processing, not data processing,

which is a big problem. And the very smart. I mean, so, I think if you think about it,

how do they come up with something that looks genuine in the first place,

at first glance? Well, basically, it's because it's taken the manuscript that you're looking at.

And it's searching around and finding, predicting, but it's going to have these,

or that it would be referencing these authors, or referencing these keywords or titles.

So, it's fabricating, it's hallucinating. And it's a really serious problem.

Yeah, thank you for providing that clarification. I think that's important, especially when we have

younger authors, want early career, they're looking at using AI more and more

flows and such. So, thank you. So, Vic, why do fabricated references often appear convincing?

Well, more to touch down it with her description of the way that LLMs are built, they scrape,

they scrape all this information and build their body, their corpus,

whatever you want to call it, they work from and incorporate real references, authors and

terminology that's coming in the literature, all those things are in there. And essentially,

when they look, I'm not a software engineer, so I might put it in the wrong terms, but this is,

this is what a copy, as a copy, this is what I see and what it feels like.

It looks like the kind of remixing, when a reference is fabricated, it feels like, as if they've

remixed these other sources and each element of the reference is, well, that sounds right?

That's a real author that's well published in that area of veterinary medicine,

but maybe they've never worked with this other author that I can find and it turns out that

this article title, it sounds right, but doesn't actually exist. And this journal certainly exists,

but if you go to that issue where that volume, the pagination is totally off and that reference

turns out not to exist at all. So yeah, it's because they're using all these elements to

spit something back at you that I guess may or may not be right. Yeah, the bad thing ever is

pretty good. Oftentimes, I think, you know, the outputs are accurate. Yeah, they sure look good.

So, Vickins, they look so good. How are these hallucinated references discovered during peer review

and by our fabulous copy editing team, including you?

Well, our peer reviewers might just know the material well enough.

Honestly, the copy editors, we use in the course of our editing, we use software to format our

references and then in the course of checking the outputs from that software to make sure they're

formatted correctly, we can see if the DOI has a link, for example, or if there's a DOI at all,

and if there is not a DOI, for example, or any kind of a P-Mid or any of these other indicators,

that's usually a red flag that this might not be a real reference. It oftentimes is still a real

reference, then we dig a little bit deeper and we can find, okay, this is maybe just a not very

well-known journal or a journal that, you know, for whatever reason doesn't use DOIs, but

in 2026, it's very rare that there's just no sign of an article at all. That's a real article.

So, that's pretty much what it is in the course of styling the references that we,

that we, you know, some red flags pop up that might tell us that that's not a real reference.

I found a study, a recent study in scientific reports by, this is September 2023,

Walters and Wilder, found that 55% of GPT 3.5, that version of GPT citations were fabricated

versus 18% for GPT 4.0, and likewise 43% of the real GPT 3.5 citations, this is the actual real

citations included substantive citation errors versus 24%. So, I could, of the, of the later version,

the GPT 4 citations. So, I could see that Vic would be kept incredibly busy, just with the

formatting issues, let alone, you know, the verification issues, because when you think about,

you know, every article seems to have at least 20 references, you know, and that's a lot of checking.

It's true, the authors do keep us busy, but that's why we love them.

You know, I think over the weekend, you were talking about how reviewers can pick this up

as well, Vic and you were saying, so we had an article that was submitted

about chatbot in veterinary medicine, and a reviewer who is an expert in AI,

looked at it and thought, you know, the language wasn't perfect, but, you know, sometimes it

isn't perfect, especially if English isn't their first language. So, it got past me, and it got

past one of our various student associate editors at Javima. It got assigned to this reviewer,

and this reviewer got back and said, this is an entirely fabricated manuscript, and it's our first

at Javima and AJV are that we know of. And this reviewer, as you were saying, but they just know

the field, this reviewer actually looked at this person's affiliation and went to the affiliation

website and didn't find them. They're, they're listed their affiliation. Yeah. And like, and then

that reviewer then took that author and plugged that author into PubMed and looked at all that author's

references, and they actually listed the AVMA as one of their affiliations and one of their

manuscripts. So, terrifying world, but we're trying to keep up. That is super scary.

Well, that's our doings episode. So hopefully we can bring some education,

shed some light on this really important issue that we're facing in scientific publishing.

So, Vick, what roles do copy editors play in safeguarding reference accuracy?

Well, I guess I think of us as the last line of defense. But we shouldn't be the first.

Really, you know, I would like authors to be more vigilant on this type of thing too.

Reviewers are great. I don't know that I can, that I think of this as like a primary responsibility.

There's nothing to catch stuff like this. So, yeah, I mean, really it's just become,

it's just become part of a part of our standard, you know, standard role, I guess.

You know, we have a new copy editor coming in. I plan on talking with this copy editor extensively

about looking for indicators of AI use. And look, AI, generative AI use even isn't necessarily

out of bounds. It's just using it responsibly. So, yeah, I mean, you know, we're all thinking this

through as we go. But, but yeah, I think of it as part of our normal responsibilities now.

So, Vick, are you working on papers that have been accepted? So, you see them after acceptance?

Yes. So, you're really the last resort to catch something.

Yeah, we're the, we're the bulwark, I guess. So, yeah, I hope we hold up. I think we usually do.

So far, the technology evolves quickly soon.

Yeah, Vick, I like how you're referring to, you know, AI is great, but really still needs

a human element. There's, you know, as you know, we've tested multiple AI detection softwares,

and none of them are great. And so, why isn't even the automated case in checking software sufficient?

Yeah, I've discussed this with software engineers even. And to some extent, at least with what we

use, it might just be an issue of the way that software engineers think. They've been like,

I guess that might be useful. I'll talk about that with the team, that type of thing.

And so, when I say this, they might flag more questionable references.

Yeah, I can see it's questionable reading it, but, you know, it doesn't, you know, what we use

doesn't, doesn't flag unnecessarily. But, you know, when a reference is, is hallucinated,

you know, or there's some problem with AI, with AI has caused a fake reference.

You want to be very sure, or you reject someone's paper. I would never want even if we had reference

software that, that flagged everything and said, this is, you know, this, even something like this

journal doesn't exist or whatever, something very obvious. I would still want to follow up on that.

This is, this is people, you know, this is people's life's work that they're trying to publish with

us. And so, you know, like you mentioned, we always want a human to make, to make that final call.

So, before I, before I come, you know, come to you, Lee, so with, with the paper saying,

these appear to be, you know, fabricated references or some portion of the reference list, you know,

we want a human looking at that, not just feed it through software, it says, I was suspect, and that's it.

So, we would always double check no matter what.

So now we get to turn the tables a bit, and I get to ask Lisa a question. So, Lisa,

I'm with your insider's editor and chief for Java and AJVR. How do you editorial workflows influence

when and how citation issues are caught? I think it's been touched on, but it's great to always

reiterate this. You know, the first, the first stop app are the reviewers. You know, and if,

if I'm reviewing a manuscript, and I think, wow, I haven't read that article. I'm going to click on

the link or go look for it, and then you're like, that's weird. It doesn't exist. So most of the time,

those reviewers are right to us and say, I'm concerned that this is a fake reference hallucination.

But then they're caught by our copy editors. So, in our workflow, we have, as you know,

taken a heart stance, and whether you have a AI hallucinated reference during the review process,

or as you just heard in the exchange between Mornin and Vic, even if it's after acceptance,

and it's a manuscripts accepted, it goes to Vic and his team for copy editing. Even then,

we will rescind the accept and reject the manuscript. Because of all the things that we've talked

about, it's super important that it's important to the integrity of the entire article

that these references are correct. Most of the time, in other journals that aren't self-published,

like we are, don't take the personal approach that we do, you would get an author query on your

galley proof, and it would just say, please check this reference. So I think this is happening,

obviously, in every journal area, in the way that the editorial works well, and then what they,

what the journal ultimately does, once they discover an artificial,

intelligence-generated reference is highly variable, but we've taken a heart stance on this.

Yeah, thank you for sharing. I think it's important just to have that stated out there,

so people know. And so, when they go in, see if you've been drafting a manuscript,

they can have this in their mind, is there doing that literature search?

You think about how much distrust of science there is nowadays, and all the misinformation

that's out there. So, journals, reputable journals like Javman, AJVR, really have to be very,

very strict in upholding the standards of accuracy. I mean, it's sort of a question of accuracy.

And if you're publishing inaccurate information, which is what a hallucination is,

this is a terrible disservice to science, especially in this day and age, when there's so much

skepticism and so much misinformation out there, so many conspiracy theories and so on and so forth

about various, about so many things. And I'm sure true in veterinary medicine as well as in human

medicine. That's really well said, Mourna. And so, AI's here, and people using it, what does

responsible use of AI look like in scientific writing? I think the first and most important thing

is disclosure of how it's used, and that it has been used. I think that we have to have authors

really understand that it's not enough to say, well, I check the references if they actually used

AI to generate some or all of the references. I mean, I think it's, it's disclosure is the number one

tool. And then the description how exactly the AI was deployed. And I think the final step is

that the authors have to be asked to sign off on saying that they have fully vetted any content

for which AI was used in any way. So it's really up to the author. And I think that's why when the

author fails to follow these rules, why we have to take such a hard line. So going back to authors,

think what verification habits should authors adopt when using AI tools? Well, proofread people.

That would be my advice to authors submitting to jabma and anywhere else really. Our editors are

great, but you know, there's an old old saying and copy editing, comparing it to a car wash where

the car is only going to, the car is only going to come out so clean. So the dirtier the car is going

in, you know, it's not going to be as clean as you like coming out. And so if you submit stuff to us

that has been proofread better and that, you know, you've checked your AI outputs more closely,

then we're all going to publish a better paper together. And obviously avoid things like

fabrications and hallucinations and the various torture logic phrases and things like that, the AI

generative AI can produce. And so yeah, in particular, you know, more to mention, you know, check

every reference yourself. Yes, our copy editors will too, but you know, we also will reject the paper

if we say, as you mentioned, if we find a hallucinated reference. So don't just generate a list and

say that looks right and send it to us. And you don't just delegate maybe to someone else on the

author group too. I mean, I guess, you know, everybody's got to stake in it, obviously. So

Yeah, I think the DOIs are really useful as well here, because if you, I mean, I would really

encourage people to click on the links. And if there comes back with it on the DOI link, the digital

object identifier link from CrossRough, I think it would be a dead giveaway if you came back with a

DOI not found, which happens when you have a bad reference. Yeah. Yeah, that piece wouldn't even

take that long for authors to do. Yeah, if they just go through and click on the DOI, that would

be, that would be a nice stuff for them to take. All right, Zick, our next editorial is going to be

proofread people. I love it. It is kind of old school. And if you're not taught by somebody who's

old school. And I agree, it's everybody's responsibility as an author, not just the first or

corresponding. So this is evolving and it's not going to stop, right? We need to evolve and meet

people where they are with AI. So Vick, us in the publishing community, how should you respond

to this rise in AI assisted writing? Yeah, I would say, like it's easy to say just be vigilant.

And it's true who we need to be vigilant, but it might not be clear what that looks like,

right? So for authors, check your outputs closely as we've mentioned.

Make sure the data are accurate, the conclusions follow from the results if you've used AI

for helping the composition of the paper. Obviously, you want to be thorough in checking your

references as we've discussed for editors and production, you know, be thorough.

Really want to slow down. It sounds counter and counter and to counter and to it because AI is

for speeding things up. But really, that goes for author too. You want to slow down these days and

make sure that AI has given you good outputs. And so on the editor slash production side,

you want to take it more slowly. And when you're reading a manuscript, I guess this is very,

very copied or specific, but you know, it almost listen for notes of the uncanny, make sure the

logic tracks. And then yeah, just always be very scrupulous as you go through the references.

And so yeah, that would be my take away for everybody is slow down. And

uh, you know, look, uh, with AI, I would say it's a shortcut. Well, it's a shortcut that will

be best used by people who know how to do it the long way. And so, so, you know, you keep that in

mind, I guess, when when you're working with AI. I like that. More than how about you, what do you

think? I think that really it's about education. And I think, uh, how do we educate potential

authors or actual authors? Well, so much communication occurs at conferences and and scientific

meetings. And I think it would be be hoovous to have sessions on topics like, you know, the

responsibilities of authorship and what happens. I mean, maybe just pointing out the bad consequences

if the literature gets loaded up with hallucinations, what is why is that such a problem? And maybe

we have to reinforce the basic principles of scientific exploration and research and publications.

So going back to the the basics, I think reaching early career researchers where they are is

very important. And I think some of these, you know, nice things like boards of early career

researchers involved with journals in some way, um, maybe publishing some tutorials on

the the whole topic would be helpful. It's it's about really, I think,

teaching people about the seriousness of the issue because it doesn't seem very serious. I mean,

it seems kind of funny to talk about a machine hallucinating. That just doesn't seem like

something that we would take terribly seriously. So I think giving it they do amount of

serious consideration and building it into our process for educating authors would be a key thing.

I don't know. Of course, I mean, I'm talking about early career people. It probably is just as

prevalent with seasoned authors who can't be bothered to check their efforts. So maybe

not so much emphasis on early career, but are unequal emphasis on on later career, which we

don't talk about much. You don't hear all about LCRs. It's all ECRs. So I think just, you know,

finding ways to get that message across. Yeah, I think I like to add to you as publishers also

just updating our AI guidelines for authors. As we talked about AI is rapidly advancing. So making

sure that we're staying abreast of that too and updating our guidelines to stake in temporary,

what's what's currently happening with AI as well. It's a great one, sir. You might have to start

putting some of that in your committee already are in your communications with authors as their

paper progresses through the peer review process. You know, just last reminder, bad consequences

if you have hallucinations here. And, you know, that we can withdraw the acceptance if we find

this. So, you know, just really putting it very upfront in author communications, I believe.

For sure. That's why we're doing this too. So, Mornavik, thank you so much for joining us.

You really appreciate you sharing your insight on this really important topic.

Thanks for having me. Thanks for having me. I'm Sarah Wright here at Lisa 48.

Be sure to tune in next week for another episode of Veterinary Vertex. And don't forget to

leave us a rating and review on Apple Podcasts or wherever you listen.