2.5 Admins 316: BGP Clipboard Hijack

0:00Thanks for choosing to listen to this late night Linux family podcast. We're only able to do this because of the people who support us. None of our showers are pay-walled, but our Patreon supporters can get most episodes early and never hear any ads. Learn more at latenightlinux.com slash support, or support us by telling a friend or colleague about everything we do. 2.5 Admin's episode 316. I'm Joe. I'm Jim. And I'm Alan. And here we are again. And before we get going, we've got a plug, but it's not one from Alan. It's one from all of us. And that is send us more free consulting emails essentially. It feels like lately the news has been getting a little grim and everybody knows we don't like AI and we're tired of oligarchs. So by all means, let's answer some questions you folks have. I think it's probably going to be more fun for all of us. Yeah, and not necessarily just free consulting questions. Maybe it's just a topic that you'd like our thoughts about, for example. Yeah, you know, we do this show for the listeners. So the listeners can tell us what they would like to hear about.

1:00So you know, I'd love to hear Jim rant about food or what does Joe think of bar? But nobody wants to hear your opinions on bad, Alan. I'm sorry. Exactly. Yeah. So anyway, show at 2.5admin.com. Send us an email. BGP hijack, infecting networks caused by a comedy of errors that's not funny at all. Yeah, this one involves a whole cast of characters. But the big one is a software company called Softaculous, which makes a virtualization management software called Virtualizer that is used by a bunch of places to manage their virtual environments. Basically had the IP range they host at Hetzner hijacked by a different provider and then malicious software updates for their software deployed to their customers. So any customer who tried to update their virtualizer software would basically get routed to the right IP address and the right domain name, but pointing to the wrong server because

2:00of the BGP hijack and then download malware in place of the update for the software they were trying to install. Of course, this is only possible because Softaculous doesn't sign their updates in any way and basically relied only on TLS to verify that the other side was actually their server. But because of the BGP hijack, the attackers were able to actually issue a legitimate Let's Encrypt Certificate for the domain because the domain was not configured to lock down the issue and sub certificates to certain Let's Encrypt accounts or keys. And so using the normal well known path thing that lets you issue Let's Encrypt Certificates, they were able to get a legitimate certificate for the domain name because they actually controlled the server where the domain name pointed right now because they hijacked the IP address. Of course, they were only able to hijack the IP address because of BGP security issues, although in this case, it's not the standard one. So Hetzner was actually deploying the routing PKI stuff to say, here's a sign list of places

3:03that are actually allowed to publish routes for this block of IP addresses. The issue turned out to be that Hetzner's rule did allow more specific routes within that block. So they have a slash 16 that they announced and they have it our PKI, but the attackers were allowed to broadcast a slash 24 that was more specific and hijack only that one part of the subnet. And because they listed Hetzner's AS number in the path, it was seen as legitimate, whereas if Hetzner had their rules more locked down, then it wouldn't have been allowed. But at the same time, if Hetzner publishes two locked down rules, if they ever want to make a more specific rule, for example, to deal with a denial of service attack and limit the impact of it, then they have to wait for new rules to roll out. And so it's a bit of a double edge sort of like, if you lock it down too tight, you can't respond easily in emergency, but if you don't lock it down tight enough, then the hijacking

4:04can still take place. The comments from Stautaculous are pretty telling our product update clients did not yet cryptographically verify update packages. So a modified package would not have been rejected on this basis, meaning we didn't bother trying, but maybe we will now. I don't think I would bet anything I couldn't afford to lose on that. I did a little bit of looking into this. And by the way, folks, if you're not quite sure when Alan first pronounced this as virtualized or that was correct, it's spelled to rhyme, if you will, with burn in a tour. I looked into virtualizer a little bit and it's basically a web front end for managing virtual machines. Stautaculous, I am assuming is an Indian design company because prices are given in dollars and rupees on the website. And there is no explanation of the symbol for rupee. You just have to know what that is. So that tells me this is probably an Indian dev team.

5:06And the final thing that I will mention is that screenshots of the product for all the world look like the open sense router interface. It looks like somebody's managing VMs with open sense. This BGP hijack was interesting in that it got around our PKI and also interesting in that they were able to get around normal TLS rules by actually being able to issue a lesson crypt certificate, which may suggest you might want to look into applying extra rules to your own domains using the CAA records to limit how TLS certificates can be issued and look at only allowing DNS based rather than just dot well known path based certificates for your domains, specifically because that way if you get hijacked, it'll be a little bit more difficult. Although if they can hijack your whole IP range, they might be able to get your DNS servers as well. I believe this is the most targeted BGP hijack I've ever seen. They're not usually used to spread malware at all.

6:08And I believe in the past, typically when we see BGP hijack spreading malware, it seems to be aimed at a broader attack surface for lack of better phrasing. Whereas this appears to have been a very clear and clean use of BGP hijacking to target a small subnet that was exactly what they were aiming for to do exactly what the attackers wanted to do, which was plant malicious updates to one particular product. You have to wonder what the target was because that's not the kind of attack that is made in order to see what you can get. Those attackers wanted something and it seems really likely they probably got it. Yeah, like they likely had one very specific site or something they were trying to get in the first place to use that as a watering hole attack or something. Yeah, this is very specific. Other amusing thing is it got fixed and then they got hijacked again and then got fixed, which also raises some interesting questions about how that worked. Windows bug incorrectly tells users that Microsoft Defender Antivirus is turned off.

7:11That's not a big deal, right? It wasn't turned off, no problem. On the surface, it doesn't sound all that bad, but Microsoft's advice to users is to ignore the giant warning that their protection is turned off, which if they do and then at some point in the future, it is legitimately off when it's not supposed to be. That's a whole issue. I would have to agree here. It's bad that Microsoft pushed an update that caused Defender to falsely report itself as disabled when it isn't. That's very much not good, but it's within the expected range of, well, company pushes bad software update, has to recall and fix it. News 11, advising people, don't worry about it. Just ignore errors while we figure out what's going on. No, no, no, no, no, no, no, no. That's very bad, juju. Aren't people just trained at this point to dismiss any warning like that? When you try and install something on Windows, you get this big pop-up, are you sure you

8:12want to do it? People are just so blind to it at this point. They just click yes, yes, yes. That popping up when you weren't trying to install something does tell you something, right? True. In this case, it's a notification that comes in like a new message in an app or something. It's like, hey, this isn't working, but it's one that you can't dismiss because it's critical. If this thing is not supposed to be turned off and it's not working, that's a problem. This is essentially the check engine light in a late 1990s Ford. It's on from the factory and you always ignore it. It's not a big deal when the check engine light usually is just telling you nonsense that doesn't matter. In this case, it's like, what if the check engine light was also the frag grenade detected light? It is a very, very, very big deal when your Windows Defender installation actually does suddenly refuse to turn on. That is one of the biggest primary symptoms of some kind of malware infection is because

9:15that's one of the first things they do. They use a vulnerability to get in and then the first thing they do immediately is try to disable any antivirus services because the thing is the zero day that got malware on your system isn't going to stay a zero day forever. If they just leave the antivirus process as running, then once a new library update comes down the line that does detect this malware infestation, then the antivirus service is going to start fighting it and that's not what they want. Yeah, you don't want whether or not Defender is enabled to turn into that check engine light. That is an indication that you really need to be paying attention to. I saw a very interesting post I'm mastered on recently from Simon Tatham. Simon wrote, I noticed today that an update to the Linux Zoom client has made it start proactively reading everything written to the X11 clipboard. Simon, of course, is the author of PUTTY. So I think we can take this seriously. And I do think we absolutely should take the matter of clipboard security seriously.

10:17If it's not already immediately obvious why that could be a problem, if you're using a password manager that likes to put passwords in the clipboard where you can easily paste them into other applications, well, that's something that Zoom is going to be getting a copy of all of that data now. Generally speaking, people very frequently put a lot of sensitive data in the clipboard and all that really means is copying and then pasting. It can be anything from passwords, to financial information, to even images that you wouldn't want to get leaked. All of these things can make their way through your clipboard. When you've got an application that's just rating it for anything and everything it can get, that is a major problem that should be taken seriously. What's interesting here partly is how Simon found it. Because of the way the X11 pace buffer works, he actually uses an app that does a one-shot pace buffer. So basically, when you paste, it sends the message and then if you try to paste again, it doesn't have anything else. Or in this case, it's actually the second time you hit paste, it sends the second message.

11:21And then the third time, the third thing, he was using this to fill out a form sequentially by just hitting paste a bunch of times in it, provided the right answer for each field separately. But he found that every time you tried to use it after upgrading Zoom, it was empty because Zoom had already taken the one-shot paste. One wonders if Linux is the only OS where Zoom is monitoring the clipboard like this. I don't actually know how the Windows or Mac OS clipboard work to know whether it's possible for them to be doing this on those operating systems. This is why I use Diceware passwords so that I can both remember and type in the secrets to my various applications. And it's also why I keep my vault on a separate device where I bring up the vault and I read it with my mechanical Mark 1 eyeballs and type it in with my mechanical Mark 1 fingers. Essentially, at that point, you're going to need a cyberjack if you want to hack that process. Or a key logger. Over to those microphones that can tell which button you're pressing.

12:22But yes, I definitely see that the advantages to that type of Diceware password instead of the randomly generated password manager passwords. I wonder if we'll see operating systems or things like Whalen actually introduce something being like, hey, I've noticed this app is trying to paste the pace buffer a thousand times. Maybe you want to ask that app what the hell they're doing. That actually does seem like a really good molly guard. Yeah, just kind of warn you that some app is obviously snooping on your pace buffer. And that seems like something relatively easy to implement and that shouldn't have a lot of false positives. And it might get certain applications to be more well behaved when there's backlash from it, which I think is the main reason we're covering this story is so that we will help Simon ramp up the backlash against this client doing something that is obviously very bad. I'm guessing that this is just a guess. This is just idle speculation. I would not personally bet that Zoom was deliberately doing something gross here.

13:25This to me has the earmarks of junior dev or God help us all vibe coding. Did something dumb that has potential problems? This is not necessarily an actual attempt by Zoom to get all of your precious, precious data's, but it's not okay either. Yeah, I assume this was somebody filed a bug report that copying pace doesn't work reliably on Linux. Well, fixed. Yeah, it sounds about right. Yeah, exactly. I mean, Hannan's Razerman, it's almost certainly in competence rather than manist. Why would they do something that is so obvious to detect? I don't know that it's that obvious to detect. Simon is running a really esoteric app here where it's the one shot pace buffer thing. And it's probably the only reason anybody detected this. The only other time I see this is if someone was like profiling a system and trying to figure out why something was slow and noticing that, hey, Zoom's calling the paste function like every 50 milliseconds or something. I've been running the native Zoom client every time that we do one of these episodes

14:28and I hadn't noticed it attacking the pace buffer. So yeah, I think that this is genuinely something that you would expect could go for a little while before getting detected, which just raises Alan's point again about, you know, maybe operating systems ought to be monitoring how frequently apps are accessing the pace buffer. You know, just kind of look for warning signs of this kind of attack. Yeah, even if this one wasn't nefarious and was just incompetence, it is absolutely something that could be nefarious and should be detected. You're right. Right. And we have absolutely seen this same technique used to varying nefarious purposes, especially in the mobile world, prior to NL have issues with clipboard hijacking on Android have cropped up pretty frequently over the years and with actual malice of forethought. So while I don't think that Zoom were deliberately being bad guys here, this is absolutely some bad guy stuff and it needs to get fixed. Obviously for late night Linux, we put out a call for questions that were kind of on topic

15:31ask us anything type questions. And Gabriel had one that I thought was more suited to this show. So the question is this, what is missing to have a non US and ideally open certificate authority that delivers free automated TLS certificates? Why don't we have that yet? So the ACME protocol that's used by Let's Encrypt is an open standard and there are actually multiple implementations of it now. So beyond Let's Encrypt, there's also zero SSL, which a number of the ACME client support is basically just changing the URL that you request the Let's Encrypt certificate from to this other one and you get kind of the same thing, but from a different provider. And I think having some diversity there is good. It seems Google has one that will also do like 45 day certificates. And I've also started to see some of the commercial TLS providers offering, you know, use the ACME protocol to renew your paid certificate. So instead of buying your certificate, doing the CSR once and then, you know, having to redo it manually every year, you would use your normal, like, Let's Encrypt ACME client

16:37and just give it your credentials or API key or whatever for the paid provider and have that. So I think this concept is definitely going there. I think as we're seeing the certificate browser cabal clamping down on this and getting to shorter and shorter certificates, we're going to see the kind of automation tooling and I think kind of settling on this ACME protocol, being the thing that's going to happen. And so other than getting added to the trusted list, I don't think there's anything stopping a non-US open certificate authority from doing that. It would just be the security audits and so on cost money. So we have to find a way to fund it. But beyond that, I don't think there's anything stopping us from having a third entry, like Let's Encrypt in zero SSL, that is open and non-US. Maybe the EU could fund it. Yeah, something like the sovereign tech agency out of Germany or just somewhere in the EU could fund a third entity to do EU style ACME protocol.

17:39Almost seems like something the NL net guys would do. Let's do some free consulting then. But first, just a quick thank you to everyone who supports us with PayPal and Patreon. We really do appreciate that. If you want to join us, people can go to 2.5abbins.com slash support. And remember that for various amounts on Patreon, you can get an advert free RSS feed of either just this show or all the shows in the late night Linux family. And if you want to send any questions for Jim and Alan or your feedback, you can email show at 2.5abbins.com. Another perk of being a Patreon is you get to skip the queue, which is what Ryan has done. I have a small true nurse box I'm using for replication once a week. I'm hoping to keep it at a friend's house, but wanting to save on power, I'm hoping this way to power it on and off when needed. I know I could just use Cron, but that may interrupt replication or scrub. Is there a way I can do it once it's finished and shut down, safe and cleanly? So on FreeBSD, which I assume is not what you have because you mentioned newer true nurse, but there is a tool called Lock-F that you can use basically as a wrapper around

18:41the command and it creates a lock file and maintains that the entire time that command that you wrapped is running and it doesn't clean it up until the command exits. And the original use case I had for this was just making sure the replication doesn't try to run twice at once. If you Cron have it to run once an hour, but if it takes more than an hour, it means make the second one not start while the first one's still running. And you can use that either to do that to prevent duplicate commands. But the other thing you could do is you can have a second command, for example, shut down that won't be able to run while the lock is held by some other command that in this case be the replication or it could be a scrub with ZFS. There's a command Zpool wait that based on the flags you give it will wait until a certain thing is done. So you can actually say Zpool wait until scrub is finished and the command will sit there and just sleep and wait until the scrub is finished. And so you could basically have a lock-F on that that would again make sure that and you could probably even just do this using that with the script.

19:43So you're trying to have a script to shut down the machine, but it waits for any running scrub to finish first. And then that way it wouldn't shut down while you were still scrubbing the pool. I think the Linux equivalent of that is Flock, SpeltFlock. Anyway, if you have that, you can basically take care of shutdown so that it doesn't happen while you're replicating and it doesn't happen while you're scrubbing. And then you can shut the machine off. That solves part of the problem, but then you still have waking it up. One thing that a lot of computers have in the BIOS is a setting for wake on RTC or real time clock where basically you can configure an alarm clock to make the computer turn itself on at 7 a.m. every morning. Using that, you can basically have the machine boot up every morning and then a cron tab runs at boot and says, you know, oh, it's not replication day shut back down. But when it is replication day, it'll stay up and replicate and only shut down once the replication is done. You could do that, but that's a lot of ways that could go wrong. Well, apart from anything, that's spinning up your disks every single day and then spinning them down again.

20:44Once a day isn't the same as all the time, but yeah. So instead, what I would probably recommend is getting a good smart plug or smart power strip. Obviously, a real power distribution unit for server grade stuff would be better, but those are very expensive compared to a reasonable smart power strip. And that would basically allow you to turn the power to the computer on and off. Ideally, you're not, you know, turning the power off unexpectedly on the machine. That's not good for the hard drives. But if you can turn the power on remotely, then you can configure the BIOS that, hey, on power loss when the power comes back immediately turn on instead of the default, which is stay in the state you were already in or stay off. So with that, even if the computer shut itself off nicely, if you turn off the power, wait five seconds, turn the power back on, the computer will boot up on that signal. So I have this with some of my machines that are at a remote location. And then there's a wire guard tunnel back to my house so that I can actually control those PDUs with the homocyc in at my house.

21:44So there's a button on the control panel on my phone where I can just cycle the power on the remote thing. In my case, it's even just the modems for the internet connection at the second location to make sure I can power cycle those when they're misbehaving. I would warn folks that you know, the smart plug approach occurred to me pretty much immediately as well, just only turn the machine on when you need it basically is what you're looking for there. That's probably going to be the right answer if your true concern is about conserving power as well, because the idle power on those drives is probably not going to add up to as much as the idle, you know, like desktop idle power consumption of the machine itself. The big thing that I have to warn you about it, if you're thinking about doing this thing with the smart plug and sitting your BIOS to always boot the machine, the first time it's been given power is you may only do this if you've got a UPS behind that smart plug. Because otherwise the next time you have an actual power out of junior area and the lineman do you their usual, you know, deal of turn the power on, oh wait, it's not quite

22:45right. You know, open the breaker again. Oh wait, we've got low voltage for the next, you know, 10, 15 seconds. Well, we figure out what's going on. Well, let's try again on and on and on that will make the magic smoke come out of your server. I will also warn you that the idle power consumption on each drive is most likely around a watt, maybe two watts. And the total power consumption of, you know, I don't remember you said how many hard drives you had, but even if you got say 12 of them, you're looking at like 12 to 20 watts. And it's, it's just not a lot of power that you're saving by, you know, jumping through all these hoops. And not really something I would recommend, honestly. Well, yeah, I know Alan, you said, you know, once a day is not too bad, but in my mind, hard drives do not like turning on and off again. Like ideally you spin them up and you keep them spinning until they die, if possible, if you want them to last as long as they are going to last.

23:48All the mind that has lasted a very long time have had less than 50 power cycles. That is true. But I don't think once a day is going to kill it, but definitely the ones where, you know, they park and I'm park all the time. And you know, you look at the smart stats and it's like, this drive has started and stopped 8,000 times or 12,000 times. Yeah, that definitely eventually adds up. I don't think I've ever seen more than a couple of thousand power cycle events on a healthy hard drive before, which strongly implies to me that if you're power cycling once a day, essentially, that strongly implies to me that you're only going to get, you know, six or seven years out of those drives, like best case scenario. The question is, is six or seven years of all you're really expecting to get out of those drives anyway in which case maybe once a day or so is not bad? I think it's kind of on the edge. I don't think it's the worst thing in the world, but I do think it's going to take a year or two off of the life of most people's hard drives as they use them.

24:49I think power cycling once a week or so is probably not going to add sufficient load to be concerned about once a day. If you really, really, really want to, but my position on this entire endeavor has been, if you really want to, we can't stop you, but I don't think you're getting enough out of this to make it worth it. Yeah, in this case, it's it may be just a little bit of smoking mirrors in order to keep the friend who's hosting the box for free, happy with the amount of power it's using. One of the advantages is if you get a slightly more expensive smart power strip, it will actually tell you how many watts the computer used each week for the amount of time that was powered on versus not and so on. That may help prove to your friend that it's fine to leave it on all the time because that number of watts or kilowatt hours times their eight cents or whatever electric rate means you buy the wikofi once a quarter and it's paid for. Now we're into the territory that I'm interested in and my suggestion would be, you know, if the concern is the friends concern about power consumption, put a kilowatt on it, leave

25:53it there for a month. Look at the total power consumed compared with the power costs. Show that to your friend. Hey, it's not going to add up to enough money to really be worth worrying about one way or the other, you know, reimbursing between the two of you and be hopefully it will really make the point that even if friends concern is more environmental than it is, you know, directly out of their own pocketbook that this is really not a thing that needs a whole lot of careful attention. Just for reference, I have got one of those anchor power station things we talked about, a Solix C300 and I've got plugged into that my onsite backup box, which is like just an old HP desktop with a couple of hard drives in it. And I've got all of my network gear, including all of my access points and everything and that is currently idling at 50 watts total for the whole lot, which in the grand scheme of things isn't that much. So one small box, if it's only got a couple of hard drives in it at a friend's house, it

26:56really isn't going to be using that much power in the grand scheme of things. It's roughly 1.2 kilowatt hours a day, which means in a month that 50 watt load is going to eat, you know, roughly 30 to 35 kilowatt hours, which at least where I live means about three bucks. Right, well, we better get out of here then. Remember show at 2.5abbins.com if you want to send any questions or feedback or any topic suggestions, you can find me at joarest.com slash master. You can find me at mercenarycessadmin.com. And I'm at allandjude.com. See you next week.