2.5 Admins 317: Need Less

0:00Thanks for choosing to listen to this late night Linux family podcast. We're only able to do this because of the people who support us. None of our showers are pay-walled, but our Patreon supporters can get most episodes early and never hear any ads. Learn more at latenightlinux.com slash support or support us by telling a friend or colleague about everything we do. Two and a half admins episode 317. I'm Joe. I'm Jim. And I'm Alan. And he rare again. Before we get started, your customary, Clara article plug Alan is freeBSD's release cycle. What it means for your upgrade schedule. Yeah. So in this article, we break down how the freeBSD release cycle actually works. The different branches and what that means, how releases actually get shipped, how often they get shipped, and when they end. From the article there, there's a bunch of end dates coming up real soon. Like freeBSD15.0 goes end of life at the end of September, which is less than a month away when we're recording this and even closer when you're listening to it.

1:00So gotta keep an eye on that and keep up to date. And so the fact that there's a schedule and you can tell ahead of time, it can be a great help. Right. Well, Lincoln, the show notes as usual. It cannot have escaped anyone's attention that hardware prices have gone through the roof recently. Specifically RAM and storage are just ridiculous now. And so I thought it might be an idea to talk about some practical tips and advice for optimizing systems in the face of these ridiculous hardware costs. Rather than upgrading hardware, what can we do software-wise to make the most of what we have, I guess is what I'm asking you. Now we did do this on Linux AfterDark episode one, two, three. But that was kind of more with a focus on buying janky old Dell, Optiplexes and stuff. I'm thinking a bit more high-rend than that this time. Well, I think everybody will be absolutely shocked when I, and I assume Alan,

2:02you know, one of our first sets of advice is we'll just use the FS because then you can turn compression on and very likely save up to about 30% of your storage needs. Yeah, the compression built into ZFS works very well and kind of hides the fact that there's any compression from the application. But it also does double duty. Compressing the data will take less space on disk and that means you don't run out of space as fast. But ZFS will also use that compressed version in its cache, meaning that the same amount of RAM can cache more data if that data is compressed. And that can help you a lot there. By default, ZFS uses LZ4 compression, but you can opt to one of the newer heavier compressors like ZStandard. Basically, if LZ4 will give you 3X compression, then ZStandard will give you 4X. So that'll give you that little bit more. And that can help. And then starting with ZFS 2.3 and later, so especially if you're on Ubuntu 2604, you'll have this, you can use the ZFS rewrite command to basically troll through an existing data set

3:05and say rewrite all this data and compress it with the new compressor. So you can enable, you know, switch your compression from on or LZ4 to the default to ZStandard-3 or even ZStandard-5 or something like that. And it'll go through one file at a time and recompress them in place without you needing to, you know, copy all the data off and copy it back or duplicate it all with R-Sync or something like that. What kind of data is that the best at then? Basically, anything that's not already compressed, so it doesn't do well with images or video or music, but anything that's not already being like lossy compressed or is already compressed as part of the file format, but basically anything with a lot of text in it will definitely compress, but even binary files will compress like executables and so on. Anything that's not media should likely compress pretty well. A lot of executables these days are already compressed right from the factory, but text is really a very good answer here. And in particular, log files. I actually had to help a client just today who they had a malfunctioning application

4:05that was spamming incredible amounts of log entries over an error that this vibe-coded app was, you know, producing multiple times per second. And the 68 gigabytes of log files that had completely filled the hard drive on this particular VM, well, they, they, uh, they guns up to down to less than two gigabytes. So when we talk about, you know, this compression having a real impact not only on drive space, but also, you know, on your, uh, on your cache and RAM, well, if you were working with those log files and they're working their way through the operating systems kernel cache, then the difference between that 68 gigs uncompressed versus two gigs compressed, that's the kind of difference that we're talking about in terms of. Now this doesn't necessarily mean that you don't have to spend more money, but it definitely means that you get a lot more bang for the buck out of the RAM that you actually have on hand. Yeah. So for example, we have to keep a bunch of like web server log files for,

5:08in case we need to redo billing, slightly differently or something. And we compress those with ZSanter-9. And we get a 24.7 to one compression ratio with that. So 27 terabytes of logs compresses to 1.24 terabytes. And now that raises an additional point. We usually give the advice, look, turn compression on, set it to something that's not going to hurt you, and then forget about it. But if you're in this mode of like, you know, I need to get the absolute most I can out of the hardware that I have. And in particular, you're looking to conserve RAM, and storage via compression. It's worth noting that yes, we do have multiple compression algorithms. And it's not just one compression algorithm for the entire pool. You can set a really, really meaty compression algorithm on the data set that holds all of your log files. And maybe you don't want to use that compression algorithm. Be it, you know, GZ9 or, you know, one of the really hairy, you know, Z standard settings. Maybe you don't want that on the entire pool.

6:09But on the specific data set, you keep these gigantic log files in. It can be a much bigger win than you might think. Yeah. And with the ZSanter-Visory-Rite command, you can also do stuff like write a little script that says, find files I haven't modified or even accessed in the last six months and compress them really heavily. Because I know I'm not reading them all the time. I'm not writing them all the time. I'm not going to pay the tax. But if I want my data, it'll still be there. So you can basically make anything that's only an archive or just a backup or whatever, get compressed more while things that you're still using have the later compression. So you don't notice the impact of having to spend a bit more CPU to access them. Now, if we use ZSanter-Visory-Rite to recompress these less frequently accessed files, what impact is that going to have on replication, Alan? So in all versions of ZFS, when you use rewrite, it is going to grow your snapshots. If you have ZFS 2.4, when you run the rewrite command, you can add the capital P flag, and it will not modify the birth time of the data when it rewrites it, so it will not get replicated

7:16to the other side. We didn't actually change the data. We just recompressed it. So the birth time hasn't changed, and so it will avoid resending it to the other side. So if you're only on 2.3, you don't have that feature. But on 2.4, you can pass the capital P for physical flag, and it will rewrite the data on your machine, but not cause it to get replicated to the other side. That can save you a lot of replication if you're just both rewriting everything. Although sometimes you might actually want to replicate it to the other side, so that it stays compressed, and it takes less space on the backup as well. So you'll have to decide if you want to suffer the pains of re-replicating it in order to save the space on the backup as well. But you do have the option with ZFS 2.4 and later to do a physical rewrite, which will make it not actually appear as a dip when you're doing a send. So let's say I'm on 2.4, and I use ZFS 3.5 with the dash capital P flag, and I recompressed data on some not very frequently used files. But I don't change the birth dates as you've been talking about. And I go on with my incremental replication scheme to a backup target.

8:21If a few months from now, I wind up needing to do an incremental replication backwards from what's normally my replication target back to production to the source. Is this going to cause me any issues because I've done this kind of hack basically, and hidden the fact that they've changed? No, because the data didn't actually change. They changed on disk, but the content of the file is still the same. I think the file was locked during it to make sure that it's not actually modified. So at worst, if you roll back to an older version of the file, it will be uncompressed, or compressed with the older compressor or whatever. But in general, no, because the file didn't actually change, it'll be fine. Just locally your snapshots will be bigger. So if you rewrite everything and you have a bunch of snapshots of it, you're going to pay for that file twice until you clean up the old snapshots, which can impact your incremental replication if you delete all your snapshots, then you're having to reboot strap everything. So don't get too carried away with deleting your snapshots until your replication is cut up. Now this snapshots thing has hit me recently, right? So I've got image running on my NAS, and it's using about 100 gigabytes roughly

9:26for all of the media and the database and everything. Now on my NAS, because I prune the snapshots more frequently, because I've got it as my production template in your wonderful Sanoid application gym, it's using about 116 gigabytes. So about 16 gigabytes extra snapshots on my backup, which keeps a lot more of the older snapshots is using 365 gigabytes. And so I'm thinking maybe I need to be a little bit more zealous about pruning those older snapshots. Are you deleting that many photos? Like what is changing so much in your photo archive? I looked into it briefly and I think it's to do with thumbnails and stuff. I don't know. It seems odd to me that it's doing that, but I see it in the snapshots. An interesting tidbit maybe for this because specifically, yes, if you find your snapshots are bigger and you think they should be, you can run ZFS diff snapshot one snapshot two, and it will tell you what change between those

10:28snapshots. You know, we added this file, we deleted this file, we renamed that file, all the diffs that are in there. And maybe looking at that will tell you what files like changing every snapshot, because even if it's a database and less is rewriting the entire database from scratch every weekend or something, I wouldn't expect to see the level of churn you're talking about. Yes, so the database I've got on a child data set and that doesn't change much. So it is the main one. So why would your photos that you didn't change change? Two words, Alan, temp files. Well, I did the diff as you suggested. I did find that in documentation somewhere and it was thumbnails, basically, that were changing. Well, in one thing, can you configure it to put thumbnails in a child data set like you did the database so that those thumbnails maybe have a different pruning policy for the snapshots? Because I can always regenerate the thumbnails. Maybe I don't even want to back those up. Hmm, that's true. I should look into that. Yeah, there's a great tip for saving hardware space. Figure out the files you don't need to be keeping all the time. Yeah, exactly.

11:31For so long, my policy has been it is cheaper to just buy more hard drives than to figure out what it is safe to delete, but the math may have flipped on that one. Yeah, your time maybe isn't quite more valuable than hard drives anymore. Well, it's certainly not more valuable than enterprise grade solid state drives and promise you that. Rust, I haven't seen go up so much that I'm like, oh, no, this is just intolerable. But good Lord, you go to buy like a 3.8 terabyte DC 600M solid state drive and you're going to feel that pinch really quickly. We did quote for a company last summer and then they actually bought it this summer. And over the course of that, basically one year, the price of a 15.36 terabyte enterprise NVMe that does like a million IO so like a really fancy good drive went from like $2600 to $10,000 apiece. And so maybe all flash arrays don't make as much sense as they once did. Well, I think

12:35they can still make sense sometimes, but the number of use cases that require that are probably not as many anymore. You can, yeah, I think your point is a small amount of flash as a special VDIV instead of S or some other acceleration type can make a hard drive pool tolerable enough that you can avoid having to pay for all flash, which in fact was the next tip that we were scheduled to bring up anyway. The idea that you can use a little bit of flash to accelerate a rust pool. Normally, Alan and I are both, you know, more frequently giving the advice, you know, look, just build everything with SSDs in the first place. You don't have these problems. You strip away a lot of the complexity. It makes life a little easier. Save the special VDIVs and the L2 arc and you know, this and that and the other for, you know, these really big enterprise applications where you've got so much data, you have to have a massive rust pool and you need ways to accelerate dealing with that. But, you know, these big price hikes, they make us question that sort of general

13:35principle. There are definitely still a lot of applications that require all flash arrays and that hasn't changed and money isn't really going to change that. However, if you're running a home lab, you know, if you're managing a smaller business that doesn't have an immense amount of load, then maybe the calculus has changed and it has become worth saying, hey, do I get enough I apps out of, you know, maybe a pool of six or 12 drives in mirrors with a special or an L2 arc? Yeah, like actually the quote I was mentioning earlier with that because of the price change at the NVMe drives, they switch their design from a big hard drive pool with a couple of metadata drives and a separate dedicated NVMe pool for their VMs to just putting all of the NVMe as special devices and setting their VM data sets to have special small blocks so they would store on the VM. But if they run out of space on the NVMe, it can still fall back to the hard drive and they don't

14:38just like, sorry, your VM has to pause because we're out of space and kind of making different trade-offs there of like, we'll use up to 75% of the NVMe for VM data and only the last 25% for metadata because we can't afford enough SSDs to do a dedicated SSD only pool. Now realistically, this is not a magic bullet. There's a reason why Alan and I haven't just always been advising, hey, save money, you know, use Rust and just a little bit of flash and it'll all be great. This is not ever going to be as fast as an all SSD pool, little on all NVMe. Even when you're using special small blocks to divert blocks below a certain size onto the NVMe or you know, other flash torches you have and not directly to the rust, even in terms of throughput, little on latency, it's not as fast as just having an all flash pool in the first place. It's also going to be less predictable. It's not going to be as easy to figure out when you're going to have a latency problem because with all this additional complexity, you're going to see a lot more interaction in between

15:42workloads popping up at unexpected times. But that doesn't necessarily mean it's completely unlivable. So basically, I think the calculus really is, is there enough money on the line here to make it worth paying the extra money for the greater speed and the greater predictability and better consistency or is there not enough money to make it worth that? That was always what the calculus was. It's just maybe the value some of the variables have changed. Yeah, but you could also get the the Dassy surprise, you know, we helped a different unrelated customer set up special VDEVs and the performance increase they got from it, they loved it. They were like, oh, this is awesome. Can we do special small blocks too? And we're like, yeah, and they did that. And they were like, oh my god, this is amazing. We love it. Until two days later is like, our special VDEV is full and now even the metadata is going back to the hard drive. It's like, yeah, this side effect of going to special small blocks and going overboard with it is, you know, even those ZFS reserves to last 25% of the space only for metadata so that special doesn't like the small blocks don't

16:44completely take over the drive. Their pool was so big that basically they had undersized the amount of special they had. And then when they made it worse by adding special small blocks suddenly they're out of space. And yeah, you get that performance cliff where the metadata was all great because it was on the SSD and now the SSD is full. So the metadata is back to being on the hard drive as if you didn't have a special VDEV. And you know, if you read this file as fast but if you read that file, it's slow. And then, you know, I should delete files. Some new file will randomly be on the SSD but the rest won't and yeah, it can be very unpredictable. But you know, it's opportunistic, at least some of it is faster is better than everything being slow. So what about any rate then? This is a feature that your company built right that allows you to mix hard drive sizes in a pool. Is that going to be of any help when that finally arrives? Yeah, so they're hoping that'll be in ZFS 2.5. It's such a much about mixing drive size in a pool but in a VDEV. So if you're making, you know, a RAID Z2 out of six drives and they're not all the same size,

17:46normally ZFS would use the smallest common denominator size of each drive and the rest of that space would be wasted. With any RAID, you get the majority of that extra space as usable. And it means, for example, if you have a pool where you built it originally out of all four terabyte drives, and then over time you started replacing those drives with bigger ones, you know, whatever is the most cost efficient when you were buying it or whatever, you don't get any of that space until the smallest drive size ratchages up each time. With any RAID, you would get some more space every time you replaced one drive and that could make a big difference, especially if even in my case, you know, it didn't make sense to buy 12 terabyte drives anymore. So when some of those drives started dying after their five plus years in service, we bought 16 terabyte drives as replacement. But until we place every drive in the VDEV, we don't get any of that extra space whereas with any RAID, we would get some of that extra space every time we upgraded even just one drive. But, yeah, that one's probably still a couple of months from actually being in upstream ZFS, it's still out

18:47for review. Yeah, and by the time I get that on Ubuntu is going to be 28 or 4. Well, I guess if you're really to run a non-LTSU button, it'll be in earlier versions and you can install newer ZFS whatever Ubuntu you have. But it's then you're missing out on most of the benefit of the fact that Ubuntu ships ZFS in their repo that's actually aligned with their kernel and you're messing with all the DKMS stuff by yourself. Yeah, which I absolutely do not want to do. I think another thing to look at from this is maybe buying any hardware is not out of the question. So then it becomes what hardware to buy. And as much as the prices maybe make you think otherwise, if performance is the goal, then even a little bit of RAM is often worth it. You have to look at your ARC hit ratio and some of the other things to tell which thing is actually the bottleneck on your system. But sometimes a couple hundred dollars for an extra stake of RAM or two can make a big enough

19:47difference to be worth it where switching a bunch of the drives from hard drives to SSDs or something in the same price range might have made some of it faster. But if you can keep the whole database you're worried about in RAM is always going to be faster than whatever storage you have. But at the same time, you know, if the problem is I need more storage, it's either buy more storage or get a bigger CPU so you can use more compression. But again, that depends on do you have data that's actually compressible? Yeah, or in my case, if I need additional storage, maybe just to let some of my old crap. So at the end of the day, we are down to the sage advice of RAM and storage are expensive. So try to need less of those guys. Let's do some free consulting then. For first, just a quick thank you to everyone who supports us with PayPal and Patreon. We really do appreciate that. If you want to join those people, you can go to 2.5abbins.com slash support. And remember that for various amounts on Patreon, you can get an advert free RSS feed of either just this show or all the shows in the late night Linux family. And if you want to send any questions

20:50for Jim Allen or your feedback, you can email show at 2.5abbins.com. Huan asks, I work in a small company in Australia with a hundred employees. Its internal network is a flat slash 24 from 20 years ago when the company was much smaller. Our IT provider is recommending creating a VLAN for each department with its own slash 24. The argument is to improve security and traffic efficiency. Reduce ARP traffic. But since at the end of the day, all these business units have to communicate with one another. Wouldn't it be easier to just do a slash 16? I wouldn't recommend an entire slash 16 for an organization of the size that you're describing. With a hundred employees, yeah, a slash 24, which means you've only got 255 hosts. That's going to get cramped pretty quickly because a lot of people are going to be bringing in one or more personal portable mobile devices that they want to attach and each one of those is going to want its own IP address. You're going to run out pretty quickly. But a slash 16 is truly enormous. And it's tempting to say, well, why do I care? It's playing your room to grow into and it's private.

21:55It's non-routable. Why shouldn't I? And what you find out pretty quickly is when you allocate an entire slash 16 to a hundred employee network, what you've done is made it very, very difficult to bridge that network to other private networks because there are only so many non-routable slash 16s to go around. I think that's something along the lines of like a 22, a 21, or a 20, you know, essentially somewhere between four and 10 slash 24s all in one big flat land is probably going to be the better answer here. Security-wise, I don't think the VLANs are going to do anything for you. You've already said that all of these departments have to communicate with one another. If they're all sharing like the same server infrastructure and and you know, so on down the line, then you're not really going to get any benefit to speak of security-wise out of multiple VLANs. The one big argument can be made for VLANs here is, you know, the idea of reducing ARP traffic. ARP is address resolution protocol. This also has some impact on decreasing,

22:58general network chatter from various things like DHCP and, you know, what have you. But, again, there's only so much it can do because we're talking about VLANs here. We're not talking about an actual physically segmented network with separate switches for separate departments. So all the traffic is going over the same hardware, whether you're doing it via VLANs or not. The idea of breaking it up with VLANs to reduce the network chatter like ARP traffic basically just means that, you know, your HR department is an arping machines in engineering and vice versa. But with only, you know, a hundred people, a few hundred devices, that's just not going to add up to enough to matter. Yeah, like unless the departments are in basically separate buildings, it probably doesn't make sense to actually isolate them that way. Now, when you start talking about Wi-Fi, it gets a little more interesting. Every bit of airtime, you don't waste on all that ARP traffic can actually make a difference. But unless, you know, the HR department is going to have a different Wi-Fi than the other departments, which it doesn't sound like they would,

24:01it's not actually going to help you in the end anyway. And so, yeah, just one bigger subnet, like Jim said, a slash 21 or something likely is the easier answer and doesn't involve all kinds of weirdness of just this machine can't reach this machine anymore. Why not? Or this protocol that depends on broadcasts suddenly can't see the machines in the other department and that causes some headache that we didn't anticipate. Now that we've said why we don't think WAN should use the lens for security. Let's talk about why the lens are actually a security feature in some instances with a really large organization that has essentially close to independent departments, each of which has its own server infrastructure, its own applications that all are cleanly separated and nobody in the engineering should be accessing somebody in accounting's application and nobody in HR should be accessing somebody in engineering's application. And these things live in actual separate servers or at the very least different VMs with different IP addresses. Now you're looking

25:03to the situation where using VLANs to segment this traffic can actually be worthwhile because basically, what you're doing is your firewalling each department off from incidental traffic from the other departments. So for example, if somebody in HR clicks the shiny link and you know, gets some kind of a worm that then wants to reach out and try to touch everything in the business organization, that worm is not going to be able to get to the servers that engineering needs to get its job done. But when everybody's using the same servers and the same IP addresses and the same sets of resources, then there's just not really any security to be gained. Yeah, basically by having this separate VLANs you force cross department traffic to go through a router, which means you can apply some policies of what's allowed. But the downside is if your policies just going to be all of them, we can get to all of them, then you're just making that router now a bottleneck for the throughput. Whereas if it was just letting the switch do it, it would be that much better. And so it makes sense if you need to enforce policy between departments. But if

26:04you don't, then it's a lot of management for no reason. Right, well, we better get out of here then. Remember show at 2.5abince.com if you want to send any questions or feedback or any topic suggestions. You can find me at jorest.com slash master. Don you can find me at mercenarycessadmin.com. And I'm at allandjude.com. We'll see you next week.