Còpia remota entre amics amb TrueNAS, ZFS i Tailscale

Nobody wants to lose their stuff, right? Well, that's exactly what almost happened to me when not one, but two of the drives in the NAS right here that stores all of our current projects, as well as every past video I've ever made, they just decided to crap out on the same day, pretty much at the same time. Now, by a stroke of I don't know, pure distilled luck, they happened to be in separate RAID groups, which means the single drive parity that I use in here right now, it saved us because I don't actually have a backup for like 95% of what's on this NAS. And to add insult to injury, before those drives died, I was already in need of an expansion. We're like running out of space. So now, not only do I need to buy hard drives to add capacity, but I also have to buy hard drives to replace dead ones, which given the current global energy, global AI, global storage, well, there's a lot of crises going on, storage has gotten really expensive. Fortunately, somebody has come to our rescue. The folks over at Tailscale, the fantastic VPN/secure connectivity platform, who sponsored this project, as well as the Linux god himself, Wendell from Level One Techs. Thanks to those people, not only are we going to be able to solve, or at least band-aid my local storage woes, but we're also going to build something that I have been gravely missing, a real geo-redundant, I'm talking on the other side of the heckin' continent buddy backup. We'll effectively be each other's disaster plan, AND THAT'S KIND OF SICK. But before I can give Wendell any space to back his stuff up, we need to have space here just just to store files in general. But not in here

because we ain't got no space left. There are at least still some ways to get storage without having to bet the farm. What have I got myself into? I'm talking about used and recertified hard drives. But before you scream in horror, I promise you, there's a way that you can hedge this bet a little bit in your favor. You see, I got these 18 terabyte Seagate Exos SAS hard drives from server parts deals. I'll have a link down in the description. Thanks to those guys for sending these over. These might have 30,000 hours on them, but they come with a 2-year warranty. So, if there's a problem, they're going to replace them for you. And at a price of around 350 USD, I I can't believe I'm saying this, that's still in the neighborhood of like 30 to 60% cheaper than you can buy hard drives like this new right now. So, you know what? I'll take it. And if you're crafty, you can even find better deals, albeit usually without a warranty. I got these six drives from another seller on eBay. That's a good way to internationally ship sensitive hard drives. I'm sure they're all going to work perfectly fine. Mhm, 16 terabyte Exos, very nice. These ones are at least new though, so uh if they are damaged, it's even worse. But, whatever you do, you want to make sure that you actually test the drive so that if there's problems with them, sit your bum down, fluffy boy. Sit. You can actually take advantage of eBay's fantastic buyer protections and maybe get your money back. You can also look for new recertified drives from Seagate and WD. Those are drives that as far as I'm aware are usually decommissioned data center drives that have then been

tested, repaired if needed, reset, and relabeled for resale. And again, there's a warranty there. So, as long as you use decent parity protection, maybe a little better than I've been using, it can be a great way to save some money during these terrible terrible bit starved times. So, why don't we test mine? Perfect. I'm going to clean this thing anyways because uh Pool update. It's kind of worse. Do you think he's going to strike at us? Let him go. Holy. He's speedy. Very demure. How did this get so overgrown? Oh, no. Uh I'm just going to die from nature. What was I doing again? Oh yeah, right. >> Many deaths later. That's where this JBOD comes into play. It's just a place for you to store a bunch of discs. Just a bunch of discs. So, no computer inside here, but that's actually intentional because this box, it's meant to hold drives for another computer. Like maybe my NAS, huh? You just slap one of these HBAs into your computer. I'm using this LSI 9305-16E, good for 16 channels of SAS, into your system, and boom! You've got, well, 16 or 24 with an expander like this, 60, maybe even 90 more bays for all the SAS hard drives you shouldn't buy because they're so expensive right now. Now, you can usually snag the NetApp version of this JBOD, typically called a DS4246, on eBay for around 300 bucks. Holy WHOA! MAYBE THAT'S why this was 300 bucks. Fixed. Ow.

This one just happens to be like a branded one by not a very nice company, but it's what was available on Canadian eBay and it's essentially the exact same thing. Just try to make sure that if you do buy one, that it has all the trays. Mine has what looks like some, but they're actually just filler trays. They're not even ones that work because we like manufacturing e-waste. I mean, I get it. It's meant to block the front so the air flow is not terrible. BUT FEAR NOT, CUZ I 3D PRINTED SOME. It might have been a good idea to not print these in draft quality. You take the tray and you just slide it on. What? Why are they so long? Oh my god, they're all useless. We like manufacturing e-waste. Maybe for now I could just make this a later J problem. I hate that guy. Oh, man. Look how mint that is. Perfecto. Definitely going to work. No problem. Boom. >> [screaming] >> Mini SAS form MS02 ultra. My NAS doesn't actually have any more available PCIE slots. So, it's not really going to be taking part in this project for now. Get subscribed so you don't miss whatever ominous thing I'm talking about. >> [cough] >> Pedabyte. The nice thing about this thing is despite its size, it packs a serious punch. It's got space for multiple caching SSDs. It's got a 25 gig NIC in the back and heck, 24 CPU cores for calculating parity and doing other shenanigans that we're going to be doing. Like maybe some VPN traffic encryption what with Tailscale?

Can you tell I bought this on eBay? Cuz it didn't come with the bracket. Buyer beware, check for that if you need a low profile bracket. But for now, we're stuffing this in here. We're raw dogging her. It's even secured in there now. That's perfect. You can't even You can tell. Now all that's left is to connect it with these mini SAS HD external cables. If you ended up picking up a NetApp JBOD that's an older version, you might end up with QSFP connections on the back of it. In that case, you need to get a mini SAS to QSFP adapter cable to connect your HBA to your JBOD. It's also worth noting that since this one uses IOM 12 controllers which run at SAS 12 gigabit, if you want to use SATA drives, you would need to use interposers which I don't even think that this Oracle chassis supports, but the NetApp ones do. But if your JBOD has IOM 6 or IOM 3 controllers, you should be okay to just run SATA drives in it directly. I just um I'm not 100% sure on which port we're supposed to plug into on this one. I'm kind of thinking zero makes sense. The second power power doesn't work. Why isn't it working? See that? I got to go check the eBay pictures cuz I don't remember seeing a giant dent in the thing. Maybe I have to use that eBay buyer protection after all, hmm? One TrueNAS install later. Hey, there you go. We've got 6 16 TB and 6 18 TB. There's even two 4 TB Sabrent Rocket 4s and an 8 TB Sabrent Rocket 4 that I left in there from a previous project. So, that's cool. We can use those as cache SSDs. What should we name it? We'll call it the the fixer. Solving my problems. And then for the

configuration of our pool, since this is TrueNAS, we're going to be using ZFS. It's fantastic, incredibly resilient, super performant. It's basically, you know, the bee's knees, everything you could want. In my perfect world, I would maybe have a few more drives so that I could make the VDEVs that are essentially little RAID groups within our overall storage pool a bit bigger, maybe 10 wide for instance, but I don't. So, I'm going to go with the exact same setup that I have in my current NAS, which is six wide. And honestly, it kind of works out with the drives we have. We'll have a six wide of 16 TB, and then you'll see a six wide of 18 TB. And that looks like we're going to have another 150 TB of storage we can play around with. Sweet. A spare? We don't need that where we're going. I thought drives are just going to die. And then I'm going to add a cache SSD since we have it. Might as well set an 8 TB to hold a bunch of extra data that it spills out of the RAM cache that ZFS uses. Why not, right? Then we'll make a data set to test with. We'll call it test. Give ourselves permission to it. User Jake. 148 TiB. Woohoo! And let's see if this works. Beautiful, beautiful. Look at that. Look at that. We got test. Copy a file. 200 MB a second. Let's go! Now, I'm not going to be able to like do a perfect speed test of this right now because the built-in Linux SMB mounting thing is not very performant, but That honestly on its own is a great sign, and the fact that we don't have any errors in TrueNAS also a fantastic sign. And from here, I want to see how many hours are on these drives, and you can check that with their smart data. You're first going to want a list of

your drives by typing lsblk. You can see all of them here, sdb down to sdm. And then you just do smartctl -a /dev/sdb. Manufactured in week 22 of year 2023. That means those drives have just been sitting on a shelf somewhere for like 3 years, but they only have an hour and 40 minutes on it. So, I think that they were not bamboozling us. And then, there's the drives we got from ServerPartDeals. Those Whoa, these actually have less hours than I think they told me they did. Those are 15,000 hours. Oh, sick. Better than I thought. I mean, usually even in a business environment, you could probably say that they'll last at least 5 years before you might want to think about replacing them, but I have drives that have like 70,000 hours on them that are still just ripping. But these ones have lots of life left in them, and that's fantastic. But despite that we now have some storage, we don't have a backup. And while I could rely on cloud storage for that, maybe Backblaze, some AWS Glacier, the costs on that really start to add up over time. And I'm a freaking data hoarder, man. I need that data, and so do my data hoarder friends. So, what better way to satiate both of our desires for data than to back up our data to each other? That way, we have double data, a heckin buddy backup powered by Tailscale. It's really the star of the show here because it's going to allow us to connect our storage servers, which are on the other side of the continent from each other like they're in the same room. Fully encrypted, super fast, and it's easy. You just have to make an account on tailscale.com, which is free. No credit

card required. You just make the account. You can even use it for your small business if you've got six users or less. It's It's kind of insane, to be honest. I like If you're not familiar with Tailscale, by the way, it's basically tunneling a private internet connection from me to him in a way that doesn't require port forwarding on either of our sides, and it's encrypting everything. It's just like a VPN, but it's so much easier to set up. But for our setup, it's pretty basic. >> We're just connecting two computers at either end directly to each other, and that is like a default supported use case here. But if you wanted to, you could use them for things like maybe sharing devices on your local network. Like, I don't know, a a Jellyfin server. You want to share your media outside of your network in a secure manner? This is a way you can do it. You can even just expose your entire home network to say your phone, your laptop, whatever. For work, secure access to your work's network. Boom. Tailscale servers, everything like that. It can do it. >> Now, we just need to get a Wendel. Hi. Greetings. Despite the fact that me and Wendel, I would like to think are friends, and I would say I would trust Wendel, it's generally best to operate on a principle of least access, right? We can agree on this. Uh yes, absolutely. >> We just want to share storage space for backups without giving the other person access to our entire machine. And we can do that. ZFS even has this really cool zoning feature that allows you to grant a container access to a single data set, which would be kind of perfect. That's separation on an OS and a data level,

but that feature, it's a little bit primitive right now. >> [laughter] >> And to get that to work, you'd need to DIY some scripts. Exactly the sort of thing that Wendel would love to do, and and why you should check out his channel, because I guarantee you he's going to have a video talking about exactly that. But for the purposes of making this something that even a low-headache tier enthusiast would want to do, we're going to do it with a VM instead of a container, since that just works without fussing about I mean there's going to be a little bit of fussing about. >> It's a turducken of ZFS. We're going to have a ZFS volume and then inside the ZFS volume we're going to put another data set. Except that data set is managed by Jake and the volume is managed by me. It's going to work great. I guarantee it. >> [laughter] >> It's a sin against mankind, but it's okay. Look, the point is it keeps our top-secret host system separate from what our buddy gets access to. And to do so, we're going to install TrueNAS VM. So I will have TrueNAS running inside of TrueNAS and then what are you running? You're running Proxmox? Yeah, it's Proxmox. So you get a little TrueNAS VM for your volume. >> Honestly, TrueNAS inside of Proxmox makes a little bit more sense, but I didn't think this far ahead beforehand about the fact that it was going to be TrueNAS-ception. So we're just going to live here now. And once those are up and running, we can each create a ZFS volume on our host with an amount of space we want to share. How much space are we

sharing? Uh I think it's 50 or 75 terabytes. Something in that neighborhood. I don't know if I want to give you that many terabytes. >> Okay. I mean, uh space is really, really expensive these days. Did I tell you I just bought a couple petabytes? Yes, yeah. That's exciting. It's hard to find a good deal, but you found a good deal. >> Wendell made a connection. I also rated the same good deal. >> We bought some petabytes. Get subscribed so you don't miss that. Then we add our volume to the VM, create a single drive pool within the TrueNAS GUI. >> [laughter] >> Boom! 50 terabytes. That's a BIG HARD DRIVE. NOW THE NEXT THING IS I GOT TO SSH and install Tailscale. >> I was just going to install it as an app in TrueNAS. Oh, living dangerously, are we? It's going to be fine. You'll excuse me if I'm a little gun-shy about trusting apps in TrueNAS. >> Don't worry, they're going to change the whole architecture of the app system by the time this video comes out and then we'll have to remake it. >> [laughter] >> Anyways, to get Tailscale installed in our VM, I'm just going to search for it in the TrueNAS app store. If you're not using TrueNAS, you can just install it. They have a a of great information and documentation that you can use, but for this, it's really easy. You just go in here and the only option that we need to set in here is to add an off key so that actually logs into your account. And you get that in your account, go to settings, hit keys, generate an off key, we'll call it Wendel. It's not going to matter because it's a one-time use thing

anyways. Copy, paste, and that's it. There's a bunch of other features in here if you were doing some fancy stuff, maybe like using this to advertise some other devices on your network, which you can totally do. You just might be giving Wendel access to it as well then, which you probably wouldn't want in this instance, okay? You definitely also want to think about, you know, I'm adding that to my Tailnet. So, Jake would have access to my Tailnet through that machine. >> Incorrect. >> What? Because we're going to make use of Tailscale's sharing feature, which by default quarantines that device. So, you have access to whatever I give you on that device, maybe everything, definitely everything in this case, but it doesn't have access to any of your stuff. It's really cool. And then you just do the opposite thing to me. In that scenario, that VM for me is actually on your Tailnet. >> Yeah. And your VM on mine is the one that's on my Tailnet that I share out to you. Yeah. I didn't actually know you could do this before. I thought it was going to be way more annoying with access control and whatnot, but the sharing feature is actually like kind of perfect for this. Oh, shared out, look at that. Your machine has been shared out with one user outside your network. Cool. Oh, it's loading the GUI. Uptime 13 minutes, woo. It's an all good. The really mind-blowing thing about this too is that if you're on carrier-grade NAT, like Starlink or something like that, where you literally don't have a public IP address, they have their own universal plug-and-play like thing, and

Tailscale can actually punch a direct connection through that in that scenario. And then you get actually good speeds. They have one of the best write-ups on the internet for how NAT works and the algorithms for figuring out how a port should be forwarded. >> Can you send send me some of your your data, Hal? Uh through rsync. You know, kick the tires, see what we're looking at in terms of bandwidth, and you know, is it a direct connection? And everything's looking pretty good. The UPnP is uh saving [music] us. You're telling me there's data on here already, so I should look. Oh, yeah. Is this a Linux ISO? Uh actually, it's a TV show. The joke is that it's a pirated TV show, so it's like a Linux ISO. Oh, right. Yeah, I forgot that WAS A THING. >> [laughter] >> HE'S HE IS too You are too in I've got this on DVD. It's fine. The other fun thing that I can back up is literally every OS/2 program ever written. That sounds like very important data that we must maintain. And the reason I can see this stuff is because this is just random data Wendel is just rsync sending to the server, but we should try ZFS send and receive because then we can actually send data fully encrypted, and it'll just be gobbledygook to me. Now, since our JBOD server and the storage VM Wendel is sharing to me over Tailscale, both run TrueNAS, then we can use TrueNAS's snapshot feature, to basically like you just like take a picture of all your data at a at a given time, and then just just throw it at Wendel, bit by bit. And it does so incrementally. So, if we run backups like every night, for instance,

only the differences, the extra files actually have to get sent [music] and stored on the other side. So, it's really space-efficient and transfer time-efficient. If you're doing backups that are not incremental, you are like you suck, basically. I don't know what to tell you. Um actually, it's block level. It automatically knows what blocks have changed. I just got nerded. >> [laughter] >> It's great, though, because like if you get a big database that's, you know, a terabyte, and you only change 50 meg inside that 1 terabyte database file, it only has to send 50 meg cuz it knows what changed at a block level. >> It's even better than I said, basically, is what Wendel's saying. We just have to add Wendel's the VM's credentials within my TrueNAS installation on the JBOD server, the minisforum, and then you just create a replication task. Now, if you don't care about your buddy being able to see the data you're backing up, you can just go through the wizard and make a normal replication task for your normal ass data. And speaking of which, um normal ass Wendell, can you see the the normal ass files I gave you? Uh let's go look. Hello. So it worked. That's fantastic. And every night any data that I add in there is just going to get yucked right at Wendell's face exactly as we want. But what if you don't want Wendell to get smacked in the face with your data? And instead, you want him to get hit with incomprehensible encrypted bits that were originally composed of your data? Well, then you need to back up from an encrypted data set. And if you do so, it actually maintains the

encryption into the backup. So your buddy won't be able to see anything. You just need to enable full file system replication after creating the task and be backing up from an encrypted data set. And just to be sure that Wendell can't read any of the data I'm sending him, I'm leaving a secret message in here. And once you've seen it, make sure you comment on his video about this setup and tell him what that message said because he's not going to be able to read it. So here you go. It's just locked. I can't see it. >> Not all of the Linux wizardry in the entire world could break this encryption unless you're maybe Claude Mythos and then maybe you'll figure it out. Even if there were a side channel attack, it it's useless here because you're not decrypting the data on my system. And you don't even get the encryption key. Like there's actually just nothing there you can do with that data. And that's what makes this so cool. And if you're planning a buddy backup cluster with your friends, you want to access your NAS from anywhere in the world in a secure manner, or heck, you just want to check your security cameras and you don't want to port forward, then what better time than the present to check out Tailscale? Their personal plan will be always free for now up to six users and [clears throat] unlimited devices without having to supply a credit card. So check it out at tailscale.com/jack you to get signed up right now or or at the link in the description. And if you want to try Tailscale at work, which you really should, you can use code jacku for three free months on any of their paid plans. Thousands of companies like Nvidia, Lovable, and Duolingo have

already made the switch, and so can you. So, should you. And I hope you thought this video was pretty sweet. Thanks to Wendell for helping. Bye. Make sure you subscribe to Level One Techs, leave a comment on the video telling him what was actually in the encrypted file I backed up. That'll definitely make him feel better. He's got FOMO of the century. For my video, I'm not going to be like, "Look, I'm putting these good, you know, 12 to 22 terabyte hard drives in it." I'm going to just go through all the drawers in the basement and just be like, "This is what Jake's backing up to." And it's a pile of like WD greens and like 6 terabyte hard drives cuz it's like an average of 7 terabytes per drive times 60 bays is like 300 terabytes of storage. 300 terabytes of garbage. >> [laughter] >> You can now approach your tech hoarder friends and say, "In these uncertain economic times, look what we're going to do. And we're going to not pay for cloud services, and it's going to literally run on garbage." What if we got like, I don't know, Jeff Geerling or something? We could have a a threesome of garbage. Yeah, a garbage threesome. Oh, and then we take it to like another level. Forget ZFS replication. No, no, no. We have like a geo-redundant S3 cluster, and then we run, I don't know, we just run the Level One Techs forum off of it, you know? Should be fine, right? It's a distributed foruming. Yeah, subscribe to my channel. Check out the Patreon if you want TO SUPPORT US. BYE.