Title is TLDR. More info about what I’m trying to do below.
My daily driver computer is Laptop with an SSD. No possibility to expand.
So for storage of lots n lots of files, I have an old, low resource Desktop with a bunch of HDDs plugged in (mostly via USB).
I can access Desktop files via SSH/SFTP on the LAN. But it can be quite slow.
And sometimes (not too often; this isn’t a main requirement) I take Laptop to use elsewhere. I do not plan to make Desktop available outside the network so I need to have a copy of required files on Laptop.
Therefor, sometimes I like to move the remote files from Desktop to Laptop to work on them. To make a sort of local cache. This could be individual files or directory trees.
But then I have a mess of duplication. Sometimes I forget to put the files back.
Seems like Laptop could be a lot more clever than I am and help with this. Like could it always fetch a remote file which is being edited and save it locally?
Is there any way to have Laptop fetch files, information about file trees, etc, located on Desktop when needed and smartly put them back after editing?
Or even keep some stuff around. Like lists of files, attributes, thumbnails etc. Even browsing the directory tree on Desktop can be slow sometimes.
I am not sure what this would be called.
Ideas and tools I am already comfortable with:
-
rsync is the most obvious foundation to work from but I am not sure exactly what would be the best configuration and how to manage it.
-
luckybackup is my favorite rsync GUI front end; it lets you save profiles, jobs etc which is sweet
-
freeFileSync is another GUI front end I’ve used but I am preferring lucky/rsync these days
-
I don’t think git is a viable solution here because there are already git directories included, there are many non-text files, and some of the directory trees are so large that they would cause git to choke looking at all the files.
-
syncthing might work. I’ve been having issues with it lately but I may have gotten these ironed out.
Something a little more transparent than the above would be cool but I am not sure if that exists?
Any help appreciated even just idea on what to web search for because I am stumped even on that.
You can mount remote with rclone and fine tune caching to your liking: https://rclone.org/commands/rclone_mount/#vfs-file-caching
I have a very similar setup to you, and I use SyncThing without issue for the important files (which I keep in my Documents directory to make it easy to remember).
Hey I’m replying again directly to your post in the hopes that I can push against some of the advice you’re getting. My intent is to do an end run around arguing with the people making these suggestions because they’re very smart and made them for good reasons but their ideas aren’t necessarily good for you and I don’t want you to have to go through a troublesome recovery like I did and many people on the internet have.
Do not under any circumstances set up raid or zpools for your data drives once you get them inside a case and on the pcie bus somehow.
In these configurations accessing a file requires spinning up all the drives in the array or pool. Not only is that putting wear and tear on your drives, it increases the temperature of the case and draws much more power. Those conditions lead to drive failure. When your drive fails and you have a spare to use in its place, resilvering (the process of using extra data called parity to rebuild the contents of the failed drive on the spare one) will put those exact conditions on your remaining drives.
For people like us, who may not have a hot spare, or great cooling, or an offsite backup, an array like that will set us up for failure rather than resilience.
Please consider using mergerfs or something like it and a snapshot parity system like snapraid instead.
There are very good use cases for the raid and zpool systems that have been brought up, but you aren’t there. I got there at moderate expense and moved away from them.
thanks I appreciate it. I’ve been around the block enough times to expect maximalist advice in places like this. people who are motivated to be hanging around in a forum just waiting for someone to ask a question about hard drives are coming from a certain perspective. Honestly, it’s not my perspective. But the information is helpful in totality even though I’m unlikely to end up doing what any one person suggests.
RAID is something I’ve seen mentioned over and over again. Every year or two I go reading about them more intentionally and never get the impression it’s for me. Too elaborate to solve problems I don’t have.
good luck. what’d you end up figuring out to access your files quicker?
I think others have mentioned TrueNAS as an OS for your Desktop. TrueNAS uses ZFS which is nice cuz it uses RAM to help with speeding up file operations. TrueNAS makes it easy to set up NFS shares if your laptop is running a Unix-like OS, or Samba, for anything else. IDK how much RAM is in your desktop, but if you can get that to 16 or 32 GB, you’d be set on that front.
For dealing with an OS you don’t want to change, I’d think about the following first:
What is the speed of your network card in the Desktop? If its 100mbit, you may want to look into upgrading that, if possible to at least a Gigabit card for PCIe. That would speed up anything you do with it.
As for needing local sync for when you’re away from your home network, Syncthing could maybe do what you want. TrueNAS can run Syncthing pretty easily as well, but it can be installed on anything, though, idk how this works if you set Syncthing to track a folder thats also one mounted on your local machine via NFS/Samba. Syncthing will just sync the most recent changes to a file to the server, so you can sync when you’re on your home network. Assuming that no one will modify the same files on your desktop when you’re away.
Thanks! I have gone to look at TrueNAS or FreeNAS a few times over the years. I am dissuaded because hardware-wise they seem expensive. Then on the other hand, they are limited in what they can do.
Comprehension check. Is the below accurate?
- TrueNAS is an OS, it would replace Debian.
- Main purpose of TrueNAS is to maintain the filesystem
- There are some packages available for TrueNAS, like someone mentioned Syncthing supports it
- But basically if I run TrueNAS, I will likely need a second computer to run services
Also for comprehension check:
- The reason many people are recommending NAS (or WebDAV, NFS, VPN etc) is because with better storage and network infrastructure I would no longer be interested in this caching idea.
- Better would be to have solid enough file sharing within the LAN that accessing files located on Desktop from Laptop would work.
- The above would be completely plausible
How’m I doing?
There’s one thing I forgot to consider in my original reply and I’m sorry for that. With TrueNAS you’d probably have to copy your data off of the existing drives to somewhere else because they will have to be reformatted to create a ZFS pool. I don’t know if that is practical for you, so please don’t feel like the following is something you must do or anything.
I think you’re doing great. Sorry for the late reply. To answer your questions:
- TrueNAS Scale is an OS that is built on top of Debian. Using TrueNAS makes set up simpler to set up, but you could implement what you want with a Debian install, but if you were to install TrueNAS, it would replace whatever existing OS you have installd.
- Yes, TrueNAS would manage your filesystem. It can manage your hard drives for you. Its UI isn’t too hard to understand, and it can be accessed and managed through a web browser on your Laptop.
- TrueNAS has some software packages in the form of docker containers, they are managed through the TrueNAS UI. You can browse them though their website here. My advice with these apps would be to set up your NAS with all of the drives in storage pools first before installing these. If theres something you want that isn’t supported, TrueNAS can also set up Virtual Machines, and you can use one of those to run those services, provided your CPU supports it and its turned on in the BIOS. If you go need to go down this route, you will have to set up a bridge network in TrueNAS in order to get the VM to communicate with TrueNAS over your network, but that’s not particularly hard or anything.
- You will need to run TrueNAS on its own computer, yes. What I was suggesting was installing TrueNAS as the OS on your Desktop. Idk if thats practical for you or not, since doing so would need you to wipe everything on the boot drive of your Desktop, so idk if you have a place to copy any important data off of it to.
In terms of comprehension, yeah I think you’ve got it. I think a NAS system would handle your caching idea for you, if I’m understanding you correctly. Having a good file sharing setup over LAN, whether its using NFS, or Samba would allow you to mount a folder from your Desktop on your Laptop and access them.
For files that you want to have access to when not on your home network, you could set up a folder that Syncthing tracks on your Desktop, sync that with your Laptop, and then have access to them that way.
thanks for all the details! I’ve fairly recently done an FS migration that entailed moving all data, reformatting, and moving it all back. Mega pain in the ass. I know more now than I did at the start of that project, so wouldn’t be as bad but not getting into something like that lightly.
Though it might be the excuse I need to buy another 12 tb hdd…
