I have been looking around the forum for all these topics and I’m not sure what is the status today. So I know Syncthing is ok on Rasp pi .There was some issue with speed at one point but I haven’t seen recent issue on that. So I’m assuming all is good. Large size repo I mean 1.5 T with all aspect of repo structure (many little files ,many directories, large files). This last part worries me more. The latest talk were referencing 800 go on fast computer. Haven’t found topic since. Finally it will be between 3 remote sites. One “master” and 2 “slaves”(all R-pi). i use to use rsync/ssh. But I like the p2p approch for it’s flexibility.
I’m just looking for advices/config advices or share experiences. i’m just hoping to save time…maybe i’ll have to use something else.
Yeah, I think your RPI will melt indexing 1.5T of files.
I know someone managed to get something up and running by doing indexing on a powerful machine (with the same content, same mtimes, same everything), then copying the index directory on RPI so that it wouldn’t need to do indexing.
I was refering to this thread i think Large repositories -- general questions and some issues
But it was 500 GO and the RAM use was 850MB … But eitherway It sounds like I’m looking at the wrong tool for this job with my setup.
The biggest issue is getting it all indexed.
Once that is there, it should more or less work.
It might be the wrong tool, but you can always give it a try.
You are talking about the first indexation of course…let say It index it all once after few days. What are the event that could accidentally trigger a full re-indexation afterwards ?
Deletion of the index via -reset or via the index folder disappearing.
But 1.5T might take years to index on a RPI.
Also, mtimes or filesize changing will cause files to be reindexed, so if you have a lot of large files changing often, your RPI won’t be able to handle it.
If the files are static and your additions are small … then it should be fine.
Alright So i would have to index with a “desktop”. Then copy the index over like a guy in your exemple did. Simple enough.It’s mostly storage .So additions only. 100 files at the time. Ok , i might try , but only if i can’t find anything more suitable.
On a side note (sorry in advance for my un-educated question): Why the indexation is so resource intensive and seems to be a major bottle neck for low end hardware with syncthing indextation system, when a program like rsync doesn’t seems to suffer one bit from the under powered device ? i might be comparing apple and oranges, i have no clue.I do understand Synthing and rsync have 2 differents purpose and finality but both use indexation.Just simple curiousity.
Additions are not 100 files at a time, but rather 1GB at a time or something like that.
It is comparing apples and oranges.
rsync uses rolling non-cryptographic hash, where as syncthing uses cryptographic hashes (which are naturally more expensive).
rsync access model is based on ssh credentials and is uni-directional
syncthing is peer to peer.
If you use a non-cryptographic hash, an attacker can easily generate some random garbage content which matches the hash, and then when a request comes in for that hash, he can send the random content (which from the receivers perspective will be valid, as the hash matches) this way corrupting the file and causing DoS.
I understand since I had issues of the sort a while ago. i do not recall the exact application/circumstances. but i remember a similar issue where the hashes had to be encrypted for “it” to function and the pi could not go thru with it. humm it sounds like p2p is off the table in my case. Thank you for the help and the explainations.
One thing that might make things even slower but that’s probably important to avoid syncthing using too much memory: use a lower GOGC setting,
e.g. GOGC=20 ./syncthing