Maximum Repository Size

Support
1

Is there a maximum limit to the size of the repository?

The first repository I attempted was my pictures folder on my macbook pro to my Windows 7 server on a local LAN. 177,000 photos 500GB. I figured it might not work, but thought I would give it a shot. It said scanning for over a day and never sent a file.

This same pictures folder has 20 sub-folders, I setup a couple of them as repositories and they will transfer in minutes. So do I have to setup 20 repositories to get this synced?

2

It should be fine to be honest. Can you set STTRACE=scanner environment variable to see what’s happening? I assume you are on a modern PC, which means that most likely the bottleneck is the disk speed, but even then on a spinning disk you should get 500GB hashed in roughly two hours.

I guess you might want to set the rescan interval to something large and see if that helps?

3

Yes, node 1 = macbook pro with 7200 rpm hybrid drive so it is a fast drive. node 2 = Windows 7 PC with SSD main drive.

I looked briefly thru the forum, but couldn’t find how to set the environment variable on OSX or actually on any OS. Is there a document you can point to?

4

Hi there I think you can find the answer for the logging and the environment variable here: https://pulse-forum.ind.ie/t/debugging-syncthing/222/1

2 Likes
5

Thanks, worked. Was able to set STTRACE=scanner environment variable and I could see it was getting stuck on one file. It was a single 9GB 1080p .mov video that I had filmed for an event. When it got to that large file I got the dreaded spinning Mac beach ball and wouldn’t respond so wound up power cycling the mac (rare on a mac). I moved that large 9GB file out of the sync area and it completed scanning all 189,213 files and 491 GB. It did take just under 5 hours to finish scanning. It is now synchronizing, which looks like it will take awhile to do 500GB.

For those searching the Pulse forums on how to set the syncthing debug environment variable this is how it worked for me on Mac OSX (I have version 10.10 but it should be the same on any version). Thought I would share, and save some others some look-up time.

  1. click on spotlight, then type terminal, and click it, now inside of this terminal type the following:
  2. cd to the folder that the syncthing binary is located. For me the command was: cd /Users/Your username Here/Download/syncthing-macosx-amd64-v0.10.5/ but for you this will probably be different, it is where ever you downloaded syncthing
  3. STTRACE=scanner ./syncthing
  4. let it run till it is done. If you want to remove the Debug environment variable this is what I did:
  5. control-C stop syncthing (its not running now) – not sure if this is the proper way to stop syncthing or not, but it worked for me.
  6. ./syncthing run syncthing without any environment variables
2 Likes
6

Another question now that I have fixed the scanning issue. The next issue is that sync is going very slow, at a snails pace for this same large repository. The sync has been running for a couple of days and it has only synchronized 45GB out of 491GB, at this slow rate it will take another 20 days. I turned off compression and that didn’t help. I connected both computers wired to the router and that didn’t help. I know syncthing can transfer much faster because I was able to sync a 5GB repository in just a few minutes.

1) Which of the STTRACE environment variables will help me debug this slowness issue? 2) What else can I try?

7

Try increasing the scan time. I think you are retransferring the full index after each scan. (I am not sure about that, maybe the full index is sent only if you change a file inside) Since the index is quite large, this might be a limitation. If this might me the problem, see also: https://pulse-forum.ind.ie/t/protocol-retransfer-of-the-index-based-on-actual-change/971 Edit: Sorry for the shameless self-referrer

1 Like