Preparing to Sync is taking days. Creating folders but hasn't synced a single file.

Support
1

Like the title says, Syncthing has been on “Preparing to Sync” for almost 2 days now. The folder structure is being created, but no files have moved over. Out of 93,000 folders about 35,000 have been created in that time.

I don’t think it’s frozen but it’s definitely going really slow.

Is it normal that Synching would create the full folder structure first before syncing any files? And is it normal for it to take this long with that amount of data?

Total number if files is 782,608 totaling ~2.48TB

The setup is a little odd so I’ll explain the details below:

Client 1: (current source of all files)

  • Synology DS918+, 16GB Ram, 5Gb/s NIC
  • This end of Syncthing is fully indexed and ready to go.
  • I have verified all permissions are correct.
  • The sync target is a bind mount of the user home folders (There is a reason I did it this way).
  • Structure is /volume1/ST-Bind/homes/{username}
  • homes is the sync target so the synching files don’t get written into the users home folder.

Client 2: (Current Destination of all files)

  • Unraid, Ryzen 5 2600+, 64GB DDR4, 10Gb/s NIC.
  • Syncthing Docker.
  • Verified permissions in the container match the destination folder.
  • The structure is very similar. with /homes/{user}. Homes is a “real” Dir and is the sync target to that the syncthing files are written there instead of the user folder.
  • The User folders on this end are RClone mounts.
  • The Rclone mounts are connected to my Seafile server, as this is the only way to expose Seafile as a R/W directory outside of using its own apps.
  • The Seafile storage is located on a 24TB ZFS pool of 3 groups of 2 mirror devices each.
  • The Rclone vfs_cache is located on a mirror ZFS pool of 2 nvme 2TB SSDs and is a full read & write cache.
  • The Syncthing container, the homes dir and the Rclone mount all have the same permissions.
  • The RClone mount started as empty when setting up syncthing.

Im not seeing any errors in the logs, I haven’t noticed any permission errors, and I have seen no notifications within Syncthing. Unless I’m just not enabling the correct “Debugging Facilities”.

The Syncthing files were created properly in the /homes directories on both sides. And empty directories are slowing being created through the RClone Mount and are visible in Seafile.

My only concern is that after almost 2 days only ~35,000 directories have been created out of ~93,000 and zero files have been synced yet, however more and more directories are slowly but steadily still being created. Is this normal or is something wrong?

2

Yep.

For what you describe with the data being behind some rclone mount to some other service, I expect so. You’re being limited by how long it takes to do synchronous I/O to that destination.

What we usually say is, run Syncthing where the files are stored.

3

Ok, well it’s goo to know that it is doing what it should be doing then. I was starting to worry that it was some kind of permission issue where it could create directories but not files. Unix permissions are a pain to begin with, and having 4 different points where I needed to set them didn’t make it any easier.

What we usually say is, run Syncthing where the files are stored.

Honestly I wish I could, but in this case I cant. Seafile stores data where the files are just “blocks” and the metadata gets stored in a database. So there is no “filesystem” to sync to. Your expected to use their clients to access your data.

They have a FUSE plugin, but it’s currently read only, they also have a WebDav plugin that is read/write but you have to specify “library” you want to sync. Libraries are basically just the folders at the root of your home directory. So In this case I would have to create about 30 individual webdav sync tasks on my synology, across 2 different users. Plus I’d have to manually do that any time a new “library” gets created by any user.

But RClone Mount can speak Seafiles native language, and even automatically create new libraries as needed. So that is definitely the best way to go if I want access to my Seafile contents as if it were a filesystem.

Incase your wondering why I’m willing to deal with all that…Seafile is insanely fast! Even when accessing my files over the internet through my horrible home internet service it still blows me away how fast it is. Plus it is one of the only self-hosted platforms with iOS apps that actually integrate properly into the Files App and actually shows thumbnails. It shows them almost instantly too. Every other app I have tried either wont show thumbnails at all, or it will literally take 10-15min of waiting with my phone unlocked for all the thumbnails to slowly pop in.

Having said all that, I still don’t want the sole source of my data to be on a single server, much less in a open source but sort of “proprietary” format.

1 Like