partial sync with dynamic ignoring

Support
1

I’m using syncthing several ways, now I’m trying to do a partial one way sync. Here is the setup and requirements:

  • two servers: local and remote, remote do not have enough disk space to download all data
  • local server is primary, no data should be modified from remote
  • new data generated on daily basis
  • remote server should have last N days of data

So I set all this way: local server shares data dir in master mode with some ignores:

*.md5
**.md5
.*
!/data-days*
!/days.dat
!/envs.dat
*
**

The dirs to sync looks like:

...
data-days(20160311)
data-days(20160312)
data-days(20160313)
data-days(20160314)
data-days(20160315)

Global State 11665 items, ~3337 GiB

To perform a partial sync I made this ignores on remote server:

**.md5
*.md5
.*
!/days.dat
!/envs.dat
#include .ignore-daily
*

With daily ignore

...
!data-days(20160312)
!data-days(20160313)
!data-days(20160314)
!data-days(20160315)

So every night syncthing is stopped, new .ignore-daily is generated and oldest data-days dir removed. Then syncthing starts again and should download new data. But here comes the trouble – syncthing stops at ‘out of sync’ state. I think main issue is global state visible to remote server:

 Global State          11665 items, ~3337 GiB
 Local State	        3666 items, ~1426 GiB
 Out of Sync Items	7999 items, ~1911 GiB

Isn’t it supposed to limit ‘Global state’ according to ingnore patterns?

P.S. Couldn’t find a way to paste configs with correct newlines. Edited by calmh to fix that.

2

There are known issues around state reporting and ignored files… It should hopefully do the right thing though.

3

Should ‘Override changes’ on master work in my case with ignored files?

4

You should set up the ignores on the local server, so the ignored files will never be in global state and the remote server will never know, they exist.

And (I think) when you change the ignores on local server to include the new daily and exclude the old daily, the old daily will be deleted on the remote server, because it is in the index (was included before) but isn’t “seen on disk” anymore (excluded now). But @calmh needs to confirm that.

5

The file will not be deleted in the case it is missing from another device’s index. The other device has to explicitly announce the file as deleted which it won’t if it is ignored.

The global state will show what the folder would look like if all reported content existed in one place… So if you are ignoring 2 files that are not ignored on other devices, the global state will show 2 files more than local state when your status is in sync.

6

So server with ignored files should always have out of sync items?

7

If the files are created elsewhere, yes.

That’s why i proposed you use the ignores on the local server, so they never get into the index and the servers are reported as in sync.

8

Unfortunately yes. It honestly bugs me a little but I don’t have any time to look at it for now and my skill level isn’t there to fix it quickly.

I would follow @wweich 's advice and put the ignores on the master. Once the local machine restarts, I think, you will be able to use “overwrite changes” to delete the old remote data. This is completely untested and assumption on my part. I am assuming the global state not matching local state on a master will cause it to prompt for you to “overwrite changes”.

9

There are second remote server planned, I want to limit remote servers individually. Also restart with current config makes local server to scan files for about 40 min, I don’t want to generate this load every night.

10

Yea, with additional remote servers, you need the ignores on the remote servers.

The new ignore system, which is currently in planning, the sync status will probably be reported more “appropriate”.

As for the restarts. ignores are not config. changing the .stingore file and triggering a rescan via API (or wait till rescan interval) should suffice.

1 Like
11

Thanks for the clarification. The topic actually reminded me to ask the question, I was setting up a new device for my father a couple of weeks ago and kept getting prompted for restarts when changing ignores even though they were being applied immediately.

12

Is this kind of rescan faster than rescan on startup?

13

No, it’s always the same kind of rescan. Although if it’s just to update ignores, you can ask for a rescan of a limited part of the directory, even no part of it (I think).

14

Also, a restart will rescan all folders, not just that one. And the WebUI and API is only accessible after the initial scan.

15

I’ve got my remote server stuck in ‘Out of sync’ state with ‘Failed items’ not showing in web GUI (non-stop spinner), while local one says ‘Up to date’. There is no override button for local server, how do I force it to perform override? I think there should be some REST API method, but it is not documented.

16

I built SyncthingFUSE for this exact use case.

Getting the share to have N days of data may or may not be difficult, depending on how consistent the amount of data per day.

It also isn’t clear what the remote server is doing with the data. SyncthingFUSE won’t pull the data until it is requested. If you want it to be pre-pulled, you might need to write a script that reads the data each evening.

If using ignores still isn’t quite working for you, you may consider SyncthingFUSE.

1 Like
17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.