How to check how much has changed after a rescan (i.e. how much data needs to be transferred)?

[tomasz86]

I’d like to check how much data exactly needs to be transferred after a rescan has been performed. The GUI doesn’t help because even if a just a tiny bit has been modified in a very large file, it still displays the size of the whole file which supposedly is to be transferred to the other side.

For testing purposes, I’ve ignored everything else, leaving only 1 file to monitor.

This is what the logfile shows with db,model debugging enabled:

[DHK2G] 2023/07/19 09:27:04.061701 set.go:385: DEBUG: u2cgj-pf5ua IndexID(7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4)
[DHK2G] 2023/07/19 09:27:04.064704 set.go:408: DEBUG: u2cgj-pf5ua MtimeOption()
[DHK2G] 2023/07/19 09:27:04.064704 model.go:421: INFO: Ready to synchronize "folder" (u2cgj-pf5ua) (sendreceive)
[DHK2G] 2023/07/19 09:27:04.064704 model.go:545: INFO: Unpaused folder "folder" (u2cgj-pf5ua) (sendreceive)
[DHK2G] 2023/07/19 09:27:04.064704 folder.go:149: DEBUG: sendreceive/u2cgj-pf5ua@0xc000558800 starting
[DHK2G] 2023/07/19 09:27:04.065703 folder.go:200: DEBUG: sendreceive/u2cgj-pf5ua@0xc000558800 Scanning due to timer
[DHK2G] 2023/07/19 09:27:04.065703 folder.go:422: DEBUG: sendreceive/u2cgj-pf5ua@0xc000558800 scanning
[DHK2G] 2023/07/19 09:27:04.066703 folder.go:1018: DEBUG: Started filesystem watcher for folder "folder" (u2cgj-pf5ua)
[DHK2G] 2023/07/19 09:27:04.070703 set.go:170: DEBUG: u2cgj-pf5ua Snapshot()
[DHK2G] 2023/07/19 09:27:04.072229 set.go:170: DEBUG: u2cgj-pf5ua Snapshot()
[DHK2G] 2023/07/19 09:27:04.073704 set.go:268: DEBUG: u2cgj-pf5ua Get(folder1)
[DHK2G] 2023/07/19 09:27:04.073704 set.go:268: DEBUG: u2cgj-pf5ua Get(folder1\file1)
[DHK2G] 2023/07/19 09:27:04.090705 set.go:170: DEBUG: u2cgj-pf5ua Snapshot()
[DHK2G] 2023/07/19 09:27:04.090705 progressemitter.go:285: DEBUG: progress emitter: bytes completed for u2cgj-pf5ua: 0
[DHK2G] 2023/07/19 09:27:05.134888 set.go:170: DEBUG: u2cgj-pf5ua Snapshot()
[DHK2G] 2023/07/19 09:27:05.134888 progressemitter.go:285: DEBUG: progress emitter: bytes completed for u2cgj-pf5ua: 0
[DHK2G] 2023/07/19 09:27:09.079075 set.go:170: DEBUG: u2cgj-pf5ua Snapshot()
[DHK2G] 2023/07/19 09:27:09.079075 progressemitter.go:285: DEBUG: progress emitter: bytes completed for u2cgj-pf5ua: 0
[DHK2G] 2023/07/19 09:27:14.020436 set.go:281: DEBUG: u2cgj-pf5ua GetGlobal(folder1\file1)
[DHK2G] 2023/07/19 09:27:14.020436 set.go:373: DEBUG: u2cgj-pf5ua WithBlocksHash("35572cd485ac92a22ff616758b97d1cb070b99ce958044a02da654d5ddab5c52")
[DHK2G] 2023/07/19 09:27:14.020937 set.go:116: DEBUG: u2cgj-pf5ua Update(7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4, [1])
[DHK2G] 2023/07/19 09:27:14.021438 lowlevel.go:378: DEBUG: removing sequence; folder="u2cgj-pf5ua" sequence=27 folder1/file1
[DHK2G] 2023/07/19 09:27:14.021438 lowlevel.go:239: DEBUG: insert (local); folder="u2cgj-pf5ua" File{Name:"folder1/file1", Sequence:55, Permissions:0644, ModTime:2023-07-19 09:26:59.8379645 +0200 CEST, Version:{[{DHK2GDE 1689751624}]}, VersionHash:, Length:9609150464, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:false, BlockSize:8388608, NumBlocks:1146, BlocksHash:35572cd485ac92a22ff616758b97d1cb070b99ce958044a02da654d5ddab5c52, Platform:{<nil> <nil> <nil> <nil> <nil> <nil>}, InodeChangeTime:1970-01-01 01:00:00 +0100 CET}
[DHK2G] 2023/07/19 09:27:14.021936 transactions.go:635: DEBUG: update global; folder="u2cgj-pf5ua" device=7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4 file="folder1/file1" version={[{DHK2GDE 1689751624}]} invalid=false
[DHK2G] 2023/07/19 09:27:14.021936 transactions.go:649: DEBUG: new global for "folder1/file1" after update: {{Version:{[{DHK2GDE 1689751624}]}, Deleted:false, Devices:{7777777}, Invalid:{}}}
[DHK2G] 2023/07/19 09:27:14.021936 lowlevel.go:260: DEBUG: adding sequence; folder="u2cgj-pf5ua" sequence=55 folder1/file1
[DHK2G] 2023/07/19 09:27:14.025936 set.go:170: DEBUG: u2cgj-pf5ua Snapshot()
[DHK2G] 2023/07/19 09:27:14.025936 set.go:234: DEBUG: u2cgj-pf5ua WithPrefixedHaveTruncated(7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4, "")
[DHK2G] 2023/07/19 09:27:14.026437 folder.go:486: DEBUG: sendreceive/u2cgj-pf5ua@0xc000558800 finished scanning, detected 1 changes
[DHK2G] 2023/07/19 09:27:14.026437 folder.go:905: INFO: Completed initial scan of sendreceive folder "folder" (u2cgj-pf5ua)
[DHK2G] 2023/07/19 09:27:14.026437 folder.go:308: DEBUG: sendreceive/u2cgj-pf5ua@0xc000558800 next rescan in 1h9m46.201561338s
[DHK2G] 2023/07/19 09:27:14.026437 set.go:170: DEBUG: u2cgj-pf5ua Snapshot()
[DHK2G] 2023/07/19 09:27:14.026437 set.go:192: DEBUG: u2cgj-pf5ua WithNeed(7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4)
[DHK2G] 2023/07/19 09:27:14.026437 set.go:170: DEBUG: u2cgj-pf5ua Snapshot()
[DHK2G] 2023/07/19 09:27:14.026437 set.go:192: DEBUG: u2cgj-pf5ua WithNeed(7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4)
[DHK2G] 2023/07/19 09:27:15.056139 set.go:170: DEBUG: u2cgj-pf5ua Snapshot()
[DHK2G] 2023/07/19 09:27:15.056139 progressemitter.go:285: DEBUG: progress emitter: bytes completed for u2cgj-pf5ua: 0

Is the reported 1689751624 number the number of bytes to be transferred?

[calmh (Jakob Borg)]

There is no way to get this number from the sending side. Only the receiving side can figure it out, and only after looking for each block it needs in the database. The sending side doesn’t have enough information to know.

If you were asking about how much data is in blocks changed compared to the version we think the other side has, we could in principle compute that – but we don’t, and it’s anyway not guaranteed to be related to how much data will need to be transferred. They might have the missing blocks elsewhere that we don’t know about and be able to copy them without transfer. They may be missing the blocks we think they have when they finally get around to doing the sync. Etc.

@tomasz86

Is the reported 1689751624 number the number of bytes to be transferred?

That’s a timestamp-based version counter. Not related.

[tomasz86]

Oh, I was only testing locally without even sharing the folder. This is useless, right?

I’ve now connected another device, let the folder sync, made changes on one side, then synced them to the other.

The GUI did provide some useful information there.

This would mean that ~400 MB have actually been transferred.

Below is a db,model debug log from the receiving side. Is there an easy way to calculate the same number from there?

syncthing.log (231.8 KB)

[calmh (Jakob Borg)]

You can can calculate it after the fact based on debug logs, but as mentioned not beforehand.