Some deletions and modifications getting stuck in sync since v1.27.5

tomasz86 · April 3, 2024, 9:15pm

There’s a possibility that the issue unrelated to the newest version of Syncthing, however since I upgraded to v1.27.5 on multiple devices yesterday, I’m starting to see some deletions getting seemingly “stuck” in sync as below.

So far I’ve got rid of them by recreating the deleted files manually, then deleting them again. However, this is happening in multiple folders, and I’ve never seen such behaviour before v1.27.5.

I’m going to update the topic if I either manage to gather more information or find the actual culprit.

Edit:

I’ve gone through the commits between v1.27.4 and v1.27.5 and I’m thinking that maybe it’s related to lib/model: Prevent infinite index sending loop (fixes #9407) (#9458) · syncthing/syncthing@73cc555 · GitHub but I’m not sure, especially without testing. However, I’ve also now checked all my devices, and I’m seeing files that are seemingly stuck in sync in multiple folders, on multiple devices. They are also not all deletions, as some of them are just modifications.

For the record, it seems that they are only “stuck” in the GUI, as the actual changes have synced, i.e. the deleted files themselves have been deleted, etc.

tomasz86 · April 4, 2024, 6:55am

I reverted lib/model: Prevent infinite index sending loop (fixes #9407) (#9458) · syncthing/syncthing@73cc555 · GitHub in my personal build a few hours ago and installed it on all devices, and since then nothing got “stuck” in sync, so I’d say the linked commit is the culprit, but the question still remains why…

calmh · April 4, 2024, 7:10am

Since you’re in there hacking, could you add a print for the case where the two sequence numbers mismatch and see if that triggers for you?

tomasz86 · April 4, 2024, 9:51pm

I am into hacking but also, to be honest, I’ve got little idea what I’m doing when it comes to the Go code.

Should I add something like this?

	if s.prevSequence != f.Sequence {
		l.Warnln("current sequence:", f.Sequence, ", previous sequence:", s.prevSequence)
	}

I’m going to play with this in my test environment, as I can confirm 100% now that the problem is gone with the commit reverted, however it was a major pain to get rid of all the files that were “stuck” in my real Syncthing setup (as I had to track each of them separately, recreate, then delete again, etc.) and I really don’t want to do it again.

calmh · April 4, 2024, 10:08pm

if snap.Sequence(protocol.LocalDeviceID) != f.Sequence { ... badness }

note that some badness is expected as per the comment in the commit, e.g., you might see this trigger for reverted files in receive only folder or whatever.

tomasz86 · April 4, 2024, 10:36pm

Thanks, I’ve added it like this:

	if snap.Sequence(protocol.LocalDeviceID) != f.Sequence {
		l.Warnln(snap.Sequence(protocol.LocalDeviceID), f.Sequence)
	}

I’ve set two local Syncthing instances that sync the real folder where the problem happened but only between each other. If I can’t reproduce the issue this way, then I’m probably going to do it in my real configuration again, dealing with the consequences.

tomasz86 · April 5, 2024, 9:53am

As often is the case when trying to reproduce an issue, nothing has happened in my test environment which ran for about ~10 hours. I’ve now began testing the “debug” version on two of my real devices, which suffered from the original issue previously.

The whole code looks like this:

	// Use the sequence of the snapshot we iterated as a starting point for the
	// next run. Previously we used the sequence of the last file we sent,
	// however it's possible that a higher sequence exists, just doesn't need to
	// be sent (e.g. in a receive-only folder, when a local change was
	// reverted). No point trying to send nothing again.
	if snap.Sequence(protocol.LocalDeviceID) != f.Sequence {
		l.Warnln(snap.Sequence(protocol.LocalDeviceID), f.Sequence)
	}
	s.prevSequence = snap.Sequence(protocol.LocalDeviceID)

So far, I’ve only noticed that upon starting Syncthing, there are always a few of these recorded.

imsodin · April 5, 2024, 10:44am

Edit: I wrote this before noticing the latest post by @tomasz86 .

There are two bits of info here I can’t really reconcile:

If the actual changes propagate, and it’s “just” the web UI being out of sync, to me that would indicate an issue with the api or events, not actual index handling. If the index information doesn’t propagate properly, the syncing shouldn’t either. Unless a 3rd device with the same folder is involved, then then the information can propagate that way.
And reverting is a pretty strong indicator, however given the non-reproducible nature I don’t think it’s out of the question that the reset incurred by the rebuild and restart had an impact and it’s a red herring. Not saying it is, just an option.
Maybe you can be more specific as to what happened that indicated it was UI only?

imsodin · April 5, 2024, 10:46am

~~These are send-receive folders?~~

Never mind, these are expected: We call sendIndexTo once always when establishing a new connection. If there is nothing to send, what’s seen in those logs is expected: Snapshot shows the current sequence, while there is no file and thus seq zero there.

tomasz86 · April 5, 2024, 11:10am

I’m sorry if I wasn’t clear enough. It was only about the GUI, because the GUI had the files still “stuck” in sync, while in reality, the deletions did propagate and the files themselves were already gone from the disk. In other words, it seemed as if Syncthing was trying to sync the deletions twice. The same was true for the modifications.

The test environment, where nothing unusual happened, consisted of three devices connected as A → B → C, while the normal setup, where the issue was happening, is a much more complex one with 15+ devices, but not all are interconnected. I was seeing the same issue on multiple of them, in several folders.

tomasz86 · April 5, 2024, 11:56am

I think I’ve got something.

There appears to be one modification that the GUI thinks is stuck in sync, however the file itself has already been synced onto the disk.

What do you think? I’ve got no additional debug logging enabled right now. Should I enable anything specific that could be helpful for troubleshooting?

Edit:

More screenshots from multiple devices and more deletions/modifications getting “stuck” in sync.

calmh · April 5, 2024, 1:46pm

So, when taking a “snapshot” we don’t hold the update lock:

github.com

syncthing/syncthing/blob/bdfd0f0548d2f6fc4b5500690dbd383baa3b0561/lib/db/set.go#L168-L184


      
          func (s *FileSet) Snapshot() (*Snapshot, error) {
          	opStr := fmt.Sprintf("%s Snapshot()", s.folder)
          	l.Debugf(opStr)
          	t, err := s.db.newReadOnlyTransaction()
          	if err != nil {
          		s.db.handleFailure(err)
          		return nil, err
          	}
          	return &Snapshot{
          		folder: s.folder,
          		t:      t,
          		meta:   s.meta.Snapshot(),
          		fatalError: func(err error, opStr string) {
          			fatalError(err, opStr, s.db)
          		},
          	}, nil
          }

So there’s nothing in principle that prevents the sequence number from increasing between us creating the database snapshot and taking the metadata snapshot.

Now wether that matches the symptoms or not I don’t think (it would result in files not sent, so always out of sync on the peer from the sending point of view), but it’s not great.

calmh · April 5, 2024, 1:49pm

@tomasz86 see if this has any effect

 lib/db/set.go | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/lib/db/set.go b/lib/db/set.go
index 84cfb3452..fb4a9fda5 100644
--- a/lib/db/set.go
+++ b/lib/db/set.go
@@ -168,6 +168,10 @@ type Snapshot struct {
 func (s *FileSet) Snapshot() (*Snapshot, error) {
 	opStr := fmt.Sprintf("%s Snapshot()", s.folder)
 	l.Debugf(opStr)
+
+	s.updateMutex.Lock()
+	defer s.updateMutex.Unlock()
+
 	t, err := s.db.newReadOnlyTransaction()
 	if err != nil {
 		s.db.handleFailure(err)

(next we’ll see if we ever try to take a snapshot from within a write transaction :p)

imsodin · April 5, 2024, 3:21pm

Oh what a great find. It nicely fits that the reverted commit made this worse, as we before didn’t care about the snapshot sequence. Also the overall symptoms seem plausible:

DB write in progress.
Index sending at the same time: DB snapshot: index at A
DB write commits new stuff.
Index sending: Meta snapshot: seq at A+1 (I’d have expected A + index-batch-size ?).
Index sending: Iterate over DB until index A.
Index sending: Update last seq to index from meta snapshot: A+1.

File with seq A+1 is never sent to remote.

tomasz86 · April 5, 2024, 8:16pm

I’ve added the lock and I’ve also added f.Sequence > 0 to the other code to reduce the noise:

if f.Sequence > 0 && snap.Sequence(protocol.LocalDeviceID) != f.Sequence {
	l.Warnln(snap.Sequence(protocol.LocalDeviceID), f.Sequence)
}

It’s been running like that on all devices for a few hours now, and there have been zero warnings so far.

calmh · April 5, 2024, 9:44pm

I dropped an RC with the probable fix, for anyone who feel they be hit by this. Will roll it out on a couple of days unless someone screams.

nikow · April 7, 2024, 9:30pm

I think i am affected by the same bug, but in my case files are not synced for few days already (files itself has been sent, but not deleted after). I use syncthing to synchronize mainly WriteAheadFiles sender to receiver, but i have send&receive on both sides. From sender it looks like follows:

From receiver it looks more complicated. The folder sees different size and number of files:

And device view is stuck too:

And out of sync tab:

Lucky for me date of creation of file contains datetime in YYYYMMDDHHmmss format, so i know exacly when files has been created. First one should be 40MB and second one 36MB.

I recovered the log audit and log files from the backup. The main log says it was synced and changed:

syncthing.log.2024-04-03T18:36:25:[MKOIB] 2024/04/03 22:53:16 VERBOSE: Started syncing “utym6-2eckq” / “fn1_hmd_20240403225301.data” (update file) syncthing.log.2024-04-03T18:36:25:[MKOIB] 2024/04/03 22:55:01 VERBOSE: Finished syncing “utym6-2eckq” / “fn1_hmd_20240403225301.data” (update file): Success syncthing.log.2024-04-03T18:36:25:[MKOIB] 2024/04/03 22:55:01 VERBOSE: Remote change detected in folder “utym6-2eckq”: modified file fn1_hmd_20240403225301.data syncthing.log.2024-04-03T18:36:25:[MKOIB] 2024/04/03 22:56:53 VERBOSE: Local change detected in folder “utym6-2eckq”: deleted file fn1_hmd_20240403225301.data

The audit log says:

audit-20240403-183625.log:{“id”:7204,“globalID”:7204,“time”:“2024-04-03T22:53:16.714886061+02:00”,“type”:“ItemStarted”,“data”:{“action”:“update”,“folder”:“utym6-2eckq”,“item”:“fn1_hmd_20240403225301.data”,“type”:“file”}} audit-20240403-183625.log:{“id”:7206,“globalID”:7206,“time”:“2024-04-03T22:53:21.714575666+02:00”,“type”:“DownloadProgress”,“data”:{“utym6-2eckq”:{“fn1_hmd_20240403225301.data”:{“total”:314,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:1,“pulling”:123,“bytesDone”:130870,“bytesTotal”:41093096},“fn1_hmd_20240403225302.data”:{“total”:286,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:6,“pulling”:136,“bytesDone”:784738,“bytesTotal”:37405800}}}} audit-20240403-183625.log:{“id”:7209,“globalID”:7209,“time”:“2024-04-03T22:53:26.715074855+02:00”,“type”:“DownloadProgress”,“data”:{“utym6-2eckq”:{“fn1_hmd_20240403225301.data”:{“total”:314,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:4,“pulling”:125,“bytesDone”:523479,“bytesTotal”:41093096},“fn1_hmd_20240403225302.data”:{“total”:286,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:12,“pulling”:134,“bytesDone”:1569475,“bytesTotal”:37405800}}}} audit-20240403-183625.log:{“id”:7212,“globalID”:7212,“time”:“2024-04-03T22:53:31.716091223+02:00”,“type”:“DownloadProgress”,“data”:{“utym6-2eckq”:{“fn1_hmd_20240403225301.data”:{“total”:314,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:7,“pulling”:130,“bytesDone”:916089,“bytesTotal”:41093096},“fn1_hmd_20240403225302.data”:{“total”:286,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:24,“pulling”:129,“bytesDone”:3138949,“bytesTotal”:37405800}}}} audit-20240403-183625.log:{“id”:7226,“globalID”:7226,“time”:“2024-04-03T22:53:41.717661779+02:00”,“type”:“DownloadProgress”,“data”:{“utym6-2eckq”:{“fn1_hmd_20240403225301.data”:{“total”:314,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:36,“pulling”:129,“bytesDone”:4711311,“bytesTotal”:41093096},“fn1_hmd_20240403225302.data”:{“total”:286,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:48,“pulling”:130,“bytesDone”:6277897,“bytesTotal”:37405800}}}} audit-20240403-183625.log:{“id”:7229,“globalID”:7229,“time”:“2024-04-03T22:53:46.717860338+02:00”,“type”:“DownloadProgress”,“data”:{“utym6-2eckq”:{“fn1_hmd_20240403225301.data”:{“total”:314,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:54,“pulling”:126,“bytesDone”:7066966,“bytesTotal”:41093096},“fn1_hmd_20240403225302.data”:{“total”:286,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:61,“pulling”:133,“bytesDone”:7978161,“bytesTotal”:37405800}}}} audit-20240403-183625.log:{“id”:7234,“globalID”:7234,“time”:“2024-04-03T22:53:51.718637369+02:00”,“type”:“DownloadProgress”,“data”:{“utym6-2eckq”:{“fn1_hmd_20240403225301.data”:{“total”:314,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:68,“pulling”:125,“bytesDone”:8899142,“bytesTotal”:41093096},“fn1_hmd_20240403225302.data”:{“total”:286,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:73,“pulling”:134,“bytesDone”:9547635,“bytesTotal”:37405800}}}} audit-20240403-183625.log:{“id”:7237,“globalID”:7237,“time”:“2024-04-03T22:53:56.72185827+02:00”,“type”:“DownloadProgress”,“data”:{“utym6-2eckq”:{“fn1_hmd_20240403225301.data”:{“total”:314,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:78,“pulling”:126,“bytesDone”:10207840,“bytesTotal”:41093096},“fn1_hmd_20240403225302.data”:{“total”:286,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:86,“pulling”:133,“bytesDone”:11247898,“bytesTotal”:37405800}}}} audit-20240403-183625.log:{“id”:7240,“globalID”:7240,“time”:“2024-04-03T22:54:01.722073015+02:00”,“type”:“DownloadProgress”,“data”:{“utym6-2eckq”:{“fn1_hmd_20240403225301.data”:{“total”:314,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:89,“pulling”:130,“bytesDone”:11647407,“bytesTotal”:41093096},“fn1_hmd_20240403225302.data”:{“total”:286,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:103,“pulling”:129,“bytesDone”:13471320,“bytesTotal”:37405800}}}} audit-20240403-183625.log:{“id”:7243,“globalID”:7243,“time”:“2024-04-03T22:54:06.722428279+02:00”,“type”:“DownloadProgress”,“data”:{“utym6-2eckq”:{“fn1_hmd_20240403225301.data”:{“total”:314,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:97,“pulling”:130,“bytesDone”:12694365,“bytesTotal”:41093096},“fn1_hmd_20240403225302.data”:{“total”:286,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:112,“pulling”:129,“bytesDone”:14648426,“bytesTotal”:37405800}}}} audit-20240403-183625.log:{“id”:7246,“globalID”:7246,“time”:“2024-04-03T22:54:11.723127876+02:00”,“type”:“DownloadProgress”,“data”:{“utym6-2eckq”:{“fn1_hmd_20240403225301.data”:{“total”:314,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:98,“pulling”:131,“bytesDone”:12825234,“bytesTotal”:41093096},“fn1_hmd_20240403225302.data”:{“total”:286,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:114,“pulling”:128,“bytesDone”:14910005,“bytesTotal”:37405800}}}} audit-20240403-183625.log:{“id”:7249,“globalID”:7249,“time”:“2024-04-03T22:54:16.723350306+02:00”,“type”:“DownloadProgress”,“data”:{“utym6-2eckq”:{“fn1_hmd_20240403225301.data”:{“total”:314,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:117,“pulling”:136,“bytesDone”:15311759,“bytesTotal”:41093096},“fn1_hmd_20240403225302.data”:{“total”:286,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:135,“pulling”:123,“bytesDone”:17656584,“bytesTotal”:37405800}}}} audit-20240403-183625.log:{“id”:7262,“globalID”:7262,“time”:“2024-04-03T22:54:26.723801696+02:00”,“type”:“DownloadProgress”,“data”:{“utym6-2eckq”:{“fn1_hmd_20240403225301.data”:{“total”:314,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:141,“pulling”:134,“bytesDone”:18452633,“bytesTotal”:41093096},“fn1_hmd_20240403225302.data”:{“total”:286,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:156,“pulling”:125,“bytesDone”:20403164,“bytesTotal”:37405800}}}} audit-20240403-183625.log:{“id”:7267,“globalID”:7267,“time”:“2024-04-03T22:54:31.724550224+02:00”,“type”:“DownloadProgress”,“data”:{“utym6-2eckq”:{“fn1_hmd_20240403225301.data”:{“total”:314,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:149,“pulling”:135,“bytesDone”:19499591,“bytesTotal”:41093096},“fn1_hmd_20240403225302.data”:{“total”:286,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:163,“pulling”:123,“bytesDone”:21318691,“bytesTotal”:37405800}}}} audit-20240403-183625.log:{“id”:7280,“globalID”:7280,“time”:“2024-04-03T22:54:36.725334503+02:00”,“type”:“DownloadProgress”,“data”:{“utym6-2eckq”:{“fn1_hmd_20240403225301.data”:{“total”:314,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:164,“pulling”:149,“bytesDone”:21462637,“bytesTotal”:41093096},“fn1_hmd_20240403225302.data”:{“total”:286,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:177,“pulling”:109,“bytesDone”:23149744,“bytesTotal”:37405800}}}} audit-20240403-183625.log:{“id”:7283,“globalID”:7283,“time”:“2024-04-03T22:54:41.726431435+02:00”,“type”:“DownloadProgress”,“data”:{“utym6-2eckq”:{“fn1_hmd_20240403225301.data”:{“total”:314,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:178,“pulling”:136,“bytesDone”:23294813,“bytesTotal”:41093096},“fn1_hmd_20240403225302.data”:{“total”:286,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:193,“pulling”:93,“bytesDone”:25242376,“bytesTotal”:37405800}}}} audit-20240403-183625.log:{“id”:7286,“globalID”:7286,“time”:“2024-04-03T22:54:46.726644436+02:00”,“type”:“DownloadProgress”,“data”:{“utym6-2eckq”:{“fn1_hmd_20240403225301.data”:{“total”:314,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:193,“pulling”:121,“bytesDone”:25257859,“bytesTotal”:41093096},“fn1_hmd_20240403225302.data”:{“total”:286,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:208,“pulling”:78,“bytesDone”:27204219,“bytesTotal”:37405800}}}} audit-20240403-183625.log:{“id”:7289,“globalID”:7289,“time”:“2024-04-03T22:54:51.728088034+02:00”,“type”:“DownloadProgress”,“data”:{“utym6-2eckq”:{“fn1_hmd_20240403225301.data”:{“total”:314,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:218,“pulling”:96,“bytesDone”:28529602,“bytesTotal”:41093096},“fn1_hmd_20240403225302.data”:{“total”:286,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:231,“pulling”:55,“bytesDone”:30212377,“bytesTotal”:37405800}}}} audit-20240403-183625.log:{“id”:7295,“globalID”:7295,“time”:“2024-04-03T22:54:56.729047872+02:00”,“type”:“DownloadProgress”,“data”:{“utym6-2eckq”:{“fn1_hmd_20240403225301.data”:{“total”:314,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:248,“pulling”:66,“bytesDone”:32455694,“bytesTotal”:41093096},“fn1_hmd_20240403225302.data”:{“total”:286,“reused”:0,“copiedFromOrigin”:0,“copiedFromOriginShifted”:0,“copiedFromElsewhere”:0,“pulled”:254,“pulling”:32,“bytesDone”:33220536,“bytesTotal”:37405800}}}} audit-20240403-183625.log:{“id”:7301,“globalID”:7301,“time”:“2024-04-03T22:55:01.260517205+02:00”,“type”:“ItemFinished”,“data”:{“action”:“update”,“error”:null,“folder”:“utym6-2eckq”,“item”:“fn1_hmd_20240403225301.data”,“type”:“file”}} audit-20240403-183625.log:{“id”:7302,“globalID”:7302,“time”:“2024-04-03T22:55:01.267887766+02:00”,“type”:“LocalIndexUpdated”,“data”:{“filenames”:[“fn1_hmd_20240403225301.data”],“folder”:“utym6-2eckq”,“items”:1,“sequence”:2332,“version”:2332}} audit-20240403-183625.log:{“id”:7303,“globalID”:7303,“time”:“2024-04-03T22:55:01.267929542+02:00”,“type”:“RemoteChangeDetected”,“data”:{“action”:“modified”,“folder”:“utym6-2eckq”,“folderID”:“utym6-2eckq”,“label”:“offload”,“modifiedBy”:“FLM3W3O”,“path”:“fn1_hmd_20240403225301.data”,“type”:“file”}} audit-20240403-183625.log:{“id”:7407,“globalID”:7407,“time”:“2024-04-03T22:56:53.290183259+02:00”,“type”:“LocalIndexUpdated”,“data”:{“filenames”:[“fn1_hmd_20240403225301.data”,“fn1_hmd_20240403225302.data”],“folder”:“utym6-2eckq”,“items”:2,“sequence”:2336,“version”:2336}} audit-20240403-183625.log:{“id”:7408,“globalID”:7408,“time”:“2024-04-03T22:56:53.290217626+02:00”,“type”:“LocalChangeDetected”,“data”:{“action”:“deleted”,“folder”:“utym6-2eckq”,“folderID”:“utym6-2eckq”,“label”:“offload”,“modifiedBy”:“MKOIBGQ”,“path”:“fn1_hmd_20240403225301.data”,“type”:“file”}}

So basically files has been transfered correctly, then other automation processed said file and deleted it, but on sender side file exists and deletion of file has not been synchronized with “source” Freenet-Debian host.

Clicking on Rescan button or restarting both nodes does not help, so i suspect that’s not only GUI glitch.

chri-bra · April 8, 2024, 8:30pm

I also wanted to give some feedback on v1.27.6-rc.1.

My syncthing installation was also affected by the bug this week that the systems went out of sync. Unfortunately, before I found the relevant threads in the forum, I had rebuilt everything from scratch. And ran into the same errors/problems.

Now that I have installed v1.27.6-rc.1 everywhere, Syncthing is working fine again.

Thanks for the awesome tool. And the bugfix!

lightrush · April 16, 2024, 5:37am

I was trying to sync a large dataset on a new machine and I upgraded to 1.27.5 right before that. I was constantly getting strange out of sync issues. I thought I was losing my mind. Syncthing has been so solid for me over the years that I never considered it could be a bug. Time to donate!

Are existing databases alright having run on 1.27.5? Is there any action required after upgrading to 1.27.6? Trigger a manual rescan? Something else?

calmh · April 16, 2024, 6:20am

Should resolve automatically on upgrade.