Raspberry Pi sync-conflict with rapid changes

Support
1

I have a Raspberry Pi running 1.23.6, and a Windows computer also on 1.23.6 When rapidly editing files on the windows machine, there are frequent sync conflicts with the Pi. They all have the windows computer’s ID in them, as if the Pi is altering the file, then the change comes in. At times, I get stacks of conflicts, especially when quickly changing and saving again.

On the windows machine, the log looks like this: (From top to bottom: Conflict arrives, Second Modification, First Modification)

Pi       	modified 	file 	Documents 	FileName.sync-conflict-20230718-083545-5EQESM2 	2023-07-18 08:35:43
Windows 	modified 	file 	Documents 	FileName 	2023-07-18 08:35:43
Windows 	modified 	file 	Documents 	FileName 	2023-07-18 08:35:23

On the Pi, the log looks like this, with an extra file modification in there:

Pi       	modified 	file 	Documents 	FileName.sync-conflict-20230718-083545-5EQESM2 	2023-07-18 08:35:43
Windows 	modified 	file 	Documents 	FileName 	2023-07-18 08:35:43
Pi          modified 	file 	Documents 	FileName 	2023-07-18 08:35:43
Windows 	modified 	file 	Documents 	FileName 	2023-07-18 08:35:23

I tried the thread about the Raspberry Pi RTC clock, but it didn’t help.

2

After the RPi has booted, timekeeping is generally not an issue.

Also, even if the RPi’s system clock is off, there won’t be file conflicts caused by the last modified time because it’s mirrored by Syncthing from the sender rather than being set based on the local system time.

The problem is that the last modified time resolution is 2 sec for FAT vs. 100 ns for NTFS vs. 1 ns for ext4. Then Windows also imposes its own rules such as delaying updating the access time by up to 1 hour. (Raspberry Pi OS / Raspbian / RasPiOS uses ext4 for its root partition.)

So depending on the combination of the hardware on the Windows PC, the version of Windows, plus the filesystem format, the modTimeWindowS setting might need to be adjusted.

2 (seconds) is usually sufficient, but I’ve run across some wonky Windows systems where 4 seconds was needed. Try 1 second first to see if it clears the conflicts, then increment until it does.

(The --modify-window parameter in rsync serves the same purpose.)

1 Like
3

Excellent. Which computer needs the setting applied? The Pi or the Windows machine?

4

Okay, I have tried modifying it for the folder on both computers, and even 4 seconds does not prevent the problem.

5

Try the –reset-deltas command-line option to force a complete re-indexing.

If that doesn’t help, then going to need to know more about the setup:

  • Where are the synced files being stored on the RPi?
  • Is 5EQESM2 the Windows device?

Any other details, screenshots of Syncthing’s GUI, etc. would also be helpful.

6

Okay, did a --reset-deltas on both machines, and no joy.

  • Communicating over TCP LAN
  • RPi has a USB HDD mounted using NTFS
  • 5EQESM2 is indeed the Windows device.
  • There are other devices online, but they do not synchronize this folder.
  • Windows 10, and the RPi is Linux (64-bit ARM)
  • Both Syncthing 1.23.6
  • New symptom just noticed: Once sync-conflicts are being generated, any additional change and save produces a new one, even if significantly after the previous one, see example log and note timestamps.
  • Running IPv6
  • config for this folder below log.

Example log after doing --reset-deltas:

Pi 	modified 	file 	Documents 	FileName.sync-conflict-20230718-142120-5EQESM2 	2023-07-18 14:21:20
Windows 	modified 	file 	Documents 	FileName 	2023-07-18 14:21:19
Pi 	modified 	file 	Documents 	FileName.sync-conflict-20230718-141428-5EQESM2 	2023-07-18 14:14:28
Windows 	modified 	file 	Documents 	FileName 	2023-07-18 14:14:28
Pi 	modified 	file 	Documents 	FileName.sync-conflict-20230718-141408-5EQESM2 	2023-07-18 14:14:08
Windows 	modified 	file 	Documents 	FileName 	2023-07-18 14:14:08
Windows 	modified 	file 	Documents 	FileName 	2023-07-18 14:13:58
Pi 	modified 	file 	Documents 	FileName 	2023-07-18 14:12:56

What I’m doing is adding spaces to a text file, saving, waiting until I see it show up in the Recent Changes on the webgui, then removing some spaces and saving again. Repeat as needed.

<folder id="<removed>" label="Documents" path="<removed>\Documents" type="sendreceive" rescanIntervalS="1800" fsWatcherEnabled="true" fsWatcherDelayS="10" ignorePerms="false" autoNormalize="true">
    <filesystemType>basic</filesystemType>
    <devices removed></device>
    <minDiskFree unit="%">1</minDiskFree>
    <versioning>
        <cleanupIntervalS>3600</cleanupIntervalS>
        <fsPath></fsPath>
        <fsType>basic</fsType>
    </versioning>
    <copiers>0</copiers>
    <pullerMaxPendingKiB>0</pullerMaxPendingKiB>
    <hashers>0</hashers>
    <order>random</order>
    <ignoreDelete>false</ignoreDelete>
    <scanProgressIntervalS>0</scanProgressIntervalS>
    <pullerPauseS>0</pullerPauseS>
    <maxConflicts>10</maxConflicts>
    <disableSparseFiles>false</disableSparseFiles>
    <disableTempIndexes>false</disableTempIndexes>
    <paused>false</paused>
    <weakHashThresholdPct>25</weakHashThresholdPct>
    <markerName>.stfolder</markerName>
    <copyOwnershipFromParent>false</copyOwnershipFromParent>
    <modTimeWindowS>4</modTimeWindowS>
    <maxConcurrentWrites>2</maxConcurrentWrites>
    <disableFsync>false</disableFsync>
    <blockPullOrder>standard</blockPullOrder>
    <copyRangeMethod>standard</copyRangeMethod>
    <caseSensitiveFS>false</caseSensitiveFS>
    <junctionsAsDirs>false</junctionsAsDirs>
    <syncOwnership>false</syncOwnership>
    <sendOwnership>false</sendOwnership>
    <syncXattrs>false</syncXattrs>
    <sendXattrs>false</sendXattrs>
    <xattrFilter>
        <maxSingleEntrySize>1024</maxSingleEntrySize>
        <maxTotalSize>4096</maxTotalSize>
    </xattrFilter>
</folder>
7

Ah…

So it sounds like the conflicting file(s) are actually different from the one(s) on the RPi. What’s unintentionally happening is a sort of versioning retention partly caused by a changed file plus conflicting modification time stamps.

I’m very interested in seeing what the RPi says when you stat the conflicting file from the Windows device and also the one sitting on the USB HDD.

Chances are if the USB HDD is reformatted as ext4 / xfs / btrfs or some filesystem type other than FAT / exFAT / NTFS, the file conflict issue would go away (plus a side benefit would be much better read/write throughput).

If you’re running a recent version of RasPiOS with a 6.x Linux kernel, there are actually two NTFS drivers. The FUSE-based “NTFS-3G” and a newer kernel module contributed by Paragon Software. The latter has significantly faster read/write performance, but I haven’t looked into how mtimes are affected.

8

It would be nice if I didn’t have to re-format everything and copy all that data back over. I’ll start looking into how to do it though.

Here are the stat blocks for the files.

  File: FileName
  Size: 1046            Blocks: 8        IO Block: 4096    regular file
Device: 801h/2049d      Inode: 200       Links: 1
Access: (0777/-rwxrwxrwx)  Uid: ( 1000/<>)    Gid: ( 1000/<>)
Access: 2023-07-18 16:36:48.529778500 -0500
Modify: 2023-07-18 16:36:48.529778500 -0500
Change: 2023-07-18 16:36:59.246432500 -0500
 Birth: -

  File: FileName.sync-conflict-20230718-163659-5EQESM2
  Size: 1042            Blocks: 8        IO Block: 4096    regular file
Device: 801h/2049d      Inode: 194       Links: 1
Access: (0777/-rwxrwxrwx)  Uid: ( 1000/<>)    Gid: ( 1000/<>)
Access: 2023-07-18 16:36:48.553664100 -0500
Modify: 2023-07-18 16:36:48.553664100 -0500
Change: 2023-07-18 16:36:59.237965800 -0500
 Birth: -
9

There are tools to do an in-place migration, but I wouldn’t recommend it unless there’s a full backup, or at the very least there’s a UPS and other precautions to avoid any interruption while the process is ongoing.

In a nutshell, it involves defragmenting and shrinking the NTFS partition → creating a Linux partition for ext4 or some other filesystem in the freed space → copying files/folders from the NTFS partition to the Linux partition → deleting the NTFS partition → expanding the Linux partition to occupy the entire drive.

Of course, it’s only an option if the drive is less than 50% full.

If you don’t already have a full backup, then starting with a second drive that’s formatted as ext4 is the best option. Then the NTFS-formatted drive becomes the backup.

Yeah, there’s a difference in the “Modify” timestamps.

10

Okay, converting to ext4 seems to have worked. Thank you for your help.

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.