Clean sync to new device does not work and stops at 99%

Support
1

Syncthing is one of my favorite applications – I have been using it for years and to great success across half a dozen devices. I’ve run into a quirk or two at times but never into anything that could not be fixed by restarting Syncthing or rebooting. This time, however, it’s different:

  • A clean sync to a new phone, on which Syncthing has just been set up, always stops around 95-99% with various items being “out-of-sync” (and in fact missing on the device) for no good reason at all.
  • Moreover, it seems syncing to my old phone hasn’t been working properly for some time, either: There is one folder which, too, displays a handful of out-of-sync items.

For the details (and screenshots) see this issue on Github – @AudriusButkevicius asked me to discuss the issue here in the forums until a clear bug emerges.

I should mention, I have searched the forum and while the following threads seem to discuss very similar issues, no clear solution has been found so far:

Anyway, I would greatly appreciate any help! Please do let me know if you need any additional information from me to debug this!

Thanks so much in advance!

2

Could you describe the situation more, devices/os’es/versions involved, is it a completely clean folder or sharing some old existing one etc?

3

For the details, including devices/OS’s/versions, see the linked Github issue. :slight_smile: Please let me know if anything is missing there!

The folders in question are “old” folders in the sense that they have been around on my laptop, my old phone and my even older devices for some time (all of which have Syncthing installed) and syncing had always worked like a charm. They are “new” folders in the sense that on the new phone they had not been present previously, so – as indicated by the thread title – it should have been a full & clean sync from my laptop to my new phone.

4

You should get the logs (for syncthing, not the android app) from the phone that is having issues.

On the surface it seems like enabling ignore permissions might.

5

For convenience, from the issue by OP:

Background

I generally only ever use one laptop and one phone on a day-to-day basis. I have various old devices (multiple old laptops, phones and tablets) which I only use rarely if ever (they basically serve as backup in case of a hardware failure). My current laptop more or less serves as the single source of truth when it comes to my documents & files – meaning that it always has a copy of all my files –, while my phone generally only receives a selected subset of folders & files due to size constraints.

The way I set up a new device (a new phone, a new laptop) has always been the following: I use Syncthing to transfer all relevant files to the new device and make sure all files got transferred properly; then I stop using the old device and rarely (if ever) turn it on again. (If I do turn it on again I make sure all files/changes get sync’ed before turning it off again.)

Device overview

  • old phone: Aquaris X Pro
  • new phone: Pixel 5
  • laptop: sokrates

For completeness’s sake, there are also some other devices showing up in some of the screenshots but I believe they should not play a role for the issue at hand:

  • very old phone: HTC One M8
  • very old phone 2: Xiaomi Redmi Note 4
  • old tablet: SM-T805
  • old laptop 1: philosoph
  • old laptop 2: diogenes
  • old laptop 3: aristoteles

(As one might be able to tell from their names, the laptops are all Thinkpads :))

The bug

I recently got a new phone (a Pixel 5) and had my other devices (my old phone, my laptop) share their folders with it. None of these folders had existed on the new phone yet, so it should have been one huge clean sync of ~70 GiB.

After a few hours, I looked at the new phone: For pretty much every folder, syncing had stopped at 99% (or even less) and the UI said “out of sync”. In the app’s web UI it was listing the names of the files that were out-of-sync. Meanwhile, on my laptop, Syncthing’s web UI was showing the same status: It was listing the very same files as “out-of-sync items” for my new phone and, in addition, it also said which device was the “modifying device” of the file in question: This was either the very same laptop (sokrates), my old phone (Aquaris X Pro) or my old laptop 1 (philosoph).

Here’s the thing, though: Both on my old phone and my laptop (and also on my old laptop), the files in question are present and identical. Indeed, my laptop’s Syncthing instance also says that all folders (on my laptop) are up-to-date, confirming that it is indeed the “single source of truth” mentioned earlier. In short, there is no reason for Syncthing not to sync them to my new phone. Yet, they are not present on my new phone.

Further investigation

I have since repeated this process multiple times (i.e. I deleted and re-installed Syncthing on the new phone): Every single time, in every folder some (apparently) random files don’t get transferred to my new phone. Meanwhile, the overall percentage at which syncing stops tends to vary a bit but it’s usually around 95% - 99%.

Also, I have tried the whole sync process with my old phone not known to my new phone (i.e. my new phone only knew about my laptop) and, another time, also with my new phone knowing and being connected to both my laptop and my old phone. The result in both cases was roughly the same in that syncing didn’t go through 100%.

Actually, now that I’ve looked into this more closely, for my old phone my laptop is now also showing 7 items in one particular folder (out of a dozen folders) as being “out of sync” and, indeed, those file are not present on my old phone. I’m not sure how long this has been the case but I don’t think it’s been like this for a long time. So, basically, the issue seems to be present both on my new and on my old phone, though to varying degree.

Screenshots

Laptop (sokrates)

Old phone (Aquaris X Pro)

New phone (Pixel 5)

Miscellaneous questions / notes

  • Not sure if it’s of any relevance but I’ve noticed a second marker folder .stfolder (1) with a file DO_NOT_DELETE.txt show up in the relevant (out-of-sync) folders. (Meanwhile, .stfolder is also present but empty.) I have since deleted the .stfolder (1) and it hasn’t shown up again.
  • Some files show a last-modified date of 1970 / Unix Epoch
  • Many files that are out-of-sync (though not all) contain whitespace somewhere in their path
  • In how far could file permissions on Android play a role here? I noticed that I enabled ‘ignore permissions’ for some folders on some devices but not all/everywhere.
  • Outside of the “out-of-sync items”, scanning & syncing still seems to work. It seems Syncthing somehow arrives at the (IMO wrong) conclusion that something is wrong with the particular files it lists as “out-of-sync” but everything else seems fine and new files or changes to existing files propagate to the other devices as expected.

Version Information

New phone (Google Pixel 5):

Syncthing Version: v1.18.4
Android Version: Android 12 / GrapheneOS (SPA1A.211105.004)

Laptop:

OS: Ubuntu 21.04
Syncthing Version:  v1.18.4

Old phone (BQ Aquaris X Pro)

Syncthing Version: v1.18.4
Android Version: Android 9 / LineageOS

@codethief

6

@codethief You need to click on “Failed Items” to get a list of the failures/error messages indicating what has gone wrong.

Maybe we should add a link to “Failed Items” at the top of the “Out of Sync Items” list if there is any, a lot of people seem to be looking at that. Maybe also because it’s first, so we might also want to put failed items above out-of-sync items in the folder details.

1 Like
7

I agree with the failed item, I had somewhat the same problem and when I clicked failed items it showed a file with a ? symbol i got rid of the ? and all was fine not sure if the same applies to this thread though…

8

First of all, thanks everyone for such quick, elaborate and helpful responses! I’m starting to feel much more confident now that my issue can be solved. :slight_smile:

@imsodin

Maybe we should add a link to “Failed Items” at the top of the “Out of Sync Items” list if there is any, a lot of people seem to be looking at that.

Oh wow, the “Out of Sync Items: 7 items” link and the “Failed Items: 7 Items” one looked so similar to me that I silently assumed they would lead me to the same information. Now I know why I remembered from a couple years ago that I had seen specific error information somewhere in the UI. It’s just that, when Syncthing is working perfectly for years, you don’t spend a lot of time with the UI and tend to forget these things. (As an aside, this is probably the only critique I would have: The UI could be more intuitive in general. Are there any changes planned in this direction?)

Maybe also because it’s first, so we might also want to put failed items above out-of-sync items in the folder details.

How about combining the two popups into one? Or could there generally be a difference between out-of-sync items and failed items? I mean sure, not all out-of-sync items have necessarily failed. But could there be failed items that are not out-of-sync? If not, i.e. the failed items are always a subset of out-of-sync items, I would suggest combining the two popups and changing the link to something like “out of sync: 7 items (7 failed)”, where “7 failed” gets displayed in red.

You should get the logs (for syncthing, not the android app) from the phone that is having issues.

For anyone else reading this, who – like me – didn’t know at first where to find those logs: In the web UI they can be found in the dropdown menu at the top. The second tab (“debugging facilities”) then allows you make the logs more verbose.

In my case, however, it turns out the logs contain exactly the same information as the list of “failed items”. Moreover, enabling e.g. the “fs” debugging facility didn’t provide any additional information that seemed valuable.

Anyway, here are the logs / failed items:

Failed items: Old phone

Aquaris X Pro - 05 - folder details - failed items - censored
Aquaris X Pro - 05 - folder details - failed items - censored1080×1920 262 KB

All 7 failed items fail for the same reason that can be seen in the screenshot: A symlink cannot be created. In all cases, the symlink points to a relative path within the synchronized folder. (Only in one case the symlink is in fact a dead link, i.e. the target doesn’t exist. In all other cases it does.) As per the FAQ, my understanding had always been that synchronizing symlinks themselves (as opposed to following them) should not be an issue on non-Windows systems? (Besides, in my particular case those symbolic links have existed for years and I’m pretty sure that they used to get sync’ed just fine.)

Failed items: New phone

If I remember correctly, initially, after turning on the phone (with my other devices all offline), the “failed items” popup said that no available device had an up-to-date version of the files in question (or no device had the desired version or something). Now that all my devices are online and connected to each other, though, I’m seeing a different error message (though the same for all failed items in all folders):

pixel5 - 03 - folder details - musik - failed items
pixel5 - 03 - folder details - musik - failed items1080×2340 423 KB

Again, the logs with the “fs” debug facility enabled didn’t provide any additional information on why creating the temporary .syncthing.*.tmp file failed.

@AudriusButkevicius

On the surface it seems like enabling ignore permissions might.

I just had another look: On my both my phones “ignore permissions” is enabled for every folder, so that doesn’t seem to be it. Meanwhile, only on my laptop I had apparently enabled it for some folders and disabled for others – not sure why. In any case, I now think it’s rather unlikely that this is the issue here.

9

Some android versions/filesystems/roms simply do not support symlinks, hence the issue.

The second issue is quite vague, but if I had to bet it will be either:

  1. The location is not writable. Android has a lot of restrictions where files can and cannot be written.

  2. It doesn’t like the German characters in the filenames.

Again, all of these restrictions are imposed by android, not us, hence why kind of frown upon people using syncthing on Android, as it’s generally going to be a poor experience.

I guess I’d rule out 2 before I try to understand 1.

1 Like
10

The location is not writable. Android has a lot of restrictions where files can and cannot be written.

Well, as mentioned before, everything works fine for the vast majority of all files. Moreover, I also haven’t noticed a pattern when it comes to which files fail and which don’t. (Every time I remove the shared folder and add it to my new phone again, different files seem to fail.)

As for my old phone (where Syncthing is complaining about symlinks), I’ll look into whether it’s possible to create symlinks on my phone at all (e.g. via adb).

It doesn’t like the German characters in the filenames.

Most “failed items” don’t carry German umlauts or any other special characters in their filename. I just picked some files for the screenshot where I wouldn’t have to blur anything. :slight_smile:

11

The error itself is coming from the operating system, which is not being helpful explaining why its failing, rather than syncthing.

To be honest, I don’t have a good suggestion, other than see is common between those files that are causing issues.

Are they all empty, have weird permissions, very long paths, unusual characters, etc?

2 Likes
12

So I just had a look at the file paths of the failed items:

Observation 1: The failed items on my old phone (the symlinks) are the only symlinks contained by directories I am trying to sync. They also fail on my new phone with the same error message.

Conclusion 1: Symlinks on Android don’t work, at least on Android 9 and 12 (or, more precisely, the corresponding LineageOS/GrapheneOS versions). Note that I’m talking about Android internal storage here which AFAIK is ext4-formatted! According to Android.SE it might be possible to create symlinks from the terminal, though(?)

→ Whether we can solve this or not: Would it be possible for Syncthing to at least provide a proper error message like “Android does not support symlinks”?

Observation 2: For every single failed item on my new phone, which is not a symlink, the following statement holds for every character of its file path (not necessarily just the name):

If the character is not contained in the set of “standard” file path characters described by the regular expression /[a-z0-9.,-_/ ]/i (note the whitespace), it will be contained in the following set of “unusual” characters:

:&()?>;@!<ÄÖÜäöü#[]áíóúñø|=¿³ºï"'

where I have roughly tried to sort by the number of occurrences in descending order. That is, the most common characters (by far!) are :&()?>, due to some huge directories containing at least one of them.

Notably, there are many cases where one of these predominant characters is the only unusual character in the file path, meaning that each of them alone must make synchronization fail if our hypothesis is correct.

I will try to reproduce this by attempting to synchronize files whose paths contain only a single one of these characters, so that we will know for sure which characters cause synchronization to fail and which don’t.

Possibly someone could try to reproduce this on AOSP 12, so that we can rule out GrapheneOS as the cause?

13

I think at this point the conclusion is that syncthing is not the problem here, but the platform in question is.

14

So, on my computer I created a file <character>.txt for each of the characters in the set :&()?>;@!<ÄÖÜäöü#[]áíóúñø|=¿³ºï"' and then tried to have Syncthing sync them to my phone. The result: For the characters ":<>?| Syncthing failed to create the corresponding file on the phone, probably because Android 12 now considers these characters illegal. I will try find out upstream why this (in my eyes completely ridiculous) breaking change was made post-Android 9.

Notably, for ÄÖü syncing also failed but for a different reason:

remote Ä.txt uses different upper or lowercase characters than local `ä.txt; change the casing on either side to match the other.

Meanwhile äöÜ did get created on my phone, therefore explaining the error message above. This seems to be a bug in Syncthing, though, because Ä and ä are simply different characters and should not be treated as equivalent.

15

They are equivalent if the underlying filesystem is case insensitive, which Syncthing assumes that it is. There’s an advanced option case sensitive fs you can set if that assumption is wrong.

1 Like
16

Do not use that option in this case!
Syncthing only emits that error if the filesystem is indeed case-insensitive, which apparently the fileystem layer on android is. Do not expect any one behaviour from filesystem interaction on android - it does various different things depending on both the android version, the phone vendor and the target api of the app. I wouldn’t be surprised at all if it was indeed some kind of case-insensitive, meaning you can’t have a.txt and A.txt at the same time.

17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.