item has UTF8 encoding conflict with another item

These files are in failed items. How do I do to fix this issue programmatically or is there a way to do so?

I can rename manually to something else such as from

  • 小ゼミ予定表2019年度前期.xlsx
  • to 小ゼミ予定表2019年度前期.xls and
  • change it back again to 小ゼミ予定表2019年度前期.xlsx

Then syncthing will be able to work properly. However, I have many files with this problem. If you have any suggestion, please share.

Note: the problem started when I moved syncthing folders to another disk. Syncthing v1.3.1 Linux (64-bit) on Docker, macOS 10.13.6.

There is a convmv tool which might help.

Thanks, but what would be the encoding syncthing wants? utf-8?

Well, I thought all my files are already utf-8, to be honest, but I might be wrong; that’s why I’m here asking. I have tried to convert from utf-8 -> utf-8-strict (whatever that means) convmv said that already converted. However, syncthing doesn’t seem to be happier.

Yes, we use utf-8. Do you have the actual error?

Log

2019-11-30 09:01:33 Scanner (folder qybnk-g4mmg, item "docs/大ゼミ計画20180619.xlsx"): normalizing path: item has UTF8 encoding conflict with another item

If you need anything else, please let me know.

The problem is that you have two files, in two encodings, so if we normalize the non-utf8 filename, we end up with a duplicate file with the same name. You need to remove one of the files.

You mean the other file on the other machine, right? How could I know which machine I have to delete? This directory is shared among 5 machines or so.

No, on the same machine.

There is only one file per one “failed item.”

What OS/filesystem is this on? You mention both Linux and macOS but it’s not obvious to me which side is seeing the error. Are you doing a network mount of some kind? Are you saying the you’re running the Linux version of Syncthing under Docker on Mac? If so, don’t do that. The Mac has peculiar requirements and behavior on normalization on the filesystem, and Syncthing expects this and knows how to handle it when running on a Mac. When running under Linux Syncthing doesn’t expect a Mac filesystem.

The file is in Mac OS Extended (Journaled) file system, but I run syncthing in docker container and mount the directory to that container (probably as fuse.osxfs)

Basically, I run like

docker run \
    --name my-syncthing \
    -v /Users/test/syncthing/data:/var/syncthing \
    syncthing/syncthing:latest

You are saying that I should not do this, but still is there a way to understand the issue? I mean I run this way since syncthing 0.xx and I kinda accept the complication. To me, changing filename back and forth would solve the problem immediately, but that’s not why I’m here. For example, maybe syncthing might need to have better error messages for this sorta problem.

The issue is that Syncthing tries to enforce normalization form C which Linux and most other systems use. The Mac filesystem enforces normalization form D, so this doesn’t work. The check Syncthing does to find out whether the two different normalization forms refer to the same file also fails, not sure precisely why but due to some factor in the Docker filesystem translation I guess.

In short, use the Mac version on Mac. I don’t think we want to try to detect this really odd situation and produce a better error message. In principle the underlying OS is lying to Syncthing about the expected normalization form and how filenames work.

When I tried convmv I use both --nfd and --nfc which is normalization form D and C consecutively. I waited for syncthing to do its thing before trying another thing too, but as I reported, nothing changed.

If this doesn’t seem like anything to concern, then it’s fine. :slight_smile: I guess, I could rename them all then.

Thank you all :+1: