German Umlaute like äöü seems to make trouble

calmh · May 15, 2021, 8:46am

Looks like fixing incorrectly normalized names on macOS doesn’t work any more; whether that’s a change in Syncthing or macOS I’m not sure.

bt90 · May 15, 2021, 9:22am

Is the absolute path guaranteed to be normalized at this point?

imsodin · May 15, 2021, 9:28am

It’s in the normalization step of the scanner, i.e. there’s no absolute path involved at all - it’s all relative paths on top of fs.Filesystem. And it occurs while normalizing, so path is not normalized, normPath is.
I know where the error occurs because of the “normalizing path” context in the log:

github.com

syncthing/syncthing/blob/main/lib/scanner/walk.go#L336


		ignoredParent = ""


		return nil
	}
}


func (w *walker) handleItem(ctx context.Context, path string, info fs.FileInfo, toHashChan chan<- protocol.FileInfo, finishedChan chan<- ScanResult, skip error) error {
	oldPath := path
	path, err := w.normalizePath(path, info)
	if err != nil {
		handleError(ctx, "normalizing path", oldPath, err, finishedChan)
		return skip
	}


	switch {
	case info.IsSymlink():
		if err := w.walkSymlink(ctx, path, info, finishedChan); err != nil {
			return err
		}
		if info.IsDir() {
			// under no circumstances shall we descend into a symlink

Then in normalizePath there’s only two places where a filesystem error can be returned:

github.com

syncthing/syncthing/blob/0d054f9b647cc3bd50a1f16382da50055141bf8b/lib/scanner/walk.go#L515


	if !w.AutoNormalize {
		// We're not authorized to do anything about it, so complain and skip.


		return "", errUTF8Normalization
	}


	// We will attempt to normalize it.
	normInfo, err := w.Filesystem.Lstat(normPath)
	if fs.IsNotExist(err) {
		// Nothing exists with the normalized filename. Good.
		if err = w.Filesystem.Rename(path, normPath); err != nil {
			return "", err
		}
		l.Infof(`Normalized UTF8 encoding of file name "%s".`, path)
		return normPath, nil
	}
	if w.Filesystem.SameFile(info, normInfo) {
		// With some filesystems (ZFS), if there is an un-normalized path and you ask whether the normalized
		// version exists, it responds with true. Therefore we need to check fs.SameFile as well.
		// In this case, a call to Rename won't do anything, so we have to rename via a temp file.

github.com

syncthing/syncthing/blob/0d054f9b647cc3bd50a1f16382da50055141bf8b/lib/scanner/walk.go#L530


	}
	if w.Filesystem.SameFile(info, normInfo) {
		// With some filesystems (ZFS), if there is an un-normalized path and you ask whether the normalized
		// version exists, it responds with true. Therefore we need to check fs.SameFile as well.
		// In this case, a call to Rename won't do anything, so we have to rename via a temp file.


		// We don't want to use the standard syncthing prefix here, as that will result in the file being ignored
		// and eventually deleted by Syncthing if the rename back fails.


		tempPath := fs.TempNameWithPrefix(normPath, "")
		if err = w.Filesystem.Rename(path, tempPath); err != nil {
			return "", err
		}
		if err = w.Filesystem.Rename(tempPath, normPath); err != nil {
			// I don't ever expect this to happen, but if it does, we should probably tell our caller that the normalized
			// path is the temp path: that way at least the user's data still gets synced.
			l.Warnf(`Error renaming "%s" to "%s" while normalizating UTF8 encoding: %v. You will want to rename this file back manually`, tempPath, normPath, err)
			return tempPath, nil
		}
		return normPath, nil
	}

This operation returning no such file is weird, given path comes from walking, i.e. has been stat-ed without error before.

opensource21 · May 15, 2021, 9:31am

rsync with iconv produce the same directory like mkdir

calmh · May 15, 2021, 9:59am

So, currently file names must be in NFD (macOS standard) for Syncthing to pick them up on macOS.

bt90 · May 15, 2021, 11:04am

So OP can fix this by converting his remaining NFC normalized files to NFD first.

e.g using rsync locally to a temp folder with rsync --iconv=UTF-8,UTF-8-MAC

calmh · May 15, 2021, 11:20am

github.com/syncthing/syncthing

lib/fs: Handle normalization on macOS (fixes #7677)

syncthing:main ← calmh:normalize

opened 11:16AM - 15 May 21 UTC

calmh

+70 -1

This one was somewhat tricky; macOS does normalization-insensitive lookups so Ls…tat will find a file regardless of the normalization on disk vs in lookup, but out caseFS is more sensitive. Initially I thought we should simply normalize in caseFS on macOS, but that means we return normalization issues as case conflicts. That then means we should allow renames over case conflicts but that's risky as macOS can also have case sensitive filesystems. Instead I changes checkCase to handle "not found" from the case insensitive lookup in the same manner as "not found" from Lstat. This works and I don't see any obvious negative side effects...

opensource21 · May 15, 2021, 11:33am

Must install a newer version of rsync for that. Local it work remote the name isn’t converted.

opensource21 · May 15, 2021, 12:17pm

Can we check if this is the root-cause for the issues? Or does it sound reasonable to you?

Then I would remove the complete folder on the mac and transport it again, via syncthing. Seems a cleaner approach to me with no garbage in the database. Or is the conversion with rsync the better approach?

imsodin · May 17, 2021, 11:21am

Everything points at the normalization as the root cause. The fix has just been merged to the main branch, and an v1.17.0-rc.4 will be out soon containing that fix. You can wait for that or check on your test setup already now with the current build artifact from main: https://build.syncthing.net/buildConfiguration/Syncthing_BuildMac/103268?buildTab=artifacts

opensource21 · May 21, 2021, 6:01pm

I download the version. Get at about versionnr: v1.17.0-rc.3.dev.11.g97437cad, macOS (64-bit ARM) Still no success. console-out.log (66.9 KB) syncthing.log (319.2 KB)

Testscenario is still the same. Create at commandline a file with echo “asd” > äöüß.cox.txt

opensource21 · May 21, 2021, 6:06pm

Forgot to activate debug, so once again syncthing.log (351.0 KB) console-out.log (32.3 KB)

imsodin · May 22, 2021, 10:57am

The logs are still from v1.16.1:

[start] 13:28:27 INFO: syncthing v1.16.1 “Fermium Flea” (go1.16.3 darwin-arm64) teamcity@build.syncthing.net 2021-05-05 07:37:23 UTC [stnoupgrade]

opensource21 · May 22, 2021, 2:53pm

Base on console-out.log it’s [start] 2021/05/21 20:04:39.938450 main.go:571: INFO: syncthing v1.17.0-rc.3.dev.11.g97437cad "Fermium Flea" (go1.16.4 darwin-arm64) teamcity@build.syncthing.net 2021-05-17 10:35:03 UTC The syncthing.log contains the whole log of all test. I didn’t create a complete new environment. Only new files. Look at line 2113.

If it’s necessary to create a complete new environment. I can do this too.

calmh · May 22, 2021, 4:16pm

This is new and amazing:

folder.go:1080: INFO: Scanner (folder "testSyncFolder" (testSyncFolder), item "äöüß.cox.txt"): normalizing path: file does not exist

opensource21 · May 22, 2021, 8:14pm

We have such message at the beginning, so for me, it looks like nothing has changed. What can I do to enable further improvements of syncthing?

imsodin · May 22, 2021, 8:51pm

You caught a bug in the fix that greatroar noticed and fixed already: Fix UnicodeLowercaseNormalized on lowercase NFD by greatroar · Pull Request #7692 · syncthing/syncthing · GitHub. With a newer build that includes said fix, this issue should finally be gone: https://build.syncthing.net/buildConfiguration/Syncthing_BuildMac/103620?buildTab=artifacts

opensource21 · May 23, 2021, 10:10am

Made a retest, now everything seems okay. Many thanks! Will the 1.7 released in the first week of July?

imsodin · May 23, 2021, 5:06pm

great. ~yes~ no

calmh · May 23, 2021, 6:10pm

No, June, as per the header here on the forum.