### Purpose
Many commonly used filesystems do not allow certain characters or… names in file and directory names. For example, the NTFS, exFAT, FAT32 and reFS filesystems do not allow the characters `/\"*:<>?|` in the file name or directory name component of a path. On Windows. [reserved filenames](https://ss64.com/nt/syntax-filenames.html#reserved) such as `CON` and `Nul.txt`, and filenames ending with a period, or space are also not allowed. Syncthing currently rejects these files causing out-of-sync errors.
Windows Subsystem for Linux (WSL), Cygwin, Msys2, git-bash, Linux's [CIFS driver][1], and other similar systems solve this issue by encoding these characters, to allow filenames containing these characters to be saved to disk. They do this by replacing the reserved characters with UNICODE characters in the private use area (\uf000-\uf0ff). This conversion system was first developed in [Interix][2] over 25 years ago.
This PR adds an advanced folder option titled "Encoder Type" which has three options: `passthrough`, `fat`, and `standard`.
1. The `passthrough` encoder sends and receives encoded filenames without decoding them. This is how Syncthing currently works, and is the default option.
2. The `fat` option encodes as described above.
3. The `standard` option does not encode, it only decodes. While the `standard` encoder doesn't encode, it ignores all other encoded filenames, in case a user sets the encoder type, and then sets it back to `standard`.
Here's a table of the different encoders:
| Name | Filesystem(s) | Encodes | Decodes? | Reserved Filenames on Windows | Comments |
| ------- | ------------- | -------- | --------- | --------------------- | ------------|
| `passthrough` | Any | (none) | No | Fail | Filenames containing encoded characters are sent & received and read/written on disk as is. |
| `fat` | [NTFS][5], [exFAT][6], [FAT32][7], [reFS][8] | `"*:<>?\|`, `\x01-\x1f` | Yes | Fail | 100% compatible with WSL/Cygwin/git-bash/Msys2, etc. On Windows, file or directory names with trailing periods or spaces are still rejected. |
| `standard` | [APFS][10], [Btrfs][4], [ext4][3], etc. | (none) | Yes | Fail | Filenames containing encoded characters are not sent or received, and are ignored on disk. |
### Windows reserved filenames<a name="Windows-reserved-filenames"/>
If this PR is accepted, we may want to address Windows reserved filenames such as `CON` and `Nul.txt`, and filenames ending with a period, or space, via two new encoders:
| Name | Filesystem(s) | Encodes | Decodes | Reserved Filenames on Windows | Comments |
| ------ | ------------- | -------- | --------- | --------------------------------- | ------------|
| `windows` | [NTFS][5], [exFAT][6], [FAT32][7], [reFS][8] | `"*:<>?\|`, trailing spaces and periods, `\x01-\x1f` | Yes | Succeed (`con` encoded as `co\uf06e`) | On Windows, reserved filenames can be opened in File Explorer, CMD, etc. <br/>Reserved filenames are incompatible with WSL/Cygwin/git-bash/Msysw, etc, as they are encoded, whereas WSL saves reserved filenames as is. |
| `wsl` | [NTFS][5], [exFAT][6], [FAT32][7], [reFS][8] | `"*:<>?\|`, `\x01-\x1f` | Yes | Succeed (`con` saved as `con` via [`\\?\`](https://ss64.com/nt/syntax-filenames.html) prefix) | 100% compatible with WSL/Cygwin/git-bash/Msys2, etc. <br/>On Windows, reserved filenames cannot easily be opened in File Explorer, CMD, etc. (The user would need to prefix the full path with [`\\?\`](https://ss64.com/nt/syntax-filenames.html)). |
<!--
| `ios` | [APFS][10] | `:`, leading periods (except for `.st{filter,ignore,versions}`) | Yes | Fail | The [Files][11] app rejects files with colons and leading periods. |
| `plan9` | [Fossil][9] | `\x01-\x1f`, `\x80-\x9f` | Yes | Fail | See [here][13]. |
| `auto` | Any | | | | Auto-selects `fat` on Windows and Android systems, `ios` on iOS systems, `plan9` on Plan 9 systems, and depending on the underlying filesystem, selects either `standard` or `fat` on all other systems. |
-->
### Unencoded characters
None of the encoders encode NUL (`\x00`). While some filesystems, such as [HFS+][12] allow file and directory names to contain slash (`/`) characters, none of the encoders in this PR encode this character. On Windows, this PR does not encode file and directory names containing backslashes (`\`). These unencoded characters will result in out-of-sync errors on systems that don't allow these characters.
### Testing
All the tests pass. `go vet` is clean. `go lint` is clean. `go fmt` produces no output.
I have tested it with Windows, Linux, Android 11, macOS and [iOS][14] clients pushing changes to a Windows client. Directory listings showed the correct filenames in WSL, Cygwin, Msys2, and git-bash environments. I also created filenames containing reserved characters in WSL, Cygwin, Msys2 and git-bash, and these transferred successfully to Windows, Linux, Android 11, macOS, and iOS clients.
### Documentation
If this PR is accepted, I will draft a PR to update the documentation.
## Testing
To test this PR, simply switch to this branch on all nodes. The `passthrough` encoder is enabled by default.
To test the other encoders, all nodes need to be set to either `standard` or `fat`. Typically, unix like systems will use `standard` and Windows (NTFS), and FAT-based systems such as Android, will use `fat`.
While `standard` can't be used on FAT/NTFS-based systems without errors, `fat` can be used on any system, but it makes little sense, if the underlying file system is not FAT/NTFS-based.
## Authorship
My name and email are already in the AUTHORS file.
## Links
For reference, see:
https://cygwin.com/cygwin-ug-net/using-specialnames.html
http://msdn.microsoft.com/en-us/library/aa365247%28VS.85%29.aspx
https://en.wikipedia.org/wiki/Filename#In_Windows
https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file
For implementations, see
https://github.com/mirror/newlib-cygwin/blob/fb01286fab9b370c86323f84a46285cfbebfe4ff/winsup/cygwin/path.cc#L435
https://github.com/billziss-gh/winfsp/blob/6e3a8f70b2bd958960012447544d492fc6a2f1af/src/shared/ku/posix.c#L1250
https://github.com/torvalds/linux/blob/master/fs/cifs/cifs_unicode.h#L27
[1]: https://github.com/torvalds/linux/blob/761c6d7ec820f123b931e7b8ef7ec7c8564e450f/fs/cifs/cifs_unicode.h#L27
[2]: https://en.wikipedia.org/wiki/Interix
[3]: https://en.m.wikipedia.org/wiki/Ext4
[4]: https://en.m.wikipedia.org/wiki/Btrfs
[5]: https://en.m.wikipedia.org/wiki/NTFS
[6]: https://en.m.wikipedia.org/wiki/ExFAT
[7]: https://en.m.wikipedia.org/wiki/File_Allocation_Table#FAT32
[8]: https://en.m.wikipedia.org/wiki/ReFS
[9]: https://en.m.wikipedia.org/wiki/Fossil_(file_system)
[10]: https://en.m.wikipedia.org/wiki/Apple_File_System
[11]: https://en.m.wikipedia.org/wiki/Files_(Apple)
[12]: https://en.m.wikipedia.org/wiki/HFS_Plus
[13]: https://9fans.github.io/plan9port/man/man9/intro.html
[14]: https://github.com/syncthing/docs/pull/654