Is it possible to delete .stfolder and stignore file?

[rasa (Ross Smith II)]

Instead of Syncthing using an .stfolder marker to identify a folder under it’s care, what if it recorded the DeviceId/FileId of the folder’s root directory in the folder’s config? This information is available in all environments that Syncthing compiles for. See https://github.com/rasa/godupless/blob/b8ec113b2ecd4d1c72d99384389ed5b3a0f86e91/file/file.go#L58 . This would actually be much safer than using a generic .stfolder marker across all folders.

For example, let’s say a user has two USB disks mounted at D: and E:. The user shares two folders, the first as D:\ and the second as E:\. Both now have an .stfolder marker. If the user (or Windows!) decides to swap the drive letters, both folders would have all their files removed without warning. Or if a new drive D: was added, and the old D:\ now is mounted as E:\, all of the second folder’s files would be deleted.

The same thing could happen on Linux, if a user unknowingly swapped /mnt/d and /mnt/e.

I could whip up a PR to store the DeviceId/FileIds pretty quickly. The tricky part would be how to handle when Syncthing detects the IDs changing.

When that happens Syncthing should probably set the folder as “read only”, then scan the folder and compare it to what’s in its database, and if the two are different, ask the user how to proceed. Right?

[tomasz86]

Personally, I think a simpler solution would be to instead of .stfolder use something like .syncthing_folderID (e.g. .syncthing_udmbc-qrwmp). Changing .stfolder to .syncthing has already been proposed on the forum previously (for better clarity of which software the folder belongs to), and adding the folder ID would make the marker unique to the specific folder.

The largest problem with any such changes though would be backwards compatibility, and also possible breakage of other 3rd tools that may rely on .stfolder being present.

This is just my personal opinion, unconsulted with anyone else, so I have no idea what others may think about a change like this . I’m not a fan of gluing the folder to a specific device though, as I myself happen to move folders around quite a bit, and doing so would make my own use case quite annoying.

[calmh (Jakob Borg)]

FileId is also sometimes fictitious (assigned on the fly when the file is opened), like in some encrypted / compressed filesystems on Windows. This made it not reliable for purposes of recursion detection for example.

Indeed .syncthing-$folderID would solve some issues, especially if ignores and stuff where then stored in there as well. But migrating to it is painful, we’ve had a couple of false starts on it.

[acolomb (André Colomb)]

@calmh

Indeed .syncthing-$folderID would solve some issues, especially if ignores and stuff where then stored in there as well. But migrating to it is painful, we’ve had a couple of false starts on it.

Wouldn’t it be more robust to keep a default filename, but put the folder ID inside as content (.syncthing/folder-id contains abcde-fghij)? That would allow to distinguish between “folder marker missing” and “another ST-shared folder happened to show up under this path”.

I agree that migration is painful, but coupled with other changes like ignore patterns it might become worth it when enough good reasons have been collected.

In general, I think it’s good to distinguish files that are required for internal functionality (folder marker) from files that contain user preference (.stignore). Collecting the former under a common path is wise, but telling users to edit some files inside it but definitely leave others alone is confusing. As a reference, Git has exactly this distinction, cluttering the top-level with e.g. .gitattributes, .gitignore etc. which are all preferences, and maintaining internal data in .git/.

[calmh (Jakob Borg)]

One reason a folder ID specific directory would make sense is because a single directory on disk could be shared as several distinct folders, potentially with separate ignore patterns.

[acolomb (André Colomb)]

What’s the allowed character set for folder IDs? When a file is named after it, that must be restricted accordingly.

[calmh (Jakob Borg)]

There’s no character restriction for folder IDs. However we could use a short hash of the folder ID, or base32 encode it, or something.

[tomasz86]

Doesn’t there already exist some kind of mechanism that is applied when folder IDs are used instead of labels for the root path? What I mean is that when “Auto Accept” is enabled, but a folder with the exact same name already exists in the path, then Syncthing uses the folder ID for the automatically accepted one, instead of its original label. I haven’t tested what happens if the ID is custom and uses funky characters though.

[calmh (Jakob Borg)]

Yeah there’s a filter that creates a reasonable directory name from a folder ID or label, but that doesn’t result in something that’s provably unique.

(That said, if you wanted to share a directory both as abcd*^*1234 and abcd<:>1234 and complained that it resulted in the same on-disk folder ID representation of abcd_1234 then “don’t do that then” is close to hand.)

Our auto generated folder IDs don’t suffer from these ambiguities, you’d have to actively go looking for it.

[tomasz86]

@calmh

(That said, if you wanted to share a directory both as abcd*^*1234 and abcd<:>1234 and complained that it resulted in the same on-disk folder ID representation of abcd_1234 then “don’t do that then” is close to hand.)

It turns out that Syncthing is quite bulletproof .

Auto Accept is enabled, but if automatically added, the folder would end up using the very same path as the existing one, so it actually asks what to do.

[tomasz86]

@acolomb

Wouldn’t it be more robust to keep a default filename, but put the folder ID inside as content (.syncthing/folder-id contains abcde-fghij)? That would allow to distinguish between “folder marker missing” and “another ST-shared folder happened to show up under this path”.

I actually like this one, because it keeps the root folder clean, with only one Syncthing-related folder there. Ignores (and versions) could also potentially be stored inside those, e.g. using .syncthing/.abcde-fghij/.stignore or something in this manner.

[rasa (Ross Smith II)]

Here’s a visual of what’s been discussed, with a few of my ideas thrown in:

/!syncthing/ # not hidden, so user can easily find ignore files?
 |
 + folde-rid01/ # replaces .stfolder, folder marker, not hidden
   |
   + .versions/ # replaces /.stversions, file versions, hidden
   |
   + .tmp/ # stores all temp files, per #2208, hidden
   |
   + ignore.txt # replaces /.stignore, default ignore file, not hidden (user created, optional)
   + included.txt # included ignore files (user created, optional)

The !syncthing folder is named such, so it will stand out from other directories, and sort first in a listing. If we’d rather it sort last, we can name it ~syncthing.

But it bears asking: Do we want to allow different folders to share the same folder root, but with different ignore patterns? Unless the ignore patterns are mutually exclusive, will Syncthing play nice with itself handling concurrent folder changes in a single directory?

[AudriusButkevicius]

Ignore includes are relative to folder root, and probably should stay that way.

We should probably drop the folder- prefix as it serves no purpose.

We should not permit folder ids to be non [a-z0-9]. Folders have labels as the display value so there is no point having anything else than auto-generated values in the id.

I don’t think we can easily move the temp file creation location, it probably requires more thinking.

And yes, sharing the same folder root is an existing usecase, which could be improved by permitting separate ignores etc.

[tomasz86]

@rasa

The !syncthing folder is named such, so it will stand out from other directories, and sort first in a listing. If we’d rather it sort last, we can name it ~syncthing.

Isn’t it better to keep it hidden though? Normal users shouldn’t really be messing about inside those folders directly. The GUI offers control over both ignore patterns and versioning already, so I’m not sure if there is any actual need for keeping them visible. More advanced users, who need direct access, won’t have problems accessing hidden folders anyway.

On the other hand, if the top .syncthing folder is hidden, there’s not really any need to hide the subfolders, is it?

How about something more like this?

/.syncthing/ # hidden
 |
 + folde-rid01/ # replaces .stfolder, folder marker
   |
   + versions/ # replaces /.stversions
   |
   + tmp/ # stores all temp files, per #2208 <-- for consideration, implemented later
   |
   + ignore.txt # replaces /stignore, default ignore file

However, there is also a problem with those, whose .stfolder is a file rather than a folder right now (although it could probably be automatically converted), and also those that use custom folder markers, which is a much bigger problem, because you cannot really store any Syncthing-related stuff there.

@rasa

But it bears asking: Do we want to allow different folders to share the same folder root, but with different ignore patterns?

I’m not sure if there exists an issue on GitHub about this, but there have been some discussions in the forums for sure. There could be a variety of use cases for having different ignore patterns for the same root path, e.g. like my own in https://forum.syncthing.net/t/solved-large-index-size-with-nothing-synced-but-a-lot-of-ignores/14284. Also, they would be useful when sharing the same folder with different people without wanting to give them access to everything.

@rasa

Unless the ignore patterns are mutually exclusive, will Syncthing play nice with itself handling concurrent folder changes in a single directory?

Syncthing can do this already (without different ignore patterns). You can add the same folder even 100 times, and it will work, with understandable performance impact, that is.

@AudriusButkevicius

We should not permit folder ids to be non [a-z0-9]. Folders have labels as the display value so there is no point having anything else than auto-generated values in the id.

Backward compatibility will be a huge problem with this one, won’t it? There surely are some users with very funky custom folder IDs currently .

[calmh (Jakob Borg)]

Moving ignore patterns into the config, perhaps retaining #include directives from there, would probably solve some concerns. You could even retain the legacy behavior if you wanted by having the in-config ignore patterns be just #include .stignore.

[rasa (Ross Smith II)]

If folderId != strings.ToLower(SanitizePath(folderId)),1 we could use the base32 (or even a modified base41) of the folder id, and then include a file in .syncthing/, say folderid.txt, that contains the actual id, to allow multi-folder users to determine which is which.

[acolomb (André Colomb)]

Moving ignores to the config, while allowing to include other files, sounds very nice.

Regarding temp files I would not try moving those to a central location. There may be different file systems mounted underneath one Syncthing folder (even on Windows IIRC), so renaming the temp file to its final name will require another copy operation, which is not atomic. Keeping them right beside the target is the only sensible way I see here.

I’d vote for keeping the .syncthing folder hidden, as it it’s more an internal data store not relevant to most basic usage patterns.

The folder marker is configurable for a reason, namely to support read-only filesystems. The same should hold for the hypothetical .syncthing hierarchy IMHO. So everything in there should be optional as well. That’s definitely true for versioning, as that is even disabled by default. Or if it needs to go outside the folder, external versioning can be used for the special case. Changing the default location to .syncthing/versions/ sounds fine though.

If the ignores were stored in the config, the whole point of keeping several folder markers with different IDs becomes redundant. EDIT: Not really, if we want to solve the issue of detecting swapped USB disks…

[tomasz86]

@acolomb

If the ignores were stored in the config, the whole point of keeping several folder markers with different IDs becomes redundant. EDIT: Not really, if we want to solve the issue of detecting swapped USB disks…

Yeah, although frankly, if ignores are no longer the problem, we can also utilise the current custom folder marker option to solve the risk (of reusing the same path) already. On a similar vein, the versions path is also configurable right now, so it’s possible to place it even inside .stfolder, although I don’t think the path is relative to the root folder, so it likely needs to be absolute.

@acolomb

Regarding temp files I would not try moving those to a central location. There may be different file systems mounted underneath one Syncthing folder (even on Windows IIRC), so renaming the temp file to its final name will require another copy operation, which is not atomic. Keeping them right beside the target is the only sensible way I see here.

It would require more effort to physically move finished downloads to their proper location, although as far as I know/remember, other services like Dropbox do just that, so this may not necessary be a deal breaker (since it seems to have been working quite well for them).

[calmh (Jakob Borg)]

The only sane way to move a file across fs boundaries is using a temp file, which brings us back to square one. I don’t really see temp file placement as being an issue though, in particular not the same issue as marker and ignore files.