For any scenario where local reads (as in listing directories and similar, not necessarily raw storage bandwidth) take a little longer (real life examples: MacOS (example from the forum, although I’m not familiar with the mechanism that slows it down in particular), GlusterFS (that’s clear), modern Android (clear too)) the scanning can get 100x, possibly 1000x slower or more. From tens of seconds or minutes to days or weeks. I presume if you have a regular block device with full access, cached by the OS everything goes quickly and no drop is noticeable. The extra ReadDir() mentioned in one of the threads below that it’s done for each file is most likely growing linearly with the number of files, so we increase our scanning time with the square of the number of files, not linearly, which can be very bad.
I think there should be some mechanism that makes it clear what’s going on, not some cryptic advanced configuration with a VERY misleading description in the documentation: “will provide a small improvement in performance”.
Examples:
any thread from various places about “stuck at scanning” Android WhatsApp backup/media directory (there is nothing wrong with syncing that, not an issue with the files being updated all the time or kept busy, it’s just that).
While of course finding a way to speed this up would be preferable (if it can be done safely and efficiently at the same time at all, with no way of just getting a consistent atomic snapshot of a directory from the OS) I’d say to not let perfect be the enemy of good.
Just general awareness would go a long way here. Ideally making it into the documentation instead of the laughable “will provide a small improvement in performance” and maybe even some runtime warning if the scan takes more than some time (5 minutes? 1h? 12h?).
The last use case I mentioned with WhatsApp directory is present in some places (one example below) but I’ve found NOWHERE the simple fix to just flip caseSensitiveFS. People go on all the wrong tangents while investigating this, including recommending other tools. While there’s nothing wrong with it, it would work quickly, reliably and efficiently if only this extra (completely unnecessary in this case, as in most) safety check is disabled. This would also benefit people thinking it works fine for them if the scanning takes an hour (but don’t know it could finish in 5s), and who don’t realize “works for me it doesn’t work for you” hits a user for which it would finish in 1 minute but it takes a week (and it actually never finishes if it gets restarted from time to time on the phones).
As always, there’s the question of what’s representative. On my Mac laptop, which is one of those environments where this is deemed problematic, my Lightroom database folder is ~7000 files, ~8000 directories. That’s not enormous in any way at all, but it’s the same magnitude as the median user according to usage stats.
Scanning that folder with default settings takes 5.6 seconds. Scanning it in case-sensitive mode takes 3.7 seconds. For sure it’s a significant difference percentage wise, but it’s still insignificant in either case.
When writing docs, and code, I can usually only base it on what I see.
The problem is how it scales (it seems quadratically with the number of files or even worse if most of the files are in one large directory instead of a spread structure).
Is there enough empirical information provided and/or known from the inner-workings of the software to be able to consider now this thing “seen” or are there some more specific tests needed?
We have no data on how this performs in the wild, really, although we could add some timing statistics to the usage reporting data. I’m pretty sure there’s nothing quadratic about what it does, but I do think it scales with the width times depth of a given directory structure, though the cache is intended to handle the depth aspect (all the parent lookups). It’s quite possible the cache is ineffective when the lookups themselves become slow enough; I’ve suggested increasing the cache time.
Ideally, someone who actually has a problem with this could figure out what the problem is and see if tweaking the existing values has an effect. That’s harder to do when it’s working as intended on all the systems I see.
What we do have, though, is the metrics endpoint which among other things show the number of FS operations we do and the time spent doing them. If the problem is long listdirs, for example, that’d show up there. Restarting Syncthing, letting it run one of those problematic scans (or as much of one as you have patience for) and pasting a copy of the metrics output would show what Syncthing sees when talking to the filesystem, to begin with.
So you say, but my feeling is the CPU cost is negligible compared to the system call cost/latency. Listing a large directory can take seconds, normalising a string will be microseconds even on a raspberry…
We can in the sense of just not doing so many syscalls, not that I prefer to prioritize speed over safety of the data but at least to have it spelled out - a factor of 35x speedup as in the last example is nothing so sneeze at, versus some over cautious protection that might not even be applicable for that setup.
I am testing how this scales with the number of the files when listdir (or whatever else similar) is slow and the only platform from the mentioned ones which I have easily available is Android. But I think it’s for the best, as it’s popular (if not for syncthing but still billions of devices deployed), and syncthing is one of the VERY rare programs that actually syncs both ways (as mostly anything from Nextcloud to Google Drive are just a pimped up web view of the server, not actual sync with the local files). In any case it takes a while to clock the long runs, by definition - I’ll come back with the results.
And here are the results. Each line shows the times in seconds for 3 consecutive rest/db/scan on the same files with the number of files shown at the end (4 files, 8 files, and so on). The 2nd and 3rd scan should do the same (just walk the unchanged directory) and they do give the same results repeatedly (usually better than 1%). While I took extreme care to have such repeatable results the absolute numbers won’t be relevant for anyone else for sure. However, my initial hunch that the default scenario is quadratic while the caseSensiveFS=true is I hope 100% confirmed. I’m not sure why that was disputed before, if we do a listdir per file and the listdir (time) grows linearly with the number of files, and the number of files grows linearly with the number of files (doh) the result is quadratic.
Once we get “out of the noise” doubling the number of files gets 4x larger scanning times with the default and 2x with the tweaked “caseSensiveFS=true”
I strongly hope this won’t be read as “who’s going to do 40 times the median number of files and then complain it’s slow”, the point is that taking 40x longer can be PERFECTLY acceptable (and achievable with a click, if only people know about it), but taking 1600x longer isn’t fine (and 40x speedup, the FACTOR scaling with the number of files, certainly isn’t “a small improvement in performance” as documented).
@a24 could you test this PR with caseSensitiveFS=false ? Note that it will only improve performance for ASCII-only paths. This might or might not be the case depending on your test data.
I’ll be trying that, although it’s more complicated than it seems, the numbers above were for Android and last time I tried to build something for Android it took some work, so I’m doing regular Linux tests now for this patch but can’t get reproducible results yet even with vanilla build. Here it just bogs down all over the place, and there can be 20x factor time jump when just doubling the number of file.
In contrast nothing was loaded with the Android runs, and everything was like clockwork, probably delayed precisely with whatever timing SAF (or what we’re using to access the files/listdir) allows.
I’m not sure if I can’t edit my previous post or I just can’t find the button, but I said well not to take the numbers in absolute, there is a small error in the script and the correct number of files is the one before the line, as in all FILE NUMBERS should be halved. For example when it says 2048 it should be read 1024 (read the number from above, or just divide what you see by 2).
