Last week I tried tail -f on some synchronized file, but it didn’t behave like a local file I would append to: tail didn’t detect any update in the file.
Is this expected? Would that be a suitable feature request?
This is expected. Syncthing doesn’t append to the file, it replaces the file. (In some tail implementations tail -F might do what you want, or it might not.)
1 Like
As a feature request, would it make sense to append to the file instead of replacing it when the old version of a file is a prefix of the new version?
Actually thinking about it, this might not be possible for big files: if several writes have to happen before the next version is fully downloaded, the user shouldn’t be exposed to some intermediate version.
But would it make sense for smaller files for which there will be more or less one “write”?
We always rewrite the whole file, as modifying files in place is not atomic.
For example, we might have to write ABCD to a file, crash having written AB, then on next start-up would detect the file as having changed, because we crashed and did not manage to record the fact we changed the file in the database, assume that the new correct content is just AB, and propagate that out causing data loss everywhere.
If you are doing something silly like trying to tail log files transported by syncthing, it’s a bad idea in genera because it ends up being n^2 complexity, for every write, you re-read everything you’ve written before. If it’s a log file that doesn’t change often, it’s probably fine, but if it’s something you are trying to tail, it’s probably something that changes often, hence a bad idea, because syncthing is probably constantly burning cpu cycles hashing that file.
In the general case Syncthing can’t even know it’s an append. For Syncthing to know that a given change is an append we’d have to have stashed a full copy of the previous version of the file and compare them byte for byte. Syncthing deals with blocks, and it can see that the block as a whole has change because the hash changed.
Ok I forgot that part, I guess that’s the main argument.
Yes that’s my use-case: I’m just saving myself a ssh and possible wrappers around it to handle disconnects and I just synchronize the logs I’m interested in so that I can have a look at them locally. I’m happy with this for now, even without the change I mentioned.
I assume you’re talking about the sender, since the receiver already has this previous version.
Maybe I’m misunderstanding, but if a sender S is trying to synchronize a file to a receiver R, they first figure out that the first few blocks are the same, so that they don’t exchange them again, right?
At this point, the updated file could be considered an appended version iff it consists of new blocks as well as at most one changed block that was at the end of the previous version and which is related to the first block of the new changes. For instance, the changed block could consist of 32KiB of old data and 96KiB of new data if the block size is 128KiB.
If this is correct, determining whether this sync could be considered an append by R would only require to compare the last block of the previous version with its updated content.
Am I missing something?
But anyway as mentioned before, without atomicity there’s no way to make it work.
It could work that out, but its not atomic, hence we don’t do that.
1 Like
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.