Why send an index of a receiveonly (IsReceiveOnlyChanged) file?

(Xjtdy888) #1

On the master we have a new receiveonly folder type, read the code, I see the file added or modified for the receiveonly folder, When sending the index information, we will always set LocalFlags to 0, so consider the following scenario:

  1. On machine A, add a send receive folder (empty directory) and share it to machine B.
  2. On machine B, add the directory to receive only
  3. Add some files to the directory of machine B.
dd if=/dev/urandom of=10m.bin count=10 bs=1M

Now on machine A, we can see that the global size is 10M and the local size is 0M. If there are more nodes like machine B, our global size will be very confusing.

I don’t understand why we want to send receiveonly file index to other nodes, and force LocalFlags to 0, which interferes with global index.

0 Likes

(Audrius Butkevicius) #2

If A (send receive) adds a file, and B (send only) modifies the file, we need to let C (send receive) know that the version of the file B has is fudged, and it should not go asking for it.

I guess it still has to send information about the files only existing on B, so that if a file with the same name is ever added on A, C knows that B has the file fudged.

Why is that counted towards the global size, I am not sure. I think it shouldn’t, as these files cannot be retrieved.

0 Likes

(Xjtdy888) #3

Thanks @AudriusButkevicius , now I know why want to send this kind of index. For why is that counted towards the global size, because we forced LocalFlags to 0 when sending, the receiver does not know what happened. 0 is a regular state file.

0 Likes

(Audrius Butkevicius) #4

That’s strange, I’d assume it would advertise it as invalid, hence would not be counter towards the global size.

Summoning @calmh and @imsodin

0 Likes

(Audrius Butkevicius) #5

Actually LocalFlags is always zero in global index, as the flags are local. The only thing that should matter is the invalid bit being set (which is not local), and invalid things should not be counted towards size in the global index.

0 Likes

(Jakob Borg) #6

Files on a device with a recvonly folder should count in both the local and global size, on that device, regardless of whether that file is locally changed or not. If I’ve missed counting them in the local size that’s a bug.

When announcing locally changed flags we should 1) set the invalid bit, 2) clear the version vector to zero. To other devices this will look like we have an infinitely old, broken version of the file and need to get in sync. This is by design. No other device should feel the desire to download our changed file, and it should not affect the calculation of which is the latest version of that file.

0 Likes

(Audrius Butkevicius) #7

This does not look right:

Send only:

image

Send receive:

image

I’d expect that very invalid file that exists on send only not to be counted towards global state?

1 Like

(Jakob Borg) #8

Yes, maybe. Perhaps the corner case is that when a file exists only in invalid variants it should not count toward then global state - although I think that will be no fun to implement. Otherwise this falls under the usual invalid-files-go-into-the-global-set case. I expect the effect is the same if I add a file, let it scan and announce, and then ignore it at the source.

Perhaps we should remove the global state as a concept in the UI, and just deal with local state plus needed items… It’s not particularly “global” any more when every device can make five different types of exceptions from it.

0 Likes

(Audrius Butkevicius) #9

I assume that would result in others not finding a place where to get the file, which can lead to forever out of sync scenario?

0 Likes

(Jakob Borg) #10

Well, it should get re-announced with the invalid bit set when it’s scanned as ignored, and look the same to other devices as above. (Assuming they don’t have the file already.)

I’m not saying this is a good look, but that’s it’s a bit more general, affecting all files announced as invalid, afaik.

0 Likes

(Jakob Borg) #11

Ideally we might want to skip announcing files that were created on a recvonly side and hasn’t been modified by anyone else. But there’s no way to know that in general without adding more flags to the mix.


Actually, that could be solved without too much ugliness I think…

0 Likes

(Simon) #12

Before the local flags and metadata bucket stuff, we simply discarded invalid files in both local and global buckets. I don’t see why we should do that differently now, hence I think this is a regression.

0 Likes

(Jakob Borg) #13

Hm. I feel you have a hand in this. :wink:

I don’t think I changed the accounting of this in the last few changes.

0 Likes

(Simon) #14

With no doubt am I trowing stones in a glass house. And to keep at it: I was thinking about this change: https://github.com/syncthing/syncthing/pull/5027/files#diff-2cb6309d901390f04df8595abde667f4L90
However I haven’t actually looked into it, it might just be perfectly fine - that’s just what came to mind.

0 Likes

(Jakob Borg) #15

No, you’re right, that’s a bug. The change correctly handles local files, which now get some sort of local flag and get accounted in another bucket which is then not counted in the global/local state. (Except for local-changed on recv-only folders.) But the handling of remote files with invalid bits changed and is wrong.

0 Likes