Improved Block Distribution?

rumpelsepp · March 30, 2016, 5:37am

Just on mobile phone, will edit later. There are two major issues about that:

github.com/syncthing/syncthing

Temporary indexes

syncthing:master ← AudriusButkevicius:dprog

opened 05:47PM - 08 Sep 15 UTC

AudriusButkevicius

+1609 -231

Last commit is not finished, but throwing this out here to discuss the general a…pproach before I spend more effort. Also needs tests fixed and so on. ## TODO: - [X] Send block index as an integer, rather than the whole block. - [X] Add a flag to indicate this feature is off, don't send stuff if it's off. - [X] Fallback to Request() in-case temporary request fails (no longer applicable, given we don't do validation, so it's just a if branch) - [X] Address lint/vet/naming suggestions - [X] Fix tests - [X] Add tests ## Maybe: - Limit number of entries we allow to receive, though people should just disable it. _purpose_ > Files that are not fully downloaded are not shared, means that the initial seeding is always done from a single device which originates the change. > Others start participating only when they have the full file. > > This changes that by advertising partially downloaded files (temporary).

Temp indexes is blocked by:

github.com/syncthing/syncthing

Refactor pulling

syncthing:master ← calmh:changeset

opened 09:07PM - 30 Nov 15 UTC

calmh

+6037 -2465

## Current progress - [x] Files - [x] Symlinks - [x] Directories - [x] Permissio…ns - [x] Modification times - [x] Forcing rescan (interface) - [ ] Forcing rescan (actual implementation) - [x] Versioning - [x] Virtual mtimes (interface) - [x] Virtual mtimes (actual implementation) - [x] Progress updates - [x] Database updates (via Progresser interface) - [x] Network pulling parallellization - [x] Conflict handling - [ ] Tricky test cases! - [x] Integrate with rwfolder - [ ] Benchmark to ensure we don't regress too badly - [x] Proper queue status / progress in GUI - [x] Manual queue reordering ## Issues - We need the entire set of files to change in memory. We need this in order to properly sort stuff, but it may require large amounts of memory. Perhaps we can get away with doing it in segments anyway, and retrying failures... - Manual reordering of the pull queue is no longer allowed. The reason is that we enforce an ordering on the entire change set so that we can properly do things like reuse blocks while renaming, make sure to delete a directory that is in the way for a file, and so on. However we retain as much as possible of the original ordering so newest-first, smallest-first etc still work mostly as you'd expect.