Reconnection index transfer

robertookode · July 24, 2015, 12:59pm

hi,

We are currently working with a master repository synchronization and two nodes ( slave).

I can synchronize large files , so Syncthing generates a lot of indexes. Each block size is 128KiB generating many blocks in each file.

We have problems when we perform the reconnection to the calculation of indices , Syncthing has 50MB up and down every time. you can change the size of each block?

calmh · July 24, 2015, 1:05pm

Currently no^*. But,

and

are thoughts/efforts toward this.

_{*) You can change the block size by altering the constant, rebuilding, and not interacting with anyone else.}

AudriusButkevicius · July 24, 2015, 5:09pm

If you want you can compile a special version with a different block size. But it will only be compatible with the same custom version.

robertookode · July 28, 2015, 9:22am

Thanks calmh,

In this moment I have the version of the deltas, but the problem persist. I view transfer all indexes.

thanks

AudriusButkevicius · July 28, 2015, 9:41am

How do you know? Did you add print statements in the index sender routine?

robertookode · July 28, 2015, 11:03am

I’m using iptraf, is a program from network monitoring.

I have seen that transfers indexes remain in the same proportion 1/3500~ with all nodes synchronized.

AudriusButkevicius · July 28, 2015, 11:43am

So given you compiled that version, I suggest you actually add print staements to verify your observations. Furthermore, you’d have to usr this version on most nodes, and it might only be in effect after second startup on all nodes (after all nodes has seen each other for the first time).

robertookode · July 28, 2015, 1:01pm

If I create a new version with modified block size to (1GB) if the connection is down will start downloading that block or continue ?

thanks!

AudriusButkevicius · July 28, 2015, 1:26pm

It reuses whole blocks, not part of blocks. Also, blocks are kept in memory (multiple of them), hence 1GB is too much.

Also, if 1 bit changes in a block, we will have to redownload the whole block.

robertookode · July 29, 2015, 9:37am

In the query API ( http://192.168.241.129:8443/rest/db/status?folder=default ) , I see the version “version” : 287544554 . This number is related to the indexes?

Because version number changes on other devices when a device is offline ?

thanks!

AudriusButkevicius · July 29, 2015, 10:36am

I think its documented here:

robertookode · July 29, 2015, 11:05am

Then the value indicated in the following figure, corresponds to maximum local version ??

thanks

AudriusButkevicius · July 29, 2015, 11:24am

github.com

syncthing/syncthing/blob/master/cmd/syncthing/gui.go#L431




func (s *apiService) CommitConfiguration(from, to config.Configuration) bool {
	// No action required when this changes, so mask the fact that it changed at all.
	from.GUI.Debugging = to.GUI.Debugging


	if to.GUI == from.GUI {
		return true
	}


	if to.GUI.Theme != from.GUI.Theme {
		s.statics.setTheme(to.GUI.Theme)
	}


	// Tell the serve loop to restart
	s.configChanged <- struct{}{}


	return true
}


func getPostHandler(get, post http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

I have no clue why its a sum, @calmh will have to answer that.

calmh · July 30, 2015, 7:56am

Basically so that the “total” version changes if either the local or remote indexes has changed. The RemoteLocalVersion is a sum of all the other devices version counters if I remember correctly.

robertookode · July 30, 2015, 8:43am

Thanks Calmh,

I’m now again tested version of deltas. I get this in the logs :

What is "Database cache capacity 16020 KiB block "?

On the right you can see how all of the indexes are transferred .

Thanks!

AudriusButkevicius · July 30, 2015, 10:01am

So database cache capacity, is database cache capacity. I don’t think iptraf is the right tool to decide if indexes were sent as a whole or not. I’d add logging to go code to verify instead. Also, are you sure you are using code from the right pull request?

calmh · July 30, 2015, 12:05pm

I suggest not using the delta index pull request code for the moment. It’s not merged for a reason, there’s an outstanding issue that needs resolving for it to be reliable.

robertookode · July 31, 2015, 7:25am

Thanks AudriusButkevicius,

I used the version of deltas (https://github.com/calmh/syncthing/commit/97019f767fb0dbc94578d14c3f86e1c0c7a470fc).

I work with virtual machines, I have installed only syncthing and iptraf for network monitoring.

What software do you recommend for network traffic ?

Thanks!

robertookode · July 31, 2015, 7:50am

Thanks Calmh and good job.

We are particularly interested in improving the deltas, we would like to know when will be available .

We 'd like to help in testing the deltas when they are available even if isn’t merged.

Thanks!

AudriusButkevicius · July 31, 2015, 7:53am

So iptraf for traffic is fine, but traffic might not always tell you wether the index stuff is working or not.