REST API - correct procedure

Hi,

I’m developing a system that controls syncthing via the REST interface in java.

I currently have 3 servers in my test environment:

syncmaster ( all folders on here are set to readonly )

sync1 ( only connected to master )
sync2 ( only connected to master )

I have the logic down to sync files to the master then sync to the slave machines however i am now building the “add folder” part and am experiencing weird behaviour.

My current logic is:

1) make sure all syncthings are connected to the master
2) get config on each node
3) add new folder to each config
4) post this config back to each node

now i’m guessing this is a timing thing somewhere as sometimes this works completely fine. other times either sync1 or sync2 rest interface will stop responding indefinitely however the web interface will still load just everything stays in “unknown” state.

I look at the pid stack and its sitting on futex_wait:

root@sync2:/proc/3857# cat stack
[<ffffffff810d853d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8a82>] futex_wait+0x182/0x290
[<ffffffff810db67e>] do_futex+0xde/0x760
[<ffffffff810dbd71>] SyS_futex+0x71/0x150
[<ffffffff81734a9d>] system_call_fastpath+0x1a/0x1f
[<ffffffffffffffff>] 0xffffffffffffffff

I have tried versions: 0.12.20 0.12.22 0.12.23 0.13.0-beta4 ( latest master as of today )

I enabled the STRACE=all but it doesn’t give me much information as when it happens i see it disconnects from the master then no more log entries happen.

What can i do to debug this and figure out why this is happening? Is there a set procedure i have to follow to add a folder, ie do i need to add it on each node and wait for it to connect again?

Cheers

Antony

Are you restarting Syncthing after you change the config? If not, changes might not be applied. Otherwise, it’s hard to say without seeing the code.

You can also check out the code from syncthing-android here. You won’t be able to use it directly because it needs some Android classes, but it should be pretty easy to rewrite for plain Java.

Sounds like a bug. If you can reproduce it, hit the “innermost” Syncthing process with a SIGQUIT to produce a full stack trace that we can use to diagnose it. That process has the higher PID of two. That is:

$ ps aux | grep syncthing
# get either one PID, or two. If two, take the highest
$ kill -QUIT $pidOfSyncthing

Then post the backtrace somewhere :slight_smile:

Where does the backtrace go once i have done this? :slight_smile:

To stderror (or stdout, I don’t remember), or wherever that is redirected to by the thing that starts syncthing :slight_smile:

BTW there was a bug in at least the v0.13 beta that would deadlock, fixed by a commit last evening, so if you’re on that do try the newest newest.

I tried checkout out master but it still happens.

i checkout out master @ commit level: 21e116aa45a3c09953a51ba2a4aead1be7360ed0

ran:

./build.sh assets
./build.sh deb
dpkg-deb --build deb syncthing_0.13.4_amd64.deb

then installed this onto the 3 servers

the version string according to the interface is: v0.13.0-beta.4+18-g21e116a, Linux (64 bit)

Gotcha. Now grab the stack trace and we’ll fix it. :wink:

heres a gist of the stacktrace:

add https:// as new users can only have 1 link :slight_smile:

Erm…

This commit doesn’t exist in my git?

You want 31f6418, current master. I think you’re suffering from the thing fixed yesterday. At least it looks similar from the trace, and the line number don’t line up with current.

Edit: Sorry, I looked wrongly. Your commit looks fine. But do ensure you are actually running what you’re building, because I don’t think you are?

does the build.sh script lock to a certain commit?

# git log
commit 21e116aa45a3c09953a51ba2a4aead1be7360ed0
Author: Jakob Borg <jakob@nym.se>
Date:   Mon May 9 18:25:39 2016 +0000

then ran:

./build.sh assets
./build.sh deb
dpkg-deb --build deb syncthing_0.13.4_amd64.deb

The version looks good:

v0.13.0-beta.4+18-g21e116a, Linux (64 bit)

i assume this means git @ 21e116a

That looks fine. But the line numbers in your backtrace don’t match that commit. Perhaps one more try, one more backtrace?

strange:

$ /usr/bin/syncthing --version
syncthing v0.13.0-beta.4+18-g21e116a "Copper Cockroach" (go1.6 linux-amd64) root@Ubuntu-Trusty 2016-05-09 18:25:39 UTC

I’ll get another one now.

new gist here including --version output:

Nice, that illustrates it.

Excellent :slight_smile:

For what it’s worth, the deadlock is in calling /rest/system/connections in parallel with posting the config. Not doing that, and not having the GUI up, should work around the deadlock.

Ah brillliant,

I’ll just disable the gui for now and purely use the rest api with single thread ( apart from getEvents ).

many thanks!

fyi: this works fine with the beta version ( including the locking fixes ) without the gui open

1 Like