What is syncthing telling me in these screens part 2

continuation of:

(Sorry, was not dilligent enough to respond and keep previous thread alive).

I waited for several days for completion of the scanning/sync. It doesn’t seem to have ever finished. I removed the original “receive only” folder, created a new one, added many “ignore” folders and let syncthing run for several days.

Thus current state on the receive-only end is this:

I am perplexed and clearly not understanding the fucntion of syncthing. In this applicatoin (a backup scheme with one sendonly folder on one device and one recevie only folder on another device on the same lan):

  1. Where does the “exclude” file list get configured? Send end, recieve end or both? I’ve done both, and inexplicably, the subfolder structure is created (in some cases, several levels of the tree), though those folders “excluded” contain no files. (So it’s like the exclude “files” function works, but doesn’t exclude the folder itself.

  2. You can see the current status (the screen shot is the recieve only end), which indicates out of sync and a substantial number of “out of sync items” and “locally changed” items. Plus the “revert changes button” is available. When i look at the list of “out of sync items”, i see many files that I would have expected were to be EXCLUDED. When i look at the “locally changed items” i see many files that i expect SHOULD remain. Yet if i click “revert”, the list of files deleted are ones i expect to remain. (and can verify still exist at the “Send only” end).

  3. I believe i am unclear on the effect of “scanning” and “watching” regarding which “end” of the “send only” “receive only” pair. Who is scanning who and who is watching who. If you note on the screenshot, despite the “out of sync” message, you can see “This device” indicates no data transfer in either direction. Sniffing the IP also indicates no packet traffic at this time. (though there was previously).

  4. Although it was suggested that i upgrade both syncthing instances, I am unable to at the “send only” end as it is winxp, so I’m as far along the development chain as i can be on that device (if i understand the documentation i read), if it’s possible that these issues are related to the version mismatch, then I may have no hope of resolving.

I truly appreciate the outstanding work that has gone into this amazing product. I have struggled with the limited documentation, and If i can get this issue i’m having solved, I commit to fully documenting it for others to benefit from.

again, thank you.

I’ve also made a presumption that it is not necessary for the GUI to be running on either end for synchronization to occur. Is this correct?

1 Like

Firstly: There’s of course been a lot of changes and bugfixes since 1.0.1. And there’s also plenty of reasons to not use XP anymore besides Syncthing. Whether or not your issues are connected to that I cannot say at this point, but it means there’s no point doing lots of debugging for such an old version.

What kind of NAS is it? In any case it’s highly likely you should check the ignore permissions option - have you already done so?

For only two devices it’s almost always the right thing to have the exact same patterns on both sides (aka I can’t imagine a use-case for differing patterns, but my imagination is limited :slight_smile: ).

If you have ignore patterns set up but they don’t take effect, clearly something is wrong. Please post the paths that are created and your ignore patterns for reference.

  1. Your mixing lots of things here that aren’t directly connected. Scanning means it’s currently looking for local changes on disk. “watching” isn’t a folder state, that’s a feature to trigger scanning of changed directories only. In out-of-sync state, it’s normal that there’s no traffic, that would occur during “syncing” state (not exclusively though, you can have traffic while up-to-date too).

Yes.

Thank you!

  1. yes something other than xp woudl be lovely. not possible in this situation.
  2. NAS is Dlink with Alt-f firmware and syncthing.
  3. I had not selected “ignore perms”, but have and will see. In the meanwhile, here is the: Source directory tree:

Here is the exclude list: (Root is “a21msync”, these are subfolders of that)

.android

.kde

.oracle_jre_usage

.stfolder

.swt

Cookies

Favorites

IETldCache

Local Settings\History

Local Settings\Temp

Local Settings\Temporary Internet Files

Recent

NetHood

PrintHood

PrivacIE

SendTo

Sync

UserData

WINDOWS

Application Data\logitech

Application Data.kde

Application Data.purple

Application Data\adobe

Application Data\adobeum

Application Data\ahead

Application Data\apple computer

Application Data\avs4you

Application Data\blackberry desktop

Application Data\brother

Application Data\cch

Application Data\winrar

Application Data\u3

Application Data\downloaded installations

Application Data\dropbox

Application Data\enchant

Application Data\epson

Application Data\fileopen

Application Data\filezilla

Application Data\fritzing

Application Data\garmin

Application Data\gnupg

Application Data\gnupgold

Application Data\gtk-2.0

Application Data\help

Application Data\identities

Application Data\imgburn

Application Data\installshield

Application Data\intuit canada

Application Data\ipodder

Application Data\jam software

Application Data\krksoft

Application Data\leadertech

Application Data\logishrd

Application Data\wireshark

Application Data\macromedia

Application Data\media player classic

Application Data\mozilla

Application Data\mp3tag

Application Data\mpeg streamclip

Application Data\mussio ventures ltd

Application Data\nero

Application Data\nitro

Application Data\nitro pdf

Application Data\nvu

Application Data\obsidium

Application Data\oracle

Application Data\photoscape

Application Data\real

Application Data\research in motion

Application Data\tomtom

Application Data\tgcmlog

Application Data\teracopy

Application Data\sandisk

Application Data\telefonica

Application Data\skype

Application Data\skypepm

Application Data\sling media

Application Data\steelbytes

Application Data\sumatrapdf

Application Data\sun

Application Data\syncthing

Application Data\teamviewer

(spaces not in actual file but added to prevent paragraph display in this interface)

This is the resulting destination file tree:

Earlier today i also pre-pended (?i) to each line in case it was a case issue (pun intended). But there has been no change in action.

To me, without exhaustive analysis, it looks like some, but not all folders were created in destination despite being excluded.

Thoughts?

one more clarifying question.

In a send receive application like this. Enabling “watch” and “scan interval- (say 3600)”. What is the process flow when a file changes on the “send” end. Who notices? how does the file get updated on the receive only end? (pushed? pulled? requested? sent?).

It would seem logical in a normal (bidirectional) installation, each end would be responsible to watching their own end for changes, and when any occurred to push an update to other nodes. (Or notify other nodes that a new file was available and should be pulled). I didn’t find a technical explanation of file flow in the docs anywhere. Maybe i can write one once i understand it. Thank you!

sorry, my adhd gets more than the best of me sometimes. for the sake of completeness, here is the source end gui display:

Have you read Understanding Synchronization — Syncthing documentation and Folder Types — Syncthing documentation ? That should cover the question you just asked. If not, please clarify what is unclear.

Checking ignore permissions had no effect then?

The global states vary greatly and the nas has a higher local than global state - something is clearly broken. I’d update the NAS to v1.6.1, besides the improvements this will also cause a database consistency check, which might clear things up.

You can put a line with three backticks (```) before and after a block of pre-formatted text like this.

Thank you. I took the advice and upgraded the NAS to 1.6.1. The system did rescan all the files, and transfered 70+Gb of data from the “send” (windows computer) to the “receive” (Linux Arm NAS).

It has been running since the above post, and still both the “send” gui and the “recive” gui indicate “synching”. Neither reached 100%.

I added some extra logging choices in the log config of both ends. Here is the NAS log, most of the intial makes sense, but then an interesting entry:

2020-06-06 13:56:55 My ID: TJDVXBI-IXWLS2O-V3CT3YF-NS5JROZ-EJKWELC-GD4WFFK-6ZNOOX2-66BCUQX
2020-06-06 13:56:56 Single thread SHA256 performance is 6.0 MB/s using crypto/sha256 (6.0 MB/s using minio/sha256-simd).
2020-06-06 13:56:59 Hashing performance is 5.54 MB/s
2020-06-06 13:56:59 Migrating database to schema version 10...
2020-06-06 13:58:00 Migrating database to schema version 11...
2020-06-06 13:58:07 Compacting database after migration...
2020-06-06 13:58:45 Detected upgrade from v1.4.0 to v1.6.1
2020-06-06 13:58:45 Checking db due to upgrade - this may take a while...
2020-06-06 13:59:53 Repaired 23294 sequence entries in database
2020-06-06 13:59:53 Overall send rate is unlimited, receive rate is unlimited
2020-06-06 13:59:53 ...
2020-06-06 13:59:53 TCP listener (0.0.0.0:22000) starting
2020-06-06 13:59:53 Ready to synchronize "a21msync" (zo7pr-qtwjg) (receiveonly)
2020-06-06 13:59:54 GUI and API listening on 0.0.0.0:8384
2020-06-06 13:59:54 Access the GUI via the following URL: http://127.0.0.1:8384/
2020-06-06 13:59:54 My name is "Ford-NAS"
2020-06-06 13:59:54 Device XS4X5IL-3OZBI35-JJAJ76W-MTOJLKR-B2RNYSP-JVACQRY-PLDALOO-CR5XNAC is "a21m" at [tcp4://192.168.1.30:22000]
2020-06-06 14:00:24 Listen (BEP/tcp): TLS handshake: write tcp4 192.168.1.9:22000->192.168.1.30:4943: i/o timeout
2020-06-06 14:01:00 Established secure connection to XS4X5IL-3OZBI35-JJAJ76W-MTOJLKR-B2RNYSP-JVACQRY-PLDALOO-CR5XNAC at 192.168.1.9:50372-192.168.1.30:22000/tcp-client/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
2020-06-06 14:01:00 Device XS4X5IL-3OZBI35-JJAJ76W-MTOJLKR-B2RNYSP-JVACQRY-PLDALOO-CR5XNAC client is "syncthing v1.0.1" named "a21m" at 192.168.1.9:50372-192.168.1.30:22000/tcp-client/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
2020-06-06 14:01:00 Device XS4X5IL-3OZBI35-JJAJ76W-MTOJLKR-B2RNYSP-JVACQRY-PLDALOO-CR5XNAC folder "a21msync" (zo7pr-qtwjg) has mismatching index ID for us (0x3F6539986FC54E6E != 0x2C5B7104F54AE770)
2020-06-06 14:01:00 Device XS4X5IL-3OZBI35-JJAJ76W-MTOJLKR-B2RNYSP-JVACQRY-PLDALOO-CR5XNAC folder "a21msync" (zo7pr-qtwjg) has a new index ID (0x7EC02B4956758514)
2020-06-06 14:05:24 Connection to XS4X5IL-3OZBI35-JJAJ76W-MTOJLKR-B2RNYSP-JVACQRY-PLDALOO-CR5XNAC at 192.168.1.9:50372-192.168.1.30:22000/tcp-client/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 closed: reading message: read tcp4 192.168.1.9:50372->192.168.1.30:22000: read: connection reset by peer
2020-06-06 14:05:57 Listen (BEP/tcp): TLS handshake: write tcp4 192.168.1.9:22000->192.168.1.30:1075: i/o timeout
2020-06-06 14:06:05 Listen (BEP/tcp): TLS handshake: write tcp4 192.168.1.9:22000->192.168.1.30:1076: write: connection reset by peer
2020-06-06 14:06:13 Listen (BEP/tcp): TLS handshake: write tcp4 192.168.1.9:22000->192.168.1.30:1078: write: connection reset by peer
2020-06-06 14:06:27 Established secure connection to XS4X5IL-3OZBI35-JJAJ76W-MTOJLKR-B2RNYSP-JVACQRY-PLDALOO-CR5XNAC at 192.168.1.9:22000-192.168.1.30:1079/tcp-server/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
2020-06-06 14:06:27 Device XS4X5IL-3OZBI35-JJAJ76W-MTOJLKR-B2RNYSP-JVACQRY-PLDALOO-CR5XNAC client is "syncthing v1.0.1" named "a21m" at 192.168.1.9:22000-192.168.1.30:1079/tcp-server/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
2020-06-06 15:04:21 Completed initial scan of receiveonly folder "a21msync" (zo7pr-qtwjg)
2020-06-07 14:40:00 Puller (folder "a21msync" (zo7pr-qtwjg), item "My Documents/mail/mike_archive.pst"): pull: generic error
2020-06-07 15:49:35 Puller (folder "a21msync" (zo7pr-qtwjg), item "My Documents/mail/barn.pst"): pull: generic error
2020-06-07 15:56:22 Puller (folder "a21msync" (zo7pr-qtwjg), item "My Documents/mail/mike_main.pst"): pull: generic error
2020-06-07 15:56:22 Puller (folder "a21msync" (zo7pr-qtwjg), item "My Documents/mail/joke.pst"): pull: generic error
2020-06-07 20:46:24 Puller (folder "a21msync" (zo7pr-qtwjg), item "My Documents/mail/mike_archive.pst"): pull: no such file
2020-06-08 02:14:54 Puller (folder "a21msync" (zo7pr-qtwjg), item "My Documents/mail/mike_archive.pst"): pull: generic error
2020-06-08 02:14:54 "a21msync" (zo7pr-qtwjg): Failed to sync 1 items
2020-06-08 02:14:54 Folder "a21msync" (zo7pr-qtwjg) isn't making sync progress - retrying in 35h11m32.316158005s.
2020-06-08 04:32:11 Connection to XS4X5IL-3OZBI35-JJAJ76W-MTOJLKR-B2RNYSP-JVACQRY-PLDALOO-CR5XNAC at 192.168.1.9:22000-192.168.1.30:1079/tcp-server/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 closed: reading length: read tcp4 192.168.1.9:22000->192.168.1.30:1079: read: connection reset by peer
2020-06-08 04:33:49 Pausing XS4X5IL-3OZBI35-JJAJ76W-MTOJLKR-B2RNYSP-JVACQRY-PLDALOO-CR5XNAC
2020-06-08 04:51:00 Listen (BEP/tcp): TLS handshake: write tcp4 192.168.1.9:22000->192.168.1.30:1218: i/o timeout
2020-06-08 04:51:11 Listen (BEP/tcp): TLS handshake: write tcp4 192.168.1.9:22000->192.168.1.30:1219: i/o timeout
2020-06-08 04:51:25 Listen (BEP/tcp): TLS handshake: write tcp4 192.168.1.9:22000->192.168.1.30:1220: i/o timeout
2020-06-08 04:51:35 Listen (BEP/tcp): TLS handshake: write tcp4 192.168.1.9:22000->192.168.1.30:1228: write: connection reset by peer
2020-06-08 04:51:50 Established secure connection to XS4X5IL-3OZBI35-JJAJ76W-MTOJLKR-B2RNYSP-JVACQRY-PLDALOO-CR5XNAC at 192.168.1.9:22000-192.168.1.30:1229/tcp-server/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
2020-06-08 04:51:50 Device XS4X5IL-3OZBI35-JJAJ76W-MTOJLKR-B2RNYSP-JVACQRY-PLDALOO-CR5XNAC client is "syncthing v1.0.1" named "a21m" at 192.168.1.9:22000-192.168.1.30:1229/tcp-server/TLS1.2-TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
2020-06-08 07:49:25 Puller (folder "a21msync" (zo7pr-qtwjg), item "My Documents/mail/mike_archive.pst"): pull: no such file

********

The entry:

2020-06-08 02:14:54 Folder “a21msync” (zo7pr-qtwjg) isn’t making sync progress - retrying in 35h11m32.316158005s.

Seems out of place. What does it mean “isn’t making sync progress”.

QUESTION 1: what would cause this?
QUESTION 2: why such a strange (and large) retry interval?

I noticed the LINUX ARM NAS fan was running a lot. TOP:

Mem: 122436K used, 2528K free, 0K shrd, 2992K buff, 721660K cached
CPU:   0% usr   8% sys  91% nic   0% idle   0% io   0% irq   0% sirq
Load average: 2.30 2.06 1.61 2/82 2363
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 1378  1372 syncthin S N   795m 651%  92% syncthing -home=/var/lib/syncthing
 2362  2343 root     R     1200   1%   8% top -bn1
 1372     1 syncthin S     793m 649%   0% syncthing -home=/var/lib/syncthing
  906   862 root     S    11388   9%   0% smbd -D
  862     1 root     S    11340   9%   0% smbd -D
  859     1 root     S    11064   9%   0% nmbd -D
  686     1 root     S     1936   2%   0% smartd -i 1800
 2343  2342 root     S     1272   1%   0% {sys_utils_proc.} /bin/sh sys_utils_proc.cgi
  544     1 root     S     1224   1%   0% syslogd -C -m 0 -D
 2342   780 root     S     1216   1%   0% httpd -ifh /usr/www
 1247     1 root     S     1212   1%   0% {dns320-temp.sh} /bin/sh /usr/sbin/dns320-temp.sh
    1     0 root     S     1208   1%   0% init
  780     1 root     S     1204   1%   0% inetd
  698     1 root     S     1204   1%   0% crond
 1246     1 root     S     1204   1%   0% /bin/sh --
 1248     1 root     S     1200   1%   0% {watch-inetd.sh} /bin/sh /usr/sbin/watch-inetd.sh
  547     1 root     S     1192   1%   0% klogd
 1300  1248 root     S     1188   1%   0% sleep 180
 2255  1247 root     S     1188   1%   0% sleep 15
  600     1 root     S      592   0%   0% sysctrl

Syncthing appears to be using substantial portion of available resource. Could this explain anything (besides the running fan :wink: )

The actually interesting lines answering the question are just above that line:

2020-06-07 15:56:22 Puller (folder "a21msync" (zo7pr-qtwjg), item "My Documents/mail/mike_main.pst"): pull: generic error

It can’t get the psts from the other device. Those are outlook database files if I remember correctly, so likely they changed since last scan.

Either the sync operation already failed a few times or it took a long time. In any case if something relevant changed, syncing happens immediately. This retry only takes affect if nothing happens during that interval.

Screenshots please. That statement is ambiguous and potential conflicts with the logs below (i.e. failed pull means folder should be out-of-sync with failed items, not syncing).

Yes: It’s doing something. If it’s not evident from the web UI (screenshots), you could get a cpu profile: Profiling — Syncthing documentation

After purging the installation and files and starting from scratch with new shares (at both ends - send & recieve), and ending up in a similar place (never “sync” status on receive and the red not in sync) i made a hypothesis that maybe the version missmatch between send (syncthing 1.01) and recieve (syncthing 1.6) might be worth trying to resolve.

With help, i installed syncthing 1.0.1 on the arm NAS. However, it fails silently.

The log says:

[TJDVX] 23:17:38 INFO: syncthing v1.0.1 "Erbium Earthworm" (go1.11.5 linux-arm) teamcity@build.syncthing.net 2019-01-18 10:34:18 UTC
[TJDVX] 23:17:38 INFO: My ID: TJDVXBI-IXWLS2O-V3CT3YF-NS5JROZ-EJKWELC-GD4WFFK-6ZNOOX2-66BCUQX
[TJDVX] 23:17:39 INFO: Single thread SHA256 performance is 6.2 MB/s using crypto/sha256 (6.2 MB/s using minio/sha256-simd).
[TJDVX] 23:17:40 INFO: Archiving a copy of old config file format at: /var/lib/syncthing/config.xml.v30
[TJDVX] 23:17:43 INFO: Hashing performance is 5.11 MB/s
[TJDVX] 23:17:43 FATAL: Database schema: Syncthing v1.6.0 required
[monitor] 23:17:43 INFO: Syncthing exited: exit status 1
[monitor] 23:17:44 WARNING: 4 restarts in 45.550844701s; not retrying further

What does " 23:17:43 FATAL: Database schema: Syncthing v1.6.0 required" mean?

Syncthing 1.6 was installed on the device, and i installed 1.01 overtop of it using the following commands (at the NAS cli) provided by a friend:

wget https://github.com/syncthing/syncthing/releases/download/v1.0.1/syncthing-linux-arm-v1.0.1.tar.gz # download from github
tar xzf syncthing-linux-arm-v1.0.1.tar.gz # extract to temporary location
rcsyncthing stop # stop an existing syncthing. *Don't* stop it other way
cp -a syncthing-linux-arm-v1.0.1/* /opt/syncthing/ # copy over existent
chown -R syncthing:sync /opt/syncthing # change user and group
rm -rf syncthing-linux-arm-v1.0.1* # remove download and tmp extracted files
rcsyncthing start # start syncthing. *don't* start syncthing other way

Have I ignorantly made an error in my method? The mention of syncthing 1.6 in the failure message makes me think that my “downgrade” was not completed fully or sucessfully, even though syncthing identifies that it is v 1.01.

Question 2: Must i uninstall syncthing before trying to install a new version, and is there advice somewhere on how to do that, or a command built in to do it?

Thank you for an incredible product. I am learning a lot while trying to implement and get it functioning.

Sorry, why did you install a 18 month old version? I can’t see it being recommended anywhere.

Yuu most likely can’t downgrade that far back and you’d have to start from scratch, but there really isn’t a good reason for running an ancient version.

1 Like

In this case, the good reason is that it is running on ancient hardware. I know the development world is all about upgrade upgrade upgrade, but it’s not reasonable/practical in some cases. I’m sorry for asking, but do appreciate your hint that i should start from scratch. Thank you

No need to be sorry for asking. Downgrading is very often a dangerous thing to do, as it’s not “the intended path”.

Out of curiosity: What are the problems with your hardware that affect recent versions but not 1.0.1?

You built a great product, in a way it’s an insult to be using an old version (it disrespects the continued love you pour into the product improving it).
Please don’t laugh, the reason i must use 1.01 is that the OS is XP, and the reason the OS is xp is that is the newest OS the intel based hardware will support. I think in the software world you call this a “dependency”. :wink: Sometimes in the world, we must do things that are not the optimum, and of course, we must accept the consequences thereof.
however, you could also say that 18 months ago, V1.01 was the best there ever was! So i live a little in the past and hopefully get this simple implementation of an old version working to solve the need i have. It is true that these “rabbit holes” also broaden ones knowledge, so if time is available (covid?) then learning is a reasonable time expense. Again thank you for an excellent product.

Just a side note, but if you indeed need to use the old version, it may be better to use the same version everywhere. Otherwise you may encounter various compatibility issues, which no-one here will be able to resolve, even if theoretically the old and the new versions should work with each other.

Downgrading in itself is always a potentially risky procedure, regardless of the software.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.