"empty" folders and .syncthing.*.tmp files


(Ben) #1

hi,

i recently started using syncthing and thought i got into it pretty well… however, i have just encountered a massive problem:

device a: sync folder /Users/me/* device b: sync folder /home/sync/me-macbook

device a contains 111188 files 22426 folders ~316 GiB data (after ignoring certain files via .stignore)

device b contains 0 files 22426 folders ~2.74 MiB data

most folders do actually contain the files they are supposed to, but don’t seem to be recognized by syncthing itself. all those files’ name have this pattern .syncthing.*.tmp, i.e. .syncthing.1-11 Alkohol.mp3.tmp.

therefore, the ~2.74 MiB of data are a mistake; according to device a, ~190 GiB of data have been synced yesterday (and i’m sure if i added up all those .syncthing.*.tmp files, that would be what i’d get).

device a states that device b is syncing (0% since the beginning and still at that point) device b states that device a is up to date

device a keeps switching between scanning and up to date for that specific folder, device b keeps the scanning status up.

am i just impatient and all files will be visible in their regular form once the entire amount has synced? but if so, why does device b state that 130584 items, ~315 GiB are currently Out Of Sync Items?

just trying to make sure i don’t spend the entire day letting my devices sync this enormous amount of data just to find out later that something went wrong and i have yet to go back to square one…

thanks in advance for your help :slight_smile:


(Audrius Butkevicius) #2

Your console or failed items list should have a ton of errors explaining why it can’t finish syncing.


(Ben) #3

When running syncthing -logfile= i get [UHUIW] 10:44:56 FATAL: Cannot open database: resource temporarily unavailable - Is another copy of Syncthing already running?

running top shows two instances of syncthing, which i don’t unterstand. i did pkill one of them.

i cannot find failed items on the web interface; there is only Out Of Sync Items. when opening it, i get an empty list (no files shown) with no errors at all. state of all those files is “sync”. there are color labels and the top of that box is purple, which indicates copied from elsewhere" but each individual file is no purple so i guess the error does not apply here?

also, i just checked the most recent file that has been synced without an error; then cded into that folder to check if the file was there. the file is not regularly via ls, but ls -la shows the file as syncthing.myfile.tmp o.O


(Audrius Butkevicius) #4

Dot files are not shown in ls without the -a option.

Anyways, syncthing logs to stdout, so the logs should already be visible and not require -logfile when running from commandline.

Also, you cannot run syncthing twice, hence the error you are seeing.


(Kluppy) #5

One process is the controller that allows syncthing to restart itself after update etc. The other is syncthing itself.

Depending on how you installed syncthing package maintainers sometimes do not keep the logs.

Without the logs it is really hard to be sure what the issue might be, I am guessing it is permissions related but that is just wild speculation.


(uok) #6

@AudriusButkevicius @calmh would it be possible to give each process a separate name?


(Ben) #7

ugh… thanks. this problem was permissions related. i recreated the entire folder structure in order to start from scratch and when that didnt work either realized pi was running as a different user than it was supposed to. idk why but now that i fixed that, everything works the way its supposed to.


(Ben) #8

ok, nevermind. even though i corrected permissions, i still only get folders and invisible tmp files.

the device causing these errors is a raspberry pi. i am syncing my android and my macbook with it.

for some reason, all files from my android device are being synced (android <-> raspberry pi <-> macbook) the raspberry pi syncs them to an smb mount on my synology nas.

the android sync works in each direction. i can delete folders in my android-dcim folder on my macbook and soon after they will be removed from my android device. so far, so good.

i have an “syncall” folder on my macbook, which (weirdly enough) can also successfully be synced with all devices. this folder is a subfolder of my home folder; the home folder itself, however, still causes the permission problems.

should i delete all syncthing files from all devices (mainly talking about ~/.config/syncthing on my pi and the according syncthing folder on my macbook) and resync everything? or is there another way figuring out how to fix this?

actually, i have tried nextcloud before in a similar setup and had the same permission problems before (nextcloud was running, but could not successfully write any data to my smb mount). however, the “pi” user itself can write to these folders and syncthing is running as pi. anybody else encountered this?


(Audrius Butkevicius) #9

I think this could be related to smb and it having acl’s, oppose to just standard permissions. You could also try setting ignore permissions.


#10

Why not just use syncthing directly on your NAS instead of the Pi?


(Ben) #11

I think this could be related to smb and it having acl’s, oppose to just standard permissions. You could also try setting ignore permissions.

I did that as well. It still does not sync correctly, though. All it does is create those .syncthing.something.tmp files. When not setting ignore permissions, at least I get an error log (seems like this log is hidden otherwise) in the webgui. (and since syncthing is started as a daemon upon boot, I do not see the error log in my terminal)

Why not just use syncthing directly on your NAS instead of the Pi?

Until recently, I had used synology’s own Cloud solution (which was installed on my NAS). This resulted in constant cpu usage around 98%, even when no syncing was happening. That is the main reason why I tried switching to another solution (first nextcloud, now syncthing). There is an inofficial syncthing repo for my DS, but I was not sure whether or not it could increase my synology’s cpu usage again.

I am planing to build a home server / nas from scratch, so syncthing will run on the same hardware that is connected to my hdds; just haven’t figured out a hardware setup yet for my needs.


(Jakob Borg) #12

If you told us what the error is, we might be able to understand the issue and suggest reasonable workarounds. Running it manually once, to see the logs on the terminal, might be also be useful.


(Ben) #13

How do I access the logfile? cat /home/me/.config/syncthing/syncthing.log only displays a few lines (looking exactly like the ones I get when first running syncthing manually). There are no errors in this file, except [26ESY] 10:20:34 FATAL: Cannot open database: resource temporarily unavailable - Is another copy of Syncthing already running?

I have been told before that two instances of Syncthing cannot be running at the same time; and even if possible, I am running syncthing trough a preconfigured startup script and did not start another instance manually, so not sure what this is supposed to mean.

This is the script

### BEGIN INIT INFO
# Provides: syncthing
# Required-Start: $local_fs $remote_fs
# Required-Stop: $local_fs $remote_fs
# Should-Start: $network
# Should-Stop: $network
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Multi-user daemonized version of syncthing.
# Description: Starts the syncthing daemon for all registered users.
### END INIT INFO

# Replace with users you want to run syncthing clients for
syncthing_USERS="pi"
DAEMON=/etc/syncthing/syncthing

startd() {
  for stuser in $syncthing_USERS; do
    HOMEDIR=$(getent passwd $stuser | awk -F: '{print $6}')
    config=$HOMEDIR/.config/syncthing/config.xml
    if [ -f $config ]; then
      echo "Starting syncthiing for $stuser"
      start-stop-daemon -b -o -c $stuser -S -u $stuser -x $DAEMON
    else
      echo "Couldn't start syncthing for $stuser (no $config found)"
    fi
  done
}

stopd() {
  for stuser in $syncthing_USERS; do
    dbpid=$(pgrep -fu $stuser $DAEMON)
    if [ ! -z "$dbpid" ]; then
      echo "Stopping syncthing for $stuser"
      start-stop-daemon -o -c $stuser -K -u $stuser -x $DAEMON
    fi
  done
}

status() {
  for stuser in $syncthing_USERS; do
    dbpid=$(pgrep -fu $stuser $DAEMON)
    if [ -z "$dbpid" ]; then
      echo "syncthing for USER $stuser: not running."
    else
      echo "syncthing for USER $stuser: running (pid $dbpid)"
    fi
  done
}

case "$1" in
  start) startd
    ;;
  stop) stopd
    ;;
  restart|reload|force-reload) stopd && startd
    ;;
  status) status
    ;;
  *) echo "Usage: /etc/init.d/syncthing {start|stop|reload|force-reload|restart|status}"
     exit 1
   ;;
esac

exit 0

(Jakob Borg) #14

That startup script is crap and you should request a refund from whoever sold it to you. :slight_smile: Sorry. I suggest stopping Syncthing and running it manually.

$ /etc/init.d/syncthing stop
$ pkill syncthing                    # to be sure(-ish)
$ /etc/syncthing/syncthing | tee logfile.txt

(Ben) #15

so far, so good… it seems like files can be synced now. device a (syncing from):

  • Global Status 54672 files 22513 folders ~281 GiB
  • Local Status 54669 files 7714 folders ~281 GiB device b (syncing to):
  • Global Status: 54672 files 22513 folders ~281 GiB
  • Local Status: 3 files 22513 folders ~2.77 MiB

BUT device a: Remote Devices:

  • Pi (syncing 0%) Upload 0 B/s (605 B) Download 0 B/s (499 B)

device b: Remote Devices:

  • Macbook (syncing 100%) Upload 9 0 B/s Download 0 B/s

Terminal displays an error like this: [26ESY] 10:50:43 INFO: Puller: final: chmod /home/pi/sync/ben-macbook/Documents/Bücher/Miran Lipovaca/Learn You a Haskell for Great Good! (37)/.syncthing.cover.jpg.tmp: operation not permitted

However, all folders on my pi are chowned to pi:pi while syncthing is running as pi. What could be the reason the operations are not permitted?

EDIT: also, I do not understand how suddenly 3 files (before was zero files) could have been synced, but all the other files (which are within the same subdirectory) could not have sufficient permissions…


(Jakob Borg) #16

Probably some interesting SMB setting or interaction.

What happens when you set “ignore permissions”, like you mentioned earlier?


(Ben) #17

This produces an error like this [26ESY] 11:05:57 INFO: fsync “/home/pi/sync/ben-macbook/Audio/Logic/Aktuell/Schroedingers Death/Audio Files” failed: sync /home/pi/sync/ben-macbook/Audio/Logic/Aktuell/Schroedingers Death/Audio Files: invalid argument

My mount settings are like this: //10.0.0.2/_sync /home/pi/sync cifs auto,credentials=/home/pi/mountcreds,domain=WORKGROUP,nodfs 0 0

Like I’ve mentioned before, my android device does sync all its files successfully on the pi in the same subdirectory (/home/pi/sync). Only the macbook seems to cause problems. However, it seems like I can sync single folders (i.e. /Users/ben/folder1) without any trouble, while /Users/ben/* cannot be synced.

Also, the ignore pattern for my macbook home folder does not work either. It does (or attempts to) sync folders that I excluded, for example .*, Library, Desktop.


(Jakob Borg) #18

Now, see, that’s a meaningful error and something we can work around. Apparently the CIFS mount doesn’t support the sync() call (to make sure file contents are on disk). You can disable that call by unchecking the advanced option fsync (Actions -> Advanced, expand the folder).


(Ben) #19

yeah, that was it. thank you so much!! now all folders and files are being synced :slight_smile:


(system) #20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.