Error syncing Joplin notes - help configuring syncthing

I am trying to sync my Joplin notes between my PC and Android phone with syncthing. At times, I will have issues with notes getting corrupted.

To reproduce the problem:

  1. Create a new note with several lines in Joplin.
  2. Delete a line from the note.
  3. Hit “Syncronize” in Joplin

Can someone suggest a way to configure the syncthing full scans and the file system watcher to avoid a problem like this?

In discussing the issue on the Joplin forum, one of the Joplin maintainers suggested that syncthing and Joplin are not working together nicely. I guess I need to learn more about how syncthing works to fully understand the issue.

What does “Synchronize” in Joplin do, precisely? What does “corrupted” mean?

By corrupted, I meant that the data at the end of the Joplin note file becomes invalid. The data is readable text. Below is an example.

The ending line “type: 1: 1” should just be “type: 1”. I think that when I deleted part of the note, extra characters got added/left to the end of the file?

edit: Joplin then gives me an “Item cannot be null” error as mentioned in the Joplin forum link I posted above.

Test syncthing and joplin

This is a note to test what is corrupting my notes.

1



id: e393a5186da742899147eb516d0c3ddd
parent_id: 132f72eb544549378dc8f7222e47fef8
created_time: 2019-12-05T13:46:36.923Z
updated_time: 2019-12-05T13:47:30.865Z
is_conflict: 0
latitude: 0.00000000
longitude: 0.00000000
altitude: 0.0000
author: 
source_url: 
is_todo: 0
todo_due: 0
todo_completed: 0
source: joplin
source_application: net.cozic.joplin-mobile
application_data: 
order: 0
user_created_time: 2019-12-05T13:46:36.923Z
user_updated_time: 2019-12-05T13:47:30.865Z
encryption_cipher_text: 
encryption_applied: 0
markup_language: 1
type_: 1: 1

When Syncthing writes to a file it first writes it out to a temporary file, all the while verifying the checksum of the data. It then moves that temp file in place over the old file. It doesn’t write in place, but what looks like it has happened here is that something has written a slightly smaller note on top of a larger one, leaving the old extra data in place at the end. I don’t know what would cause that but I have a hard time imagining Syncthing doing it.

@calmh, thanks for your responses. I am afraid this is a little over my understanding? Is it OK if I copy and paste your response over on the Joplin forum?

Sure, go ahead. Given that neither of us knows about the others’ solution and it’s all on an unreliable medium (Android) it might be tricky to sort out what’s going on here precisely.

I’m not quite sure but is it possible that syncthing is creating that temporary file while joplin is still writing to it? because that could explain the garbling at the end of the file.

If Joplin is writing to our .syncthing.whatever.tmp files that would be a problem, yes. Similarly, if Joplin writes to a file it thinks is small or doesn’t exist, but Syncthing has just replaced it with something larger. I don’t know and it comes back to

As far as I know, and I can ask the joplin devs if you’d like. The Synchronize button writes the continence of all the notes into .md files all at once. The fine names are a bit strange. Here is an example file that’s not corrupted.

file name 40e9f3f561ae4d929f4ea921f35bf217.md

test note

This is a test note.

id: 40e9f3f561ae4d929f4ea921f35bf217
parent_id: 45db4e3584714cb59f030329bf6ee834
created_time: 2019-12-05T16:11:18.648Z
updated_time: 2019-12-05T16:11:28.551Z
is_conflict: 0
latitude: 39.28740000
longitude: -80.34190000
altitude: 0.0000
author: 
source_url: 
is_todo: 0
todo_due: 0
todo_completed: 0
source: joplin-desktop
source_application: net.cozic.joplin-desktop
application_data: 
order: 0
user_created_time: 2019-12-05T16:11:18.648Z
user_updated_time: 2019-12-05T16:11:28.551Z
encryption_cipher_text: 
encryption_applied: 0
markup_language: 1
type_: 1

and the last line in the file when it’s corrupt looks like this type_: 1: 1 or

type_: 1: 1
type_: 1

Joplin can “sync” using a variety of backends. Dropbox, OneDrive, NextCloud… So I assume synchronizing in the Joplin app means between it’s internal data and the file system folder markdown files.

But this problem is happening without changes being made on another device, so syncthing would not be writing or updating info?

(from laurent, a Joplin maintainer)

One important point is that Joplin has been used with many sync clients, such as Dropbox, OneDrive, Nextcloud and many others for several years, and this particular issue only happens with Syncthing. I’m mentioning this because it would be easy to assume Joplin writes to the files in a strange way, but it’s not like this, or it would happen with all the other backends too.

I’m not surprised it works perfectly with Dropbox, OneDrive, etc. if there are specific backends for those. It would be surprising otherwise. I could counter and say that if the problem isn’t Joplin then Syncthing would be corrupting files left and right, and it generally isn’t.

However the truth is probably somewhere in the middle with some odd interaction between us. I have no idea what that would be – someone needs to sit down with a stethoscope and a magnifying glass and take a look at what actually happens. We see weirdness with just regular file I/O on Android so that’s possibly a factor as well. :man_shrugging:

1 Like

Thanks for your work on syncthing. It is a great tool and I have been using it for years now.

Unfortunately, I don’t yet have enough expertise to know where to look for more details on the problem. Maybe someone else can take it further.

2 Likes

Booth syncthing and joplin are cross platform. I’ve been testing booth on android to linux and linux to linux. Linux to linux works just fine. It’s only when I add my android device does things stop working. And it’s only when the Watch for changes toggle is on. When it’s off and I manually tell syncthing to sync the files after joplin is finished writing to them it works just fine.

1 Like

@lerker100, can you confirm or contradict what I posted on the Joplin forum thread about this issue (link in top post)? Perhaps this is not related to syncthing? I can cause this issue in Joplin (or at least a very similar issue) even when I remove syncthing from my Android device.

This is the thread on the Joplin Forums on this topic. We are trying to get to the bottom of this issue. Personalty I agree with.

And I think it has something to do with android as well because I can only cause the error on an android device.

I am thinking this has nothing to do with syncthing and might just be between Joplin and Android 10.

1 Like

Are you syncthing the Joplin data directory directly, or did you configure a “synchronization” in Joplin to some other directory on you file system, which you then sync with Syncthing? (On your Desktops and Android)

I have Joplin set to do its synchronization process to a folder in my phone storage. That folder is inside my Syncthing folder. Normally, I edit notes on my phone. I manually check syncthing status before editing notes on a different device.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.