Mixed Syncthing with Nextcloud


#1

Nextcloud always fascinated but it just never really worked in its entirety. Webdav is slow and requires re-syncing entire files every time a change occurs. The Nextcloud sync client on all platforms has issue after issue with syncing and don’t really trust it, especially as files get bigger (anything over 10 MB). If you open the Nextcloud client and need to sync say 50 GB of data good luck closing the client and opening it next day, or even next minute and expecting it to pick up where it left off. All kind of errors will be thrown (even with Apache optimization). Again this is webdav. This obviously doesn’t scale well.

But one neat thing about Nextcloud was being able to access data in browser like Google Drive. This is what liked about Nextcloud. This part of Nextcloud is a great replaced for Google Drive or Dropbox, the Nextcloud sync just never was or is. So combined Syncthing with Nextcloud. Syncthing is just amazing for syncing files. It works flawlessly and never had an issue no matter the size of file. Can close Syncthing and reopen it next day and it picks up right where left off. And its actually easier to setup Syncthing sync than Nextcloud sync.

So now run Syncthing as www-data user on Debian server on top of Nextcloud. Syncthing syncs beautifully between any device necessary and can use the web interface of Nextcloud when necessary. Wish didn’t have to run a server (the strength of Syncthing in most cases) but once set all this up (which wasn’t very hard or time consuming) it runs itself and can utilize the best of Syncthing and Nextcloud.


(uok) #2

Looks like a good combination! How does Nextcloud detect changes made by Syncthing?


#3

php /nextcloud-install-location/occ files:scan --all

via cron every day. Can run more often if use web interface more.

Scan takes about 50 minutes for about 60,000 files. Once a file is scanned it is available immediately in Nextcloud interference as the scan continues on.


#4

You could also implement a watcher, which uses the event api from syncthing to get file updates and tell Nextcloud to specifically scan that files.


(Alex) #5

I made such a watcher some time ago and I still use it, I assume it also works with nextcloud


#6

I use a similar setup, but instead of Nextcloud (or Owncloud back in those days), I use Pydio. Pydio is really powerful and as an advantage it doesnt’t bother about files being changed externally. So using Pydio I can always access my files which are being sync between my machines and a FreeNas server/


#7

@Alex Really cool.

In regards to:

scanner -ocuser=“owncloudUser” -stfolder=“cloud” -occpath="/path/to/occ" -target=“http://127.0.0.1:8384” 2>&1

-ocuser would be www-data or the actual Nextcloud user? Assume Nextcloud user name but wanted to double check.

Would have to be run for every folder that wanted to watch, correct? Probably yes in answering own question unless am completely misunderstanding the watcher.


(Alex) #8

-ocuser is the user in own/nextcloud, running it as www-data needs to be done manually

It currently only works if the syncthing folder is the complete nextcloud folder, you may need to change https://github.com/alex2108/syncthing-owncloud/blob/master/scanner/main.go#L98 to get something else (maybe you can misuse -ocuser to get some subfolder because I think the user only defines the path for own/nextcloud so for a subfolder your “user” could be user/path/to/subfolder)


#9

While @Alex program is awesome figured out a simpler way to have changes in Syncthing immediately show up in Nextcloud.

If add Syncthing folder as an external local folder in Nextcloud all changes made through Syncthing will immediately show up in a Nextcloud instance.

Pros to using @Alex program is that (think) it integrates with Nextcloud/ owncloud trash, versioning, and maybe activity so it is a more complete solution integration solution than the above method. Cons are that @Alex program adds even more complexity. Nextcloud trash and versioning still work with the external local folder method but only changes made in the Nextcloud web interface will utilize the Nextcloud trash and versioning. Syncthing changes will only utilize Syncthing file versioning.


(Zilexa) #10

Thanks a lot @schnappi, I am going to try this. I don’t have 1 single SyncThing folder, there are multiple folders like /Documents, /Pictures and also folders I do not want to share or sync like /TVshows /Movies /Downloads. But I will probably have to move /Documents and /Pictures to 1 folder, like /Personal/Documents and /Personal/Pictures and add /Personal folder to NC as external storage.


#11

@Zilexa how did it work out for you? Did you decide to move everything you wanted available to a single external folder or did you add multiple external folders in Nextcloud?


(Sunjam) #12

I’m also curious to hear updates.

You might be interested in this open feature request for Nextcloud to integrate with Syncthing as a backend.


#13

Still works well for me using external Nextcloud folders to pickup the changes that Syncthing makes. Exponentially better than using webdav (Nextcloud sync client) and again brings together best of both projects.


(Sunjam) #14

Do any of the changes in Syncthing show up in Nextcloud Activity app?


(Sunjam) #15

As far as I can tell, Syncthing currently has to be mounted as local storage, which breaks:

  • Local Folder is no longer counted as part of available Nextcloud Storage
  • Quota warning app no longer useful
  • Activity app can no longer track files

Also, local storage is not recommended in the official documentation.… due to significant security risks.


#16

Changes made by Syncthing do not show up in the activity app. Agree that the storage indicator does not work with external storage of any type. For me this is not an issue since Syncthing is “primary” for lack of a better word. Nextcloud is secondary for the sole purpose of accessing through a web browser and iPhone.

As to the local storage issue. Do not see this as a security risk. For example an attacker who has admin privileges can create new external local folders to anywhere they want. Utilizing external local folders is just using a feature which is already turned on and ready to be exploited by an attacking admin user. Not using external local folders in no way prevents another admin user from doing so unless the external storage app is turned off, but then an attacking admin user could just re-enable the external storage app. For these reason do not see using external local storage as a security issue.

Of course file system permissions also matter in terms of adding external local storage to any folder on the system as a whole. As long as Syncthing is running as www-data many folders will not be available as external local folders through the Nextcloud external storage app. What are your thoughts as to the security implications?


(Sunjam) #17

Conversation on Local Storage folders x-posted over to Nextcloud Discourse forum for more info on potential security risks.


(Jen) #18

There are talks about a bounty for the integration of Syncthing into Nextcloud to use it as fully supported syncing back end: