I have read a million posts about this and the group answer is always…you missed something…I get that, but I never see an actual answer.
I have 2 machines syncing 40TB of data. They sync a root folder and are identical. the folder path is /media
I don’t want to sync /media/Plex/Torrent_Temp or /media/hyper-v temp. /media is not actually seen but it’s the mount.
My ignore looks like this on both sides, yet all the files and folders in these directories sync. the GUI says reduced by ignore patterns but it’s not. Yes, it’s in triplicate because I am throwing everything at it to get it to ignore.
During an earlier thread, you’d mentioned having permissions and other issues, so just confirming… your ignore patterns above are written to /media/.stignore?
I know you said you were throwing everything at the problem trying to get something to work, so you might already be aware of it. But just in case, note that lines 3 and 4 won’t match anything since the root of your Syncthing folder is /media (unless you happen to have a /media/media/Plex/Torrent_Temp path).
So, in your earlier thread, it sounded like you’d already at least partially synced the 40TB volume between two machines. Were the subdirectories in your ignore list above already synced?
This is mostly an educated guess based on your earlier thread where you said you were using a SMB share…
Your Syncthing instance is in a Linux container, so I’m assuming TrueNAS is also Linux instead of the FreeBSD version. That would mean the SMB share is being served by Samba.
/media/.stignore is a dot file – and by default – Samba hides dot files in network shares (for security reasons). So there could be other settings in TrueNAS that are preventing Syncthing from opening the file?
It’s just a guess because the clues so far offer conflicting results.
(Layers of containers, bespoke app wrappers, etc. makes debugging things like this less than straightforward.)
I fixed the issue that stopped me from mounting local storage and having it work correctly. Yes, this is truenas, and yes, it’s Linux. No SMB shares are being used now.
I just set up a 3rd server. added the ignores to it and then let it start syncing, and it grabbed all the ignored directories. I have the patterns on both sending and receiving servers.
I am driving myself crazy as the main server has a 40gb connection to the internet and people add torrents all the time, and the temp directory is always churning. My local server only has 2gb connection and can’t even come close to keeping up. basically just waiting a bunch of bandwidth on constantly changing files.
Here is a quick look at the directories and .stignore file
Given how long the ignore feature has been in Syncthing, the number of people who use it, and the fact that your 3rd server is behaving the same way seems to point in the direction of a pattern matching problem.
What’s the output of the following command?
od -c /media/.stignore
And what happens if you copy the string “/media/Plex/Torrent_Temp” from your .stignore file and try to list the directory with it like so…?
I fully agree I am missing something about how to format the pattern. That is why I am asking those who have used it successfully to help with my setup.
The reason for the od output was to see if there were any characters outside the range of UTF-8 encoding supported by Syncthing. The result looked fine, so no issues with your .stignore file.
(Over the years I’ve seen users make some wild choices for file/directory names including using the bullet symbol.)
So… I’d asked for the result from the following command…
… but got a ls media/Plex/Torent_Temp/ instead…
Although it wasn’t exactly what I’d asked for, it still accomplished what I’d hoped.
The last time I checked, “Torent” is spelled with two r’s.
None of your ignore patterns below will ever match /media/Plex/Torent_Temp/:
well, you probably just fixed my first issue, as I obviously can’t spell or type. I swear I copied and pasted that path a million times, making sure I wasn’t fat-fingering it.
on the Hyper-V side. The space seems to be causing some kind of issue. when trying to navigate they if I tab it out, I get media/Hyper-v\ temp/. tested other directories with spaces and they are the same \ before the space
Been there, done that many times over. “Forest for the trees” as the saying goes.
I mitigate typos by traversing the path in a terminal, then copying and pasting the path into a .stignore file to build patterns from. So the less I have to type, the fewer mistakes I’ll make.
No, that’s perfectly normal. It’s how Unix/Linux command shells indicate that “Hyper-v temp” is being treated as a single string instead of two since whitespace is the default separator between parameters.
The command shell prefixes characters that have special meaning with a backslash (\) to “escape” the character that immediately follows.
Other than four small exceptions, almost anything else including parentheses, square/curly brackets, colons, question marks, etc. can be used in a file/directory name – e.g.:
Syncthing treats spaces in patterns as just another character, so the pattern (?i)/hyper-v temp would have matched /media/Hyper-v temp unless there are invisible characters in the path name or /media isn’t actually the root of the Syncthing folder.
At this point, I recommend pruning your list of patterns to avoid potential confusion. Toss the two lines that won’t work, and try more generic pattern first in order to iron out issues with your patterns and directory names.
I’m guessing that if there were other torrent_temp directories outside of your Plex directory you wouldn’t want them synced either, so might as well simplify the list of patterns.
If the simplified .stignore still doesn’t work as expected, you’ve got an issue with your Syncthing configuration and/or your path names.
Personally, I try to avoid spaces and punctuation in my file/directory names even though Linux supports it if there’s ever a chance that I’ll be sharing something with a macOS or Windows system.
had to add the new folder as Qbitorent always has these files open and it causes all but the main server to claim local changes.
With these 3 folders, I cut 9TB of files off the sync.
You mention cutting the ignore file to:
(?i)hyper-v temp
(?i)torrent_temp
does that work for folders? I thought folders had to end in a / and I thought they needed relative paths.
Also, you spelled torrent correctly, and my filesystem has it misspelled. not sure I am going to fix it as it’s not a user-accessible area, and I would have to find all the places I messed it up in my ARRS apps.
Might need a new post, but just have a quick question if you know off the top of your head. I now have 3 working servers. currently all on the same 40gb lan. I have all set to send and receive, and all have the folder shared with the 2 other servers, but I only see my main server uploading to the new empty server. Do they not work together? If my main went down, would the 2 others start syncing with each other? I only ask, as I actually have a second colo, and since I already built a 3rd server, I might as well do a bigger collection of syncing servers.
Sorry, I come from the world of Resilio that just kinda handles all this for you automatically and builds a big mesh network. however, they are now getting greedy and constantly upping prices, so I am down to learn something new.
Without the ending forward slash (/), those two patterns above match both files and directories anywhere within the directory tree being synced – i.e. a file/directory named torrent_temp 10 levels deep would still match.
A / at the beginning of the pattern pins the match starting at the root of the Syncthing folder, while a / at the end of the pattern limits the scope to a directory.
So the following pattern only matches a directory named “Directory” that’s sitting at the root of the Syncthing folder…
/Directory/
While this pattern expands the match to anywhere in the directory tree…
Directory/
Perhaps you have dozens of directories named “Directory” but only want to exclude one particular one…
/path/to/a/specifc/Directory/
Or perhaps you want to ignore everything except for that directory…
!/path/to/a/specifc/Directory/
*
(If you haven’t already, it’s helpful to study the Ignoring Files page.)
Yes, if the 3 working servers are in a mesh topology with each one linked to the other two. And even without file/directory changes to send/recieve, all three would be pinging each other to let the others know that they’re still online.
Recheck the configuration, verifying that the shared folder ID is identical across all three servers and that the folder hasn’t been paused on the server that’s not receiving changes.
The difference is you have to register with Resilio, so it knows the topology of your mesh, while with Syncthing each node starts out as an only child until it’s been introduced to its sibling(s).
It’s the never-ending struggle between cost and convenience.
I think technically this matches everything inside the directory, i.e. directory/ is equal to directory/*, meaning that all subfolders and files will be ignored. However, the directory itself will still be synced, just empty.
