Global state different from local state

imsodin · January 21, 2018, 9:30pm

Can you give some more details about what you do when “trying various exclude config in .stignore”? Ideally some simple steps such that it can be reproduced and debugged - that’d be great.

brunod · January 22, 2018, 7:58am

Edit the .stignore file, try various way to understand how to distinguish root files, root folders, subdir folders and file types, then save and rescan to see what happens on another device. I wanted to exclude everything, except any .Pdf anywhere in the tree. Here are collected the tested contents of .stignore. Display is a bit weird here, without CReturn replaced by /. I cannot be more precise as I don’t know which one causes the trouble exactly. Hope it helps, otherwise just ask

(?i)!/*.PdF
!/*
/*/
*

//19/0
(?i)!/*/*.pDf
(?i)!/*.PdF
/*/*/
!/*/*
/*/
//!/* sinon 21/2
*

//19/0
(?i)!/*/*.pDf
(?i)!/*.PdF
//!/*/*
//(?i)!/English
//(?i)!/FrencH
//!/*
*/*/
*/
*

//20/0
(?i)!/*.PdF
!/*/
*

//20/2
(?i)!/*/*.pDf
(?i)!/*.PdF
/*/*/
// !/*/*
/*/
!/*
*

//18/0
(?i)!/*/*.pDf
(?i)!/*.PdF
/*/*/
/*/
*

//18/0
(?i)!/*/*.pDf
(?i)!/*.PdF
*/*/
*/
*

//21/2
(?i)!/*/*.pDf
(?i)!/*.PdF
!/*/*
!/*
/*/*/
/*/
*

//21/2
(?i)!/*/*.pDf
(?i)!/*.PdF
!/*
*/*/
*/
*

//17/0
(?i)!/*/*.pDf
(?i)!/*.PdF
*/*/
*/
*

cosas · January 22, 2018, 8:38am

I never thought multiple (?i) or (?d) prefixes in the same pattern would ever work. Are you sure of that ?

(?i)*.pdf would do the job I think

brunod · January 22, 2018, 8:41am

They are not in my case; it’s displayed on one line because the carriadge return are omitted in “code display”. But anyway, I think I’ve read in doc that it’s allowed. I 'll post back if I find it. Here is how the first should look like :

(?i)!/*.PdF

!/*

/*/

(<- this is a star)

Edit : https://docs.syncthing.net/users/ignoring.html

quote : Prefixes can be specified in any order (e.g. “(?d)(?i)”), but cannot be in a single pair of parentheses (not “(?di)”).

cosas · January 22, 2018, 8:44am

Mistake. Good way :

!(?i)*.pdf
*

brunod · January 22, 2018, 8:48am

(?i)!*.Pdf do the job but when you try to exclude all rest with a star *, it cuts the subdirectories too. So you only have the root .pdf. (Spending 2 days with this…) So the target was to design generic subfolder content with something like **/*.pdf, **/**/*.pdf, but I didn’t succeed. So I had to list all directories in the exclude file. Inspired from : (searching source). Edit : found un-ignore subfolders & files of ignored folder

cosas · January 22, 2018, 8:55am

Hmmm, strange, I didn’t thought it did.

Then what about this ? :

!(?i)**.pdf
*

brunod · January 22, 2018, 9:02am

Nope, tried too, like

 *, **, /*, /**, /*/, /**/...

cosas · January 22, 2018, 9:04am

Do you insist putting (?i)!*.pdf instead of suggested !(?i)*.pdf

PS : 4 white space on lines biginning to post blocks (will preserve * also)

brunod · January 22, 2018, 9:09am

Yes, because I didn’t try “your” solution as it’s not the way it is presented into the refered online help for exclude file : https://docs.syncthing.net/users/ignoring.html A pattern beginning with a (?i) prefix enables case-insensitive pattern matching. (?i)test matches test, TEST and tEsT. The (?i) prefix can be combined with other patterns, for example the pattern (?i)!picture*.png indicates that Picture1.PNG should be synchronized.

I didn’t even though of trying it Edit : But I’ve tried with and without (?i) yesterday; it doesn’t change the behaviour concerning the subdir exploration. So it’s not the problem I think.

cosas · January 22, 2018, 9:17am

! is your major aim, i.e. do not exclude. Then any case (?i)*.pdf. Then exclude all remaining, * on a new line

What you did seems to ignore case of files named !blahblah.pdf i.e. having a ! as first character, then do exclude them.

Then the final * excludes all remaining, i.e. everything as surely you have not even a single !*.pdf in your [sub]folders

brunod · January 22, 2018, 9:32am

No, I tried with only 20 files and 3 folder to be sure what is or wasn’t included in the share. To ease the trial, I omit the (?i). When I had understood it, only then reintroduce it at last step, checking its action. That part is correct I’m 100% sure. I don’t have any file beginning with a ! and the share were ok. Here is my final version that works :

(?i)!/English/user/folder1/*.pdf
(?i)/English/user/folder1/*
(?i)!/English/user/folder1
(?i)!/English/user/folder0
(?i)/English/user/*
(?i)!/English/user
(?i)!/English/*.pdf
(?i)/English/*
(?i)!/English
(?i)!/French/*.pdf
(?i)/French/*
(?i)!/French
(?i)!/Dutch/*.pdf
(?i)/Dutch/*
(?i)!/Dutch
(?i)!/Deutsch/*.pdf
(?i)/Deutsch/*
(?i)!/Deutsch
(?i)!/Other/*.pdf
(?i)/Other/*
(?i)!/Other
(?i)!*.pdf
*

imsodin · January 22, 2018, 10:22am

@cosas Unfortunately @brunod is right, that is a limitation of the current matching: Lets take e.g. a/b.pdf, then * matches the directory a, so it is skipped and a/b.pdf is never checked at all (in which case it would be included by !(?i)**.pdf).

cosas · January 22, 2018, 10:26am

Then you are lucky

I’m more as I have the same kind of tree you have with nested sub-dirs [PicturesRoot]/YYYY/YYYYMM blahblah in a 1st computer that I fully sync with a 2nd computer.

My own day2day laptop is also involved in this share, but nearly all the time I have a single * enabled as I don’t need to waste space in my SSD. Sometimes although, I need to sync a single whole subfolder YYYY/YYYYMM blahblah what I do by inserting the same syntax you show above on top of my single * (once done the job requiring this and sync back done, I just comment the lines with // for syntax memory purpose).

To check your issue I added a single !(i)*mG0075.jpg on top : ST then created in my laptop just the needed tree to the 7 files it found in the 2 other devices.

Notice all my nodes are set R/W for this shared folder which eventually make a difference in this matter with you… maybe.

Good luck

cosas · January 22, 2018, 10:33am

[EDIT] @imsodin

He said “exclude all but any *.pdf anywhere”. Why not just use !*.pdf. This single * doesnot match the / so no subdir will be excluded out from parsing ? As shown above this works for me… strange.

I’d also try !(?i)**/*.pdf (but maybe not very usefull), not forgiving the final * all at bottom

imsodin · January 22, 2018, 10:51am

The double asterisk “**” is indeed unnecessary, but the point about skipping dirs still applies. To not ignore /a/b.pdf but everything else you need

!*.pdf
!/a
*

Something like /c/b.pdf will still be ignored though.

If you don’t mind syncing all directories, you should be able to go for this:

!*.pdf
!*/
*

cosas · January 22, 2018, 11:21am

?

Before:

(?i)(?d)*.ini
(?i)(?d)*.lnk
(?i)(?d)*Thumbs*.db
//!2013/201305 Portugal/24
//2013/*
//!2013
!2017/201712 30-31**
2017/*
!2017
!17
*

me@lap:~/Sync/Photos$ ls -ld *
drwxrwxr-x 2 me me  4096 avril 26  2017 0 ! DON'T DEL BELOW (pour éviter les bétises)
drwxrwxr-x 2 me me  4096 déc.  23 09:51 0 scripts
drwxrwxr-x 2 me me  4096 juin  23  2017 0 supprimer incrustations
drwxrwxr-x 6 me me 69632 janv.  8 16:09 17
drwxr-xr-x 3 me me  4096 janv.  8 15:06 2017
me@lap:~/Sync/Photos$ find -iname "*mg0075.jpg"    
me@lap:~/Sync/Photos$

After:

!(?i)*mg0075.jpg
(?i)(?d)*.ini
(?i)(?d)*.lnk
(?i)(?d)*Thumbs*.db
//!2013/201305 Portugal/24
//2013/*
//!2013
!2017/201712 30-31**
2017/*
!2017
!17
*

me@lap:~/Sync/Photos$ ls -ld *
drwxrwxr-x 2 me me  4096 avril 26  2017 0 ! DON'T DEL BELOW (pour éviter les bétises)
drwxrwxr-x 2 me me  4096 déc.  23 09:51 0 scripts
drwxrwxr-x 2 me me  4096 juin  23  2017 0 supprimer incrustations
drwxrwxr-x 6 me me 69632 janv.  8 16:09 17
drwxr-xr-x 3 me me  4096 janv. 22 12:03 1996
drwxr-xr-x 5 me me  4096 janv. 22 12:03 1998
drwxr-xr-x 4 me me  4096 janv. 22 12:03 1999
drwxr-xr-x 3 me me  4096 janv. 22 12:03 2001
drwxr-xr-x 3 me me  4096 janv.  8 15:06 2017

me@lap:~/Sync/Photos$ find -iname "*mg0075.jpg"
./1999/199904 Les filles en Espagne/IMG0075.jpg
./1999/199908 Parcay/IMG0075.jpg
./2001/200102 Fanny & Théo/IMG0075.jpg
./1996/199608 Océan/IMG0075.jpg
./1998/199807 Maubuisson/IMG0075.jpg
./1998/199807 Les filles à Oléron/IMG0075.jpg
./1998/199811 ESC Grenade & Contis/IMG0075.jpg
me@lap:~/Sync/Photos$

cosas · January 22, 2018, 11:33am

@imsodin to show nothing else was sync’ed (works as hoped):

ls -R ./199[0-9]/*
./1996/199608 Océan:
IMG0075.jpg

./1998/199807 Les filles à Oléron:
IMG0075.jpg

./1998/199807 Maubuisson:
IMG0075.jpg

./1998/199811 ESC Grenade & Contis:
IMG0075.jpg

./1999/199904 Les filles en Espagne:
IMG0075.jpg

./1999/199908 Parcay:
IMG0075.jpg
me@lap:~/Sync/Photos$ ls -R ./2001/*
./2001/200102 Fanny & Théo:
IMG0075.jpg
me@lap:~/Sync/Photos$

brunod · January 22, 2018, 3:54pm

I create a new share with this structure :

ls --group-directories-first -R -C
.:
Nouveau dossier  Nouveau fichiervideLvl0.pdf  Nouveau fichiervideLvl0.txt

./Nouveau dossier:
Nouveau fichiervideLvl1.pdf  Nouveau fichiervideLvl1.txt

So reality is 2 levels, 4 files, 1 pdf at each level.

The .stignore contents what you wrote :

!*.pdf
!*/
*

The share says there is 1 file and 0 directory instead of 2 pdf and 1 folder.

imsodin · January 23, 2018, 12:45am

@cosas The problem I describe only applies to scanning, i.e. if you create your something matching *mg0075.jpg it wont be added to the index and thus not synced to anywhere else. For pulling, there is no parent directory to traverse and it is thus not skipped. That may actually be fixable (the scanning side of course), I’ll keep that in mind for after the current scanner shuffle.

@brunod Unfortunately my idea is trash, as */ only matches contents of a directory, not directories themselves. So it really looks like that you need to explicitly include every directory you want (at least for the time being).