Connectivity issues between v1 and v2 anyone?

imsodin · June 15, 2025, 8:51am

tl;dr: Is anyone else having connection issues between v1 and v2?
I am specifically only interested in that scenario. If you have issues, please leave a comment, even if you have no additional context. Of course if you do (logs, network info, …) even better.

For reasons I have one device on v2, others on v1. And while the two v1 device happily connect, they don’t to v2. And one of the v1 and v2 are on the same network, the other is on a different one. I do get occasional connects for the devices on the same network, that immediately drop, no connections to the one on the other network. It’s all pretty weird and contradictory, some of it makes me suspect the multiple-connections enabled in v2, some of it looks like user/setup error. I don’t have time to jump into debuggng this right away and thus would be nice to see if anyone else has similar issues or not - then again, not sure many people run v1 and v2 at the same time, but with apt RC users being force upgraded, maybe

Side-note:
Why are some of my devices even on v1? One is android and the other is on nightlies, and somehow nightlies stopped being published a while ago (and I didn’t notice). I see that nightly CI actions are still running, but not the puslish stages maybe?! I must admit I am not really familiar yet with that setup.

Edit: Also I do plan to look into this of course, just don’t know yet when - it basically renders my syncthing setup useless. And while I probably could just upgrade to v2 everywhere, I kinda want to root-cause/fix this, so I wont do that and thus keep the pressure on me high to look into it

calmh · June 15, 2025, 9:15am

Ah, no, but in all my juggling between different cloud operators and blob storages I fumbled and left it pointed to an old place. I’ll sort that out.

As for your other problems, I did run v1+v2 for a while and didn’t notice anything bad, but, yeah, troubleshoot it man

imsodin · June 15, 2025, 9:20am

Thanks, already good to know that you had it working - so likely just my effed up setup. Still I will troubleshoot.

Oh I gotta disable auto-upgrades then, otherwise my “live-reproducer” might get auto-upgrade-destroyed

imsodin · June 15, 2025, 9:43am

@calmh On a related note: Did you remove 1.30.0 versions from apt candidate channel intentionally? Resp. would it be annoying to re-add that, alongside v2 of course?
Just for convenience of apt users needing to handle v1 vs v2 shenanigans. I just realized that I was actually on that channel with some v1.29.6 RC installed, and something else prevented an upgrade, and I’d like to go to v1.30.0 for the debugging. Of course I can do that manually too - I suspect those “apt users needing to handle v2 vs v2 shenanigans” might come down to a single person

calmh · June 15, 2025, 9:44am

Not intentionally, just the script keeps the last n versions and I guess that means there is no space for v1 rcs now

calmh · June 15, 2025, 9:46am

The debs are in the GitHub release though for manual wrangling right now

imsodin · June 15, 2025, 9:46am

Yeah then I guess I’ll try to see where that script is, and if it’s easy enough I’ll adjust it to keep the latest one from the prior major version for a longer time - seems somewhat useful, but also not something to invest much time in.

imsodin · June 15, 2025, 9:47am

No worries about manual wrangling, that’s obviously not an issue at all - plenty of options there

calmh · June 15, 2025, 9:47am

github.com/kastelo/ezapt

internal/publish/package.go

2cd886a75


      
          		packagesDir := filepath.Dir(path)
          		packages[packagesDir] = append(packages[packagesDir], pkg)
          		return nil
          	})
          	if err != nil {
          		return nil, fmt.Errorf("scanning packages: %w", err)
          	}
          	return packages, nil
          }
          
          func trimPackages(repo string, packages map[string][]*packageFile, keepVersions int) {
          	parent := filepath.Dir(repo)
          	for dir, pkgs := range packages {
          		perPkg := make(map[string][]*packageFile)
          		for _, pkg := range pkgs {
          			perPkg[pkg.Package] = append(perPkg[pkg.Package], pkg)
          		}
          		packages[dir] = nil
          		for _, pkgs := range perPkg {
          			slices.SortFunc(pkgs, func(a, b *packageFile) int {
          				return compareVersions(a.Version, b.Version)

calmh · June 15, 2025, 9:49am

The script evolved a bit when I wanted to do it all from github actions without installing a full debian packaging toolkit and gpg etc and also wanted custom logic in there… sorry

imsodin · June 15, 2025, 9:52am

Thanks for the pointer and definitely no worries at all. Lots of things evolved over the last half year or so, much for the better - you have been very busy. Just me not keeping up so far

calmh · June 15, 2025, 10:06am

Maybe we should add an oldstable or v1 section to APT as well to keep the v1 releases.

tomasz86 · June 15, 2025, 11:52am

I can only say that we run a mix of v1 and v2 devices, however I don’t think I’ve really observed any problems with the setup caused by potential incompatibilities between the two yet. Most of the devices had also had multiple (three, to be exact) connections enabled for a long time before v2 was even born, however there are one or two devices in the mix with just a single connection set up as well.

When it comes to the network connectivity, it’s also a mix of everything here, including direct LAN, WAN, and relays.

imsodin · June 15, 2025, 10:19pm

Bit of a cliffhanger: The important info I don’t have yet, but I really need to stop now - soon

After a bit of struggling I got the syncthing logs from android (they aren’t part of logcat logs anymore?!), and it looks like this is going to be a fun one - this looks like a file I didn’t touch in ages and likely is ignored (and never was supposed to be synced):

[OURLP] 2025/06/16 00:11:31 INFO: Connection to YPOXTIL at [::]:22000-192.168.1.54:22000/quic-client/TLS1.3-TLS_AES_128_GCM_SHA256/LAN-P20-629ARKGFUKBAAR57P6O2D6MD22 closed: protocol error on index-update for 793ex-zdhfr: “Win/WinNoInstall/viking-1.1.1_win_gtk/maps/t13s0z0/71731/72483”: file with empty block list

imsodin · June 15, 2025, 10:22pm

Oh dang, timing though?! What if this is my invalid bit removal PR? I did say “hopefully no regressions this time” somewhere, I was asking for it I did run PR builds, so it might be possible the issues only started with that change. A bit weird people are running in the issue tracker yet if this is it, but then again not that many RC users. I guess I don’t get to call it a day after all and will have a closer look.

Edit: Luckily seems like not: If I pause that one folder, everything works fine. If I ignore a test file, that gets properly ignored. So back to having no clue wtf this is, maybe an edge case, maybe a leftover of a bug from years ago that surfaces now (that syncthing setup/db and file are really ancient).

calmh · June 16, 2025, 5:09am

So is it being sent from a v1 or v2 side? If v2, what does it looks like in the database? If v1, yeah, toss it

Catfriend1 · June 16, 2025, 6:19am

The Syncthing(Native) log is still there. The recent release can switch to show either Android logcat or syncthing.log (see the upper right corner menu). Then hit “share” and “Save to Syncthing”, “Save to Material files” or share the log with your note taking app, for example.

The native output only helps me develop the wrapper when output in real time to logcat. The logcat buffers are “small” so we had users come around over years handing in incomplete syncthing logs when requested because they were truncated. That’s why we now get the “huge” logfile and only output the native’s log lines to the logcat in the debug build. (Release builds don’t do this) That way, reducing logcat spam, I could also remove the “Log to file” option, as you might expect, when needed users hadn’t turned this on before a problem came up.