Version-specific documentation proposal

Hi folks,

our Syncthing documentation currently only has one version deployed, the “main” branch head. I think it would make sense to keep an archive of built HTML for each tagged release. The tags are already applied on the docs Git repo, but not deployed to the webserver for accessing historic versions.

Let’s assume we have each built HTML set under https://docs.syncthing.net/v1.XX.Y/, with the same internal structure. This can coexist nicely with the generic “head” version.

Then I’d add some logic to the GUI to mangle all links and point directly to the URL for the currently running version. A proof of concept implementation is in this branch. It tries to check whether the specific version subdirectory exists, falling back to the generic “head” URL on error. (There are still some CORS issues with that part.)

In the end, we could patch in a version selector to the deployment on the webserver to allow live switching between versions, like e.g. on readthedocs.

What do you think of the idea? I can work out a PR, but need help with the deployment side of things.

  • Is it worthwhile?
  • Is it a sane implementation approach?
  • Does the automatic check HttpRequest make sense or do you see security / other concerns (CORS etc.)?

I did a thing a few years back with a versioning plugin for Sphinx, which resulted in a version selector down in the corner. There were issues, which is why it’s not currently still deployed, but I forget the specifics. It interacted badly with some other plugin? It broke some links? It did the building of all versions at every build? I don’t remember. Anyway, that’s not what you’re proposing.

Do people regularly want to browse old documentation versions? Why? :slight_smile:

Seems kind of intrusive, honestly (the link mangling and checking and stuff). As a possible alternative, we could tag on a ?version=1.2.3 and let some sort of server side magic handle the redirecting? If there’s no magic or the version doesn’t exist the server will just return the main version. The magic could even be client-side (Javascript on the docs site) with a top notice just saying “you seem to be on version Y while the docs are for version Z, you might try [this URL] instead if you want” with a link to the version-specific ones…

Or, we could include a full copy of the docs and serve it locally, so that if you have Syncthing you have the corresponding docs… (http://localhost:8384/docs/) It’s about one meg compressed, looks like.

Bonus points if we once and for all integrate the docs into the same repo and build them together with the binaries:wink:

1 Like

One problem with this approach is that you need to have Syncthing running first before being able to access the docs. Now, if the user is unable to even start Syncthing in the first place and wants to use the docs to find a solution, they will still be out of luck.

Of course, to remedy this problem, an additional hard copy of the docs could simply be added to the archive for each release, which already contains FAQ and Getting Started as PDFs.

Yeah, that’s a topic I didn’t want to touch on in a first post :rofl: But still on the agenda for the mid / long term.

I don’t think they want to browse old versions, but it’s unfortunate that if someone with an older version (for whatever reason) clicks on the link, they read descriptions that may no longer / not yet apply to the software in front of them.

Implementation wise, I wanted to keep it dead simple by just adding static HTML directories on the server. The fallback to a generic version could also be handled in the webserver config, and we just hard-code https://docs.syncthing.net/v1.18.6/... into the release’s GUI code. Or maybe the ?version=... suffix you mentioned, whatever is easier to configure on the server. Note that the version parameter needs to be retained for links inside the documentation, which is way easier by using a different base URL. Or the server rewrites it into the path and redirects to that: https://docs.syncthing.net/users?version=v1.18.6 becomes https://docs.syncthing.net/v1.18.6/users.

Do you have a simpler idea how to “patch” the links client-side? The JS method seems quite efficient to me. The pre-checking can be skipped of course, if it’s handled on the server side.

Let me think about the “local included docs copy” a bit. Not sure of that yet.

It’s not meant to replace the web site. Just the link targets when the GUI is already running. Everything else will always be served by going to the website, but then the version won’t matter much.

Looking again at your branch I don’t think it’s horrible. I don’t think we should do client side probing though, for various reasons. Using a query parameter instead of a path parameter has the advantage that it works even if the server doesn’t support the newfangled stuff (we can deploy it now), or if we decide in the future that multiple documentation versions were a horrible idea and remove it entirely (as we’ve already done once). (A complexity in the URL rewriter possibly when we have somepage#anchor links that need to be come somepage?version=1.2.3#anchor but not impossible to handle.)

If the server does support it and decides to provide a specific version, then it would need to redirect to a new base path or something, yes. One possible way of handling that while remaining a static site could be to generate a javascript as part of the documentation build, which recognizes the specific versions that exists and does the redirect when it sees that query parameter.

I never said that :wink:. I just think it would be nice to have a hard copy saved like that in addition to everything else. Websites come and go, while a local file is set in stone and can be stored by the user basically forever.

The anchor / “fragment identifier” is not sent to the server at all, it’s only a client thing. So no problem there.

I think either way (parameter or path) will be manageable in the server. Semantically, the query parameter is more ambiguous: https://docs.syncthing.net/users/untrusted?version=v0.14.0 will lead where? We’re asking for a specific version of a document, which did not exist at the time. The path variant inverts that hierarchy, matching what really is deployed: a directory per version filled with all documents that were generated at the time. The HTTP response is pretty clear for an “object which does not exist” in contrast to “I can give you that, but your query parameter cannot be satisfied”. However, that’s more or less an esoteric difference. Real Syncthing web GUIs don’t contain links for features which weren’t included in that version :slight_smile:

So let’s start with the simplest: Build and deploy every tagged version in the docs repo to a (static) subdirectory of the same name on the web server.

Then we look at the query parameter stuff server-side, either in the server config (quickest) or as a JS function (more visible). It allows great flexibility in mapping version strings to tags (beyond my simple “up to first dash” approach). Then again, rewriting path components is also possible on the server. By the way, python uses a path as well: https://docs.python.org/3.6/library/crypt.html

The client side will be easier with the static paths, because we don’t need to splice in the parameter before the #anchor. I definitely prefer it because directories (and maybe symlinks?) are dead-simple and need no debugging.

Definitely helpful. It goes a bit against the grain of “only one binary file for each release”. But we could add a PDF as another release artifact maybe?

Some context on the server side; it’s a static site served currently by Netlify, rebuilt on each push. So there’s currently no server side to reconfigure, though they have some functionality to redirect based on some patterns. I don’t think we can rebuild all versions on each deploy due to resource constraints, and anyway we’d want to keep the old built docs to retain time stamps, the effect of using the version of sphinx in use at the time, etc. So I think we’d need to stash each built version into some repo / s3 bucket / azure storage / whatnot in order to accumulate versions over time.

We might migrate to something like that to serve the site entire, then just push a new version dir and updated root on every release.

Also not to forget that we have clients out there with a myriad of unreleased version numbers which will never have corresponding docs (1.18.5-rc2.47.g1234567 and so on), which I think is one reason to prefer a query string.

I feel like a dinosaur from the age of httpd.conf, DocumentRoot and mod_rewrite where you’d just point an Apache instance at a directory tree full of HTML files and PHP scripts :wink: Been some time since I had to deal with web server deployments and I don’t have much of an idea how Netlify works. So it won’t allow to just put static files on the “webspace”?

I agree on building every historic tagged version now with the current Sphinx version, unless you happen to have snapshots of the older contents. In the future, let’s just create an HTML archive whenever the last tagged release version got built. Definitely no rebuilding of historic versions needed IMHO, neither any dynamic generation.

As for the tags, I’d limit it to the “base” version, basically only the vX.Y.Z form. Everything else is unofficial or just an enhancement over the last base version. Let’s not get lost in trying to cover every possible version string. Such things will get easier when we merge the repos, but at least until then…

My code strips down the string to the base version on the client-side right now. I don’t see much benefit in processing the complete string from a query on the server. The only thing I’d do there is to fall back to stripping the first path component when it doesn’t exist. But what can be easily implemented really depends on how the server configuration / scripting / etc. works. We might switch the server infrastructure overall someday, so I’d limit the eventual porting work to a necessary minimum.

I’ve opened gui: Links to documentation should point to matching version · Issue #8107 · syncthing/syncthing · GitHub to keep track of the feature proposal and progress. Happy to continue the discussion here though.

Yes, this is precisely what it does. What it doesn’t do is allow dynamic processing to look at the first path component and check if it looks like a version string, etc. Hence why I suggest a query string, as that allows docs to be served with or without server side intelligence that actually interprets it. It does allow redirects based on static patterns though. Including on query parameters though it’s not 100% obvious to me how their example works out. We’re not married to Netlify but I think it’s safe to say we’ll always have some similar functionality in whatever hosting environment we end up putting the docs.

BTW, in the issue,

[The docs are] a moving target however, documenting stuff that might not yet be found in the currently running version or missing features that have been removed / reworked in a subsequent release and already was removed documentation.

I mean, that’s is pretty unusual, I don’t remember last time we actually removed a feature. Our good friend ignore-deletes might be on the chopping block, but that’s not an easy removal. In the other direction, new features are usually tagged with “added in 1.15.0” and such, and I’d argue it doesn’t hurt the user to see that if they come from a version in the distant past.

(Many docs updates are also to just improve the wording or clarity of the docs, and then it’s actually a good thing that you get served the latest.)

I would like to disagree with this point.

RAM and CPU utilisation stats were removed not so long ago. Even if not removed per se, maxConcurrentScans was replaced with maxFolderConcurrency, which has effectively made the previously well-working option almost unusable for someone like myself (i.e. low bandwidth and slow I/O at the same time).

References to both of the above were wiped out of the Docs at the same time and are no longer to find in the current iteration, unless you search through the repo itself in commits and PRs.

1 Like

Fair enough. Though,

It’s been two years. You should let go and move on. :wink:

2 Likes

Yeah, to be fair, I keep reverting the commit, so that I can display the stats in the current Syncthing versions, but I don’t really look at them very often. Still, I find them neat just to verify that everything is working as intended, especially on remotely administered devices.

However, I’d say the real necessity to have access to version-numbered documentation isn’t for people like myself (who can search and find the required information anyway), but more likely for all those Debian and Ubuntu users that are stuck on very old versions of Syncthing (and, at least in some cases, can’t or don’t want to use any 3rd party repos to download their software).

2 Likes

I did some experiments with cloud storage, but it was cumbersome. Contrary to what I expected it turned out quite nicely to stash the rendered versions in a git repo (since there’s a lot of duplicated content between versions), and cloning that into production deploys. So now there are things like 1.18.6 docs and 1.0.0 docs to browse. Just need to add a step to the release process to fire off the render-and-commit-new-docs job.

3 Likes

First of all, thanks for jumping in on the effort @calmh, and working on the server-side stuff! :slight_smile:

Right now I’m leaning toward a JavaScript based solution actually. That would make it independent from any hosting logic capabilities. And easier to discover / maintain / understand what happens. I’ll see what I can come up with code wise. It would also fail gracefully with the query parameter, even for someone who has JS disabled by default (unlikely if they use the Web GUI).

That is true for the biggest chunk of the documentation, where concepts are explained and tutorials / guides given. I’m currently thinking about the more technical parts, such as configuration options and format. There may be subtle differences between versions or it could just cause confusion to find an option in the docs which won’t work, or not find any mention of an old option which the running version still has. The concurrency settings pullers, maxPending... etc. are a good example of such changes. As you may have guessed, the end goal of my link targets effort is to have actually working links for every config option, as @tomasz86 has previously tried. For such things, a 1:1 mapping is really helpful so we don’t have to worry about keeping link backward compatibility in the distant future.

Regarding the wording improvements, if you want the latest and greatest of some software, you need to update to a recent version. I’d apply the same for the docs. And I think it’s quite usual to ship a frozen docs version with each software release. Which we kind of did already by tagging the docs source repo. Thanks for making the HTML available now!

1 Like

This is a brilliant effort on all fronts! I can’t contribute much to the effort except to cheer from the sidelines… So, thanks!

I do like the idea of having a static copy of the docs included in the “install”, but I’m not sure how that would work. It could be very helpful for servers that are not internet connected (I operate a few such), but that’s probably a minor segment of the audience.

3 Likes

@acolomb Could you assist with what the current process is to do a proper docs release? For 1.19.1 I pulled the latest straight docs build (from Log in to TeamCity — TeamCity which should be equivalent to the netlify build) but I see that it didn’t get a version selector at the top. I tried the Makefile in the pre-rendered repo, but it’s not happy:

jb@ses:..ng/docs-pre-rendered % git clean -fxd
jb@ses:..ng/docs-pre-rendered % git reset --hard
HEAD is now at 1c35395 Add v1.19.1
jb@ses:..ng/docs-pre-rendered % rm -rf v1.19.1
jb@ses:..ng/docs-pre-rendered % mkdir v1.19.1
jb@ses:..ng/docs-pre-rendered % make v1.19.1
Makefile:24: Keeping temporary source checkout in /var/folders/rl/f379b1ys075ft5tb2kbp34w80000gn/T/tmp.A1LvSvvw/syncthing-docs
git clone https://github.com/syncthing/docs /var/folders/rl/f379b1ys075ft5tb2kbp34w80000gn/T/tmp.A1LvSvvw/syncthing-docs
Cloning into '/var/folders/rl/f379b1ys075ft5tb2kbp34w80000gn/T/tmp.A1LvSvvw/syncthing-docs'...
remote: Enumerating objects: 4760, done.
remote: Counting objects: 100% (511/511), done.
remote: Compressing objects: 100% (355/355), done.
remote: Total 4760 (delta 279), reused 295 (delta 152), pack-reused 4249
Receiving objects: 100% (4760/4760), 10.54 MiB | 7.71 MiB/s, done.
Resolving deltas: 100% (3015/3015), done.
cd /var/folders/rl/f379b1ys075ft5tb2kbp34w80000gn/T/tmp.A1LvSvvw/syncthing-docs && \
		git checkout -f v1.19.1 && \
		cat /Users/jb/dev/syncthing/docs-pre-rendered/_patches/0001-Redirect-to-version-specific-subdirectory-from-query.patch /Users/jb/dev/syncthing/docs-pre-rendered/_patches/0002-Fix-JS-warnings-about-not-well-formed-XML.patch /Users/jb/dev/syncthing/docs-pre-rendered/_patches/0003-Add-a-header-note-allowing-to-pick-a-specific-versio.patch | git am --3way
Note: switching to 'v1.19.1'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 636c32a rest: Document missing endpoints (#727)
Applying: Redirect to version-specific subdirectory from query parameter. (#713)
Using index info to reconstruct a base tree...
M	conf.py
Falling back to patching base and 3-way merge...
Auto-merging conf.py
CONFLICT (add/add): Merge conflict in _static/version_redirect.js
Auto-merging _static/version_redirect.js
error: Failed to merge in the changes.
Patch failed at 0001 Redirect to version-specific subdirectory from query parameter. (#713)
hint: Use 'git am --show-current-patch=diff' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
make: *** [v1.19.1] Error 128
jb@ses:..ng/docs-pre-rendered %

I think the ideal state would be that the docs by default include the version selector, and we can simply take a frozen snapshot of the docs build at release time and include that among the pre-rendered versions.

(Medium term we might want to automate some of this, perhaps with a GitHub action that runs when a tag is pushed to the docs repo…)

That’s the current state. You don’t need to use the Makefile from the docs-pre-rendered repo. In fact, v1.19.1 is already available for switching on the docs.syncthing.net deployment.

I can’t check what was built on TeamCity, lacking an account there. What I used for v1.19.0 was my local machine’s installation, documented in the commit message. But what you imported in https://github.com/syncthing/docs-pre-rendered/commit/1c35395cad940978d7ba99a2fb740fbf6fe3cb19 looks fine.

Where did you see that? It depends on the versions.json file being available in the top level path (/versions.json) of the calling URL. That is usually taken care of from netlify-build.sh, together with pulling the historic versions. And it must be served from a webserver, file:// URLs don’t work because of CORS stuff.

For automation, I think pushing the latest build after creating a tag in Syncthing/docs would work. It will get deployed to the main site on the first following commit. Which kind of makes sense, because latest and the most recent tag are identical until then. If we want the tag to be visible in the selector immediately, a commit on Syncthing/docs-pre-rendered must trigger a new build on Syncthing/docs to pick it up.

About the Makefile build failure, let me check how we can trick Git into ignoring a patch series where the desired end result matches the original state (all patches already applied, but it currently bails out with conflict on the first).