Move docs to main repo?

My proposal includes adding all current documentation contributors to the org, so extending the relevant community accordingly. It still means a contribution needs one more person to do the blessing, but it doesn’t have to be you. I think there are enough people happy to do the blessing, but as it is there is limited attention on the docs repo today PRs end up stale.

If anything, I think the docs repo is the one that is harder to find than the main one.

But we may need to just agree to disagree on this one. I’m not barging ahead with it just yet. :slight_smile:

I’m trying to quote the post of @AudriusButkevicius but could not find the same way you have commented @calmh (missing knowledge of the forum probably.

But I agree on the govern point of @AudriusButkevicius. You could also close the issue tracker in the docs repo and let all issue go into the main repository. If you look at the golang.org/x libraries it is IMHO also handled in this way. E.g https://github.com/golang/net -> has issue in golang/go.

The main issue tracker for the net repository is located at https://github.com/golang/go/issues. Prefix your issue with “x/net:” in the subject line, so it is easy to find.

I don’t understand the “harder to govern” point so I didn’t respond to it. What does it mean?

(To quote stuff, select it - a “quote” button appears.)

I think another point for moving the docs into the main repo is that a certain subset of support-like issues that are created, and now just closed, are in fact legitimate “x should be [better] documented” issues.

1 Like

Why not create a forum poll on the decision by the community instead of tossing coins :smiley:

I think a non-developer’s opinion (i.e., 98% of the users here) is less relevant to this question.

3 Likes

FWIW, this is the canonical example of a question that should have been answerable by diffing the relevant docs between two releases:

As it is I can git diff v0.14.43..v0.14.49 -- lib/config and get a feeling, but the changes are much more extensive than just the user facing stuff and harder to mentally aggregate. (I know, because I’ve done exactly this for a client.)

I’d like to revisit this topic with a new proposal. The GitBook system seems quite attractive.

It’s a cloud service with hosting, decent search, and a nice WYSIWYG editor that can be used directly on the pages themselves with just a GitHub (linked) account. The backend is proprietary, but syncs to a bunch of markdown files in the GitHub repo. Edits can also be made to those markdown files and are then directly visible on the web.

Using this would buy us a few advantages:

  • Users can edit the documentation WYSIWYG on the documentation site, instead of having to learn the fairly arcane RST format and make a pull request. (This requires people to sign up / sign in on GitBook and be given permission. I’m not sure exactly how that flow works.)

  • Anyone can still edit the “source” in the form of markdown files in the repo. Markdown is, admittedly, a little less expressive than RST. It’s also easier to deal with…

  • With the docs in the main repo, the web rendering picks up the current version from tags and you can switch between branches to see docs for older versions. At the same time, docs changes can be done at the same time as code changes, if you’re a dev.

  • The graphical design is somewhat more up to date than our current one.

  • The search actually seems to work.

Certainly there are some downsides too:

  • We need to convert from RST to markdown. For many pages this is trivial, but requires some manual hands on as well.

  • We need to tweak the process to generate man pages from the new docs. This is doable.

  • It’s a random cloud service that might go bust at any time. But they have a few large users, and we have a structure of markdown files in our repo, so worst come to worst this is trivial to again generate a static site from.

I added a proof of concept at https://gb.syncthing.net/ with just a couple of pages converted (i.e., most of the links still point back to docs.s.n.). It syncs to my syncthing fork repo.

Changes done on the web show up as meaningless but mostly harmless commits in the repo, like this. It’s no worse than what happens with the edit-on-GitHub option today (which is also still available, for that matter). Arguably better as it at least has a commit prefix where the GitHub default is just “Updated foo.md”.

Main points I see as advantagaeous is getting a WYSIWYG editor instead of RST in front of people, and getting the code and docs in sync version wise.

Theoretically, this invitation link should grant anyone write access to the pages: https://www.gitbook.com/invite/syncthing?invite=-LVSdj-7zBoQIrsT283S

2 Likes

I think everything that increases the chances for documentation being written is good. I haven looked at the tool yet, because as when I try to join via the link, I get get greeted by this nicely translated message:

On github we can change it on merge and GitBook: [master] one page modified is quite crappy - I mean of course something was modified, otherwise there wouldn’t be a commit, but what is modified (filename or something at least). However I agree that’s not a showstopper.

How is the review process going to look like? Still github PR or directly done on that external site.

“Imposible”, huh… :man_facepalming:

I agree on the commit messages. There’s a thing in the edit GUI where one can enter a change description, which I’d imagine becomes the commit message. It doesn’t, though, so that sucks.

Access wise, once you are added on GitBook you can just hit edit, save, publish and it will push to GitHub with a bot. This bot would need push access and it wouldn’t go through a PR.

But this requires being added to the organization on GitBook itself separately from GitHub, so it’s not really as casual as I’d maybe hoped. In fact the “edit on GitHub” link instead takes you to the file on GitHub as usual, and the edit flow from there would be the same as currently. So there’d be both ways, with the WYSIWYG one being less accessible to new users.

(That said, I think most new users will still have an easier time with markdown than rst. Even I hardly know how to make a link or monospace text after all this time and need to look it up by example.)

There’s also the thing that hints and warnings that are easy to do in the GUI become somewhat ugly {% hint style="danger" %} foo {% endhint %} things that I’m sure other markdown parsers will barf on…

All in all I like the rendering but I’m not sure it’s as awesome as I thought. Still, it’s no worse than a static site generated from the repo, and it still has neat things like version selection based on branches.

You can’t join, because the Syncthing organization is currently on “Free”, which only allows 2 users. We need “Startup” for unlimited, which is free for open source projects.

Right, yes. I’ve applied for such, but it hasn’t happened.

I don’t really like this, it essentially means you trust a user entirely (direct push access) or you don’t trust him at all (no invite). There have been good new user contributions, but they often needed tweaking and sometimes even had some minor misconceptions in them.

True. I’ve been unable to find something like an online wysiwyg editor that would assist in creating a PR. As far as I can tell it doesn’t exist, so the workflow would be the usual edit-on-github link or an actual source checkout.

Sticking to that, the only difference from the current setup would be the backend text format and rendering.

I’m still wondering whether a move to markdown and a standard static site generator (jekyll, hugo, …) wouldn’t be an improvement on the current sphinx setup. Both in ease of editing and in maintenance.

What’s the actual problem we are trying to solve here. Who was complaining that there isn’t a wysiwyg editor?

There’re two entirely different problems I want to solve.

The first is that I want to integrate the docs with the code more. As is, there is no good way to tie docs to a given release version and no good way to make code + docs changes (so we forget the latter). This is all for me. If I could expect API changes to also change the relevant docs in the same PR, and then later do a git diff v1.0.0..v1.5.0 -- docs/rest/ that would be hugely helpful to me.

The second is that we don’t get a lot of “external” contributions, in my opinion. There are probably lots of reasons for this and I can’t know them all, but the docs being tricky to get your head around as a newbie can’t help the situation. Currently the README talks about a weird compilation process and tools and the docs are in an odd format (that I’ve personally grown to dislike, too). I suspect a more Wiki-like approach here would help. A WYSIWYG editor probably wouldn’t hurt but I don’t think it’s essential either.

I was thinking GitBook might solve both these things in one change. I’m not necessarily convinced that’s the case any more, but those are the problem statements.

For the closed-source cloud part:

  • RST is more feature-full then all those markdown “standards”
  • Pandoc is your friend when the GitBook service goes offline or want to render e.g epub/html offline.
  • Not sure if Hugo static site generator is able to render something usefull from the GitBook markdown format (for offline use)

Just my ideas.

1 Like

I would still like to do something better with the docs.

If we don’t do something more drastic I still think moving them into the main repo is a good thing to do. The advantages would be the same as always, and I still don’t see a nightmare scenario with the issues (because there really aren’t that many opened on the docs repo).

The “community should self govern” thing seems dead in the water no matter how we do this imho, as I suspect the threshold is that people simply lack the desire / courage / empowerment to edit the docs as they are. If anything I think the added visibility from being in the main repo might help slightly, but I don’t expect it to be a measurable difference.

Doing something entirely different like having the docs on some wysiwyg wiki like editing experience on the web could increase participation. I’m not sure what platform that would be, though.

As a side note, sphinx does support markdown files. Some features don’t exist there though, notably links to named references which we use all over the place…

2 Likes

I do think most of our labels are on section headers, which can be done in markdown with automatic labels (https://github.com/readthedocs/recommonmark#linking-to-headings-in-other-files). And other sphinx niceties not natively supported by markdown (most of them I guess) can be done with an extension of the parser used by sphinx (https://recommonmark.readthedocs.io/en/latest/auto_structify.html#embed-restructuredtext).

I am in favor of moving our docs to the main repo and adding a PR policy to, if required, either update docs or add an issue labeled “docs” linking to the PR, thus there’s at least a clear backlog.

While I’d also prefer markdown over rst, converting will be some effort. And I doubt it will make a huge difference in attracting doc writers.

1 Like

I just found this topic after recently working a lot on the docs. So let me add my thoughts.

I was pretty convinced that integrating the docs repo to the main one would be a good idea. The longer I think about it, the less I’m convinced for myself. Issues may be discarded I suppose, but Git history is definitely worth keeping, so there’s challenge number one. The harder part I see is actually CI / CD setup. The docs repo has its own build and deploy pipeline, which I find appealing and pretty streamlined. Mixing that with all the checks and actions happening on the code repo means quite a lot of runtime wasted and a fair bit of complexity to keep all parts moving.

Coming from my recent endeavours on completing API docs and configuration file options, I must say I’m a bit disappointed by the lack of docs adjustments for new features from the last couple of years – even though the PR template clearly demands a link to the corresponding docs PR. Not trying to blame here, but I think this needs more attention during the review process. I for one will strictly remind people of linking a docs PR before merging anything where I see the need to mention it in the docs.

Now the attempt to find an answer to the topic’s question. Like @imsodin, I’m actually undecided. My best answer would be: Partly. With lock-step tagging of both repos and some automation we recently added, the current situation is not really terrible. I propose to make a distinction between reference documentation (essentially all our .protos and the CLI) and every other kind (FAQ, handbooks / guides, how-tos, conceptual description / whitepapers). IMHO, an API reference is utterly useless if it doesn’t describe every interface (option, endpoint, whatever). So at least mentioning new interfaces even with a crappy one-liner description should be forced for every code contribution, in the same PR. Of course, generating such rudimentary API docs automatically is the best approach and easiest when integrated with the code repo.

Cross-referencing is a big issue if we were to split the docs in such a way. I must say I have come to like Sphinx and reST for those features, otherwise I wouldn’t have started implementing a Sphinx extension for documenting Syncthing’s config file. It’s immensely useful to add cross-references from prose text to the option / API references. But we could maybe still do that by pulling the reference docs from the main repo during the docs build procedure. I have no detailed solution in mind, so just putting the thought out there for further pondering.

Regarding editing format / experience, certainly the “guides” part could use some more human-friendly approach, maybe even a curated Wiki (while taking snapshots at release tagging time). But as long as we have a procedure for importing the updated reference docs to the user docs repo / whatever (kind of like the cycle for Transifex), allowing cross-reference links there should be doable.

Finally, whatever we end up changing here, I still seriously doubt that even a Wiki will lead to more independent contributions. So overall, every contributor just needs to apply a good measure of self-discipline to make rudimentary docs adjustments when adding features. And from time to time, make the jump and do some writing to extend those stubs into usable user docs. I’m trying to lead by example on that appeal :wink:

I don’t think these are insurmountable challenges.

  • GitHub supports moving issues from one repo to another; we can move all the existing docs issues to the main repo, adding a docs tag in the process.
  • We can graft a branch from another repo into the main repo, preserving all commits and history, so we wouldn’t lose the docs change history (I would consider that unacceptable). We’ve done this before with things like the discovery server and whatnot that were originally separate repos.
  • CI builds can be no-ops or skipped if nothing relevant was changed. For example, Netlify skips builds if the directory containing the docs were unchanged in a given commit. Our regular builds can be easily taught to not do builds when only the docs dir changed, etc.

So all those things being equal I think it just boils down to our (desired) process.

I hadn’t considered this option. It could be good. One of the things I actually like about having docs in a separate repo is that it enables different commit policies (in terms of requiring reviews, who has access to do what, etc.), where I think it’s useful to be more relaxed on the docs repo than on the main repo. Doing a split could preserve some of that while still locking API docs to the corresponding code.

2 Likes