Move docs to main repo?

Development
21

True. I’ve been unable to find something like an online wysiwyg editor that would assist in creating a PR. As far as I can tell it doesn’t exist, so the workflow would be the usual edit-on-github link or an actual source checkout.

Sticking to that, the only difference from the current setup would be the backend text format and rendering.

I’m still wondering whether a move to markdown and a standard static site generator (jekyll, hugo, …) wouldn’t be an improvement on the current sphinx setup. Both in ease of editing and in maintenance.

22

What’s the actual problem we are trying to solve here. Who was complaining that there isn’t a wysiwyg editor?

23

There’re two entirely different problems I want to solve.

The first is that I want to integrate the docs with the code more. As is, there is no good way to tie docs to a given release version and no good way to make code + docs changes (so we forget the latter). This is all for me. If I could expect API changes to also change the relevant docs in the same PR, and then later do a git diff v1.0.0..v1.5.0 -- docs/rest/ that would be hugely helpful to me.

The second is that we don’t get a lot of “external” contributions, in my opinion. There are probably lots of reasons for this and I can’t know them all, but the docs being tricky to get your head around as a newbie can’t help the situation. Currently the README talks about a weird compilation process and tools and the docs are in an odd format (that I’ve personally grown to dislike, too). I suspect a more Wiki-like approach here would help. A WYSIWYG editor probably wouldn’t hurt but I don’t think it’s essential either.

I was thinking GitBook might solve both these things in one change. I’m not necessarily convinced that’s the case any more, but those are the problem statements.

24

For the closed-source cloud part:

  • RST is more feature-full then all those markdown “standards”
  • Pandoc is your friend when the GitBook service goes offline or want to render e.g epub/html offline.
  • Not sure if Hugo static site generator is able to render something usefull from the GitBook markdown format (for offline use)

Just my ideas.

1 Like
25

I would still like to do something better with the docs.

If we don’t do something more drastic I still think moving them into the main repo is a good thing to do. The advantages would be the same as always, and I still don’t see a nightmare scenario with the issues (because there really aren’t that many opened on the docs repo).

The “community should self govern” thing seems dead in the water no matter how we do this imho, as I suspect the threshold is that people simply lack the desire / courage / empowerment to edit the docs as they are. If anything I think the added visibility from being in the main repo might help slightly, but I don’t expect it to be a measurable difference.

Doing something entirely different like having the docs on some wysiwyg wiki like editing experience on the web could increase participation. I’m not sure what platform that would be, though.

As a side note, sphinx does support markdown files. Some features don’t exist there though, notably links to named references which we use all over the place…

2 Likes
26

I do think most of our labels are on section headers, which can be done in markdown with automatic labels (https://github.com/readthedocs/recommonmark#linking-to-headings-in-other-files). And other sphinx niceties not natively supported by markdown (most of them I guess) can be done with an extension of the parser used by sphinx (https://recommonmark.readthedocs.io/en/latest/auto_structify.html#embed-restructuredtext).

I am in favor of moving our docs to the main repo and adding a PR policy to, if required, either update docs or add an issue labeled “docs” linking to the PR, thus there’s at least a clear backlog.

While I’d also prefer markdown over rst, converting will be some effort. And I doubt it will make a huge difference in attracting doc writers.

1 Like
27

I just found this topic after recently working a lot on the docs. So let me add my thoughts.

I was pretty convinced that integrating the docs repo to the main one would be a good idea. The longer I think about it, the less I’m convinced for myself. Issues may be discarded I suppose, but Git history is definitely worth keeping, so there’s challenge number one. The harder part I see is actually CI / CD setup. The docs repo has its own build and deploy pipeline, which I find appealing and pretty streamlined. Mixing that with all the checks and actions happening on the code repo means quite a lot of runtime wasted and a fair bit of complexity to keep all parts moving.

Coming from my recent endeavours on completing API docs and configuration file options, I must say I’m a bit disappointed by the lack of docs adjustments for new features from the last couple of years – even though the PR template clearly demands a link to the corresponding docs PR. Not trying to blame here, but I think this needs more attention during the review process. I for one will strictly remind people of linking a docs PR before merging anything where I see the need to mention it in the docs.

Now the attempt to find an answer to the topic’s question. Like @imsodin, I’m actually undecided. My best answer would be: Partly. With lock-step tagging of both repos and some automation we recently added, the current situation is not really terrible. I propose to make a distinction between reference documentation (essentially all our .protos and the CLI) and every other kind (FAQ, handbooks / guides, how-tos, conceptual description / whitepapers). IMHO, an API reference is utterly useless if it doesn’t describe every interface (option, endpoint, whatever). So at least mentioning new interfaces even with a crappy one-liner description should be forced for every code contribution, in the same PR. Of course, generating such rudimentary API docs automatically is the best approach and easiest when integrated with the code repo.

Cross-referencing is a big issue if we were to split the docs in such a way. I must say I have come to like Sphinx and reST for those features, otherwise I wouldn’t have started implementing a Sphinx extension for documenting Syncthing’s config file. It’s immensely useful to add cross-references from prose text to the option / API references. But we could maybe still do that by pulling the reference docs from the main repo during the docs build procedure. I have no detailed solution in mind, so just putting the thought out there for further pondering.

Regarding editing format / experience, certainly the “guides” part could use some more human-friendly approach, maybe even a curated Wiki (while taking snapshots at release tagging time). But as long as we have a procedure for importing the updated reference docs to the user docs repo / whatever (kind of like the cycle for Transifex), allowing cross-reference links there should be doable.

Finally, whatever we end up changing here, I still seriously doubt that even a Wiki will lead to more independent contributions. So overall, every contributor just needs to apply a good measure of self-discipline to make rudimentary docs adjustments when adding features. And from time to time, make the jump and do some writing to extend those stubs into usable user docs. I’m trying to lead by example on that appeal :wink:

28

I don’t think these are insurmountable challenges.

  • GitHub supports moving issues from one repo to another; we can move all the existing docs issues to the main repo, adding a docs tag in the process.
  • We can graft a branch from another repo into the main repo, preserving all commits and history, so we wouldn’t lose the docs change history (I would consider that unacceptable). We’ve done this before with things like the discovery server and whatnot that were originally separate repos.
  • CI builds can be no-ops or skipped if nothing relevant was changed. For example, Netlify skips builds if the directory containing the docs were unchanged in a given commit. Our regular builds can be easily taught to not do builds when only the docs dir changed, etc.

So all those things being equal I think it just boils down to our (desired) process.

I hadn’t considered this option. It could be good. One of the things I actually like about having docs in a separate repo is that it enables different commit policies (in terms of requiring reviews, who has access to do what, etc.), where I think it’s useful to be more relaxed on the docs repo than on the main repo. Doing a split could preserve some of that while still locking API docs to the corresponding code.

2 Likes