How should support for sphinx-style crossrefs be implemented?

[echoix]

I’m seriously considering mkdocstrings to replace our sphinx-based docs, as we are merging a mkdocs site with a sphinx site, and even if we can use a material theme equivalent for sphinx, they aren’t really integrated together and it shows.

A couple of blockers for us exist, and some of it originates from the griffe backend.

One of the most obvious one is the way links and crossrefs are handled. When using sphinx, you end up using roles with domains like :py:class:`SomeClassName` quite often. On a considerable codebase, last year counted at 391000 lines of Python code, changing these references everywhere to adapt to this unique tool would only be a lot of churn, plus making it impossible to go back, or be coherent with the style (sphinx style, but not the expected sphinx syntax as markdown links are used).

So, my goal with my exploration this weekend was to familiarize myself a bit with the griffe codebase, in the hope of finding a place where this handling of the sphinx-style (reST) cross references. But I didn’t yet see an appropriate place where the contents of docstrings apply changes to the text (that isn’t for describing paramters and types).

Thus, my question is: where should this be done? What functions are involved. What’s the design that would appropriate for this.

[pawamoy]

Interesting idea, thanks for sharing your thoughts!

The code that handles cross-references is located in autorefs (as well as in mkdocstrings itself). However I don't think I'd want to support Sphinx-style cross-references there: both autorefs and mkdocstrings expect Markdown, supporting Sphinx stuff is a non-goal.

I'd instead recommend to try writing a Python-Markdown extension that runs a pre-processor on the markup (docstrings) to transform such Sphinx-style references into Markdown inline references. From:

:py:class:`SomeClassName`

...to:

[`SomeClassName`][]

The difficulty here is that by default Sphinx-style references allow relative names while our scoped and relative references are Insiders features, so even with such an extension running, the context provided by the initial references could be insufficient, and you'd have to at least make them absolute in your sources. From:

:py:class:`SomeClassName`

...to:

:py:class:`package.module.SomeClassName`

You'd also have to handle the various features of such directives (I think they allow starting the ref with ~ to display only the last part, etc.), to translate them to what autorefs supports.

Happy to try and elaborate if this recommendation isn't enough for you to try and starting hacking on a solution 🙂

[pawamoy]

That's another task I want to make the sphinx style support

Since we cannot know what markup a docstring is using, this will be brittle, but could work in most cases. However, unless you clearly have a use-case for it (i.e. lots of occurrences where things would wrongly be interpreted in a reST code block, and a reason to keep using reST), I won't accept such a change. If you're only interested in switching to MkDocs + mkdocstrings, you'll have to convert these reST code blocks to Markdown code blocks anyway. If this feature was requested for being able to use Griffe within Sphinx (for example through https://github.com/readthedocs/sphinx-autoapi), I would consider it more openly.

Some of it I can explain as being directly interpreted as markdown, but some not

Happy to try and explain if you provide examples.

Using reST is pretty much a requirement for sphinx, and having a supported docstring style named sphinx kinda gives an expectation that at least some features should work, or at least not break too much.

No other name made sense 🤷 Emphasis on Sphinx-style, i.e. supporting documenting parameters/etc. in docstrings, not supporting Sphinx features, and even less reST.

[echoix]

Thanks for all your replies, it's helpful to position griffe with respect to other projects, and for the ideas of where and when to use extensions.

I understand that you're open to enabling only a very limited part of Sphinx features, when inside tags already parsed, and not in free-form text. (For example, the types given to the :raises:).

The Python-Markdown you were talking about, you're talking about https://python-markdown.github.io?
I didn't fully think out through where some future extensions would hook up to, and at what point in the processing steps. I assume that the correct extension points are available, or would be accepted to be available if they don't exist yet.

[pawamoy]

Thanks for your understanding and patience 🙂

you're open to enabling a very limited part of Sphinx features

Yes. Basically, I'm OK to support the syntax that lets us document symbols/concepts (parameters, return types, raised exceptions, etc.), and nothing more from Sphinx/reST.

The Python-Markdown you were talking about, you're talking about https://python-markdown.github.io/?

Yes!

With the Python-Markdown extension route, the conversion of cross-references would happen when we convert docstrings from Markdown to HTML (which happens when mkdocstrings renders templates). This could be done with a pre-processor or an inline pattern, if I remember the terms correctly. I'd recommend getting inspiration from the pre-processor and inline patterns built-into Python-Markdown's own extensions. You'd then enable the extension in mkdocs.yml, like any other Markdown extension.

With the Griffe extension route, the conversion would happen before that, when the symbols are collected from the source files. This could be done by hooking onto the on_instance event, and modifying docstrings raw values in-place.

[echoix]

With the Python-Markdown extension route, the conversion of cross-references would happen when we convert docstrings from Markdown to HTML (which happens when mkdocstrings renders templates). This could be done with a pre-processor or an inline pattern, if I remember the terms correctly. I'd recommend getting inspiration from the pre-processor and inline patterns built-into Python-Markdown's own extensions. You'd then enable the extension in mkdocs.yml, like any other Markdown extension.

With the Griffe extension route, the conversion would happen before that, when the symbols are collected from the source files. This could be done by hooking onto the on_instance event, and modifying docstrings raw values in-place.

How does it compare (before or after) to when the code in the src/_griffe/docstrings/sphinx.py file is executed?
Are they still able to access the context of function of the docstring that contains it? If I ever wanted to support the . sphinx feature (kinda related to scoped/relative) links, I think the griffe route would be best. But at the markup side too could be useful, but doesn’t it still have access to the same info?

I also thought of, but didn’t dive in deeply, but to use docutils directly as a library (like sphinx) to handle the syntax. It might be hard and might not move fast enough, but at least there’s chances to have all syntax features correctly supported.

[pawamoy]

The docstring parser in sphinx.py runs right before rendering a docstring, to split it into parsed sections, but might also run before that, during collections of Python symbols from sources, depending on what Griffe extensions are enabled and what they do.

The Python-Markdown extension wouldn't have access to the Griffe context indeed.

Yep using docutils is an interesting idea 🙂