From: Vladimír Vondruš Date: Thu, 22 Aug 2019 20:18:12 +0000 (+0200) Subject: m.sphinx: implement intersphinx read support. X-Git-Url: https://www.chiark.greenend.org.uk/ucgi/~cjwatson/git?a=commitdiff_plain;h=4937298ceb278769bae07601328618a28c0210a7;p=blog.git m.sphinx: implement intersphinx read support. Intersphinx write support next. --- diff --git a/doc/documentation/python.inv b/doc/documentation/python.inv new file mode 100644 index 00000000..a08de13a Binary files /dev/null and b/doc/documentation/python.inv differ diff --git a/doc/plugins.rst b/doc/plugins.rst index f5bf471c..00b82f7d 100644 --- a/doc/plugins.rst +++ b/doc/plugins.rst @@ -64,7 +64,6 @@ below or :gh:`grab the whole Git repository `: - :gh:`m.metadata ` :label-flat-primary:`pelican only` - :gh:`m.sphinx ` - :label-flat-warning:`python doc only` For the `Python doc theme <{filename}/documentation/python.rst>`_ it's enough to simply list them in :py:`PLUGINS`. For the `Doxygen theme <{filename}/documentation/doxygen.rst>`_, diff --git a/doc/plugins/sphinx.rst b/doc/plugins/sphinx.rst index 0b65ed0c..571771fa 100644 --- a/doc/plugins/sphinx.rst +++ b/doc/plugins/sphinx.rst @@ -48,27 +48,111 @@ using external files in a way similar to `Sphinx `_ `How to use`_ ============= +`Pelican`_ +---------- + +Download the `m/sphinx.py <{filename}/plugins.rst>`_ file, put it including the +``m/`` directory into one of your :py:`PLUGIN_PATHS` and add ``m.sphinx`` +package to your :py:`PLUGINS` in ``pelicanconf.py``. The plugin uses Sphinx +inventory files to get a list of linkable symbols and you need to provide +list of tuples containing tag file path, URL prefix, an optional list of +implicitly prepended paths and an optional list of CSS classes for each link in +:py:`M_SPHINX_INVENTORIES`. Every Sphinx-generated documentation contains an +``objects.inv`` file in its root directory (and the root directory is the URL +prefix as well), for example for Python 3 it's located at +https://docs.python.org/3/objects.inv. Download the files and specify path to +them and the URL they were downloaded from, for example: + +.. code:: python + + PLUGINS += ['m.sphinx'] + M_SPHINX_INVENTORIES = [ + ('sphinx/python.inv', 'https://docs.python.org/3/', ['xml.']), + ('sphinx/numpy.inv', 'https://docs.scipy.org/doc/numpy/', [], ['m-flat'])] + `Python doc theme`_ ------------------- -List the plugin in your :py:`PLUGINS`. +List the plugin in your :py:`PLUGINS`. The :py:`M_SPHINX_INVENTORIES` +configuration option is interpreted the same way as in case of the `Pelican`_ +plugin. .. code:: py PLUGINS += ['m.sphinx'] + M_SPHINX_INVENTORIES = [...] + +`Links to external Sphinx documentation`_ +========================================= + +Use the :rst:`:ref:` interpreted text role for linking to symbols defined in +:py:`M_SPHINX_INVENTORIES`. In order to save you some typing, the leading +name(s) mentioned there can be omitted when linking to given symbol. + +Link text is equal to link target unless the target provides its own title +(such as documentation pages), function links have ``()`` appended to make it +clear it's a function. It's possible to specify custom link title using the +:rst:`:ref:`link title ``` syntax. If a symbol can't be found, a +warning is printed to output and link target is rendered in a monospace font +(or, if custom link title is specified, just the title is rendered, as normal +text). You can append ``#anchor`` to ``link-target`` to link to anchors that +are not present in the inventory file, the same works for query parameters +starting with ``?``. Adding custom CSS classes can be done by deriving the role +and adding the :rst:`:class:` option. + +Since there's many possible targets and there can be conflicting names, +sometimes it's desirable to disambiguate. If you suffix the link target with +``()``, the plugin will restrict the name search to just functions. You can +also restrict the search to a particular type by prefixing the target with a +concrete target name and a colon --- for example, +:rst:`:ref:`std:doc:using/cmdline`` will link to the ``using/cmdline`` page of +standard documentation. + +The :rst:`:ref:` a good candidate for a `default role `_ +--- setting it using :rst:`.. default-role::` will then make it accessible +using plain backticks: + +.. code-figure:: + + .. code:: rst + + .. default-role:: ref + + .. role:: ref-flat(ref) + :class: m-flat + + - Function link: :ref:`open()` + - Class link (with the ``xml.`` prefix omitted): :ref:`etree.ElementTree` + - Page link: :ref:`std:doc:using/cmdline` + - :ref:`Custom link title ` + - Flat link: :ref-flat:`os.path.join()` + - Link using a default role: `str.partition()` + + .. default-role:: ref + + .. role:: ref-flat(ref) + :class: m-flat + + - Function link: :ref:`open()` + - Class link (with the ``xml.`` prefix omitted): :ref:`etree.ElementTree` + - Page link: :ref:`std:doc:using/cmdline` + - :ref:`Custom link title ` + - Flat link: :ref-flat:`os.path.join()` + - Link using a default role: `str.partition()` -.. note-info:: +.. note-success:: - This plugin is available only for the `Python doc theme <{filename}/documentation/python.rst>`_, - not usable for Pelican or Doxygen themes. + For linking to Doxygen documentation, a similar functionality is provided + by the `m.dox <{filename}/plugins/links.rst#doxygen-documentation>`_ + plugin. `Module, class, enum, function, property and data docs`_ ======================================================== -The :rst:`.. py:module::`, :rst:`.. py:class::`, :rst:`.. py:enum::`, -:rst:`.. py:function::`, :rst:`.. py:property::` and :rst:`.. py:data::` -directives provide a way to supply module, class, enum, function / method, -property and data documentation content. +In the Python doc theme, the :rst:`.. py:module::`, :rst:`.. py:class::`, +:rst:`.. py:enum::`, :rst:`.. py:function::`, :rst:`.. py:property::` and +:rst:`.. py:data::` directives provide a way to supply module, class, enum, +function / method, property and data documentation content. Directive option is the name to document, directive contents are the actual contents; in addition all the directives have the :py:`:summary:` option that diff --git a/plugins/m/sphinx.py b/plugins/m/sphinx.py index 56a594e7..3f3555b6 100644 --- a/plugins/m/sphinx.py +++ b/plugins/m/sphinx.py @@ -22,8 +22,20 @@ # DEALINGS IN THE SOFTWARE. # +import logging +import os +import re +from typing import Dict +from urllib.parse import urljoin +import zlib + +from docutils import nodes, utils from docutils.parsers import rst from docutils.parsers.rst import directives +from docutils.parsers.rst.roles import set_classes +from docutils.parsers.rst.states import Inliner + +from pelican import signals module_doc_output = None class_doc_output = None @@ -126,7 +138,148 @@ class PyData(rst.Directive): } return [] -def register_mcss(module_doc_contents, class_doc_contents, enum_doc_contents, function_doc_contents, property_doc_contents, data_doc_contents, **kwargs): +# Modified from abbr / gh / gl / ... to add support for queries and hashes +link_regexp = re.compile(r'(?P.*) <(?P<link>[^?#]+)(?P<hash>[?#].+)?>') + +def parse_link(text): + link = utils.unescape(text) + m = link_regexp.match(link) + if m: + title, link, hash = m.group('title', 'link', 'hash') + if not hash: hash = '' # it's None otherwise + else: + title, hash = '', '' + + return title, link, hash + +intersphinx_inventory = {} +intersphinx_name_prefixes = [] + +# Basically a copy of sphinx.util.inventory.InventoryFile.load_v2. There's no +# documentation for this, it seems. +def parse_intersphinx_inventory(file, base_url, inventory, css_classes): + # Parse the header, uncompressed + inventory_version = file.readline().rstrip() + if inventory_version != b'# Sphinx inventory version 2': + raise ValueError(f"Unsupported inventory version header: {inventory_version}") # pragma: no cover + # those two are not used at the moment, just for completeness + project = file.readline().rstrip()[11:] + version = file.readline().rstrip()[11:] + line = file.readline() + if b'zlib' not in line: + raise ValueError(f"invalid inventory header (not compressed): {line}") # pragma: no cover + + # Decompress the rest. Again mostly a copy of the sphinx code. + for line in zlib.decompress(file.read()).decode('utf-8').splitlines(): + m = re.match(r'(?x)(.+?)\s+(\S*:\S*)\s+(-?\d+)\s+(\S+)\s+(.*)', + line.rstrip()) + if not m: # pragma: no cover + print(f"wait what is this line?! {line}") + continue + # What the F is prio for + name, type, prio, location, title = m.groups() + + # What is this?! + if location.endswith('$'): location = location[:-1] + name + + # The original code `continue`s in this case. I'm asserting. Fix your + # docs. + assert not(type == 'py:module' and type in inventory and name in inventory[type]), "Well dang, we hit that bug in 1.1 that I didn't want to work around" # pragma: no cover + + # Prepend base URL and add to the inventory + inventory.setdefault(type, {})[name] = (urljoin(base_url, location), title, css_classes) + +def parse_intersphinx_inventories(input, inventories): + global intersphinx_inventory, intersphinx_name_prefixes + intersphinx_inventory = {} + intersphinx_name_prefixes = [''] + + for f in inventories: + inventory, base_url = f[:2] + prefixes = f[2] if len(f) > 2 else [] + css_classes = f[3] if len(f) > 3 else [] + + intersphinx_name_prefixes += prefixes + with open(os.path.join(input, inventory), 'rb') as file: + parse_intersphinx_inventory(file, base_url, intersphinx_inventory, css_classes) + +# Matches e.g. py:function in py:function:open +_type_prefix_re = re.compile(r'([a-z0-9]{,3}:[a-z0-9]{3,}):') +_function_types = ['py:function', 'py:classmethod', 'py:staticmethod', 'py:method', 'c:function'] + +def ref(name, rawtext, text, lineno, inliner: Inliner, options={}, content=[]): + title, target, hash = parse_link(text) + + # Otherwise adding classes to the options behaves globally (uh?) + _options = dict(options) + set_classes(_options) + # Avoid assert on adding to undefined member later + if 'classes' not in _options: _options['classes'] = [] + + # Iterate through all prefixes, try to find the name + global intersphinx_inventory, intersphinx_name_prefixes + for prefix in intersphinx_name_prefixes: + found = None + + # If the target is prefixed with a type, try looking up that type + # directly. The implicit link title is then without the type. + m = _type_prefix_re.match(target) + if m: + type = m.group(1) + prefixed = prefix + target[len(type) + 1:] + # ALlow trailing () on functions here as well + if prefixed.endswith('()') and type in _function_types: + prefixed = prefixed[:-2] + if type in intersphinx_inventory and prefixed in intersphinx_inventory[type]: + target = target[len(type) + 1:] + found = type, intersphinx_inventory[m.group(1)][prefixed] + + prefixed = prefix + target + + # If the target looks like a function, look only in functions and strip + # the trailing () as the inventory doesn't have that + if not found and prefixed.endswith('()'): + prefixed = prefixed[:-2] + for type in _function_types: + if type in intersphinx_inventory and prefixed in intersphinx_inventory[type]: + found = type, intersphinx_inventory[type][prefixed] + break + + # Iterate through whitelisted types otherwise. Skipping + # 'std:pdbcommand', 'std:cmdoption', 'std:term', 'std:label', + # 'std:opcode', 'std:envvar', 'std:token', 'std:doc', 'std:2to3fixer' + # and unknown domains such as c++ for now as I'm unsure about potential + # name clashes. + if not found: + for type in ['py:exception', 'py:attribute', 'py:method', 'py:data', 'py:module', 'py:function', 'py:class', 'py:classmethod', 'py:staticmethod', 'c:var', 'c:type', 'c:function', 'c:member', 'c:macro']: + if type in intersphinx_inventory and prefixed in intersphinx_inventory[type]: + found = type, intersphinx_inventory[type][prefixed] + + if found: + url, link_title, css_classes = found[1] + if title: + use_title = title + elif link_title != '-': + use_title = link_title + else: + use_title = target + # Add () to function refs + if found[0] in _function_types and not target.endswith('()'): + use_title += '()' + + _options['classes'] += css_classes + node = nodes.reference(rawtext, use_title, refuri=url + hash, **_options) + return [node], [] + + if title: + logging.warning("Sphinx symbol `{}` not found, rendering just link title".format(target)) + node = nodes.inline(rawtext, title, **_options) + else: + logging.warning("Sphinx symbol `{}` not found, rendering as monospace".format(target)) + node = nodes.literal(rawtext, target, **_options) + return [node], [] + +def register_mcss(mcss_settings, module_doc_contents, class_doc_contents, enum_doc_contents, function_doc_contents, property_doc_contents, data_doc_contents, **kwargs): global module_doc_output, class_doc_output, enum_doc_output, function_doc_output, property_doc_output, data_doc_output module_doc_output = module_doc_contents class_doc_output = class_doc_contents @@ -135,6 +288,9 @@ def register_mcss(module_doc_contents, class_doc_contents, enum_doc_contents, fu property_doc_output = property_doc_contents data_doc_output = data_doc_contents + parse_intersphinx_inventories(input=mcss_settings['INPUT'], + inventories=mcss_settings.get('M_SPHINX_INVENTORIES', [])) + rst.directives.register_directive('py:module', PyModule) rst.directives.register_directive('py:class', PyClass) rst.directives.register_directive('py:enum', PyEnum) @@ -142,5 +298,14 @@ def register_mcss(module_doc_contents, class_doc_contents, enum_doc_contents, fu rst.directives.register_directive('py:property', PyProperty) rst.directives.register_directive('py:data', PyData) + rst.roles.register_local_role('ref', ref) + +def _pelican_configure(pelicanobj): + # For backwards compatibility, the input directory is pelican's CWD + parse_intersphinx_inventories(input=os.getcwd(), + inventories=pelicanobj.settings.get('M_SPHINX_INVENTORIES', [])) + def register(): # for Pelican - assert not "This plugin is for the m.css Doc theme only" # pragma: no cover + rst.roles.register_local_role('ref', ref) + + signals.initialized.connect(_pelican_configure) diff --git a/plugins/m/test/sphinx/page.html b/plugins/m/test/sphinx/page.html new file mode 100644 index 00000000..a8134e20 --- /dev/null +++ b/plugins/m/test/sphinx/page.html @@ -0,0 +1,94 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="UTF-8" /> + <title>m.sphinx | A Pelican Blog + + + + + + +
+
+
+
+
+
+

m.sphinx

+ + +

These should produce warnings:

+
    +
  • Link to nonexistent name will be rendered as code: nonexistent()
  • +
  • Link to nonexistent name with custom title will be just text
  • +
+ +
+
+
+
+
+ + \ No newline at end of file diff --git a/plugins/m/test/sphinx/page.rst b/plugins/m/test/sphinx/page.rst new file mode 100644 index 00000000..0ef1a044 --- /dev/null +++ b/plugins/m/test/sphinx/page.rst @@ -0,0 +1,65 @@ +m.sphinx +######## + +.. role:: ref-small(ref) + :class: m-text m-small + +- Module link: + + - :ref:`argparse` + - explicit type: :ref:`py:module:argparse` + +- Function link: + + - :ref:`open()` + - without a ``()``: :ref:`open` + - explicit type: :ref:`py:function:open()`, + - explicit without a ``()``: :ref:`py:function:open` + +- Class link: + + - :ref:`xml.etree.ElementTree.Element` + - explicit type: :ref:`py:class:xml.etree.ElementTree.Element` + +- Classmethod link: + + - :ref:`bytearray.fromhex()` + - without a ``()``: :ref:`bytearray.fromhex` + - explicit type: :ref:`py:classmethod:bytearray.fromhex()` + - explicit without a ``()``: :ref:`py:classmethod:bytearray.fromhex` + +- Staticmethod link: + + - :ref:`bytes.maketrans()` + - without a ``()``: :ref:`bytes.maketrans` + - explicit type :ref:`py:staticmethod:bytes.maketrans()` + - explicit without a ``()``: :ref:`py:staticmethod:bytes.maketrans` + +- Method link: + + - :ref:`str.rstrip()` + - without a ``()``: :ref:`str.rstrip` + - explicit type: :ref:`py:method:str.rstrip()` + - explicit type without a ``()``: :ref:`py:method:str.rstrip()` + +- Property link: + + - :ref:`datetime.date.year` + - explicit type :ref:`py:attribute:datetime.date.year` + +- Data link: + + - :ref:`re.X` + - explicit type: :ref:`py:data:re.X` + +- Explicitly typed page link with automatic title: :ref:`std:doc:using/cmdline` +- :ref:`Page link with custom link title `, + :ref:`Function link with a custom title ` +- Custom CSS class: :ref-small:`str.join()` +- Omitting a prefix: :ref:`etree.ElementTree`, :ref:`ElementTree` +- Custom query string: :ref:`os.path ` + +These should produce warnings: + +- Link to nonexistent name will be rendered as code: :ref:`nonexistent()` +- :ref:`Link to nonexistent name with custom title will be just text ` diff --git a/plugins/m/test/test_sphinx.py b/plugins/m/test/test_sphinx.py index 5a5e8fa5..9b868a80 100644 --- a/plugins/m/test/test_sphinx.py +++ b/plugins/m/test/test_sphinx.py @@ -22,4 +22,19 @@ # DEALINGS IN THE SOFTWARE. # -# This module gets tested inside documentation/test_python/. +# The directives are only for the Python theme and get tested inside it + +from . import PelicanPluginTestCase + +class Sphinx(PelicanPluginTestCase): + def __init__(self, *args, **kwargs): + super().__init__(__file__, '', *args, **kwargs) + + def test(self): + self.run_pelican({ + 'PLUGINS': ['m.htmlsanity', 'm.sphinx'], + 'M_SPHINX_INVENTORIES': [ + ('../doc/documentation/python.inv', 'https://docs.python.org/3/', ['xml.', 'xml.etree.'], ['m-flat'])] + }) + + self.assertEqual(*self.actual_expected_contents('page.html')) diff --git a/site/pelicanconf.py b/site/pelicanconf.py index a1074171..19e9408a 100644 --- a/site/pelicanconf.py +++ b/site/pelicanconf.py @@ -158,6 +158,7 @@ PLUGINS = ['m.abbr', 'm.math', 'm.metadata', 'm.plots', + 'm.sphinx', 'm.qr', 'm.vk'] @@ -179,6 +180,8 @@ M_HTMLSANITY_SMART_QUOTES = True M_HTMLSANITY_HYPHENATION = True M_DOX_TAGFILES = [ ('../doc/documentation/corrade.tag', 'https://doc.magnum.graphics/corrade/', ['Corrade::'])] +M_SPHINX_INVENTORIES = [ + ('../doc/documentation/python.inv', 'https://docs.python.org/3/', ['xml.'])] if not shutil.which('latex'): logging.warning("LaTeX not found, fallback to rendering math as code")