On Tue, Feb 28, 2017 at 06:13:15PM +0100, Daniel Vetter wrote: > From: Markus Heiser <markus.heiser@xxxxxxxxxxx> > > This patch brings scalable figure, image handling and a concept to > embed *render* markups: > > * DOT (http://www.graphviz.org) > * SVG > > For image handling use the 'image' replacement:: > > .. kernel-image:: svg_image.svg > :alt: simple SVG image > > For figure handling use the 'figure' replacement:: > > .. kernel-figure:: svg_image.svg > :alt: simple SVG image > > SVG image example > > Embed *render* markups (or languages) like Graphviz's **DOT** is > provided by the *render* directive.:: > > .. kernel-render:: DOT > :alt: foobar digraph > :caption: Embedded **DOT** (Graphviz) code. > > digraph foo { > "bar" -> "baz"; > } > > The *render* directive is a concept to integrate *render* markups and > languages, yet supported markups: > > * DOT: render embedded Graphviz's **DOT** > * SVG: render embedded Scalable Vector Graphics (**SVG**) > > v2: s/DOC/DOT/ in a few places (by Daniel). > > v3: Simplify stuff a bit (by Daniel): > > - Remove path detection and setup/check code for that. In > Documentation/media/Makefile we already simply use these tools, > better to have one consolidated check if we want/need one. Also > remove the convertsvg support, we require ImageMagick's convert > already in the doc build, no need for a 2nd fallback. > > - Use sphinx for depency tracking, remove hand-rolled version. > > - Forward stderr from dot and convert, otherwise debugging issues with > the diagrams is impossible. > > Cc: Jonathan Corbet <corbet@xxxxxxx> > Cc: linux-doc@xxxxxxxxxxxxxxx > Cc: Jani Nikula <jani.nikula@xxxxxxxxxxxxxxx> > Cc: Mauro Carvalho Chehab <mchehab@xxxxxxxxxxxxxxxx> > Signed-off-by: Markus Heiser <markus.heiser@xxxxxxxxxxx> (v1) > Signed-off-by: Daniel Vetter <daniel.vetter@xxxxxxxxx> Just to avoid confusion: Markus&I chatted a bit in private, and he volunteered to take over, and fix the few issues that need to be fixed in his original patch: - drop the convertsvg python fallback, since we need convert anyway - relative paths - stderr fwd - and the small polish I noticed in docs I think with my blind hacking I botched the job too much :-) Anyway, still hoping we could land this for 4.12, I really want the drm-side doc improvements this provides ... Thanks, Daniel > --- > Documentation/conf.py | 2 +- > Documentation/doc-guide/hello.dot | 3 + > Documentation/doc-guide/sphinx.rst | 90 ++++++- > Documentation/doc-guide/svg_image.svg | 10 + > Documentation/process/changes.rst | 7 +- > Documentation/sphinx/kfigure.py | 442 ++++++++++++++++++++++++++++++++++ > 6 files changed, 548 insertions(+), 6 deletions(-) > create mode 100644 Documentation/doc-guide/hello.dot > create mode 100644 Documentation/doc-guide/svg_image.svg > create mode 100644 Documentation/sphinx/kfigure.py > > diff --git a/Documentation/conf.py b/Documentation/conf.py > index f6823cf01275..e3f537ce2935 100644 > --- a/Documentation/conf.py > +++ b/Documentation/conf.py > @@ -34,7 +34,7 @@ from load_config import loadConfig > # Add any Sphinx extension module names here, as strings. They can be > # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom > # ones. > -extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain'] > +extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain', 'kfigure'] > > # The name of the math extension changed on Sphinx 1.4 > if major == 1 and minor > 3: > diff --git a/Documentation/doc-guide/hello.dot b/Documentation/doc-guide/hello.dot > new file mode 100644 > index 000000000000..504621dfc595 > --- /dev/null > +++ b/Documentation/doc-guide/hello.dot > @@ -0,0 +1,3 @@ > +graph G { > + Hello -- World > +} > diff --git a/Documentation/doc-guide/sphinx.rst b/Documentation/doc-guide/sphinx.rst > index 532d65b70500..b902744ce7dd 100644 > --- a/Documentation/doc-guide/sphinx.rst > +++ b/Documentation/doc-guide/sphinx.rst > @@ -34,8 +34,10 @@ format-specific subdirectories under ``Documentation/output``. > > To generate documentation, Sphinx (``sphinx-build``) must obviously be > installed. For prettier HTML output, the Read the Docs Sphinx theme > -(``sphinx_rtd_theme``) is used if available. For PDF output, ``rst2pdf`` is also > -needed. All of these are widely available and packaged in distributions. > +(``sphinx_rtd_theme``) is used if available. For PDF output you'll also need > +``XeLaTeX`` and CairoSVG (http://cairosvg.org) or alternatively ``convert(1)`` > +from ImageMagick (https://www.imagemagick.org). All of these are widely > +available and packaged in distributions. > > To pass extra options to Sphinx, you can use the ``SPHINXOPTS`` make > variable. For example, use ``make SPHINXOPTS=-v htmldocs`` to get more verbose > @@ -232,3 +234,87 @@ Rendered as: > * .. _`last row`: > > - column 3 > + > + > +Figures & Images > +================ > + > +If you want to add an image, you should use the ``kernel-figure`` and > +``kernel-image`` directives. E.g. to insert a figure with a scalable > +image format use SVG:: > + > + .. kernel-figure:: svg_image.svg > + :alt: simple SVG image > + > + SVG image example > + > +.. kernel-figure:: svg_image.svg > + :alt: simple SVG image > + > + SVG image example > + > +The kernel figure (and image) directive support **DOT** formated files, see > + > +* DOT: http://graphviz.org/pdf/dotguide.pdf > +* Graphviz: http://www.graphviz.org/content/dot-language > + > +A simple example:: > + > + .. kernel-figure:: hello.dot > + :alt: hello world > + > + DOT's hello world example > + > +.. kernel-figure:: hello.dot > + :alt: hello world > + > + DOT's hello world example > + > +Embed *render* markups (or languages) like Graphviz's **DOT** is provided by the > +``kernel-render`` directives.:: > + > + .. kernel-render:: DOT > + :alt: foobar digraph > + :caption: Embedded **DOT** (Graphviz) code. > + > + digraph foo { > + "bar" -> "baz"; > + } > + > +How this will be rendered depends on the installed tools. If Graphviz is > +installed, you will see an vector image. If not the raw markup is inserted as > +*literal-block*. > + > +.. kernel-render:: DOT > + :alt: foobar digraph > + :caption: Embedded **DOT** (Graphviz) code. > + > + digraph foo { > + "bar" -> "baz"; > + } > + > +The *render* directive has all the options known from the *figure* directive, > +plus option ``caption``. If ``caption`` has a value, a *figure* node is > +inserted. If not, a *image* node is inserted. > + > +Embedded **SVG**:: > + > + .. kernel-render:: SVG > + :caption: Embedded **SVG** markup. > + :alt: so-nw-arrow > + > + <?xml version="1.0" encoding="UTF-8"?> > + <svg xmlns="http://www.w3.org/2000/svg"; version="1.1" ...> > + ... > + </svg> > + > +.. kernel-render:: SVG > + :caption: Embedded **SVG** markup. > + :alt: so-nw-arrow > + > + <?xml version="1.0" encoding="UTF-8"?> > + <svg xmlns="http://www.w3.org/2000/svg"; > + version="1.1" baseProfile="full" width="70px" height="40px" viewBox="0 0 700 400"> > + <line x1="180" y1="370" x2="500" y2="50" stroke="black" stroke-width="15px"/> > + <polygon points="585 0 525 25 585 50" transform="rotate(135 525 25)"/> > + </svg> > diff --git a/Documentation/doc-guide/svg_image.svg b/Documentation/doc-guide/svg_image.svg > new file mode 100644 > index 000000000000..5405f85b8137 > --- /dev/null > +++ b/Documentation/doc-guide/svg_image.svg > @@ -0,0 +1,10 @@ > +<?xml version="1.0" encoding="UTF-8"?> > +<!-- originate: https://commons.wikimedia.org/wiki/File:Variable_Resistor.svg --> > +<svg xmlns="http://www.w3.org/2000/svg"; > + version="1.1" baseProfile="full" > + width="70px" height="40px" viewBox="0 0 700 400"> > + <line x1="0" y1="200" x2="700" y2="200" stroke="black" stroke-width="20px"/> > + <rect x="100" y="100" width="500" height="200" fill="white" stroke="black" stroke-width="20px"/> > + <line x1="180" y1="370" x2="500" y2="50" stroke="black" stroke-width="15px"/> > + <polygon points="585 0 525 25 585 50" transform="rotate(135 525 25)"/> > +</svg> > diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst > index 56ce66114665..e4f25038ef65 100644 > --- a/Documentation/process/changes.rst > +++ b/Documentation/process/changes.rst > @@ -318,9 +318,10 @@ PDF outputs, it is recommended to use version 1.4.6. > .. note:: > > Please notice that, for PDF and LaTeX output, you'll also need ``XeLaTeX`` > - version 3.14159265. Depending on the distribution, you may also need > - to install a series of ``texlive`` packages that provide the minimal > - set of functionalities required for ``XeLaTex`` to work. > + version 3.14159265. Depending on the distribution, you may also need to > + install a series of ``texlive`` packages that provide the minimal set of > + functionalities required for ``XeLaTex`` to work. For PDF output you'll also > + need ``convert(1)`` from ImageMagick (https://www.imagemagick.org). > > Other tools > ----------- > diff --git a/Documentation/sphinx/kfigure.py b/Documentation/sphinx/kfigure.py > new file mode 100644 > index 000000000000..8f749d389dae > --- /dev/null > +++ b/Documentation/sphinx/kfigure.py > @@ -0,0 +1,442 @@ > +# -*- coding: utf-8; mode: python -*- > +# pylint: disable=C0103 > +u""" > + scalable figure and image handling > + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > + Sphinx extension which implements scalable image handling. > + > + :copyright: Copyright (C) 2016 Markus Heiser > + :license: GPL Version 2, June 1991 see Linux/COPYING for details. > + > + The build for image formats depence on image's source format and output's > + destination format. This extension implement methods to simplify image > + handling from the author's POV. Directives like ``kernel-figure`` implement > + methods *to* always get the best output-format even if some tools are not > + installed.For more details take a look at ``convert_image(...)`` which is > + the core of all conversions. > + > + * ``.. kernel-image``: for image handling / ``.. image::`` replacement > + > + * ``.. kernel-figure``: for figure handling / ``.. figure::`` replacement > + > + * ``.. kernel-render``: for render markup / a concept to embed *render* > + markups (or languages). Supported markups (see ``RENDER_MARKUP_EXT``) > + > + + ``DOT``: render embedded Graphviz's **DOC** > + + ``SVG``: render embedded Scalable Vector Graphics (**SVG**) > + + ... *developable* > + > + Used tools: > + > + * ``dot(1)``: Graphviz (http://www.graphviz.org). If Graphviz is not > + available, the DOT language is inserted as literal-block. > + > + * SVG to PDF: To generate PDF, you need at least one of this tools: > + > + - ``convert(1)``: ImageMagick (https://www.imagemagick.org) > + > + List of customizations: > + > + * generate PDF from SVG / used by PDF (LaTeX) builder > + > + * generate SVG (html-builder) and PDF (latex-builder) from DOT files. > + DOT: see http://www.graphviz.org/content/dot-language > + > + """ > + > +import os > +from os import path > +import subprocess > +from hashlib import sha1 > +import sys > + > +from docutils import nodes > +from docutils.statemachine import ViewList > +from docutils.parsers.rst import directives > +from docutils.parsers.rst.directives import images > + > +from sphinx.directives import patches > + > +__version__ = '1.0' > + > +# simple helper > +# ------------- > + > +def mkdir(folder, mode=0o775): > + if not path.isdir(folder): > + os.makedirs(folder, mode) > + > +# def debug_handle(self, node): # pylint: disable=W0613 > +# from linuxdoc.kernel_doc import CONSOLE > +# CONSOLE() > + > +def pass_handle(self, node): # pylint: disable=W0613 > + pass > + > +# setup conversion tools and sphinx extension > +# ------------------------------------------- > + > +def setup(app): > + # image handling > + app.add_directive("kernel-image", KernelImage) > + app.add_node(kernel_image, > + html = (visit_kernel_image, pass_handle), > + latex = (visit_kernel_image, pass_handle), > + texinfo = (visit_kernel_image, pass_handle), > + text = (visit_kernel_image, pass_handle), > + man = (visit_kernel_image, pass_handle), ) > + > + # figure handling > + app.add_directive("kernel-figure", KernelFigure) > + app.add_node(kernel_figure, > + html = (visit_kernel_figure, pass_handle), > + latex = (visit_kernel_figure, pass_handle), > + texinfo = (visit_kernel_figure, pass_handle), > + text = (visit_kernel_figure, pass_handle), > + man = (visit_kernel_figure, pass_handle), ) > + > + # render handling > + app.add_directive('kernel-render', KernelRender) > + app.add_node(kernel_render, > + html = (visit_kernel_render, pass_handle), > + latex = (visit_kernel_render, pass_handle), > + texinfo = (visit_kernel_render, pass_handle), > + text = (visit_kernel_render, pass_handle), > + man = (visit_kernel_render, pass_handle), ) > + > + return dict( > + version = __version__, > + parallel_read_safe = True, > + parallel_write_safe = True > + ) > + > +# integrate conversion tools > +# -------------------------- > + > +RENDER_MARKUP_EXT = { > + # The '.ext' must be handled by convert_image(..) function's *in_ext* input. > + # <name> : <.ext> > + 'DOT' : '.dot' > + , 'SVG' : '.svg' > +} > + > +def convert_image(img_node, translator): # pylint: disable=R0912 > + """Convert an image node for the builder. > + > + Different builder prefer different image formats, e.g. *latex* builder > + prefer PDF while *html* builder prefer SVG format for images. > + > + This function handles outputs image formats in depence of source the format > + of the image and the translator's output format. This also means to > + manipulate/update the *image* dictionary of the builder (``builder.images``) > + > + """ > + fname, in_ext = path.splitext(path.basename(img_node['uri'])) > + src_fname = path.join(translator.builder.srcdir, img_node['uri']) > + src_folder = path.dirname(img_node['uri']) > + out_dir = translator.builder.outdir > + dst_fname = None > + > + # in kernel builds, use 'make SPHINXOPTS=-v' to see verbose messages > + verbose = translator.builder.app.verbose > + warn = translator.builder.warn > + > + verbose('assert best format for: ' + img_node['uri']) > + > + if in_ext == '.dot': > + > + # ---------- > + # handle DOT > + # ---------- > + > + dst_fname = path.join(out_dir, fname + '.pdf') > + > + if translator.builder.format == 'html': > + dst_fname = path.join(out_dir, src_folder, fname + '.svg') > + else: > + # all other builder formats will include DOT as raw > + with open(src_fname, "r") as dot: > + data = dot.read() > + node = nodes.literal_block(data, data) > + img_node.replace_self(node) > + > + > + elif in_ext == '.svg': > + > + # ---------- > + # handle SVG > + # ---------- > + > + if translator.builder.format == 'latex': > + dst_fname = path.join(out_dir, fname + '.pdf') > + > + if dst_fname: > + name = dst_fname[len(out_dir) + 1:] > + # the builder needs not to copy one more time, so pop it if exists. > + translator.builder.images.pop(img_node['uri'], None) > + img_node['uri'] = dst_fname > + img_node['candidates'] = {'*': dst_fname} > + > + mkdir(path.dirname(dst_fname)) > + > + if in_ext == '.dot': > + verbose('convert DOT to: {out}/' + name) > + dot2format(src_fname, dst_fname) > + > + elif in_ext == '.svg': > + verbose('convert SVG to: {out}/' + name) > + svg2pdf(src_fname, dst_fname) > + > +def dot2format(dot_fname, out_fname): > + """Converts DOT file to ``out_fname`` using ``dot(1)``. > + > + * ``dot_fname`` pathname of the input DOT file, including extension ``.dot`` > + * ``out_fname`` pathname of the output file, including format extension > + > + The *format extension* depends on the ``dot`` command (see ``man dot`` > + option ``-Txxx``). Normally you will use one of the following extensions: > + > + - ``.ps`` for PostScript, > + - ``.svg`` or ``svgz`` for Structured Vector Graphics, > + - ``.fig`` for XFIG graphics and > + - ``.png`` or ``gif`` for common bitmap graphics. > + > + """ > + out_format = path.splitext(out_fname)[1][1:] > + cmd = ['dot', '-T%s' % out_format, dot_fname] > + exit_code = 42 > + with open(out_fname, "w") as out: > + p = subprocess.Popen( > + cmd, stdout = out, stderr = subprocess.PIPE ) > + nil, err = p.communicate() > + > + sys.stderr.write(err) > + > + exit_code = p.returncode > + out.flush() > + return bool(exit_code == 0) > + > +def svg2pdf(svg_fname, pdf_fname): > + """Converts SVG to PDF with CairoSVG or ``convert(1)`` command. > + > + Uses ``convert(1)`` from ImageMagick (https://www.imagemagick.org) for > + conversion. Returns ``True`` on success and ``False`` if an error occurred > + (e.g. none of the conversion tool is available). > + > + * ``svg_fname`` pathname of the input SVG file with extension (``.svg``) > + * ``pdf_name`` pathname of the output PDF file with extension (``.pdf``) > + > + """ > + cmd = [convert_cmd, svg_fname, pdf_fname] > + p = subprocess.Popen( > + cmd, stdout = out, stderr = subprocess.PIPE ) > + nil, err = p.communicate() > + > + sys.stderr.write(err) > + > + exit_code = p.returncode > + return bool(exit_code == 0) > + > + > +# image handling > +# --------------------- > + > +def visit_kernel_image(self, node): # pylint: disable=W0613 > + """Visitor of the ``kernel_image`` Node. > + > + Handles the ``image`` child-node with the ``convert_image(...)``. > + """ > + img_node = node[0] > + convert_image(img_node, self) > + > +class kernel_image(nodes.General, nodes.Element): > + """Node for ``kernel-image`` directive.""" > + pass > + > +class KernelImage(images.Image): > + u"""KernelImage directive > + > + Earns everything from ``.. image::`` directive, except *remote URI* and > + *glob* pattern. The KernelImage wraps a image node into a > + kernel_image node. See ``visit_kernel_image``. > + """ > + > + def run(self): > + env = self.state.document.settings.env > + > + uri = self.arguments[0] > + if uri.endswith('.*') or uri.find('://') != -1: > + raise self.severe( > + 'Error in "%s: %s": glob pattern and remote images are not allowed' > + % (self.name, uri)) > + > + # Tell sphinx of the dependency > + env.note_dependency(os.path.abspath(uri)) > + > + result = images.Image.run(self) > + if len(result) == 2 or isinstance(result[0], nodes.system_message): > + return result > + (image_node,) = result > + # wrap image node into a kernel_image node / see visitors > + node = kernel_image('', image_node) > + return [node] > + > +# figure handling > +# --------------------- > + > +def visit_kernel_figure(self, node): # pylint: disable=W0613 > + """Visitor of the ``kernel_figure`` Node. > + > + Handles the ``image`` child-node with the ``convert_image(...)``. > + """ > + img_node = node[0][0] > + convert_image(img_node, self) > + > +class kernel_figure(nodes.General, nodes.Element): > + """Node for ``kernel-figure`` directive.""" > + > +class KernelFigure(patches.Figure): > + u"""KernelImage directive > + > + Earns everything from ``.. figure::`` directive, except *remote URI* and > + *glob* pattern. The KernelFigure wraps a figure node into a kernel_figure > + node. See ``visit_kernel_figure``. > + """ > + > + def run(self): > + env = self.state.document.settings.env > + > + uri = self.arguments[0] > + if uri.endswith('.*') or uri.find('://') != -1: > + raise self.severe( > + 'Error in "%s: %s":' > + ' glob pattern and remote images are not allowed' > + % (self.name, uri)) > + > + # Tell sphinx of the dependency > + env.note_dependency(os.path.abspath(uri)) > + > + result = patches.Figure.run(self) > + if len(result) == 2 or isinstance(result[0], nodes.system_message): > + return result > + (figure_node,) = result > + # wrap figure node into a kernel_figure node / see visitors > + node = kernel_figure('', figure_node) > + return [node] > + > + > +# render handling > +# --------------------- > + > +def visit_kernel_render(self, node): > + """Visitor of the ``kernel_render`` Node. > + > + If rendering tools available, save the markup of the ``literal_block`` child > + node into a file and replace the ``literal_block`` node with a new created > + ``image`` node, pointing to the saved markup file. Afterwards, handle the > + image child-node with the ``convert_image(...)``. > + """ > + > + verbose = self.builder.app.verbose > + warn = self.builder.warn > + srclang = node.get('srclang') > + > + verbose('visit kernel-render node lang: "%s"' % (srclang)) > + > + tmp_ext = RENDER_MARKUP_EXT.get(srclang, None) > + if tmp_ext is None: > + warn('kernel-render: "%s" unknow / include raw.' % (srclang)) > + return > + > + literal_block = node[0] > + code = literal_block.astext() > + > + if tmp_ext: > + hashobj = code.encode('utf-8') # str(node.attributes) > + fname = '%s-%s' % (srclang, sha1(hashobj).hexdigest()) > + tmp_fname = path.join( > + self.builder.outdir, self.builder.imagedir, fname + tmp_ext) > + > + if not path.isfile(tmp_fname): > + mkdir(path.dirname(tmp_fname)) > + with open(tmp_fname, "w") as out: > + out.write(code) > + > + image_node = nodes.image(node.rawsource, **node.attributes) > + image_node['uri'] = tmp_fname > + > + literal_block.replace_self(image_node) > + convert_image(image_node, self) > + > + > +class kernel_render(nodes.General, nodes.Inline, nodes.Element): > + """Node for ``kernel-render`` directive.""" > + pass > + > +class KernelRender(patches.Figure): > + u"""KernelRender directive > + > + Render content by external tool. Has all the options known from the > + *figure* directive, plus option ``caption``. If ``caption`` has a > + value, a figure node with the *caption* is inserted. If not, a image node is > + inserted. > + > + The KernelRender directive wraps the text of the directive into a > + literal_block node and wraps it into a kernel_render node. See > + ``visit_kernel_render``. > + """ > + has_content = True > + required_arguments = 1 > + optional_arguments = 0 > + final_argument_whitespace = False > + > + # earn options from 'figure' > + option_spec = patches.Figure.option_spec.copy() > + option_spec['caption'] = directives.unchanged > + > + def run(self): > + return [self.build_node()] > + > + def build_node(self): > + > + srclang = self.arguments[0].strip() > + if srclang not in RENDER_MARKUP_EXT.keys(): > + return [self.state_machine.reporter.warning( > + 'Unknow source language "%s", use one of: %s.' % ( > + srclang, ",".join(RENDER_MARKUP_EXT.keys())), > + line=self.lineno)] > + > + code = '\n'.join(self.content) > + if not code.strip(): > + return [self.state_machine.reporter.warning( > + 'Ignoring "%s" directive without content.' % ( > + self.name), > + line=self.lineno)] > + > + node = kernel_render() > + node['alt'] = self.options.get('alt','') > + node['srclang'] = srclang > + literal_node = nodes.literal_block(code, code) > + node += literal_node > + > + caption = self.options.get('caption') > + if caption: > + # parse cation's content > + parsed = nodes.Element() > + self.state.nested_parse( > + ViewList([caption], source=''), self.content_offset, parsed) > + caption_node = nodes.caption( > + parsed[0].rawsource, '', *parsed[0].children) > + caption_node.source = parsed[0].source > + caption_node.line = parsed[0].line > + > + figure_node = nodes.figure('', node) > + for k,v in self.options.items(): > + figure_node[k] = v > + figure_node += caption_node > + > + node = figure_node > + > + return node > + > -- > 2.11.0 > -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ dri-devel mailing list dri-devel@xxxxxxxxxxxxxxxxxxxxx https://lists.freedesktop.org/mailman/listinfo/dri-devel
