nixos-render-docs manual build failure

[evrim]

Hello,

I have a little bit of trouble with this new PR:

#212684

building '/nix/store/d4d59993gbfp4jqqhpn4dic1kglgrs2l-modules.xml.drv'...
Traceback (most recent call last):
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/bin/.nixos-render-docs-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/__init__.py", line 22, in main
    manual.run_cli(args)
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/manual.py", line 141, in run_cli
    _run_cli_db(args)
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/manual.py", line 131, in _run_cli_db
    md.add_section(section.id, chapters)
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/manual.py", line 33, in add_section
    self._sections[-1].chapters.append(self._render(content))
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/md.py", line 385, in _render
    return self._md.renderer.render(tokens, self._md.options, env) # type: ignore[no-any-return]
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/docbook.py", line 46, in render
    result = super().render(tokens, options, env)
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/md.py", line 101, in render
    return "".join(map(lambda arg: do_one(*arg), enumerate(tokens)))
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/md.py", line 101, in <lambda>
    return "".join(map(lambda arg: do_one(*arg), enumerate(tokens)))
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/md.py", line 98, in do_one
    return self.rules[token.type](tokens[i], tokens, i, options, env)
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/docbook.py", line 230, in heading_open
    (tag, attrs) = self._heading_tag(token, tokens, i, options, env)
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/manual.py", line 48, in _heading_tag
    raise RuntimeError("only one title heading allowed", token)
RuntimeError: ('only one title heading allowed', Token(type='heading_open', tag='h1', nesting=1, attrs={}, map=[120, 121], level=0, children=None, content='', markup='#', info='', meta={}, block=True, hidden=False))
error: builder for '/nix/store/d4d59993gbfp4jqqhpn4dic1kglgrs2l-modules.xml.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/a8bqisd13c3myqq9k3j5xxwjgqa3jv26-generated-docbook.drv' failed to build

nixpkgs unstable is at:

e1e1b19 nixos-unstable origin/nixos-unstable Merge pull request #213481 from lunik1/iosevka-buildnpmpackage

Ping @pennae @jtojnar

Any ideas? Ideas on debugging still welcome.

best,
evrim.

[jtojnar]

My guess is some NixOS module on your system has # at the beginning of a line of an option description. Do you use any extra NixOS modules?

I tried reproducing it as follows:

--- a/nixos/modules/services/web-apps/selfoss.nix
+++ b/nixos/modules/services/web-apps/selfoss.nix
@@ -37,6 +37,8 @@ in
           default = "nginx";
           description = lib.mdDoc ''
             User account under which both the service and the web-application run.
+
+            # foo
           '';
         };
 

But that showed the option name, not sure why it does not do that in your case:

$ nix-build nixos/release.nix -A manualHTML.x86_64-linux
these 4 derivations will be built:
  /nix/store/v8l8brjnblacrshjm4b5ljvjn6riqlpl-options-docbook.xml.drv
  /nix/store/x6j6vg1lwhwshjj4bdm8ifnzs0slb2y3-generated-docbook.drv
  /nix/store/galirrzifqp6bj471jhr1bqk7nccjgld-nixos-manual-combined.drv
  /nix/store/jaj0dbz1gnimyfz9kkhnsxqddw4gqzc4-nixos-manual-html.drv
building '/nix/store/v8l8brjnblacrshjm4b5ljvjn6riqlpl-options-docbook.xml.drv'...
Traceback (most recent call last):
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/options.py", line 144, in add_options
    self._options[name] = RenderedOption(option['loc'], self._convert_one(option))
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/options.py", line 123, in _convert_one
    result += self._render_description(desc)
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/options.py", line 182, in _render_description
    return super()._render_description(desc)
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/options.py", line 112, in _render_description
    return [ self._render(desc['text']) ]
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/md.py", line 385, in _render
    return self._md.renderer.render(tokens, self._md.options, env) # type: ignore[no-any-return]
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/docbook.py", line 46, in render
    result = super().render(tokens, options, env)
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/md.py", line 101, in render
    return "".join(map(lambda arg: do_one(*arg), enumerate(tokens)))
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/md.py", line 101, in <lambda>
    return "".join(map(lambda arg: do_one(*arg), enumerate(tokens)))
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/md.py", line 98, in do_one
    return self.rules[token.type](tokens[i], tokens, i, options, env)
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/options.py", line 154, in heading_open
    raise RuntimeError("md token not supported in options doc", token)
RuntimeError: ('md token not supported in options doc', Token(type='heading_open', tag='h1', nesting=1, attrs={}, map=[2, 3], level=0, children=None, content='', markup='#', info='', meta={}, block=True, hidden=False))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/bin/.nixos-render-docs-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/__init__.py", line 20, in main
    options.run_cli(args)
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/options.py", line 282, in run_cli
    _run_cli_db(args)
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/options.py", line 272, in _run_cli_db
    md.add_options(json.load(f))
  File "/nix/store/jmhw25z7dxrpj78h3l98bnhd3jr1wgkc-nixos-render-docs-0.0/lib/python3.10/site-packages/nixos_render_docs/options.py", line 146, in add_options
    raise Exception(f"Failed to render option {name}") from e
Exception: Failed to render option services.selfoss.user

[evrim]

Um, no idea why it did not show the trace properly, nix has poor tracing generally.

    auth_token = mkOpt types.str ''                                                                                                                                                                                                            
      Optional authentication information for token-based authentication:                                                                                                                                                                      
      <link xlink:href="https://docs.mesosphere.com/1.11/security/ent/iam-api/#passing-an-authentication-token" />                                                                                                                             
      It is mutually exclusive with <literal>auth_token_file</literal> and other authentication mechanisms.                                                                                                                                    
    '';

I've been lookin around and found similar to the above. Would it be possible for this to fail in case of anchor # in a link? There are others:

 url = mkOption {
        type = types.str;
        description = ''                                                                                                                                                                                                                       
          ServerName extension to indicate the name of the server.                                                                                                                                                                             
          http://tools.ietf.org/html/rfc4366#section-3.1                                                                                                                                                                                       
        '';
      };

[evrim]

Is there a way to avoid building this documentation without disabling man pages? I don't need any local nixos-manuals just man pages are suff.

[pennae]

this is caused by multiple h1s in a module chapter. the h1 is used as the title for a chapter in the manual (and label of the toc link), if you have multiple of thise it'll get confused.

[jtojnar]

Looks like there is no source info in the token: executablebooks/markdown-it-py#219

I wonder if we were able to add it using something like

--- a/pkgs/tools/nix/nixos-render-docs/src/nixos_render_docs/md.py
+++ b/pkgs/tools/nix/nixos-render-docs/src/nixos_render_docs/md.py
@@ -8,6 +8,7 @@ import re
 from .types import RenderFn
 
 import markdown_it
+from markdown_it.rules_core.state_core import StateCore
 from markdown_it.token import Token
 from markdown_it.utils import OptionsDict
 from mdit_py_plugins.container import container_plugin # type: ignore[attr-defined]
@@ -317,6 +318,18 @@ def _block_comment_plugin(md: markdown_it.MarkdownIt) -> None:
     md.block.ruler.after("code", "block_comment", block_comment)
 
 _HEADER_ID_RE = re.compile(r"\s*\{\s*\#([\w-]+)\s*\}\s*$")
+LINE_START_RE = re.compile(r"^")
+
+def line_nos(state: StateCore) -> None:
+    i = 0
+    def repl(match):
+        nonlocal i
+        i = i + 1
+        return f"<!-- line {i} -->"
+
+    string = LINE_START_RE.sub(repl, state.src)
+
+    state.src = string
 
 class Converter(ABC):
     __renderer__: Callable[[Mapping[str, str], markdown_it.MarkdownIt], Renderer]
@@ -333,6 +346,11 @@ class Converter(ABC):
             },
             renderer_cls=lambda parser: self.__renderer__(self._manpage_urls, parser)
         )
+        self._md.core.ruler.before(
+            beforeName='normalize',
+            ruleName='line_nos',
+            fn=line_nos,
+        )
         self._md.use(
             container_plugin,
             name="admonition",

[pennae]

there is source info, it's in the map attribute (containing first line of the token and first line not belonging to the token). we're preparing a PR to improve the error message here substantially.