Indexing (index terms) support?

[gdamore]

I'm starting to look at Djot for some technical content I have for my NNG open source project.

I have a rich reference manual for this, and I produce some nicely typeset printed editions, as well as EPUB and online HTML. Currently I'm using ASCIIDOC. I contributed to the libasciidoc golang project to try to improve it, but I've been very disillusioned with the ASCIIDOC projects failure to even start on a meaningful specification now several years after promising to start (and I volunteered to help.)

Anyway, to get to the point, one feature that I do rely upon, and find missing in Djot, is any support for indexterms, so that I can generate a printed index at the back of the document. For printed material, this is incredibly useful, at least for old school folks like me.

Has anyone given thought to this? It seems like this could be an specific kind of attribute reference I suppose, but we would need tooling to know how to grok this, and I'm probably not up to writing that portion of the code.

(On the other hand, as an author of several TreeSitter grammars, including probably the most complex TS grammar, the one for D, I might be able to knock out a TreeSitter grammar for djot, as if I switch to it I will surely want a grammar. ASCIIDOC is too badly specified and requires some very expensive stateful processing to make writing a TS grammar for it an easy effort -- although I've started one. I think having a clearly specified grammar, and one that is free of insanity, for djot will make this a vastly easier effort.)

[jgm]

I might be able to knock out a TreeSitter grammar for djot

That would be amazing. It looks daunting to me. Even though djot's grammar is simpler in some ways than commonmark's, it is still pretty complex from the point of view of TreeSitter; I think you'd still need separate phases for blocks and inlines, the way they do it TreeSitter grammars for markdown/commonmark.

I haven't given thought to built-in support for indexing, but this should be something you could achieve using generic constructs and some filters (AST transformations). For example, you could use spans:

[Burgundy]{index=Burgundy}

[massifrg]

I'm writing a visual editor for pandoc AST that adds support for indices, just using pandoc's flexibility, without the need of any new Inline or Block. I'm going to put it on github with a MIT license.

I'm using a Div with an index-term class for index terms and a Div with an index class around all the Div.index-term ones.

Index terms must have an id set to be referenced in the text.

Index references are just empty Spans put at the start or at the end of a word, with an idref matching the id of the term they refer to.

So without any filter that converts those elements in a real index, in the output

• the references' empty Spans would be erased

• the index would be just a text made of blocks, that would appear like a glossary

Here's an example I made with the editor:

This is a simple text[]{.ixs-ref index-name="subjects" idref="index"}
with a couple of references[]{.ixs-ref index-name="subjects"
idref="reference"} to an index[]{.ixs-ref index-name="subjects"
idref="index"}.

* * * *

{.index index-name="subjects"}
::::::
{#index .index-term index-name="subjects"}
:::
Index

:::

{#reference .index-term index-name="subjects"}
:::
Reference

:::

{#text .index-term index-name="subjects"}
:::
Text

:::

::::::

Here's a screenshot of the example above in the editor.

Months ago I started writing a lua filter to export those indices in ICML.
In the coming months I'll work on in, because now I have some data to do the testing.

Sorry for not posting anything more than some vaporware (though I edited and typeset a real book with the editor and ConTeXt).

[bpj]

Just out of curiosity why not enclose the indexed word in the span? I can see nothing to lose by doing so and you gain the possibility to highlight them in some way if you want to, e.g. for proofreading the rendered text.

That said I wrote a Perl script which is basically a hacky, in some ways more limited — e.g. just one level, no automatic ranges —, in some ways more feature-rich — Unicode-aware, “multilingual”, multi-index, custom sort orders using either Unicode::Collate::Locale or [Sort::ArbBiLex]¹, inserting anchors/links — replacement for Makeindex which can generate basically any format — *TeX, HTML, Markdown etc. — because it uses template strings for output and regexes for parsing. Naturally the Perl script can both read .idx files and generate .ind files, since being able to use mak3idx and similar LaTeX packages unmodified is a design goal.

Anyway when generating the raw data from a Pandoc document I abuse the Cite element, something like [SORTKEY @IDX-ID REALTEXT] where either the real text or the sort key may be left out if they coincide. Whichever is used it is stripped of any formatting (stringified) to generate the actual sort key. I use a filter to replace this with either an anchor/span or some raw code — e.g. makeidx style LaTeX along with a hypertarget — and if desired generate a simple TSV file with the info from the “citation”, the intended output file name and the current section including its identifier and “number”. Needless to say topdown parsing in Lua filters is a godsend for this. Before I had to make sure each section was enclosed in a div and use “subfilters” to process those top down and before that a Perl based filter.

If I learned anything it was that efficiently generating good indices is hard.

Footnotes

1. Since not only do I need to generate indices for texts in multiple languages; I need to sort Sanskrit and Tibetan in Latin script, which use their own sorting order. ↩

[massifrg]

Just out of curiosity why not enclose the indexed word in the span? I can see nothing to lose by doing so and you gain the possibility to highlight them in some way if you want to, e.g. for proofreading the rendered text.

Simply to overcome a limitation of the editor, which is based on Prosemirror, the best kit I know to make a rich text editor in a browser.

Prosemirror has a flattened, not tree-like, structure at paragraph level (see here), that simplifies the task of the editor.

When you have a Span over one or more words, a future edit could break it in two, resulting in two index references instead of one.

But that's about the internal coding of an index reference. What I actually do in the editor is this: I select a span of text -- say a person name for the index of names -- and then click on the names' index ref button (right in the toolbar in the image)

Suppose I selected the word "Robert". The editor:

• creates an empty Span just before the word "Robert", with an attribute indexed-text set to "Robert" and a class that is specific for the index of names (in the editor it's rendered as a light blue icon)

• makes a decoration that spans over the "Robert" word

Here's how it looks like:

Here's the dialog to modify the attribute of the Span element:

When you save the document and then reopen it, the editor looks at the indexed-text attribute to re-decorate the word "Robert".

If you edit it and change it to "Bob", the decoration would perhaps follow the editing, but you'll lose the match between the text you see and the indexed-text attribute; so the next time you open the document, you'll get a decoration over the next 3 characters after "Bob".

It's possible to correct this in real time, but it would slow the editor.
I think a better solution is having a "check button" that runs several tests on the document and highlights inconsistencies; Prosemirror has tools for that.

Index references are punctual in nature; also ranges are made of two punctual references, begin and end.
I do agree that seeing spans is useful (and that's what I did in a project that is the father of the editor).
Anyway a filter could reconstruct the non-empty Span, as long as the indexed-text attribute is accurate.

[irskep]

I added a proof of this concept to Djockey. Docs here: https://steveasleep.com/djockey/plugins/indexterms.html

Plugin source: https://github.com/irskep/djockey/blob/main/src/plugins/indextermsPlugin.ts

Would this potentially serve your needs? Djockey doesn't do PDF output yet, but I could prioritize it.

[massifrg]

Thank you @irskep for your message.

It looks like you're building an index collecting index terms the way LaTeX and ConTeXt do.

Though I'm mostly on ConTeXt for typesetting, I'm using a model with a predefined database of index terms, with references to them in the text (e.g. see here).

That's the way indices are modeled in ICML, DOCX and ODT.

But, even without a database of index terms from the beginning, you may follow a mixed path:

• put references in the text

• collect the marked texts into a database of index terms; if I understand right, this is what your plugin is doing

• refine the database into an index

• connect references to the index terms' identifiers

BTW, take a look at pandoc-export-index.
It's already usable, though

• it works only for indices with 1 level (no sub terms for now)

• it exports to ICML, DOCX and ODT only (but ConTeXt, at least, is on my agenda)