Extensibility in djot

[gavinhoward]

When I first looked at the syntax reference, I thought that djot was not extensible. However, Dr. MacFarlane corrected me.

Three design decisions together make djot extensible:

1. Attributes can be attached to any item.
2. It is possible to create any kind of container.
3. It is possible to do post-processing on the AST.

Any program that does post-processing can look for certain attributes on items and reform the AST as needed. AFAICT, this is sufficiently powerful to take away any need for extensibility in the language.

My original use case was for rich EBNF, like in Go's specification. However, I believe the sky's the limit with this, so much so in fact that it might replace all of my uses of Markdown and LaTeX.

I created this discussion as something to point to if someone asks the same question I did.

(Note: I originally emailed Dr. MacFarlane about this, and he made me aware of Discussions. I haven't used GitHub much for at least 2.5 years. I give Dr. MacFarlane permission to talk more about our email conversation if needed.)

[matklad]

so much so in fact that it might replace all of my uses of Markdown and LaTeX.

And my AsciiDoctor!

Nothing substantial to add, just my agreement on that:

• this kind of extensibility is a huuuuge deal
• this kind of extensibility is either completely missing, or is half-backed in other languages, or requires a heavy, "full" syntax like HTML
• Djot solves the problem more more or less perfectly

Basically, I've wrote a whole post on the topic and then learned that my wish is granted in the form of djot )

[gavinhoward]

Yep! Your post was what convinced me that Dr. MacFarlane was right when he told me about the possibility of post-processing. I might have rejected that otherwise.

[chrisjsewell]

I think it might be helpful here, to split up the extensibility into types and have a think about how each one can/should be achieved in djot.
To my mind, and as mentioned in #112 (comment), these are the types:

• inline, nested content
• inline, verbatim content
• inline, no content (aka leaf)
• block, nested content
• block, verbatim content
• block, no content (aka leaf)

plus ideally being able to pass (key/value) options to all

Also, comparing to reStructuredText roles (inline) and directives (block), it would be helpful for each to have a singular "name", to convey semantic meaning

An inline role :myname:`content`

A leaf block directive

.. myname::

A directive with an argument, options and content

.. myname:: argument
   :key: value

   content

I think of the name akin to a function name in programming languages

def myname(argument, key=value, content=content):
   return some_ast

Inline

This takes into account #112, i.e. possibly making the emoji syntax more generic

Inline, nested

Here, there is only really one option: [content]{name=myname key=value}

It does feel a little verbose though

Perhaps ideally it would be something like [content]{=myname key=value}

Inline, verbatim

There is kind of three options here:

1. `content`{name=myname key=value}
2. `content`{=myname}{key=value}
3. :myname:{content="content" key=value}

(2) is the most concise, although it's maybe not really the intended use for the "raw format" syntax

Inline, leaf

Here there is only one, if #112 is enacted

:myname:{key=value}

Block

Block, nested

{key=value}
:::myname
content
:::

Note, I opened #140 to potentially make this easier to distinguish myname in the AST

Block, verbatim

There are two very similar

{key=value}
```myname
content
```

{key=value}
```=myname
content
```

Were myname is either stored under lang of a code block or format of a raw block in the AST.

Block, leaf

I don't think there is really anything that can be used for this currently?

---

Looking above, I feel there could be some subtle improvements here

FYI, personally, I use Sphinx a lot, and am the author of MyST-Parser, a CommonMark extended syntax with approximate feature parity to reStructuredText.

So I'm definitely interested in looking to create a djot parser for Sphinx

[bpj]

Ideally the shorthand for [content]{name=name} should be [content]{name}. However <tag name> is actually allowed as a shorthand for <tag name=true> is actually allowed in HTML (although not in XHTML) so whichever syntax one chooses people are going to be confused between name=name and name=true. Also `content`{=format} is Pandoc Markdown syntax for raw inlines so using {=name} for something else in djot is bound to confuse anyone who ever has used both (it confused me already reading the above! 😄 ) Possibly something like

• {name=*} for {name=name}
• {name=+} for {name=true}
• {name=-} or {name=!} for {name=false}

might preclude confusion (assuming {name="*"} would mean an actual star as value)

[bpj]

I see that

```=format
raw block
```

`raw inline`{=format}

is already djot syntax!

[chrisjsewell]

Ideally the shorthand for [content]{name=name} should be [content]{name}.

Heya, I feel this is not quite what I'm after; it's not that I just want name=name, it's a single key going to variable values i.e. name=x, name=y, name=z, etc

The key name here may be misleading, it more related to role (https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/Roles) and conveys semantic meaning

I see that ... is already djot syntax

yes but this syntax's purpose is to designate content that should only be output (verbatim) for a particular output format.
It is not related to semantic meaning of the content

[chrisjsewell]

Note, in essence

```=format
raw block
```

equates in rST to the raw directive

.. raw:: format

    raw block

[jgm]

Currently

[hi]{name}

just parses as

doc
  para
    span
      str text="hi"
    str text="{name}"

That is, we don't allow bare keys without values. Some have argued that we should make these bare words class names, avoiding the need for the leading period. Giving them a special "role" function would be another option, but what if there is more than one?