Remove extra evaluations and unescapings

[Derugon]

Context

Parser functions always take wikitext arguments. Unlike tags, I would expect some behaviors to always apply to the arguments passed, whatever the function does in practice:

1. Evaluate variables within the wikitext,
2. Remove space-like characters at the start and at the end of the evaluated text.

If the function modifies an argument, it would do it after having normalized the argument. Consequently, if we want to pass wikitext that would still look the same once normalized, we can freeze the wikitext (e.g. by using {{(}} or {{!}}) or use <nowiki/> tags.

Some ParserPower functions apply an additional normalization step:

3. Replace escape sequences within the trimmed text.

Then, once all argument modifications have been applied, the function may re-evaluate variables within the unescaped wikitext.

This provides another way to circumvent normalization: we can use escape sequences within the wikitext to prevent variable syntax from being recognized, and spaces from being removed.

An example of the issue

Let Template:Quote be a template that would print its 1st argument within quotes. We do not want spaces inside the quotes, so we would basically want to use:

"{{#trim: {{{1}}} }}"

Suppose we want to pass {{!}} as argument, but we want it to be printed as-is. From the solutions introduced above, we could:

1. freeze the wikitext, e.g.:

{{quote | {{((}}!{{))}} }}

2. use <nowiki/> tags, e.g.:

{{quote | <nowiki>{{!}}</nowiki> }}

3. use escape sequences, e.g.:

{{#uescnowiki: {{quote | <esc>{{!}}</esc> }} }}

Outputs can be found here. While both the 2nd and 3rd approaches yield "{{!}}", the 1st one yields "|". This is because the #trim function evaluates variables within its argument (by expanding the given pre-processor node), then trims spaces, and finally returns it while telling the parser to evaluate variables (by setting the 'noparse'=false flag).

The issue

Various parser functions and tags in ParserPower evaluate variables within wikitext twice without having unescaped anything between, or unescape wikitext twice.

While changing it may not particularly allow us to do more things than we can already do, each variable evaluation that a parser function tries to do generates and evaluates additional pre-processor nodes. This takes some small additional parsing time, and artificially makes parser reports larger than they should.

Proposed changes

Remove extra variable evaluation, trimming, and unescaping steps. More precisely:

• Do not evaluate a wikitext string if:
  • under normal conditions it should not be standard wikitext with any variables/escape sequences (e.g. a raw error string, a stringified number, an escaped wikitext), or
  • variables have already been evaluated, and no following operation added any variables to the string (e.g. unescaping braces or angle brackets should lead to new variables, replacing by evaluated wikitext within evaluated wikitext should not).
• Do not unescape a wikitext string if:
  • under normal conditions it should not be standard wikitext with any escape sequences (e.g. a raw error string, a stringified number), or
  • it has already been unescaped at least once by the parser function/tag.

To achieve this, a few unrelated changes have been made:

• Small code refactorings are made in this PR to move parameter evaluation/trimming/unescaping to the outer functions as much as possible.
• For functions that accept named arguments, values of unknown arguments are no longer evaluated.

Below is a list of all variable evaluation/unescaping steps that were removed when the wikitext may have contained frozen variable syntax or wikitext escaped multiple times (that was previously evaluated/unescaped twice and is no longer in this PR), i.e. all potentially breaking changes:

• {{#trim:}}
  • The 1st argument (evaluated a 2nd time once trimmed).
• {{#or:}}
  • Any argument (evaluated twice if non empty and returned).
• {{#follow:}}
  • The 1st argument (evaluated twice).
• {{#listfilter:}}
  • The list and default arguments (evaluated twice).
  • The pattern argument (evaluated thrice if the list is not empty).
  • Any other argument (evaluated twice if the list is not empty).
  • When using a pattern as predicate, the pattern after token replacements (evaluated twice then unescaped twice).
• {{#listmap:}}
  • The list and default arguments (evaluated twice).
  • The pattern argument (evaluated thrice if the list is not empty).
  • Any other argument (evaluated twice if the list is not empty).
  • When using a pattern to generate keys, the pattern after token replacements (evaluated twice).
• {{#lstmap:}}
  • The pattern argument (evaluated twice if the list is not empty).
  • The pattern after token replacements (evaluated twice).
• {{#listunique:}}
  • The list and default arguments (evaluated twice).
  • The pattern argument (evaluated thrice if the list is not empty).
  • Any other argument (evaluated twice if the list is not empty).
  • When using a pattern to generate keys, the pattern after token replacements (evaluated twice then unescaped twice).
• {{#listsort:}}
  • The list and default arguments (evaluated twice).
  • The pattern argument (evaluated thrice if the list is not empty).
  • Any other argument (evaluated twice if the list is not empty).
  • When using a pattern to generate keys, the pattern after token replacements (evaluated twice then unescaped twice).
• {{#listmerge:}}
  • The list and default arguments (evaluated twice).
  • The matchpattern and mergepattern arguments (evaluated thrice if the list is not empty).
  • Any other argument (evaluated twice if the list is not empty).
  • When using a pattern to merge pairs of values, the pattern after token replacements (evaluated twice then unescaped twice).

[Derugon]

It seems list functions that return unescaped wikitext parse it twice, so I'm gonna work on it a little more.

[RheingoldRiver]

Sounds good, thanks so much for your contributions already!!

[Derugon]

Well, thank you all for still maintaining it, and for taking the time to review these PRs. :)

[Derugon]

I wrote this had no impact on code, but it is not true.

The base issue

Using the changes from this PR with templates from an existing wiki it caused a (subtle) change, a beneficial one in my case, but still a breaking change: generated wikitext with transcluded wikitext syntax will have it evaluated.

For example, {{#trim: {{(}}{{(}}!{{)}}{{)}} }} yields |, while I would expect it to produce the same result as {{(}}{{(}}!{{)}}{{)}}, i.e. {{!}}.

This means, this PR (as of now) is a breaking change for the #trim and #or parser functions (and only these 2).

The issue with unescaping

This is a side-effect we have to deal with when unescaping: after the text is unescaped, we need to parse it again, so we parse {{#uesc: \{\{!\}\} }} the same way as {{#uesc: {{(}}{{(}}!{{)}}{{)}} }}, and both yield |.

Changing it would mean variables can no longer contribute reliably to unescaping, e.g. {{#uesc: {{X}} }} would yield {{!}} (with Template:X containing \{\{!\}\}), which would completely break the purpose of unescaping.

So I'll double down on what I said last week, and not suggest to remove the extra parsing from unescaped text in this PR.