Protocol to extend the schema

[JosePizarro3]

We talked today (@JFRudzinski @Bernadette-Mohr @ndaelman-hu), 21.06.2024, on how to prepare a protocol to extend the current schema. I propose that all the info we agreed about will be transferred to the documentation page, under a "How to extend the Simulations schema".

I think this could be summarized in the next steps:
https://docs.google.com/presentation/d/1KRXpcptMz4bZcpkPl4o4Xk2n_uHGWAqOvBDkVB3zuBE/edit?usp=sharing

I hope it is understandable. Regarding other points raised by @ndaelman-hu:

How to check the prexisting schema (we don't have any real tools there, yet).
How to add documentation, including links to IRI definitions.

Documentation page. Definitions should be in the m_def of each class, but we are waiting for Area D to finish their implementation.

Where to place extensions within the file structure.

Not sure what this means.

When to call the inherited normalizer, e.g. at the start, at the end, in between.

I'll say at the beginning to keep the logic clean.

When to add something to the simulation schema, the parser, or the workflow schema.

Just want to comment that perhaps workflow-simulations should live here, in this same repo. Any thoughts, @JFRudzinski @ladinesa ?

[JFRudzinski]

I am very much for workflow-simulations being merged in this project. There is potentially a lot of cross-over between workflow results and outputs!

[ndaelman-hu]

There is potentially a lot of cross-over between workflow results and outputs!

You're right. I raised this issue a while back for n-phonon scattering. Just some doubts about migrating everything:

• Should we now also maintain the workflow structures, serial, parallel, etc. in the same repo? This will become huge.
• What about other Areas? Do they use these schemas too? I think we prefer to keep our core work separated.

[ladinesa]

I actually started transfering the sim workflow.schema but sure, we can put it here.

[JFRudzinski]

Is there any arguments against?

[ladinesa]

None from my side. I actually prefer it lives with the sim data schema

[ndaelman-hu]

Is there any arguments against?

Just some considerations, see above.

[JosePizarro3]

I think is a good moment to simply create a sub-folder src/nomad_simulations/schema_packages/workflows and move all the python modules in there.

But for that, you would need to wait for #88 to be merged 🙂

[ndaelman-hu]

Okay, some feedback on the flowchart. A visual representation is indeed very nice, but we should first establish the actual protocol (in writing) before spending time on a visual. You'll end up with 10 versions by the end.

Regarding the contents I agree with the spirit. I'd mostly say that some questions are currently too vague. Suggested specifications:

Is the section already defined in nomad-simulations?

This mostly comes back to my remark on how the dev should go about checking existence.

@JosePizarro3 You mentioned something about the docs, right? Did you mean that Area D is working on automatically reading m_def into mkdocs? That'd be great, since the user now has text search. Are there any other tools that we could offer (via VSCode)?

Moreover, I think that only checking nomad-simulations is too reductive: they should check the parser schema (fork tree included), then the simulations schema, and lastly the core NOMAD, in that order.

Is your section conceptually related with an existing section?

Tbh, this is the vaguest of them all. Having sat through multiple of these discussions, people tend to set the bar very differently.
Btw, f we want to follow SOLID, conceptual similarity wouldn't factor into the equation.

Still, schemas are a different beast from typical classes (again, inheritance does not entail is_a). Personally, I don't know of any reference framework for schemas. Naively, I'd say having common normalization and / or quantities could be a heuristic. These still have their own issues, though.

Probably a better approach would be to distinguish when the dev should:

1. inherit (default suggestion).
2. compose / has_a relation (to be avoided most of the time).
3. keep the section separate, but use different interactions when appropriate.

Document your new sections and methods.

This applies to both answers... Is m_def enough?
We should also mention IRI linking. I guess that there is no mechanism for how to check interoperable definitions with IRI of the parent sections.

Contact nomad-simulations developers: is your section interoperable?

We know what interoperable means, but I wouldn't assume our audience does. Verifying interoperability is also time-costly, as it comes down to technical details more so than conceptual ones. I suggest that we instead prepare a place where they can "register" their plugins, if they're interest in sharing + support. They could reach out to us for a migration audit, whenever they think the plugin is mature enough.

Btw, going by your diagram semantics, this bubble should be split into a square that points to a circle.
This is also common to any plugin, not just those with sections inheriting from ArchiveSection.

[ndaelman-hu]

Where to place extensions within the file structure.
Not sure what this means.

When to call the inherited normalizer, e.g. at the start, at the end, in between.
I'll say at the beginning to keep the logic clean.

I agree that at the beginning is preferable in most cases, more so for access to the previous quantities rather than code aesthetics.
My question is rather: are there scenarios where this fails? E.g. what if they want to skip a computationally expensive step? What if they completely want to overwrite the normalization?

[JosePizarro3]

New figure: