Csse layout specfication (#359)

* atspec changes round1

* adjust py matrix and update actions

* pkg_res for 312?

* pkg_res for 312? take 3

* update isort and imports for min py38

* should be working

* changelog

.github/workflows/CI.yaml

@@ -29,7 +29,7 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
       - name: Checkout Code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
       - name: Install QCElemental (full deps)
         if: matrix.python-version != '3.8' && matrix.python-version != '3.10'
         run: pip install '.[test,viz,align,standard]'
@@ -39,9 +39,9 @@ jobs:
       - name: Run tests
         run: pytest -rws -v --cov=qcelemental --color=yes --cov-report=xml  #-k "not pubchem_multiout_g"
       - name: Upload coverage to Codecov
-        uses: codecov/codecov-action@v3 # NEEDS UPDATE TO v3 https://github.com/codecov/codecov-action
+        uses: codecov/codecov-action@v5
       - name: QCSchema Examples Deploy
-        uses: JamesIves/github-pages-deploy-action@4.1.1
+        uses: JamesIves/github-pages-deploy-action@v4
         #if: github.event_name == 'push' && github.repository == 'MolSSI/QCElemental' && ( startsWith( github.ref, 'refs/tags/' ) || github.ref == 'refs/heads/master' )
         if: false
         with:
@@ -55,7 +55,7 @@ jobs:
       fail-fast: false
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - uses: actions/setup-python@v4
         name: Set up Python
         with:
@@ -67,7 +67,7 @@ jobs:
       - name: Build Documentation
         run: sphinx-build docs/ build/docs
       - name: GitHub Pages Deploy
-        uses: JamesIves/github-pages-deploy-action@4.1.1
+        uses: JamesIves/github-pages-deploy-action@v4
         if: github.event_name == 'push' && github.repository == 'MolSSI/QCElemental' && ( startsWith( github.ref, 'refs/tags/' ) || github.ref == 'refs/heads/master' )
         with:
           branch: gh-pages

.pre-commit-config.yaml

@@ -14,10 +14,8 @@ repos:
       - id: detect-secrets
         stages: [commit]
         exclude: "raw_data/nist_data/"
-  # TODO: Update to 5.12.x once we drop Python3.7 support
-  # https://levelup.gitconnected.com/fix-runtimeerror-poetry-isort-5db7c67b60ff
   - repo: https://github.com/PyCQA/isort
-    rev: 5.11.5
+    rev: 5.13.2
     hooks:
       - id: isort
   - repo: https://github.com/psf/black

docs/changelog.rst

@@ -37,10 +37,14 @@ New Features
 
 Enhancements
 ++++++++++++
-- (536b) ``v1.AtomicResult.convert_v`` learned a ``external_input_data`` option to inject that field (if known) rather than using incomplete reconstruction from the v1 Result. may not be the final sol'n.
-- (536b) ``v2.FailedOperation`` gained schema_name and schema_version=2.
-- (536b) ``v2.AtomicResult`` no longer inherits from ``v2.AtomicInput``. It gained a ``input_data`` field for the corresponding ``AtomicInput`` and independent ``id`` and ``molecule`` fields (the latter being equivalvent to ``v1.AtomicResult.molecule`` with the frame of the results; ``v2.AtomicResult.input_data.molecule`` is new, preserving the input frame). Gained independent ``extras``
-- (536b) Both v1/v2 ``AtomicResult.convert_v()`` learned to handle the new ``input_data`` layout.
+- (:pr:`359`) ``v2.AtomicInput`` lost extras so extras belong unambiguously to the specification.
+- (:pr:`359`) ``v2.AtomicSpecification``, unlike ``v1.QCInputSpecification``, doesn't have schema_name and schema version.
+- (:pr:`359`) misc -- ``isort`` version bumped to 5.13 and imports and syntax take advantage of python 3.8+
+- (:pr:`359`) ``v2.AtomicInput`` gained a ``specification`` field where driver, model, keywords, extras, and protocols now live. ``v2.AtomicSpecification`` and ``v1.QCInputSpecification`` (used by opt and td) learned a ``convert_v`` to interconvert.
+- (:pr:`358`) ``v1.AtomicResult.convert_v`` learned a ``external_input_data`` option to inject that field (if known) rather than using incomplete reconstruction from the v1 Result. may not be the final sol'n.
+- (:pr:`358`) ``v2.FailedOperation`` gained schema_name and schema_version=2.
+- (:pr:`358`) ``v2.AtomicResult`` no longer inherits from ``v2.AtomicInput``. It gained a ``input_data`` field for the corresponding ``AtomicInput`` and independent ``id`` and ``molecule`` fields (the latter being equivalvent to ``v1.AtomicResult.molecule`` with the frame of the results; ``v2.AtomicResult.input_data.molecule`` is new, preserving the input frame). Gained independent ``extras``
+- (:pr:`358`) Both v1/v2 ``AtomicResult.convert_v()`` learned to handle the new ``input_data`` layout.
 - (:pr:`357`, :issue:`536`) ``v2.AtomicResult``, ``v2.OptimizationResult``, and ``v2.TorsionDriveResult`` have the ``success`` field enforced to ``True``. Previously it could be set T/F. Now validation errors if not T. Likewise ``v2.FailedOperation.success`` is enforced to ``False``.
 - (:pr:`357`, :issue:`536`) ``v2.AtomicResult``, ``v2.OptimizationResult``, and ``v2.TorsionDriveResult`` have the ``error`` field removed. This isn't used now that ``success=True`` and failure should be routed to ``FailedOperation``.
 - (:pr:`357`) ``v1.Molecule`` had its schema_version changed to a Literal[2] (remember Mol is one-ahead of general numbering scheme) so new instances will be 2 even if another value is passed in. Ditto ``v2.BasisSet.schema_version=2``. Ditto ``v1.BasisSet.schema_version=1`` Ditto ``v1.QCInputSpecification.schema_version=1`` and ``v1.OptimizationSpecification.schema_version=1``.

pyproject.toml

@@ -63,7 +63,7 @@ test = [
 lint = [
     "pre-commit",
     "black >=22.1.0,<23.0a0",  # if running outside of pre-commit
-    "isort ==5.11.5",  # if running outside of pre-commit
+    "isort >=5.13.2",  # if running outside of pre-commit
     # "mypy",
     # "autoflake",
     # "flake8",

qcelemental/models/v1/basis.py

@@ -1,11 +1,5 @@
 from enum import Enum
-from typing import Dict, List, Optional
-
-try:
-    from typing import Literal
-except ImportError:
-    # remove when minimum py38
-    from typing_extensions import Literal
+from typing import Dict, List, Literal, Optional
 
 from pydantic.v1 import ConstrainedInt, Field, constr, validator
 

qcelemental/models/v1/procedures.py

@@ -1,11 +1,5 @@
 from enum import Enum
-from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
-
-try:
-    from typing import Literal
-except ImportError:
-    # remove when minimum py38
-    from typing_extensions import Literal
+from typing import TYPE_CHECKING, Any, Dict, List, Literal, Optional, Tuple, Union
 
 from pydantic.v1 import Field, conlist, constr, validator
 
@@ -75,6 +69,24 @@ class QCInputSpecification(ProtoModel):
     def _version_stamp(cls, v):
         return 1
 
+    def convert_v(
+        self, version: int
+    ) -> Union["qcelemental.models.v1.QCInputSpecification", "qcelemental.models.v2.AtomicSpecification"]:
+        """Convert to instance of particular QCSchema version."""
+        import qcelemental as qcel
+
+        if check_convertible_version(version, error="QCInputSpecification") == "self":
+            return self
+
+        dself = self.dict()
+        if version == 2:
+            dself.pop("schema_name")
+            dself.pop("schema_version")
+
+            self_vN = qcel.models.v2.AtomicSpecification(**dself)
+
+        return self_vN
+
 
 class OptimizationInput(ProtoModel):
     id: Optional[str] = None
@@ -180,7 +192,8 @@ def convert_v(
         dself = self.dict()
         if version == 2:
             # remove harmless empty error field that v2 won't accept. if populated, pydantic will catch it.
-            dself.pop("error", None)
+            if not dself.get("error", True):
+                dself.pop("error")
 
             dself["trajectory"] = [trajectory_class(**atres).convert_v(version) for atres in dself["trajectory"]]
             dself["input_specification"].pop("schema_version", None)
@@ -297,6 +310,7 @@ def convert_v(
             return self
 
         dself = self.dict()
+        # dself = self.model_dump(exclude_unset=True, exclude_none=True)
         if version == 2:
             dself["input_specification"].pop("schema_version", None)
             dself["optimization_spec"].pop("schema_version", None)
@@ -355,14 +369,17 @@ def convert_v(
         dself = self.dict()
         if version == 2:
             # remove harmless empty error field that v2 won't accept. if populated, pydantic will catch it.
-            dself.pop("error", None)
+            if not dself.get("error", True):
+                dself.pop("error")
 
             dself["input_specification"].pop("schema_version", None)
             dself["optimization_spec"].pop("schema_version", None)
             dself["optimization_history"] = {
                 k: [opthist_class(**res).convert_v(version) for res in lst]
                 for k, lst in dself["optimization_history"].items()
             }
+            # if dself["optimization_spec"].pop("extras", None):
+            #    pass
 
             self_vN = qcel.models.v2.TorsionDriveResult(**dself)
 

qcelemental/models/v1/results.py

@@ -1,12 +1,6 @@
 from enum import Enum
 from functools import partial
-from typing import TYPE_CHECKING, Any, Dict, Optional, Set, Union
-
-try:
-    from typing import Literal
-except ImportError:
-    # remove when minimum py38
-    from typing_extensions import Literal
+from typing import TYPE_CHECKING, Any, Dict, Literal, Optional, Set, Union
 
 import numpy as np
 from pydantic.v1 import Field, constr, validator
@@ -619,16 +613,25 @@ def _version_stamp(cls, v):
         return 1
 
     def convert_v(
-        self, version: int
+        self, target_version: int, /
     ) -> Union["qcelemental.models.v1.AtomicInput", "qcelemental.models.v2.AtomicInput"]:
         """Convert to instance of particular QCSchema version."""
         import qcelemental as qcel
 
-        if check_convertible_version(version, error="AtomicInput") == "self":
+        if check_convertible_version(target_version, error="AtomicInput") == "self":
             return self
 
         dself = self.dict()
-        if version == 2:
+        spec = {}
+        if target_version == 2:
+            dself.pop("schema_name")  # changes in v2
+
+            spec["driver"] = dself.pop("driver")
+            spec["model"] = dself.pop("model")
+            spec["keywords"] = dself.pop("keywords", None)
+            spec["protocols"] = dself.pop("protocols", None)
+            spec["extras"] = dself.pop("extras", None)
+            dself["specification"] = spec
             self_vN = qcel.models.v2.AtomicInput(**dself)
 
         return self_vN
@@ -798,55 +801,66 @@ def _native_file_protocol(cls, value, values):
 
     def convert_v(
         self,
-        version: int,
+        target_version: int,
+        /,
         *,
         external_input_data: Optional[Any] = None,
     ) -> Union["qcelemental.models.v1.AtomicResult", "qcelemental.models.v2.AtomicResult"]:
         """Convert to instance of particular QCSchema version.
 
         Parameters
         ----------
-        version
+        target_version
             The version to convert to.
         external_input_data
             Since self contains data merged from input, this allows passing in the original input, particularly for `molecule` and `extras` fields.
-            Can be model or dictionary and should be *already* converted to the desired version.
+            Can be model or dictionary and should be *already* converted to target_version.
             Replaces ``input_data`` field entirely (not merges with extracts from self) and w/o consistency checking.
 
         Returns
         -------
         AtomicResult
-            Returns self (not a copy) if ``version`` already satisfied.
-            Returns a new AtomicResult of ``version`` otherwise.
+            Returns self (not a copy) if ``target_version`` already satisfied.
+            Returns a new AtomicResult of ``target_version`` otherwise.
 
         """
         import qcelemental as qcel
 
-        if check_convertible_version(version, error="AtomicResult") == "self":
+        if check_convertible_version(target_version, error="AtomicResult") == "self":
             return self
 
         dself = self.dict()
-        if version == 2:
+        if target_version == 2:
             # remove harmless empty error field that v2 won't accept. if populated, pydantic will catch it.
-            dself.pop("error", None)
+            if not dself.get("error", True):
+                dself.pop("error")
 
             input_data = {
-                k: dself.pop(k) for k in list(dself.keys()) if k in ["driver", "keywords", "model", "protocols"]
+                "specification": {
+                    k: dself.pop(k) for k in list(dself.keys()) if k in ["driver", "keywords", "model", "protocols"]
+                },
+                "molecule": dself["molecule"],  # duplicate since input mol has been overwritten
             }
-            input_data["molecule"] = dself["molecule"]  # duplicate since input mol has been overwritten
-            # any input provenance has been overwritten
-            input_data["extras"] = {
+            in_extras = {
                 k: dself["extras"].pop(k) for k in list(dself["extras"].keys()) if k in []
-            }  # sep any merged extras
+            }  # sep any merged extras known to belong to input
+            input_data["specification"]["extras"] = in_extras
+
+            # any input provenance has been overwritten
+            # if dself["id"]:
+            #     input_data["id"] = dself["id"]  # in/out should likely match
+
             if external_input_data:
                 # Note: overwriting with external, not updating. reconsider?
-                dself["input_data"] = external_input_data
-                in_extras = (
-                    external_input_data.get("extras", {})
-                    if isinstance(external_input_data, dict)
-                    else external_input_data.extras
-                )
+                if isinstance(external_input_data, dict):
+                    if isinstance(external_input_data["specification"], dict):
+                        in_extras = external_input_data["specification"].get("extras", {})
+                    else:
+                        in_extras = external_input_data["specification"].extras
+                else:
+                    in_extras = external_input_data.specification.extras
                 dself["extras"] = {k: v for k, v in dself["extras"].items() if (k, v) not in in_extras.items()}
+                dself["input_data"] = external_input_data
             else:
                 dself["input_data"] = input_data
 

qcelemental/models/v2/__init__.py

@@ -14,7 +14,14 @@
     TorsionDriveInput,
     TorsionDriveResult,
 )
-from .results import AtomicInput, AtomicResult, AtomicResultProperties, AtomicResultProtocols, WavefunctionProperties
+from .results import (
+    AtomicInput,
+    AtomicResult,
+    AtomicResultProperties,
+    AtomicResultProtocols,
+    AtomicSpecification,
+    WavefunctionProperties,
+)
 
 
 def qcschema_models():

qcelemental/models/v2/basis.py

@@ -1,11 +1,5 @@
 from enum import Enum
-from typing import Dict, List, Optional
-
-try:
-    from typing import Literal
-except ImportError:
-    # remove when minimum py38
-    from typing_extensions import Literal
+from typing import Dict, List, Literal, Optional
 
 from pydantic import Field, constr, field_validator
 from typing_extensions import Annotated

qcelemental/models/v2/common_models.py

@@ -1,11 +1,5 @@
 from enum import Enum
-from typing import TYPE_CHECKING, Any, Dict, Optional, Sequence, Tuple, Union
-
-try:
-    from typing import Literal
-except ImportError:
-    # remove when minimum py38
-    from typing_extensions import Literal
+from typing import TYPE_CHECKING, Any, Dict, Literal, Optional, Sequence, Tuple, Union
 
 import numpy as np
 from pydantic import Field, field_validator

qcelemental/models/v2/procedures.py

@@ -1,11 +1,5 @@
 from enum import Enum
-from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
-
-try:
-    from typing import Literal
-except ImportError:
-    # remove when minimum py38
-    from typing_extensions import Literal
+from typing import TYPE_CHECKING, Any, Dict, List, Literal, Optional, Tuple, Union
 
 from pydantic import Field, conlist, constr, field_validator