OpenFF Gen 2 Optimization (v1.0) series

[jmaat]

These jobs had errors or were incomplete in the following data sets -

'OpenFF Gen 2 Opt Set 4 eMolecules Discrepancy':
IDs for 5 Errors:
18435692
18436178
18436876
18436926
18437240

'OpenFF Gen 2 Opt Set 2 Coverage':
IDs for 5 Errors:
18433779
18433828
18434085
18434088
18434121

IDs for 16 INCOMPLETE:
18433929
18433930
18433937
18433938
18433944
18433945
18433984
18433988
18433998
18434004
18434006
18434010
18434013
18434015
18434069
18434075

[dotsdl]

Update!

OpenFF Gen 2 Opt Set 4 eMolecules Discrepancy

Current errored OptimizationRecords:
18436178
18437465
18437470
18438079
18438248

The errors for this dataset all appear to be geometric.errors.GeomOptNotConvergedErrors, with output like the following:

geomeTRIC run_json error:
Traceback (most recent call last):
  File "/opt/conda/envs/qcfractal/lib/python3.6/site-packages/geometric/run_json.py", line 214, in geometric_run_json
    geometric.optimize.Optimize(coords, M, IC, engine, None, params)
  File "/opt/conda/envs/qcfractal/lib/python3.6/site-packages/geometric/optimize.py", line 1331, in Optimize
    return optimizer.optimizeGeometry()
  File "/opt/conda/envs/qcfractal/lib/python3.6/site-packages/geometric/optimize.py", line 1301, in optimizeGeometry
    raise GeomOptNotConvergedError("Optimizer.optimizeGeometry() failed to converge.")
geometric.errors.GeomOptNotConvergedError: Optimizer.optimizeGeometry() failed to converge.

@jmaat are there changes you would like to make for these optimizations to achieve converged results?

OpenFF Gen 2 Opt Set 2 Coverage

I have resubmitted 10 of the errored OptimizationRecords for this dataset; these 10 showed only an unknown error from QCEngine. The ids for these are:
18433929
18433930
18433937
18433944
18433984
18433988
18434010
18434013
18434015
18434075

There are 4 additional errored records:
18433828
18434085
18434088
18434121

Each of these gives a convergence error as below:

geomeTRIC run_json error:
Traceback (most recent call last):
  File "/home/chodera/miniconda/envs/qcfractal/lib/python3.6/site-packages/geometric/run_json.py", line 214, in geometric_run_json
    geometric.optimize.Optimize(coords, M, IC, engine, None, params)
  File "/home/chodera/miniconda/envs/qcfractal/lib/python3.6/site-packages/geometric/optimize.py", line 1331, in Optimize
    return optimizer.optimizeGeometry()
  File "/home/chodera/miniconda/envs/qcfractal/lib/python3.6/site-packages/geometric/optimize.py", line 1298, in optimizeGeometry
    self.calcEnergyForce()
  File "/home/chodera/miniconda/envs/qcfractal/lib/python3.6/site-packages/geometric/optimize.py", line 1002, in calcEnergyForce
    spcalc = self.engine.calc(self.X, self.dirname)
  File "/home/chodera/miniconda/envs/qcfractal/lib/python3.6/site-packages/geometric/engine.py", line 873, in calc
    return self.calc_new(coords, dirname)
  File "/home/chodera/miniconda/envs/qcfractal/lib/python3.6/site-packages/geometric/engine.py", line 865, in calc_new
    raise QCEngineAPIEngineError("QCEngineAPI computation did not execute correctly. Message: " + ret["error"]["error_message"])
geometric.errors.QCEngineAPIEngineError: QCEngineAPI computation did not execute correctly. Message: QCEngine Unknown Error: Traceback (most recent call last):
  File "/home/chodera/miniconda/envs/qcfractal/lib//python3.6/site-packages/psi4/driver/json_wrapper.py", line 217, in run_json
    json_data = run_json_qcschema(copy.deepcopy(json_data), clean)
  File "/home/chodera/miniconda/envs/qcfractal/lib//python3.6/site-packages/psi4/driver/json_wrapper.py", line 312, in run_json_qcschema
    val, wfn = methods_dict_[json_data["driver"]](method, **kwargs)
  File "/home/chodera/miniconda/envs/qcfractal/lib//python3.6/site-packages/psi4/driver/driver.py", line 691, in gradient
    wfn = procedures['gradient'][lowername](lowername, molecule=molecule, **kwargs)
  File "/home/chodera/miniconda/envs/qcfractal/lib//python3.6/site-packages/psi4/driver/procrouting/proc.py", line 2064, in run_scf_gradient
    ref_wfn = run_scf(name, **kwargs)
  File "/home/chodera/miniconda/envs/qcfractal/lib//python3.6/site-packages/psi4/driver/procrouting/proc.py", line 2002, in run_scf
    scf_wfn = scf_helper(name, post_scf=False, **kwargs)
  File "/home/chodera/miniconda/envs/qcfractal/lib//python3.6/site-packages/psi4/driver/procrouting/proc.py", line 1363, in scf_helper
    e_scf = scf_wfn.compute_energy()
  File "/home/chodera/miniconda/envs/qcfractal/lib//python3.6/site-packages/psi4/driver/procrouting/scf_proc/scf_iterator.py", line 94, in scf_compute_energy
    raise e
  File "/home/chodera/miniconda/envs/qcfractal/lib//python3.6/site-packages/psi4/driver/procrouting/scf_proc/scf_iterator.py", line 87, in scf_compute_energy
    self.iterations()
  File "/home/chodera/miniconda/envs/qcfractal/lib//python3.6/site-packages/psi4/driver/procrouting/scf_proc/scf_iterator.py", line 447, in scf_iterate
    raise SCFConvergenceError("""SCF iterations""", self.iteration_, self, Ediff, Dnorm)
psi4.driver.p4util.exceptions.SCFConvergenceError: Could not converge SCF iterations in 200 iterations.

Are there changes you would like to make for these optimizations to achieve converged results?

[dotsdl]

All optimization tasks with state 'ERROR' have been restarted for the Gen 2 Optimization series of collections. All torsiondrive services with state 'ERROR' have also been restarted.

We still see many cases of 'INCOMPLETE' optimization tasks. All of these appear to be coming from Lilac. @jchodera, can you verify that the manager(s) on Lilac have been shut down?

[dotsdl]

Correction: 'INCOMPLETE's for this dataset appear to be few, and they are currently coming from PRP. It's the Gen 2 Torsion sets that feature many INCOMPLETEs that appear to be from Lilac.

[dotsdl]

OpenFF Gen 2 Opt Set 1 Roche

No unfinished Optimizations, but 4 NaNs present:

Cc1ccccc1OC2CCCCC2-0a    NaN
Cc1ccccc1OC2CCCCC2-1a    NaN
Cc1ccccc1OC2CCCCC2-2a    NaN
Cc1ccccc1OC2CCCCC2-3a    NaN

OpenFF Gen 2 Opt Set 2 Coverage

14 INCOMPLETE Optimizations; each 1 week old since restart from ERROR last week

• last known running on PRP, UCI, MolSSI resources

OpenFF Gen 2 Opt Set 3 Pfizer Discrepancy

All Optimizations COMPLETE!

OpenFF Gen 2 Opt Set 4 eMolecules Discrepancy

14 INCOMPLETE Optimizations; each 1 week old since restart from ERROR last week

• last known running on PRP, UCI

OpenFF Gen 2 Opt Set 5 Bayer

26 INCOMPLETE Optimizations; all except one a week old since restart from ERROR last week

• last known running on PRP, UCI, Galileo