Elusive malformed PDB exports.

[kyleford8]

Running BindCraft with peptide_3stage_multimer settings (I doubt the settings are the issue) I've occasionally observed malformed PDBs being saved during initial hallucination trajectory. Happens really infrequently though (like, a job will run for 40 hrs with no errors before encountering, still generating hits passing filters along the way). Typically gets caught at the "align_pdbs" function call, with the following output error:

ERROR: Assertion "count == ref_coords.size()" failed.

Attached example malformed PDB as text file here. malformed.txt

The trajectory PDB has multiple duplicate chain A and chain Bs as if it's saving the results of multiple models to one PDB file. At first I thought it was a Rosetta issue, due to the error, but since the trajectory PDB is malformed it's probably upstream. Anecdotally, has only been observed when running on multiple GPUs via a bash script, although that may be because these are usually my longest running executions (see example below):

for i in 0 1 2 3; do
export CUDA_VISIBLE_DEVICES=$i 
python -u ./bindcraft.py --settings "./settings_target/TEST_${i}.json" --filters './settings_filters/peptide_relaxed_filters.json' --advanced './settings_advanced/peptide_3stage_multimer.json' &
done

wait

These are all outputting to different folders though, and the error happens at random points for each run. My guess is this is probably a ColabDesign issue at its root? Will dig a bit further and close / move the issue to colabdesign if I can't reproduce more consistently in the next few days.

[martinpacesa]

Hm that is interesting, but I gotta admit I never tried to run it that way. Perhaps there is some shared memory within script? Gotta say this is an issue beyond something I can fix :-/

[kyleford8]

No worries, just noticed #97 which seems potentially related. Playing around with a few things to see if I can get it to happen without waiting multiple days (and burning GPU time 😄 )

Maybe related to?:

https://github.com/sokrypton/ColabDesign/blob/4c0bc6d67f8f967135ecccc135a26b3bfded25e8/colabdesign/shared/utils.py#L7

Will look a bit further to reproduce better, but getting the feeling this is either a colab design or a machine specific issue, and will close accordingly.