Detmatch satp1_det_match_240220m failing on 15 obs-ids

[msilvafe]

We've found that 15 observations from satp1 fail to load due to missing detmatch metadata. The error that is returned when calling context.get_det_info(obsid) is pasted below.

Click for traceback

ERROR: sotodlib.core.metadata.loader: An error occurred while processing a meta entry:

  spec:    {'db': '/global/cfs/cdirs/sobs/metadata/satp1/manifests/det_match/satp1_det_match_240220m/assignment_local.sqlite', 'label': 'assignment', 'det_info': True, 'on_missing': 'fail'}

  request: {'obs:obs_id': 'obs_1717265085_satp1_1111111'}


---------------------------------------------------------------------------
IncompleteMetadataError                   Traceback (most recent call last)
Cell In[13], line 1
----> 1 ctx.get_det_info(obsid)

File ~/.local/soconda/20240903_0.1.4/lib/python3.11/site-packages/sotodlib/core/context.py:495, in Context.get_det_info(self, obs_id, dets, samples, filename, detsets, meta, free_tags, on_missing)
    490 """Pass all arguments to :func:`get_meta(det_info_scan=True)`, and
    491 then return only the det_info, as a ResultSet.
    492 
    493 """
    494 if meta is None:
--> 495     meta = self.get_meta(obs_id=obs_id, dets=dets, samples=samples,
    496                          filename=filename, detsets=detsets, free_tags=free_tags,
    497                          on_missing=on_missing, det_info_scan=True)
    498 # Convert
    499 def _unpack(aman):

File ~/.local/soconda/20240903_0.1.4/lib/python3.11/site-packages/sotodlib/core/context.py:464, in Context.get_meta(self, obs_id, dets, samples, filename, detsets, meta, free_tags, check, ignore_missing, on_missing, det_info_scan)
    461     request['dets:readout_id'] = list(dets)
    463 metadata_list = self._get_warn_missing('metadata', [])
--> 464 meta = self.loader.load(metadata_list, request, det_info=det_info, check=check,
    465                         free_tags=free_tags, free_tag_fields=free_tag_fields,
    466                         det_info_scan=det_info_scan, ignore_missing=ignore_missing,
    467                         on_missing=on_missing)
    468 if check:
    469     return meta

File ~/.local/soconda/20240903_0.1.4/lib/python3.11/site-packages/sotodlib/core/metadata/loader.py:493, in SuperLoader.load(self, spec_list, request, det_info, free_tags, free_tag_fields, dest, check, det_info_scan, ignore_missing, on_missing)
    491     error = e
    492 elif _on_missing == 'fail':
--> 493     reraise(_spec, e)
    494 elif _on_missing == 'skip':
    495     # print a warning I guess
    496     logger.warning(f'Skipping failed det_info load, spec={_spec}')

File ~/.local/soconda/20240903_0.1.4/lib/python3.11/site-packages/sotodlib/core/metadata/loader.py:406, in SuperLoader.load.<locals>.reraise(spec, e)
    404         logger.error(a)
    405     e = LoaderError(e.args[0])
--> 406 raise e

File ~/.local/soconda/20240903_0.1.4/lib/python3.11/site-packages/sotodlib/core/metadata/loader.py:485, in SuperLoader.load(self, spec_list, request, det_info, free_tags, free_tag_fields, dest, check, det_info_scan, ignore_missing, on_missing)
    483 item_keys = _filter_items('dets:', item.keys)
    484 try:
--> 485     det_info = merge_det_info(
    486         det_info, item,
    487         on_missing=_on_missing)
    488 except IncompleteMetadataError as e:
    489     if check:
    490         # I guess we report this, either way.

File ~/.local/soconda/20240903_0.1.4/lib/python3.11/site-packages/sotodlib/core/metadata/loader.py:640, in merge_det_info(det_info, new_info, multi, on_missing)
    634         raise ValueError(
    635             'When reconciling new det_info, a disagreement in the value '
    636             f'of field {k} was observed.  If this is due to ambiguity '
    637             f'in the det_id, maybe add {k} to the match_keys?')
    639 if len(det_info) != len(i1) and on_missing != 'trim':
--> 640     raise IncompleteMetadataError('{len(det_info)} -> {len(i1)})')
    642 logger.debug(f' ... updating det_info (row count '
    643              f'{len(det_info)} -> {len(i1)})')
    644 det_info = det_info.subset(rows=i1)

IncompleteMetadataError: {len(det_info)} -> {len(i1)})

The obs-ids we found are: ['obs_1717265085_satp1_1111111', 'obs_1717269701_satp1_1111111', 'obs_1717274153_satp1_1111111', 'obs_1717276387_satp1_1111111', 'obs_1717280216_satp1_1111111', 'obs_1717284043_satp1_1111111', 'obs_1717288021_satp1_1111111', 'obs_1717291699_satp1_1111111', 'obs_1717296378_satp1_1111111', 'obs_1717299355_satp1_1111111', 'obs_1717303183_satp1_1111111', 'obs_1717307120_satp1_1111111', 'obs_1717310839_satp1_1111111', 'obs_1717322632_satp1_1111111', 'obs_1717326151_satp1_1111111']

In addition to fixing these it would probably be worthwhile checking this with the more recent detmatch datasets and resolve those issues as @mmccrackan reported some obsids that failed in the most recent detmatch version but not in this 240220m version. All I did to collect these was to loop over all obs-ids and catch any that errored out from a call to ctx.get_det_info(obsid):

errors = []
for obs in obslist:
    obsid = obs['obs_id']
    try:
        _ = ctx.get_det_info(obsid)
    except:
        errors.append(obsid)