Occasional sliderule errors result in different number of returned point values

[scottyhq]

I've seen this happen locally (https://github.com/uw-cryo/coincident/actions/runs/12347943206/job/34456420504?pr=34) and in CI for icesat-2 and gedi endpoints. Perhaps this happens when the public cluster is overloaded or with flaky network connections. But will leave this open and disable those tests for now...

This seems to be the key piece of info from the logs...

ERROR    sliderule.sliderule:sliderule.py:470 Exception <-1>: Failure on resource ATL03_20221030235822_06151702_006_01.h5 track 2.0: H5Coro::Future read failure on /gt2l/geolocation/segment_ph_cnt

_________________________________________________________ TestSlideRule.test_process_atl06sr __________________________________________________________

self = <tests.test_sliderule.TestSlideRule object at 0x170c9e690>
tinyaoi =                                             geometry
0  POLYGON ((-106.47848 38.78311, -106.47848 38.7...

    def test_process_atl06sr(self, tinyaoi):
        gf_is2 = coincident.search.search(
            dataset="icesat-2", intersects=tinyaoi, datetime="2022"
        )
        data = coincident.io.sliderule.process_atl06sr(
            gf_is2, aoi=tinyaoi, include_3dep=True
        )
        assert isinstance(data, gpd.GeoDataFrame)
        assert "n_fit_photons" in data.columns
        assert "3dep.value" in data.columns
>       assert data.shape == (367, 20)
E       assert (262, 20) == (367, 20)
E         
E         At index 0 diff: 262 != 367
E         Use -v to get more diff

data       =                                 h_sigma  cycle  segment_id  rms_misfit  ...  3dep.flags   3dep.value  3dep.file_id    ...97063     17      215159    0.605831  ...           0  3111.166022   17179869184  1.307927e+09

[262 rows x 20 columns]
gf_is2     =                                               assets                                               bbox  ...          ...'xmin': -109.52405, 'ymin': 26.92905, 'xmax':...  ... 2022-11-28 22:34:20.231000+00:00       332

[6 rows x 13 columns]
self       = <tests.test_sliderule.TestSlideRule object at 0x170c9e690>
tinyaoi    =                                             geometry
0  POLYGON ((-106.47848 38.78311, -106.47848 38.7...

tests/test_sliderule.py:66: AssertionError
------------------------------------------------------------------ Captured log call ------------------------------------------------------------------
INFO     pyogrio._io:raw.py:723 Created 1 records
INFO     pyogrio._io:raw.py:723 Created 1 records
INFO     sliderule.sliderule:sliderule.py:472 proxy request <AppServer.90722> querying resources for 3dep
INFO     sliderule.sliderule:sliderule.py:472 proxy request <AppServer.90722> returned 0 resources for 3dep
INFO     sliderule.sliderule:sliderule.py:472 Starting proxy for atl06 to process 6 resource(s) with 6 thread(s)
INFO     sliderule.sliderule:sliderule.py:472 request <AppServer.90726> processing initialized on ATL03_20220227235038_10341406_006_01.h5 ...
INFO     sliderule.sliderule:sliderule.py:472 request <AppServer.204684> processing initialized on ATL03_20221128223420_10571702_006_02.h5 ...
INFO     sliderule.sliderule:sliderule.py:472 request <AppServer.118247> processing initialized on ATL03_20220131125843_06151402_006_01.h5 ...
INFO     sliderule.sliderule:sliderule.py:472 request <AppServer.193333> processing initialized on ATL03_20221030235822_06151702_006_01.h5 ...
INFO     sliderule.sliderule:sliderule.py:472 request <AppServer.33460> processing initialized on ATL03_20220828151032_10341606_006_01.h5 ...
INFO     sliderule.sliderule:sliderule.py:472 request <AppServer.188682> processing initialized on ATL03_20220529193033_10341506_006_01.h5 ...
INFO     sliderule.sliderule:sliderule.py:472 request <AppServer.118247> processing of ATL03_20220131125843_06151402_006_01.h5 complete (0/0/0)
INFO     sliderule.sliderule:sliderule.py:472 request <AppServer.188682> processing of ATL03_20220529193033_10341506_006_01.h5 complete (131020/288/0)
INFO     sliderule.sliderule:sliderule.py:472 request <AppServer.204684> processing of ATL03_20221128223420_10571702_006_02.h5 complete (0/0/0)
ERROR    sliderule.sliderule:sliderule.py:470 Exception <-1>: Failure on resource ATL03_20221030235822_06151702_006_01.h5 track 2.0: H5Coro::Future read failure on /gt2l/geolocation/segment_ph_cnt
INFO     sliderule.sliderule:sliderule.py:472 request <AppServer.193333> processing of ATL03_20221030235822_06151702_006_01.h5 complete (115733/17/0)
INFO     sliderule.sliderule:sliderule.py:472 request <AppServer.33460> processing of ATL03_20220828151032_10341606_006_01.h5 complete (130798/45/0)
INFO     sliderule.sliderule:sliderule.py:472 request <AppServer.118247> processing complete (0/1/0/0)
INFO     sliderule.sliderule:sliderule.py:472 request <AppServer.90726> processing of ATL03_20220227235038_10341406_006_01.h5 complete (0/0/0)
INFO     sliderule.sliderule:sliderule.py:472 Successfully completed processing resource [1 out of 6]: ATL03_20220131125843_06151402_006_01.h5
INFO     sliderule.sliderule:sliderule.py:472 request <AppServer.204684> processing complete (0/1/0/0)
INFO     sliderule.sliderule:sliderule.py:472 request <AppServer.188682> processing complete (52/1/52/0)
INFO     sliderule.sliderule:sliderule.py:472 request <AppServer.33460> processing complete (53/1/53/0)
INFO     sliderule.sliderule:sliderule.py:472 request <AppServer.193333> processing complete (157/1/157/0)
INFO     sliderule.sliderule:sliderule.py:472 request <AppServer.90726> processing complete (0/1/0/0)
INFO     sliderule.sliderule:sliderule.py:472 Successfully completed processing resource [6 out of 6]: ATL03_20221128223420_10571702_006_02.h5
INFO     sliderule.sliderule:sliderule.py:472 Successfully completed processing resource [3 out of 6]: ATL03_20220529193033_10341506_006_01.h5
INFO     sliderule.sliderule:sliderule.py:472 Successfully completed processing resource [5 out of 6]: ATL03_20221030235822_06151702_006_01.h5
INFO     sliderule.sliderule:sliderule.py:472 Successfully completed processing resource [4 out of 6]: ATL03_20220828151032_10341606_006_01.h5
INFO     sliderule.sliderule:sliderule.py:472 Successfully completed processing resource [2 out of 6]: ATL03_20220227235038_10341406_006_01.h5

[scottyhq]

Also am seeing occaisional timeouts when running coincident.io.sliderule.sample_3dep

Timed-out waiting for response from endpoint https://sliderule.slideruleearth.io/source/samples ...retrying request