DM-48194: Deploy a dev prompt processing service for LSSTCam-ImSim #4168
Conversation
hsinfang
force-pushed
the
tickets/DM-48194
branch
5 times, most recently
from
11c1266 to
51176d7
Compare
| @@ -88,7 +89,7 @@ detectorConfig: | |||
| 8: True | |||
| LSSTComCamSim: | |||
| <<: *ComCam | |||
| LSSTCam: | |||
| LSSTCam: &LSSTCam | |||
| 0: False | |||
| 1: False | |||
| 2: False | |||
There was a problem hiding this comment.
Do these all need to be set to true? I'm not sure why they were false in the first place.
There was a problem hiding this comment.
Yes they should be true. Thank you for catching that!
(Sorry this is not fully tested yet and hence the problem wasn't caught yet)
| topic: prompt-processing-dev | ||
|
|
||
| apdb: | ||
| config: s3://rubin-pp-dev-users/apdb_config/cassandra/pp_apdb_lsstcamimsim-dev.py |
There was a problem hiding this comment.
Should this be .py or .yaml? The Confluence page says .yaml (which it looks like we support as of w.2025.04?).
There was a problem hiding this comment.
It should be .yaml. Thank you for catching it.
| # Expect to need roughly n_detector × request_latency / survey_cadence pods | ||
| # But we do not have the compute yet. This will be adjusted. | ||
| autoscaling.knative.dev/max-scale: "200" |
There was a problem hiding this comment.
I think we have values for everything in the formula? Certainly 200 is much too low.
| # @default -- None, must be set | ||
| preprocessing: "" | ||
| # -- Skymap to use with the instrument | ||
| skymap: "lsst_cells_v1" |
There was a problem hiding this comment.
I assume this is the DC2 skymap? Is patchesPerImage = 16 (which I assume was copied from ComCamSim) still valid?
There was a problem hiding this comment.
The original plan is to really use lsst_cells_v1.
Then we found out that the DC2's DC2_cells_v1 and lsst_cells_v1 are identical, just different name.
Only lsst_cells_v1 exists in repo embargo_or5 today so I'd keep it for now. Later we might change it depending on which name is chosen for actual OR5.
| # -- Maximum time that a container can send nothing to Knative (seconds). | ||
| # This is only useful if the container runs async workers. | ||
| # If 0, idle timeout is ignored. | ||
| idleTimeout: 900 | ||
| # -- Maximum time that a container can send nothing to Knative after initial submission (seconds). | ||
| # This is only useful if the container runs async workers. | ||
| # If 0, idle timeout is ignored. | ||
| responseStartTimeout: 900 |
There was a problem hiding this comment.
I'd recommend setting these to 0, since otherwise Knative may queue up extra jobs before the Gunicorn worker times out. (Until we greatly reduce the Gunicorn timeout, the actual Knative timeout is at its cap of 1200 seconds.)
| @@ -14,6 +14,6 @@ image: | |||
| repository: ghcr.io/lsst-dm/next_visit_fan_out | |||
| pullPolicy: IfNotPresent | |||
| # Overrides the image tag whose default is the chart appVersion. | |||
| tag: 2.5.0 | |||
| tag: tickets-DM-48194 | |||
There was a problem hiding this comment.
Should I be reviewing fan out as well? I see two branches but no PR.
There was a problem hiding this comment.
No, fan-out isn't ready yet.
hsinfang
force-pushed
the
tickets/DM-48194
branch
3 times, most recently
from
2e1a091 to
bafdac3
Compare
The service is started with mostly configs from ComCam, but will be tuned later.
hsinfang
force-pushed
the
tickets/DM-48194
branch
from
85ee3e6 to
227f486
Compare
hsinfang
force-pushed
the
tickets/DM-48194
branch
from
227f486 to
26bf2ed
Compare
RuntimeWarning: Cannot store all inputs in cache; dropping {DatasetRef(DatasetType('cal_ref_cat_2_2', {htm7}, SimpleCatalog), {htm7: 147130}, run='refcats/PREOPS-301', id=2cf75345-fefa-4e70-b35e-54fd57d128b6)}.
No description provided.