Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Enable a "trigger" to immediately register and replicate data for an event_id #38

Open
kaitshin opened this issue Sep 10, 2023 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@kaitshin
Copy link

Is your feature request related to a problem? Please describe.
There's a 2 day lag between baseband data being captured on-site and that data being staged+registered with datatrail to minoc so that it's accessible from CANFAR. It would be nice to be able to manually "trigger" the data for a certain event being staged+registered immediately (slack thread for context).

Describe the solution you'd like
A datatrail cli command that could be run on-site in a docker container.

Describe alternatives you've considered
A python function, a manual script, a workflow action (where you submit your name and the event ID)

Additional context
In my opinion, this feature isn't necessary for 95% of baseband users' use cases, but at this moment, the baseband pipeline doesn't run on-site at the moment and it runs quite smoothly (and quickly!) at CANFAR. It's entirely feasible that this feature won't really be necessary once there's a working installation of the baseband pipeline on site/frb-analysis, but even then, I think having the option to access the data on CANFAR within ~hours would be good for ~time-sensitive analyses (like with ATels).

@kaitshin kaitshin added the enhancement New feature or request label Sep 10, 2023
@kaitshin
Copy link
Author

kaitshin commented Jan 5, 2024

Found another use case that'd be nice: there are some baseband datasets for bursts from a pulsar that aren't being captured by current datatrail rules, and it would have been nice to be able to follow steps to manually register these events in datatrail so I could run the baseband pipeline on them without having to wait for all the data to vcp over first.

event_id data_path
312799677 /data/chime/baseband/raw/2023/08/26/astro_312799677
339081551* /data/chime/baseband/raw/2023/11/26/astro_339081551
339636297* /data/chime/baseband/raw/2023/11/28/astro_339636297
341265154* /data/chime/baseband/raw/2023/11/30/astro_341265154
341832013 /data/chime/baseband/raw/2023/12/02/astro_341832013
347513488 /data/chime/baseband/raw/2023/12/25/astro_347513488
347931331 /data/chime/baseband/raw/2023/12/27/astro_347931331
348371120 /data/chime/baseband/raw/2023/12/29/astro_348371120
348782545 /data/chime/baseband/raw/2023/12/31/astro_348782545
349258069 /data/chime/baseband/raw/2024/01/02/astro_349258069
349734016 /data/chime/baseband/raw/2024/01/04/astro_349734016

*seem to already be registered in datatrail, and are no longer on frb-analysis (though originally this wasn't the case)

If it's undesirable to have individual users register individual events on datatrail, it'd be nice to know the proper steps! Will it forever just be DM'ing Tarik?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants