Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[teraslice-cli] add earl jobs export command #3717

Merged
merged 15 commits into from
Sep 10, 2024
Merged

[teraslice-cli] add earl jobs export command #3717

merged 15 commits into from
Sep 10, 2024

Conversation

busma13
Copy link
Contributor

@busma13 busma13 commented Aug 9, 2024

This PR makes the following changes:

  • Add jobs export command to teraslice-cli
    • Creates a Jobs class using the provided jobIds, extracts the jobConfig for each job, and saves each config to the local file system.
    • Each job is saved to ./<job.name>.json by default. If a file name already exists -N will be appended to the file name.
    • --outdir flag will set a custom directory where job files are saved.
    • A job-id of all will export all jobs. This can be combined with --status to export all jobs with executions of a specific status.

Ref: #3695

@busma13 busma13 force-pushed the earl-tjm-export branch 2 times, most recently from 8eb853a to 3b30034 Compare September 5, 2024 21:16
@godber godber changed the title add earl jobs export command [teraslice-cli] add earl jobs export command Sep 5, 2024
@godber
Copy link
Member

godber commented Sep 6, 2024

Each job is saved to ~/.teraslice/export//<job.name>.json by default.

I would think the current directory makes more sense ./<job.name>.json. Or make it write to stdout by default. Hmm, though getting an already well named file is nice (<job.name>.json).

A job-id of all will export all jobs. This can be combined with --status to export all jobs with executions of a specific status. all cannot be used in conjunction with --file-name.

will this work for clusters with 100+ or 200+ jobs? Does it paginate? This feels like an anti-feature, but I can see how it would be convenient.

@busma13
Copy link
Contributor Author

busma13 commented Sep 6, 2024

will this work for clusters with 100+ or 200+ jobs? Does it paginate? This feels like an anti-feature, but I can see how it would be convenient.

It will export 4 jobs at a time. If we choose to keep this feature I can test it on a large amount of jobs and see how it goes.

@godber
Copy link
Member

godber commented Sep 6, 2024

  • by default jobs are written to ./{job.name}.json (increment suffix -N on duplicate filenames)
  • -o/--outdir - allows user to specify output directory other than .
  • lets get rid of the custom filename option

@busma13 busma13 marked this pull request as ready for review September 10, 2024 15:29
@busma13
Copy link
Contributor Author

busma13 commented Sep 10, 2024

Exporting 30 jobs with name collisions:

➜  jobs-local-tmp git:(main) ✗ earl jobs list local
 job_id                                name                    lifecycle   slicers  workers  _created                  _updated
 c8fcfb70-42ad-4596-9e67-f668de4c7e7c  ts-tmp1-test            persistent           1        2024-09-10T15:43:40.973Z  2024-09-10T15:43:40.973Z
 4fd4f265-d707-4af8-bd85-148a3edbdeea  peter-kafka-to-s3       persistent           1        2024-09-10T15:43:38.828Z  2024-09-10T15:43:38.828Z
 aa8c14de-ccd6-4555-acfc-168a735c608c  peter-kafka-to-es       persistent           1        2024-09-10T15:43:36.718Z  2024-09-10T15:43:36.718Z
 da8ee9a8-a5bf-4bf9-9856-3a97f2971848  peter-datagen-to-kafka  persistent           1        2024-09-10T15:43:34.613Z  2024-09-10T15:43:34.613Z
 c86b4aa2-7e1f-444a-8849-e50dad7e828a  peter-datagen-to-s3     persistent           1        2024-09-10T15:43:32.470Z  2024-09-10T15:43:32.470Z
 ed05d676-5e35-4424-b36f-4a330420974e  peter datagen to es     persistent           1        2024-09-10T15:43:30.366Z  2024-09-10T15:43:30.366Z
 19371404-75ac-493a-9456-0477481649bd  ts-tmp1-test            persistent           1        2024-09-10T15:43:16.927Z  2024-09-10T15:43:16.927Z
 1d66b687-9570-49d2-978f-320b0d8dd78f  peter-kafka-to-s3       persistent           1        2024-09-10T15:43:14.802Z  2024-09-10T15:43:14.802Z
 744d93e5-cbcc-4ca1-87c3-864a55189dbe  peter-kafka-to-es       persistent           1        2024-09-10T15:43:12.651Z  2024-09-10T15:43:12.651Z
 861107cf-221f-4df8-a495-76e4ecc9e136  peter-datagen-to-kafka  persistent           1        2024-09-10T15:43:10.513Z  2024-09-10T15:43:10.513Z
 1037a24a-6177-42ad-b9ee-9e6c24337e54  peter-datagen-to-s3     persistent           1        2024-09-10T15:43:08.366Z  2024-09-10T15:43:08.366Z
 c2b40ce2-d029-4df4-ba7b-e97842bc6421  peter datagen to es     persistent           1        2024-09-10T15:43:06.230Z  2024-09-10T15:43:06.230Z
 5a2edccf-0ebe-45d2-bc24-069bb9f4b866  ts-tmp1-test            persistent           1        2024-09-10T15:42:49.082Z  2024-09-10T15:42:49.082Z
 2152ff00-a6a1-48cc-acdc-c28ad4d932f9  peter-kafka-to-s3       persistent           1        2024-09-10T15:42:46.928Z  2024-09-10T15:42:46.928Z
 38d68e22-854c-402d-ad93-be3e23092621  peter-kafka-to-es       persistent           1        2024-09-10T15:42:44.806Z  2024-09-10T15:42:44.806Z
 3834d6a3-45cd-4fa8-ae15-8bcb23cf07ed  peter-datagen-to-kafka  persistent           1        2024-09-10T15:42:42.700Z  2024-09-10T15:42:42.700Z
 6c39d751-3637-4ca5-8f2a-428221c6fcd8  peter-datagen-to-s3     persistent           1        2024-09-10T15:42:40.571Z  2024-09-10T15:42:40.571Z
 c514452a-597a-4c64-a5e8-2e4e5d6b4656  peter datagen to es     persistent           1        2024-09-10T15:42:38.452Z  2024-09-10T15:42:38.452Z
 8928432f-9453-4389-8c57-eb16380615b6  ts-tmp1-test            persistent           1        2024-09-10T15:39:45.357Z  2024-09-10T15:39:45.357Z
 10b179ba-04d0-444b-adf7-13536fad4675  peter-kafka-to-s3       persistent           1        2024-09-10T15:39:43.244Z  2024-09-10T15:39:43.244Z
 95337f01-579e-4e8e-8387-7b9d354ca723  peter-kafka-to-es       persistent           1        2024-09-10T15:39:41.128Z  2024-09-10T15:39:41.128Z
 9ea13127-e3b9-4fb5-9f35-a5b3b1c3c873  peter-datagen-to-kafka  persistent           1        2024-09-10T15:39:38.983Z  2024-09-10T15:39:38.983Z
 0c41ee18-30dc-4393-9268-d5aa5d80fe0e  peter-datagen-to-s3     persistent           1        2024-09-10T15:39:36.847Z  2024-09-10T15:39:36.847Z
 6e43f36a-c7fd-4d21-a6d6-34ccdcf7e2af  peter datagen to es     persistent           1        2024-09-10T15:39:34.771Z  2024-09-10T15:39:34.771Z
 8dedeb9d-71b4-4424-a79f-5d1228cf4545  ts-tmp1-test            persistent           1        2024-09-10T15:37:53.282Z  2024-09-10T15:37:53.282Z
 b8c90788-3667-4969-a3dc-fd1fc39c121d  peter-kafka-to-s3       persistent           1        2024-09-10T15:36:35.873Z  2024-09-10T15:36:35.873Z
 e0632b99-1ba9-4c49-afd8-1367d2b9bd9d  peter-kafka-to-es       persistent           1        2024-09-10T15:36:33.752Z  2024-09-10T15:36:33.752Z
 87c7d5f5-4169-44dc-ad93-7f85c9784160  peter-datagen-to-kafka  persistent           1        2024-09-10T15:36:31.618Z  2024-09-10T15:36:31.618Z
 fca11c9d-ef8b-4b4f-b9b6-044748e413c2  peter-datagen-to-s3     persistent           1        2024-09-10T15:36:29.466Z  2024-09-10T15:36:29.466Z
 05f90f23-bd29-4a79-94a6-7d06eb3822a0  peter datagen to es     persistent           1        2024-09-10T15:36:27.104Z  2024-09-10T15:36:27.104Z
➜  jobs-local-tmp git:(main) ✗ cd export
➜  export git:(main) ✗ teraslice-cli jobs export local all
✔ Export all jobs on local … yes
Saving jobFile(s) for c8fcfb70-42ad-4596-9e67-f668de4c7e7c, da8ee9a8-a5bf-4bf9-9856-3a97f2971848, 4fd4f265-d707-4af8-bd85-148a3edbdeea, aa8c14de-ccd6-4555-acfc-168a735c608c, 1d66b687-9570-49d2-978f-320b0d8dd78f, c86b4aa2-7e1f-444a-8849-e50dad7e828a, 19371404-75ac-493a-9456-0477481649bd, ed05d676-5e35-4424-b36f-4a330420974e, 1037a24a-6177-42ad-b9ee-9e6c24337e54, 861107cf-221f-4df8-a495-76e4ecc9e136, 744d93e5-cbcc-4ca1-87c3-864a55189dbe, c2b40ce2-d029-4df4-ba7b-e97842bc6421, 2152ff00-a6a1-48cc-acdc-c28ad4d932f9, 5a2edccf-0ebe-45d2-bc24-069bb9f4b866, 38d68e22-854c-402d-ad93-be3e23092621, 3834d6a3-45cd-4fa8-ae15-8bcb23cf07ed, 6c39d751-3637-4ca5-8f2a-428221c6fcd8, c514452a-597a-4c64-a5e8-2e4e5d6b4656, 10b179ba-04d0-444b-adf7-13536fad4675, 8928432f-9453-4389-8c57-eb16380615b6, 95337f01-579e-4e8e-8387-7b9d354ca723, 9ea13127-e3b9-4fb5-9f35-a5b3b1c3c873, 6e43f36a-c7fd-4d21-a6d6-34ccdcf7e2af, 0c41ee18-30dc-4393-9268-d5aa5d80fe0e, 8dedeb9d-71b4-4424-a79f-5d1228cf4545, b8c90788-3667-4969-a3dc-fd1fc39c121d, e0632b99-1ba9-4c49-afd8-1367d2b9bd9d, 87c7d5f5-4169-44dc-ad93-7f85c9784160, fca11c9d-ef8b-4b4f-b9b6-044748e413c2, 05f90f23-bd29-4a79-94a6-7d06eb3822a0 on local
Saved jobFile(s) to /Users/<username>/testing/export
➜  export git:(main) ✗ ls -la
total 240
drwxr-xr-x  32 peterluitjens  staff  1024 Sep 10 08:45 .
drwxr-xr-x@  9 peterluitjens  staff   288 Sep 10 08:45 ..
-rw-r--r--   1 peterluitjens  staff   373 Sep 10 08:45 peter-datagen-to-kafka-1.json
-rw-r--r--   1 peterluitjens  staff   373 Sep 10 08:45 peter-datagen-to-kafka-2.json
-rw-r--r--   1 peterluitjens  staff   373 Sep 10 08:45 peter-datagen-to-kafka-3.json
-rw-r--r--   1 peterluitjens  staff   373 Sep 10 08:45 peter-datagen-to-kafka-4.json
-rw-r--r--   1 peterluitjens  staff   373 Sep 10 08:45 peter-datagen-to-kafka.json
-rw-r--r--   1 peterluitjens  staff   375 Sep 10 08:45 peter-datagen-to-s3-1.json
-rw-r--r--   1 peterluitjens  staff   375 Sep 10 08:45 peter-datagen-to-s3-2.json
-rw-r--r--   1 peterluitjens  staff   375 Sep 10 08:45 peter-datagen-to-s3-3.json
-rw-r--r--   1 peterluitjens  staff   375 Sep 10 08:45 peter-datagen-to-s3-4.json
-rw-r--r--   1 peterluitjens  staff   375 Sep 10 08:45 peter-datagen-to-s3.json
-rw-r--r--   1 peterluitjens  staff   427 Sep 10 08:45 peter-kafka-to-es-1.json
-rw-r--r--   1 peterluitjens  staff   427 Sep 10 08:45 peter-kafka-to-es-2.json
-rw-r--r--   1 peterluitjens  staff   427 Sep 10 08:45 peter-kafka-to-es-3.json
-rw-r--r--   1 peterluitjens  staff   427 Sep 10 08:45 peter-kafka-to-es-4.json
-rw-r--r--   1 peterluitjens  staff   427 Sep 10 08:45 peter-kafka-to-es.json
-rw-r--r--   1 peterluitjens  staff   435 Sep 10 08:45 peter-kafka-to-s3-1.json
-rw-r--r--   1 peterluitjens  staff   435 Sep 10 08:45 peter-kafka-to-s3-2.json
-rw-r--r--   1 peterluitjens  staff   435 Sep 10 08:45 peter-kafka-to-s3-3.json
-rw-r--r--   1 peterluitjens  staff   435 Sep 10 08:45 peter-kafka-to-s3-4.json
-rw-r--r--   1 peterluitjens  staff   435 Sep 10 08:45 peter-kafka-to-s3.json
-rw-r--r--   1 peterluitjens  staff   387 Sep 10 08:45 peter_datagen_to_es-1.json
-rw-r--r--   1 peterluitjens  staff   387 Sep 10 08:45 peter_datagen_to_es-2.json
-rw-r--r--   1 peterluitjens  staff   387 Sep 10 08:45 peter_datagen_to_es-3.json
-rw-r--r--   1 peterluitjens  staff   387 Sep 10 08:45 peter_datagen_to_es-4.json
-rw-r--r--   1 peterluitjens  staff   387 Sep 10 08:45 peter_datagen_to_es.json
-rw-r--r--   1 peterluitjens  staff   373 Sep 10 08:45 ts-tmp1-test-1.json
-rw-r--r--   1 peterluitjens  staff   373 Sep 10 08:45 ts-tmp1-test-2.json
-rw-r--r--   1 peterluitjens  staff   373 Sep 10 08:45 ts-tmp1-test-3.json
-rw-r--r--   1 peterluitjens  staff   373 Sep 10 08:45 ts-tmp1-test-4.json
-rw-r--r--   1 peterluitjens  staff   373 Sep 10 08:45 ts-tmp1-test.json

I manually verified that each file has a unique jobId.

@busma13 busma13 requested review from sotojn and ciorg September 10, 2024 16:08
Copy link
Member

@ciorg ciorg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@godber godber merged commit 65d242b into master Sep 10, 2024
66 checks passed
@godber godber deleted the earl-tjm-export branch September 10, 2024 20:31
godber pushed a commit that referenced this pull request Sep 11, 2024
A few special characters were added a documentation change in my recent
PR: #3717
After merging the docs build failed because docusuarus could not parse
the .md file properly.
This PR escapes those characters properly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants