Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[load_dataset] merge #794

Merged
merged 3 commits into from
Feb 28, 2024
Merged

[load_dataset] merge #794

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
42c41ba
[load_dataset] update image to mlrun/mlrun
aviaIguazio Feb 28, 2024
d9b27ce
[load_dataset] update image to mlrun/mlrun
aviaIguazio Feb 28, 2024
062fef9
Merge pull request #793 from aviaIguazio/development
aviaIguazio Feb 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 20 additions & 12 deletions load_dataset/function.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ kind: job
metadata:
name: load-dataset
tag: ''
hash: 878a235b92311abb244dd1e7b423b13010ab2488
hash: d05aa41d618533335eeaeab38aa434a14e3e3980
project: ''
labels:
author: yjb
Expand All @@ -12,9 +12,13 @@ metadata:
spec:
command: ''
args: []
image: mlrun/ml-models
env: []
default_handler: load_dataset
image: mlrun/mlrun
build:
functionSourceCode: IyBDb3B5cmlnaHQgMjAxOSBJZ3VhemlvCiMKIyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKIyB5b3UgbWF5IG5vdCB1c2UgdGhpcyBmaWxlIGV4Y2VwdCBpbiBjb21wbGlhbmNlIHdpdGggdGhlIExpY2Vuc2UuCiMgWW91IG1heSBvYnRhaW4gYSBjb3B5IG9mIHRoZSBMaWNlbnNlIGF0CiMKIyAgICAgaHR0cDovL3d3dy5hcGFjaGUub3JnL2xpY2Vuc2VzL0xJQ0VOU0UtMi4wCiMKIyBVbmxlc3MgcmVxdWlyZWQgYnkgYXBwbGljYWJsZSBsYXcgb3IgYWdyZWVkIHRvIGluIHdyaXRpbmcsIHNvZnR3YXJlCiMgZGlzdHJpYnV0ZWQgdW5kZXIgdGhlIExpY2Vuc2UgaXMgZGlzdHJpYnV0ZWQgb24gYW4gIkFTIElTIiBCQVNJUywKIyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KIyBTZWUgdGhlIExpY2Vuc2UgZm9yIHRoZSBzcGVjaWZpYyBsYW5ndWFnZSBnb3Zlcm5pbmcgcGVybWlzc2lvbnMgYW5kCiMgbGltaXRhdGlvbnMgdW5kZXIgdGhlIExpY2Vuc2UuCiMKaW1wb3J0IG51bXB5IGFzIG5wCmltcG9ydCBwYW5kYXMgYXMgcGQKZnJvbSBtbHJ1bi5leGVjdXRpb24gaW1wb3J0IE1MQ2xpZW50Q3R4CgoKZGVmIGxvYWRfZGF0YXNldCgKICAgIGNvbnRleHQ6IE1MQ2xpZW50Q3R4LAogICAgZGF0YXNldDogc3RyLAogICAgbmFtZTogc3RyID0gIiIsCiAgICBmaWxlX2V4dDogc3RyID0gInBhcnF1ZXQiLAogICAgcGFyYW1zOiBkaWN0ID0ge30sCikgLT4gTm9uZToKICAgICIiIkxvYWRzIGEgc2Npa2l0LWxlYXJuIHRveSBkYXRhc2V0IGZvciBjbGFzc2lmaWNhdGlvbiBvciByZWdyZXNzaW9uCgogICAgVGhlIGZvbGxvd2luZyBkYXRhc2V0cyBhcmUgYXZhaWxhYmxlICgnbmFtZScgOiBkZXNyaXB0aW9uKToKCiAgICAgICAgJ2Jvc3RvbicgICAgICAgICAgOiBib3N0b24gaG91c2UtcHJpY2VzIGRhdGFzZXQgKHJlZ3Jlc3Npb24pCiAgICAgICAgJ2lyaXMnICAgICAgICAgICAgOiBpcmlzIGRhdGFzZXQgKGNsYXNzaWZpY2F0aW9uKQogICAgICAgICdkaWFiZXRlcycgICAgICAgIDogZGlhYmV0ZXMgZGF0YXNldCAocmVncmVzc2lvbikKICAgICAgICAnZGlnaXRzJyAgICAgICAgICA6IGRpZ2l0cyBkYXRhc2V0IChjbGFzc2lmaWNhdGlvbikKICAgICAgICAnbGlubmVydWQnICAgICAgICA6IGxpbm5lcnVkIGRhdGFzZXQgKG11bHRpdmFyaWF0ZSByZWdyZXNzaW9uKQogICAgICAgICd3aW5lJyAgICAgICAgICAgIDogd2luZSBkYXRhc2V0IChjbGFzc2lmaWNhdGlvbikKICAgICAgICAnYnJlYXN0X2NhbmNlcicgICA6IGJyZWFzdCBjYW5jZXIgd2lzY29uc2luIGRhdGFzZXQgKGNsYXNzaWZpY2F0aW9uKQoKICAgIFRoZSBzY2lraXQtbGVhcm4gZnVuY3Rpb25zIHJldHVybiBhIGRhdGEgYnVuY2ggaW5jbHVkaW5nIHRoZSBmb2xsb3dpbmcgaXRlbXM6CiAgICAtIGRhdGEgICAgICAgICAgICAgIHRoZSBmZWF0dXJlcyBtYXRyaXgKICAgIC0gdGFyZ2V0ICAgICAgICAgICAgdGhlIGdyb3VuZCB0cnV0aCBsYWJlbHMKICAgIC0gREVTQ1IgICAgICAgICAgICAgYSBkZXNjcmlwdGlvbiBvZiB0aGUgZGF0YXNldAogICAgLSBmZWF0dXJlX25hbWVzICAgICBoZWFkZXIgZm9yIGRhdGEKCiAgICBUaGUgZmVhdHVyZXMgKGFuZCB0aGVpciBuYW1lcykgYXJlIHN0b3JlZCB3aXRoIHRoZSB0YXJnZXQgbGFiZWxzIGluIGEgRGF0YUZyYW1lLgoKICAgIEZvciBmdXJ0aGVyIGRldGFpbHMgc2VlIGh0dHBzOi8vc2Npa2l0LWxlYXJuLm9yZy9zdGFibGUvZGF0YXNldHMvaW5kZXguaHRtbCN0b3ktZGF0YXNldHMKCiAgICA6cGFyYW0gY29udGV4dDogICAgZnVuY3Rpb24gZXhlY3V0aW9uIGNvbnRleHQKICAgIDpwYXJhbSBkYXRhc2V0OiAgICBuYW1lIG9mIHRoZSBkYXRhc2V0IHRvIGxvYWQKICAgIDpwYXJhbSBuYW1lOiAgICAgICBhcnRpZmFjdCBuYW1lIChkZWZhdWx0cyB0byBkYXRhc2V0KQogICAgOnBhcmFtIGZpbGVfZXh0OiAgIG91dHB1dCBmaWxlX2V4dDogcGFycXVldCBvciBjc3YKICAgIDpwYXJhbSBwYXJhbXM6ICAgICBwYXJhbXMgb2YgdGhlIHNrbGVhcm4gbG9hZF9kYXRhIG1ldGhvZAogICAgIiIiCiAgICBkYXRhc2V0ID0gc3RyKGRhdGFzZXQpCiAgICBwa2dfbW9kdWxlID0gInNrbGVhcm4uZGF0YXNldHMiCiAgICBmbmFtZSA9IGYibG9hZF97ZGF0YXNldH0iCgogICAgcGtnX21vZHVsZSA9IF9faW1wb3J0X18ocGtnX21vZHVsZSwgZnJvbWxpc3Q9W2ZuYW1lXSkKICAgIGxvYWRfZGF0YV9mbiA9IGdldGF0dHIocGtnX21vZHVsZSwgZm5hbWUpCgogICAgZGF0YSA9IGxvYWRfZGF0YV9mbigqKnBhcmFtcykKICAgIGZlYXR1cmVfbmFtZXMgPSBkYXRhWyJmZWF0dXJlX25hbWVzIl0KCiAgICB4eSA9IG5wLmNvbmNhdGVuYXRlKFtkYXRhWyJkYXRhIl0sIGRhdGFbInRhcmdldCJdLnJlc2hhcGUoLTEsIDEpXSwgYXhpcz0xKQogICAgaWYgaGFzYXR0cihmZWF0dXJlX25hbWVzLCAiYXBwZW5kIik6CiAgICAgICAgZmVhdHVyZV9uYW1lcy5hcHBlbmQoImxhYmVscyIpCiAgICBlbHNlOgogICAgICAgIGZlYXR1cmVfbmFtZXMgPSBucC5hcHBlbmQoZmVhdHVyZV9uYW1lcywgImxhYmVscyIpCiAgICBkZiA9IHBkLkRhdGFGcmFtZShkYXRhPXh5LCBjb2x1bW5zPWZlYXR1cmVfbmFtZXMpCgogICAgY29udGV4dC5sb2dfZGF0YXNldChuYW1lIG9yIGRhdGFzZXQsIGRmPWRmLCBmb3JtYXQ9ZmlsZV9leHQsIGluZGV4PUZhbHNlKQo=
commands: []
code_origin: ''
origin_filename: ''
requirements: []
entry_points:
load_dataset:
name: load_dataset
Expand All @@ -35,11 +39,9 @@ spec:
- name: context
type: MLClientCtx
doc: function execution context
default: ''
- name: dataset
type: str
doc: name of the dataset to load
default: ''
- name: name
type: str
doc: artifact name (defaults to dataset)
Expand All @@ -53,12 +55,18 @@ spec:
doc: params of the sklearn load_data method
default: {}
outputs:
- default: ''
lineno: 6
- type: None
lineno: 20
has_varargs: false
has_kwargs: false
description: load a toy dataset from scikit-learn
build:
functionSourceCode: aW1wb3J0IG51bXB5IGFzIG5wCmltcG9ydCBwYW5kYXMgYXMgcGQKZnJvbSBtbHJ1bi5leGVjdXRpb24gaW1wb3J0IE1MQ2xpZW50Q3R4CgoKZGVmIGxvYWRfZGF0YXNldCgKICAgIGNvbnRleHQ6IE1MQ2xpZW50Q3R4LAogICAgZGF0YXNldDogc3RyLAogICAgbmFtZTogc3RyID0gIiIsCiAgICBmaWxlX2V4dDogc3RyID0gInBhcnF1ZXQiLAogICAgcGFyYW1zOiBkaWN0ID0ge30sCikgLT4gTm9uZToKICAgICIiIkxvYWRzIGEgc2Npa2l0LWxlYXJuIHRveSBkYXRhc2V0IGZvciBjbGFzc2lmaWNhdGlvbiBvciByZWdyZXNzaW9uCgogICAgVGhlIGZvbGxvd2luZyBkYXRhc2V0cyBhcmUgYXZhaWxhYmxlICgnbmFtZScgOiBkZXNyaXB0aW9uKToKCiAgICAgICAgJ2Jvc3RvbicgICAgICAgICAgOiBib3N0b24gaG91c2UtcHJpY2VzIGRhdGFzZXQgKHJlZ3Jlc3Npb24pCiAgICAgICAgJ2lyaXMnICAgICAgICAgICAgOiBpcmlzIGRhdGFzZXQgKGNsYXNzaWZpY2F0aW9uKQogICAgICAgICdkaWFiZXRlcycgICAgICAgIDogZGlhYmV0ZXMgZGF0YXNldCAocmVncmVzc2lvbikKICAgICAgICAnZGlnaXRzJyAgICAgICAgICA6IGRpZ2l0cyBkYXRhc2V0IChjbGFzc2lmaWNhdGlvbikKICAgICAgICAnbGlubmVydWQnICAgICAgICA6IGxpbm5lcnVkIGRhdGFzZXQgKG11bHRpdmFyaWF0ZSByZWdyZXNzaW9uKQogICAgICAgICd3aW5lJyAgICAgICAgICAgIDogd2luZSBkYXRhc2V0IChjbGFzc2lmaWNhdGlvbikKICAgICAgICAnYnJlYXN0X2NhbmNlcicgICA6IGJyZWFzdCBjYW5jZXIgd2lzY29uc2luIGRhdGFzZXQgKGNsYXNzaWZpY2F0aW9uKQoKICAgIFRoZSBzY2lraXQtbGVhcm4gZnVuY3Rpb25zIHJldHVybiBhIGRhdGEgYnVuY2ggaW5jbHVkaW5nIHRoZSBmb2xsb3dpbmcgaXRlbXM6CiAgICAtIGRhdGEgICAgICAgICAgICAgIHRoZSBmZWF0dXJlcyBtYXRyaXgKICAgIC0gdGFyZ2V0ICAgICAgICAgICAgdGhlIGdyb3VuZCB0cnV0aCBsYWJlbHMKICAgIC0gREVTQ1IgICAgICAgICAgICAgYSBkZXNjcmlwdGlvbiBvZiB0aGUgZGF0YXNldAogICAgLSBmZWF0dXJlX25hbWVzICAgICBoZWFkZXIgZm9yIGRhdGEKCiAgICBUaGUgZmVhdHVyZXMgKGFuZCB0aGVpciBuYW1lcykgYXJlIHN0b3JlZCB3aXRoIHRoZSB0YXJnZXQgbGFiZWxzIGluIGEgRGF0YUZyYW1lLgoKICAgIEZvciBmdXJ0aGVyIGRldGFpbHMgc2VlIGh0dHBzOi8vc2Npa2l0LWxlYXJuLm9yZy9zdGFibGUvZGF0YXNldHMvaW5kZXguaHRtbCN0b3ktZGF0YXNldHMKCiAgICA6cGFyYW0gY29udGV4dDogICAgZnVuY3Rpb24gZXhlY3V0aW9uIGNvbnRleHQKICAgIDpwYXJhbSBkYXRhc2V0OiAgICBuYW1lIG9mIHRoZSBkYXRhc2V0IHRvIGxvYWQKICAgIDpwYXJhbSBuYW1lOiAgICAgICBhcnRpZmFjdCBuYW1lIChkZWZhdWx0cyB0byBkYXRhc2V0KQogICAgOnBhcmFtIGZpbGVfZXh0OiAgIG91dHB1dCBmaWxlX2V4dDogcGFycXVldCBvciBjc3YKICAgIDpwYXJhbSBwYXJhbXM6ICAgICBwYXJhbXMgb2YgdGhlIHNrbGVhcm4gbG9hZF9kYXRhIG1ldGhvZAogICAgIiIiCiAgICBkYXRhc2V0ID0gc3RyKGRhdGFzZXQpCiAgICBwa2dfbW9kdWxlID0gInNrbGVhcm4uZGF0YXNldHMiCiAgICBmbmFtZSA9IGYibG9hZF97ZGF0YXNldH0iCgogICAgcGtnX21vZHVsZSA9IF9faW1wb3J0X18ocGtnX21vZHVsZSwgZnJvbWxpc3Q9W2ZuYW1lXSkKICAgIGxvYWRfZGF0YV9mbiA9IGdldGF0dHIocGtnX21vZHVsZSwgZm5hbWUpCgogICAgZGF0YSA9IGxvYWRfZGF0YV9mbigqKnBhcmFtcykKICAgIGZlYXR1cmVfbmFtZXMgPSBkYXRhWyJmZWF0dXJlX25hbWVzIl0KCiAgICB4eSA9IG5wLmNvbmNhdGVuYXRlKFtkYXRhWyJkYXRhIl0sIGRhdGFbInRhcmdldCJdLnJlc2hhcGUoLTEsIDEpXSwgYXhpcz0xKQogICAgaWYgaGFzYXR0cihmZWF0dXJlX25hbWVzLCAiYXBwZW5kIik6CiAgICAgICAgZmVhdHVyZV9uYW1lcy5hcHBlbmQoImxhYmVscyIpCiAgICBlbHNlOgogICAgICAgIGZlYXR1cmVfbmFtZXMgPSBucC5hcHBlbmQoZmVhdHVyZV9uYW1lcywgImxhYmVscyIpCiAgICBkZiA9IHBkLkRhdGFGcmFtZShkYXRhPXh5LCBjb2x1bW5zPWZlYXR1cmVfbmFtZXMpCgogICAgY29udGV4dC5sb2dfZGF0YXNldChuYW1lIG9yIGRhdGFzZXQsIGRmPWRmLCBmb3JtYXQ9ZmlsZV9leHQsIGluZGV4PUZhbHNlKQo=
commands: []
code_origin: https://github.com/daniels290813/functions.git#55a79c32be5d233cc11efcf40cd3edbe309bfdef:/home/kali/functions/load_dataset/load_dataset.py
default_handler: load_dataset
disable_auto_mount: false
clone_target_dir: ''
env: []
priority_class_name: ''
preemption_mode: prevent
affinity: null
tolerations: null
security_context: {}
verbose: false
8 changes: 4 additions & 4 deletions load_dataset/item.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,14 @@ labels:
framework: sklearn
maintainers: []
marketplaceType: ''
mlrunVersion: 1.1.0
mlrunVersion: 1.6.0
name: load-dataset
platformVersion: 3.5.0
platformVersion: 3.5.5
spec:
filename: load_dataset.py
handler: load_dataset
image: mlrun/ml-models
image: mlrun/mlrun
kind: job
requirements: []
url: ''
version: 1.1.0
version: 1.2.0
Loading