Skip to content

Releases: zenml-io/zenml

0.63.0

30 Jul 11:14
@bcdurak bcdurak
0.63.0
430953f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
0.63.0

Moving forward from the last two releases, we have further improved the 1-click deployment tool and the stack wizard by adding support for Azure.

Moreover, we implemented a new step operator that allows you to run individual steps of your pipeline in Kubernetes pods.

Lastly, we have simplified our pipeline models by removing their versions. No migration is required but if you were using the API, please take out references to pipeline versions, and reference runs directly from now on.

What's Changed

New Contributors

Full Changelog: 0.62.0...0.63.0

Contributors

strickvl, stefannica, and 7 other contributors
Assets 2
Loading

0.62.0

16 Jul 15:44
@schustmi schustmi
0.62.0
00e7200
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
0.62.0

Building on top of the last release, this release adds a new and easy way to deploy a GCP ZenML stack from the dashboard and the CLI. Give it a try by going to the Stacks section in the dashboard or running the zenml stack deploy command! For more information on this new feature, please do check out the video and blog from our previous release.

We also updated our Hugging Face integration to support the automatic display of an embedded datasets preview pane in the ZenML Dashboard whenever you return a Dataset from a step. This was recently released by the Hugging Face datasets team and it allows you to easily visualize and inspect your data from the comfort of the dashboard.

What's Changed

New Contributors

  • @whoknowsB made their first contribution in #2798

Full Changelog: 0.61.0...0.62.0

Contributors

stefannica, htahir1, and 7 other contributors
Assets 2
Loading

0.61.0

09 Jul 14:02
@schustmi schustmi
0.61.0
50334c3
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
0.61.0

This release comes with a new and easy way to deploy a simple ZenML stack from the dashboard and the CLI. Give it a try by going to the Stacks section in the dashboard:

image
image

or by running the zenml stack deploy command!

We hope this makes it super easy for existing and new users to set up the infrastructure required to run ZenML pipelines on the cloud in one click.

Note: Only a simple AWS stack using Skypilot supported for now but GCP + Azure to come!

Additionally, this release includes improvements to our documentation and bugfixes for some integrations.

What's Changed

Full Changelog: 0.60.0...0.61.0

Contributors

strickvl, stefannica, and 4 other contributors
Assets 2
Loading

0.60.0

26 Jun 19:06
@htahir1 htahir1
0.60.0
235f333
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
0.60.0

ZenML now uses Pydantic v2. 🥳

This upgrade comes with a set of critical updates. While your user experience mostly remains unaffected, you might see unexpected behavior due to the changes in our dependencies. Moreover, since Pydantic v2 provides a slightly stricter validation process, you might end up bumping into some validation errors which was not caught before, but it is all for the better 🙂 If you run into any other errors, please let us know either on GitHub or on our Slack.

Changes in some of the critical dependencies

  • SQLModel is one of the core dependencies of ZenML and prior to this upgrade, we were utilizing version 0.0.8. However, this version is relatively outdated and incompatible with Pydantic v2. Within the scope of this upgrade, we upgraded it to 0.0.18.
  • Due to the change in the SQLModel version, we also had to upgrade our SQLAlchemy dependency from V1 to v2. While this does not affect the way that you are using ZenML, if you are using SQLAlchemy in your environment, you might have to migrate your code as well. For a detailed list of changes, feel free to check their migration guide.

Changes in pydantic

Pydantic v2 brings a lot of new and exciting changes to the table. The core logic now uses Rust, and it is much faster and more efficient in terms of performance. On top of it, the main concepts like model design, configuration, validation, or serialization now include a lot of new cool features. If you are using pydantic in your workflow and are interested in the new changes, you can check the brilliant migration guide provided by the pydantic team to see the full list of changes.

Changes in our integrations changes

Much like ZenML, pydantic is an important dependency in many other Python packages. That’s why conducting this upgrade helped us unlock a new version for several ZenML integration dependencies. Additionally, in some instances, we had to adapt the functionality of the integration to keep it compatible with pydantic. So, if you are using any of these integrations, please go through the changes.

Airflow

As mentioned above upgrading our pydantic dependency meant we had to upgrade our sqlmodel dependency. Upgrading our sqlmodel dependency meant we had to upgrade our sqlalchemy dependency as well. Unfortunately, apache-airflow is still using sqlalchemy v1 and is incompatible with pydantic v2. As a solution, we have removed the dependencies of the airflow integration. Now, you can use ZenML to create your Airflow pipelines and use a separate environment to run them with Airflow. You can check the updated docs right here.

AWS

Some of our integrations now require protobuf 4. Since our previous sagemaker version (2.117.0) did not support protobof 4, we could not pair it with these new integrations. Thankfully sagemaker started supporting protobuf 4 with version 2.172.0 and relaxing its dependency solved the compatibility issue.

Evidently

The old version of our evidently integration was not compatible with Pydantic v2. They started supporting it starting from version 0.4.16. As their latest version is 0.4.22, the new dependency of the integration is limited between these two versions.

Feast

Our previous implementation of the feast integration was not compatible with Pydantic v2 due to the extra redis dependency we were using. This extra dependency is now removed and the feast integration is working as intended.

GCP

The previous version of the Kubeflow dependency (kfp==1.8.22) in our GCP integration required Pydantic V1 to be installed. While we were upgrading our Pydantic dependency, we saw this as an opportunity and wanted to use this chance to upgrade the kfp dependency to v2 (which has no dependencies on the Pydantic library). This is why you may see some functional changes in the vertex step operator and orchestrator. If you would like to go through the changes in the kfp library, you can
find the migration guide here.

Great Expectations

Great Expectations started supporting Pydantic v2 starting from version 0.17.15 and they are closing in on their 1.0 release. Since this release might include a lot of big changes, we adjusted the dependency in our integration to great-expectations>=0.17.15,<1.0. We will try to keep it updated in the future once they release the 1.0 version

Kubeflow

Similar to the GCP integration, the previous version of the kubeflow dependency (kfp==1.8.22) in our kubeflow integration required Pydantic V1 to be installed. While we were upgrading our Pydantic dependency, we saw this as an opportunity and wanted to use this chance to upgrade the kfp dependency to v2 (which has no dependencies on the Pydantic library). If you would like to go through the changes in the kfp library, you can find the migration guide here. ( We also are considering adding an alternative version of this integration so our users can keep using kfp V1 in their environment. Stay tuned for any updates.)

MLflow

mlflow is compatible with both Pydantic V1 and v2. However, due to a known issue, if you install zenml first and then do zenml integration install mlflow -y, it downgrades pydantic to V1. This is why we manually added the same duplicated pydantic requirement in the integration definition as well. Keep in mind that the mlflow library is still using some features of pydantic V1 which are deprecated. So, if the integration is installed in your environment, you might run into some deprecation warnings.

Label Studio

While we were working on updating our pydantic dependency, the label-studio-sdk has released its 1.0 version. In this new version, pydantic v2 is also supported. The implementation and documentation of our Label Studio integration have been updated accordingly.

Skypilot

With the switch to pydantic v2, the implementation of our skypilot integration mostly remained untouched. However, due to an incompatibility between the new version pydantic and the azurecli, the skypilot[azure] flavor can not be installed at the same time, thus our skypilot_azure integration is currently deactivated. We are working on fixing this issue and if you are using this integration in your workflows, we recommend staying on the previous version of ZenML until we can solve this issue.

Tensorflow

The new version of pydantic creates a drift between tensorflow and typing_extensions packages and relaxing the dependencies here resolves the issue. At the same time, the upgrade to kfp v2 (in integrations like kubeflow, tekton, or gcp) bumps our protobuf dependency from 3.X to 4.X. To stay compatible with this requirement, the installed version of tensorflow needs to be >=2.12.0. While this change solves the dependency issues in most settings, we have bumped into some errors while using tensorflow 2.12.0 on Python 3.8 on Ubuntu. If you would like to use this integration, please consider using a higher Python version.

Tekton

Similar to the gcp and kubeflow integrations, the old version of our tekton integration was not compatible with pydantic V1 due to its kfp dependency. With the switch from kfp V1 to v2, we have adapted our implementation to use the new version of kfp library and updated our documentation accordingly.

Additional Changes

  • We have also released a new version of mlstacks with Pydantic v2 support. If you are using it in your development environment, you have to upgrade your mlstacks package as well.
  • Added zenml.integrations.huggingface.steps.run_with_accelerate to enable running any step using accelerate. This function is supported by a utility that wraps any step function into a CLI script (which is required by most distributed training tools).
  • Fixed a memory leak that was observed while using the ZenML dashboard to view pipeline logs or artifact visualizations logged through an S3 Artifact Store linked to an AWS Service Connector.
  • Previously, we had an option called build_options that allowed users to pass arguments to the docker build command. However, these options were only applied when building the parent image. On macOS with ARM architecture, one needs to specify platform=linux/amd64 to the build command to leverage local caching of Docker image layers. We have added a way to specify these build options for the "main" ZenML build as well, not just the parent image build. Additionally, users can now specify a .dockerignore file for the parent image build, which was previously not possible.

What's Changed

Read more

Contributors

strickvl, stefannica, and 6 other contributors
Assets 2
Loading

0.58.2

10 Jun 16:16
@avishniakov avishniakov
0.58.2
b8970a0
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
0.58.2

The 0.58.2 minor release is packed with a set of improvements to the ZenML logging and ZenML Server.

With this release ZenML logging will:

  • Offer pagination of the logs during fetching via REST API
  • Store the full logs history on GCS Artifact Stores
  • Be performant running logging-heavy tasks, like TQDM logging or logging of training in any Deep Learning framework (also TQDM-backed)

What's Changed

Full Changelog: 0.58.1...0.58.2

Contributors

strickvl, stefannica, and 4 other contributors
Assets 2
Loading

0.58.1

06 Jun 18:54
@safoinme safoinme
0.58.1
eaadf5a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
0.58.1

The 0.58.1 release brings a set of minor enhancements and bugfix to the ZenML framework, such as the ability to delete all pipeline versions using the Client/CLI, providing greater flexibility and control over pipeline management. Users can now specify Python package installer arguments. Furthermore, a fix has been implemented for the Sentencepiece tokenizer materializer.

We are also excited to introduce breadcrumbs to our dashboard to improve your navigation experience. This new feature allows you to easily visualize the path of your Pipelines, Models, and Artifacts, providing clear orientation, quick return to any section with a single click, and effortless navigation.

We’d like to give a special thanks to @eltociear for their first contribution.

Docs re-work

We reworked the structure of our documentation pages to make it easier to find answers to your practical questions. Please do let us know if you have any feedback on the structure or the new style of the 'How To' section!

What's Changed

New Contributors

Full Changelog: 0.58.0...0.58.1

Contributors

strickvl, schustmi, and 5 other contributors
Assets 2
Loading

0.58.0

27 May 17:00
@avishniakov avishniakov
0.58.0
21edd86
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
0.58.0

New Annotators

This release brings in three new integrations for our annotator stack component: Prodigy, Argilla and Pigeon.

  • Pigeon works within Jupyter notebooks and supports a limited feature set but is great for experimentation and demos.
  • Argilla works both locally-deployed and when the annotation instance lives in the cloud (i.e. in the Hugging Face Spaces deployment which they recommend).
  • Prodigy is a powerful closed-source annotation tool that allows for efficient data labeling. With this integration, users can now connect ZenML with Prodigy and leverage its annotation capabilities in their ML pipelines.

Retry configuration for steps

This release also includes new retry configuration for the steps. The following parameters can be set:

  • max_retries: The maximum number of times the step should be retried in case of failure.
  • delay: The initial delay in seconds before the first retry attempt.
  • backoff: The factor by which the delay should be multiplied after each retry attempt.

To use this in your code:

from zenml.config.retry_config import StepRetryConfig

@step(retry=StepRetryConfig(max_retries=3, delay=10, backoff=2))
def step_3() -> None:
    # Step implementation
    raise Exception("This is a test exception")

or using a config.yaml:

steps:
  my_step:
    retry:
      max_retries: 3
      delay: 10
      backoff: 2

In addition, this release includes a number of bug fixes and documentation updates, such as a new LLM finetuning template powered by PEFT and BitsAndBytes and instructions for the new annotators.

Breaking changes

  • The interface for the base class of the annotator stack component has been updated to account for the fact that not all annotators will launch with a specific URL. So there is no longer an url argument passed in.

🥳 Community Contributions 🥳

We'd like to give a special thanks to @christianversloot who contributed to this release by bumping the mlflow version to 2.12.2

What's Changed

Full Changelog: 0.57.1...0.58.0

Contributors

strickvl, stefannica, and 4 other contributors
Assets 2
Loading

0.57.1

14 May 09:54
@safoinme safoinme
0.57.1
7f97fdc
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
0.57.1

This a minor release that brings a variety of enhancements for
the new dashboard release, a new update to the LLMOps guide (covering the use of rerankers in RAG pipelines), and an updated README. It also introduces some new improvements to the service connectors.

We'd like to give a special thanks to @ruvilonix for their first contribution.

What's Changed

New Contributors

Full Changelog: 0.57.0...0.57.1

Contributors

strickvl, stefannica, and 7 other contributors
Assets 2
Loading

0.57.0

02 May 15:15
@avishniakov avishniakov
0.57.0
dab417c
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
0.57.0

We're excited to announce that we're open-sourcing our new and improved dashboard. This unifies the experience for OSS and cloud users, though OSS users will initially see some dashboard features unavailable in this launch release.

We're open-sourcing our dashboard for a few reasons:

  • to ensure that the dashboard experience is consistent across all users, for both the open-source and cloud versions
  • to make it easier for us to maintain and develop the dashboard, as we can share components between the two versions
  • to allow OSS contributions (and self-hosting and modifications) to the new dashboard
  • to open up possibilities for future features, particularly for our OSS users

New users of the ZenML in the dashboard will have a better experience thanks to a much-improved onboarding sequence:

Dashboard 2
The dashboard will guide you through connecting to your server, setting up a stack, connecting to service connectors as well as running a pipeline.

We’ve also improved the ‘Settings’ section of the dashboard and this is the new home for configuration of your repositories, secrets, and connectors, along with some other options.

Dashboard 3

What It Means for You

If you're already a cloud user, not much will change for you. You're already using the new dashboard for pipelines, models and artifacts. Your experience won’t change and for the moment you’ll continue using the old dashboard for certain components (notably for stacks and components).

If you're an open-source user, the new dashboard is now available to you as part of our latest release (0.57.0). You'll notice a completely refreshed design and a new DAG visualizer.

Dashboard 4

Unfortunately, some dashboard features are not yet ready so you'll see instructions on how to access them via the CLI. We hope to have these features returned into the product soon. (If you have a strong opinion as to which you'd like to see first, please let us know!) Specifically, secrets, stacks, and service connectors are not yet implemented in the new dashboard.

How to use the legacy dashboard

The old dashboard is still available to you. To run with the legacy dashboard pass the --legacy flag when spinning it up:

zenml up --legacy

Note that you can’t use both the new and old dashboard at the same time.

If you’re self-hosting ZenML instead of using ZenML Cloud, you can specify which dashboard you want to use by setting the ZEN_SERVER_USE_LEGACY_DASHBOARD environment variable pre-deployment. Specifying a boolean value for this variable will determine which dashboard gets served for your deployment. (There’s no dynamic switching between dashboards allowed, so if you wish to change which dashboard is used for a deployed server, you’ll need to redeploy the server after updating the environment variable.)

If you’re using ZenML Cloud, your experience won’t change with this release and your use of the dashboard remains the same.

What's Changed

Full Changelog: 0.56.4...0.57.0

Contributors

strickvl, stefannica, and 5 other contributors
Assets 2
Loading

0.56.4

24 Apr 13:21
@safoinme safoinme
0.56.4
17d6209
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
0.56.4

This release brings a variety of bug fixes and enhancements, including a new Comet Experiment Tracker integration, additional support for the uv package installer for zenml integration ... commands which significantly improves the speed of integration installations and dependency management, and a new evaluation section in the LLMOps guide.

In addition, it includes a number of bug fixes and documentation updates, such as a fix for cached artifacts produced via save_artifact inside steps linkage to the MCP.

🥳 Community Contributions 🥳

We'd like to give a special thanks to @christianversloot who contributed to this release by bumping the mlflow version to 2.12.1

What's Changed

Full Changelog: 0.56.3...0.56.4

Contributors

strickvl, stefannica, and 6 other contributors
Assets 2
Loading