Skip to content

Latest commit

 

History

History
49 lines (37 loc) · 1.69 KB

README.md

File metadata and controls

49 lines (37 loc) · 1.69 KB

NOOP Spark Transform

Please see the set of transform project conventions for details on general project conventions, transform configuration, testing and IDE set up.

Summary

This project wraps the noop transform with a Ray runtime.

Configuration and command line Options

Noop configuration and command line options are the same as for the base python transform.

Running

Launched Command Line Options

In addition to those available to the transform as defined in here, the set of spark launcher are available.

Running the samples

To run the samples, use the following make targets

  • run-cli-sample - runs src/noop_transform.py using command line args
  • run-local-sample - runs src/noop_local_spark.py
    • Requires prior installation of minio, depending on your platform (e.g., from here and here and invocation of make minio-start to load data into local minio for S3 access.

These targets will activate the virtual environment and set up any configuration needed. Use the -n option of make to see the detail of what is done to run the sample.

For example,

make run-cli-sample
...

Then

ls output

To see results of the transform.

Transforming data using the transform image

To use the transform image to transform your data, please refer to the running images quickstart, substituting the name of this transform image and runtime as appropriate.