[EPIC] create deploy/local_batch_job.py

[JeroenVerstraelen]

• investigate batch_job.py, can it run locally?
• Remove local_batch_job.py and use batch_job.py

local_batch_job.py is a script that takes in a process graph as input and writes its output to local disk. It can be started as a local spark driver via spark-submit. It runs inside a docker container that provides e.g. gdal support.

[JeroenVerstraelen]

• test local_batch_job.py with a simple 1+2 process graph, see if output is written correctly to local job directory.

  • Handled in: test local_batch_job.py with a simple process graph #839
  • add this as a test
  • This has to work inside the docker container for local deployments

• ensure all integrations are in-memory or not available (=no errors)

  • Handled in: local_batch_job.py ensure all integrations are in-memory or not available (=no errors)  #840
  • zookeeper, ejr, etl, elastic, hadoop, yarn, kubernetes, sentinelhub, aws, custom processes, layercatalog enrichment

• test local_batch_job.py with load_stac, see if output is written correctly

  • Handled in: test local_batch_job.py with load_stac, see if output is written correctly #807
  • Add simple test that runs process graph using local_batch_job.py and hcecks if the output is valid.
  • We will use load_stac for all datacube loading in local batch jobs
  • add documentation

• test complex process graph on local run

  • This has to work inside the docker container for local deployments
  • Add a test that runs a process graph using local_batch_job.py and checks if the output is valid.

• Can it start via spark-submit (batch_job)

  • as a local driver with local executors (threads) saving files on local disk
  • as a local driver with remote executors (cluster) saving files on local disk

• investigate logging for executors/driver on local run. Should be clear for the user

  • same file?
  • json logging?
    • This is already handled via an environment variable.
  • Handled in local_batch_job.py investigate logging #838

OPTIONAL

• add helper script in geopyspark that allows user to submit python-client datacube as batch_job

  • Users will want to use the openeo-python-client API to construct a process graph
  • local_batch_job.py is a script that takes in a process graph + some extra parameters and writes its output to disk.
  • We should make it easy to write a process graph to a file so it can be passed in as a parameter

• check UDP from url (locally)

• does random_forest work when running locally

[EmileSonneveld]

Further developments: #894