Provide "progress" in batch job status

[soxofaan]

In the category "nice to have"

GET /jobs/{job_id} has a field progress:

number [ 0 .. 100 ]
Indicates the process of a running batch job in percent. Can also be set for a job which stopped due to an error or was canceled by the user. In this case, the value indicates the progress at which the job stopped. The Property may not be available for the status codes created and queued. Submitted and queued jobs only allow the value 0, finished jobs only allow the value 100.

This progress indicator can then be shown in clients (python client, web editor).

[soxofaan]

in Open-EO/openeo-python-driver@16f324a I already added progress 0 to status "created"/"queued" and 100 to status "finished"

[soxofaan]

I toyed with some back-of-the-envelope math to get a very simple approximation of the "progress" fraction.

Assume the wall clock run time of batch jobs follow an exponential distribution. While this is a very simple (one-parameter) model, it's still reasonably close to the actual "long tail" distribution in reality. The single parameter (typically called lambda or $\lambda$) directly relates to the average run time $\overline{t}= 1 / \lambda$, e.g. say 10 minutes (600 seconds). So at the start of the batch job, the expected run time is $1 / \lambda$.

Interestingly, because of the mathematical properties of the exponential distribution, the expected remaining run time of a job that already has been running for some time, say $t_c$, is still $1/\lambda$. So the total expected run time is $t_c + 1/\lambda$, which results in an (expected) progress fraction of $t_c / (t_c + \frac{1}{\lambda})$

This gives a pretty simple progress estimate (you just need the actual run time and the average wall clock time for batch job completion). While you could argue that it is somewhat fake, I think it still gives a better user experience than giving no progress at all.
So pretty simple to implement (you just need an average wall clock time for batch job completion),

[soxofaan]

did a quick proof of concept PR Open-EO/openeo-python-driver#340