Redundant ERA5 preprocessing?

[bnubald]

---

icenet version: v0.4.0_dev
icenet-pipeline version: v0.4.0_dev
download-toolbox version: icenet_0.4

---

The preprocessing of ERA5 seems to be duplicated?

Running through the icenet-pipeline prep_osisaf_training_data.sh script, when downloading era5 data using following command, it downloads both ERA5 and ERA5T data (without distinguishing between the two).

https://github.com/icenet-ai/icenet-pipeline/blob/82bbf848d35450d35898b8e04291d78beccd839c/prep_osisaf_training_data.sh#L22

But, further down the script (below link), it tries to combine both ERA5 and ERA5T data (which was the approach for the older CDSAPI):

https://github.com/JimCircadian/icenet-pipeline/blob/86cf3a18589adbfbb9284734c89f1999ef742dc3/prep_osisaf_training_data.sh#L80-L84

so, having the following seems redundant within the icenet since its covered by download-toolbox:

icenet/icenet/data/processors/cds.py

Lines 8 to 13 in 6caa234

	if 'expver' in da.coords:
	logging.warning("expvers {} in coordinates, will process out but "
	"this needs further work: expver needs storing for "
	"later overwriting".format(da.expver))
	# Ref: https://confluence.ecmwf.int/pages/viewpage.action?pageId=173385064
	da = da.sel(expver=1).combine_first(da.sel(expver=5))

There's similar logging here in download-toolbox:

https://github.com/environmental-forecasting/download-toolbox/blob/9ab7b1aeb806b2bd71ff351c08ea6c2a6cca0d26/download_toolbox/data/cds.py#L195-L206

[JimCircadian]

Thanks for highlighting @bnubald, we'll make sure it only happens once. ;)