-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
0 parents
commit 15679f1
Showing
49 changed files
with
6,597 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
# Sphinx build info version 1 | ||
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. | ||
config: 08e903fa7a557a713623c2c456b8447e | ||
tags: 645f666f9bcd5a90fca523b33c5a78b7 |
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,68 @@ | ||
|
||
# The openEO Aggregator: federated openEO processing | ||
|
||
The openEO Aggregator is a software component to group multiple openEO back-ends together | ||
into a unified, federated openEO processing platform. | ||
|
||
|
||
```{mermaid} | ||
flowchart LR | ||
|
||
U("👤 User") --> A("openEO Aggregator") | ||
|
||
subgraph federation ["Federated openEO Processing"] | ||
|
||
A --> B1("openEO Back-end 1") | ||
A --> B2("openEO Back-end 2") | ||
A --> B3("openEO Back-end 3") | ||
end | ||
``` | ||
|
||
|
||
## Core openEO API | ||
|
||
The [openEO API](https://openeo.org/) is an open, standardized API for Earth Observation data processing, | ||
connecting openEO-capable clients at the user side with openEO-capable back-ends at the (cloud) processing side. | ||
Not only does it decouple the clients requirements from the technology stack of the back-ends, | ||
it also allows the user or client to switch between back-ends with minimal or even no code changes. | ||
Multiple openEO back-end implementations have been developed and are available today, | ||
each based on different processing technologies and each providing a different set of data collections. | ||
|
||
The freedom to choose a back-end and avoiding lock-in is one of the key features of openEO, | ||
but it also implies that the user is required to make a choice, as there is no default back-end. | ||
Moreover, the user might want to combine data or processing functionality from different back-ends, | ||
which is not directly supported by openEO's core API. | ||
Note that Earth Observation data is fast-growing and diverse, | ||
making it unsafe to assume that a single provider will be able to host all EO data. | ||
|
||
## Federated openEO processing | ||
|
||
The "openEO Aggregator" project aims to address this problem through a proxy-like component to | ||
build a federated openEO processing platform. | ||
The openEO Aggregator allows to group multiple openEO back-ends together | ||
and to and expose their combined power as a single, openEO-compliant API endpoint to the user, | ||
including, but not limited to: | ||
|
||
- merging and unification of general resource metadata such as data collections and openEO processes | ||
- unified listing of batch jobs of a user across multiple back-ends | ||
- dispatching of simple processing requests (both for synchronous processing and batch jobs) to the appropriate back-end | ||
- handling of more complex processing requests that require data from multiple back-ends | ||
|
||
|
||
|
||
```{toctree} | ||
:caption: Documentation | ||
:hidden: | ||
pages/installation.md | ||
pages/configuration.md | ||
pages/usage.md | ||
``` | ||
|
||
```{toctree} | ||
:caption: Development | ||
:hidden: | ||
pages/changelog | ||
GitHub <https://github.com/Open-EO/openeo-aggregator> | ||
openEO.org <https://openeo.org/> | ||
pages/license | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
# Changelog | ||
|
||
|
||
```{include} ../../CHANGELOG.md | ||
:start-after: <!-- start changelog --> | ||
:end-before: <!-- end changelog --> | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
|
||
|
||
# Configuration | ||
|
||
:::{Attention} | ||
This documentation page is work in progress... 👷 | ||
::: | ||
|
||
|
||
The openEO Aggregator can be configured through a `AggregatorBackendConfig` class | ||
|
||
|
||
## Gunicorn configuration | ||
|
||
TODO |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
|
||
# Installation | ||
|
||
:::{Attention} | ||
This documentation page is work in progress... 👷 | ||
::: | ||
|
||
|
||
The openEO Aggregator can be installed via pip: | ||
|
||
```shell | ||
python -m pip install openeo-aggregator | ||
``` | ||
|
||
|
||
## Requirements | ||
|
||
- Optional: a Zookeeper cluster for caching and partitioned job db |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
# License | ||
|
||
```{literalinclude} ../../LICENSE.txt | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
|
||
# Usage | ||
|
||
:::{Attention} | ||
This documentation page is work in progress... 👷 | ||
::: | ||
|
||
|
||
## Run with Flask in development mode | ||
|
||
```shell | ||
export FLASK_APP='openeo_aggregator.app:create_app()' | ||
export FLASK_ENV=development | ||
flask run | ||
``` | ||
|
||
## Run with Gunicorn | ||
|
||
```shell | ||
gunicorn 'openeo_aggregator.app:create_app()' | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,101 @@ | ||
// @ts-check | ||
|
||
// Extra JS capability for selected tabs to be synced | ||
// The selection is stored in local storage so that it persists across page loads. | ||
|
||
/** | ||
* @type {Record<string, HTMLElement[]>} | ||
*/ | ||
let sd_id_to_elements = {}; | ||
const storageKeyPrefix = "sphinx-design-tab-id-"; | ||
|
||
/** | ||
* Create a key for a tab element. | ||
* @param {HTMLElement} el - The tab element. | ||
* @returns {[string, string, string] | null} - The key. | ||
* | ||
*/ | ||
function create_key(el) { | ||
let syncId = el.getAttribute("data-sync-id"); | ||
let syncGroup = el.getAttribute("data-sync-group"); | ||
if (!syncId || !syncGroup) return null; | ||
return [syncGroup, syncId, syncGroup + "--" + syncId]; | ||
} | ||
|
||
/** | ||
* Initialize the tab selection. | ||
* | ||
*/ | ||
function ready() { | ||
// Find all tabs with sync data | ||
|
||
/** @type {string[]} */ | ||
let groups = []; | ||
|
||
document.querySelectorAll(".sd-tab-label").forEach((label) => { | ||
if (label instanceof HTMLElement) { | ||
let data = create_key(label); | ||
if (data) { | ||
let [group, id, key] = data; | ||
|
||
// add click event listener | ||
// @ts-ignore | ||
label.onclick = onSDLabelClick; | ||
|
||
// store map of key to elements | ||
if (!sd_id_to_elements[key]) { | ||
sd_id_to_elements[key] = []; | ||
} | ||
sd_id_to_elements[key].push(label); | ||
|
||
if (groups.indexOf(group) === -1) { | ||
groups.push(group); | ||
// Check if a specific tab has been selected via URL parameter | ||
const tabParam = new URLSearchParams(window.location.search).get( | ||
group | ||
); | ||
if (tabParam) { | ||
console.log( | ||
"sphinx-design: Selecting tab id for group '" + | ||
group + | ||
"' from URL parameter: " + | ||
tabParam | ||
); | ||
window.sessionStorage.setItem(storageKeyPrefix + group, tabParam); | ||
} | ||
} | ||
|
||
// Check is a specific tab has been selected previously | ||
let previousId = window.sessionStorage.getItem( | ||
storageKeyPrefix + group | ||
); | ||
if (previousId === id) { | ||
// console.log( | ||
// "sphinx-design: Selecting tab from session storage: " + id | ||
// ); | ||
// @ts-ignore | ||
label.previousElementSibling.checked = true; | ||
} | ||
} | ||
} | ||
}); | ||
} | ||
|
||
/** | ||
* Activate other tabs with the same sync id. | ||
* | ||
* @this {HTMLElement} - The element that was clicked. | ||
*/ | ||
function onSDLabelClick() { | ||
let data = create_key(this); | ||
if (!data) return; | ||
let [group, id, key] = data; | ||
for (const label of sd_id_to_elements[key]) { | ||
if (label === this) continue; | ||
// @ts-ignore | ||
label.previousElementSibling.checked = true; | ||
} | ||
window.sessionStorage.setItem(storageKeyPrefix + group, id); | ||
} | ||
|
||
document.addEventListener("DOMContentLoaded", ready, false); |
Large diffs are not rendered by default.
Oops, something went wrong.
Oops, something went wrong.