Computing Cloud as a perfect environment for Big EO data repositories
Copernicus as a flagship European space programme
Copernicus is the European Union's Earth Observation (EO) programme managed by the European Commission. The main goal of the programme is to observe our planet and its environment to benefit all European citizens. The programme has revolutionized the European space industry by making access to sattelite data is easier than never before. The enormous amount of EO data available as a public good for anyone at no cost.Huge amount of archival data and the continuous growth of large EO data sets made it necessary not only to create a cloud environment, but also to develop the usability and exploit the potential of EO data - the DIAS platforms (Data and Information Access Services). For this reason, EO platforms were launched, thanks to which we have gained a powerful tool for EO data access and scalable processing.
DIAS and EO platforms
EO platforms are an essential element to enhance the EO data usability, making it possible to change data into information. The platforms provide users with a computing power that enables fast and efficient processing and allow users to access, process and analyse Copernicus products in the cloud directly connected to an EO archive. The availability of data, storage, and processing in a single place makes it easier to develop and distribute scalable digital services based on the EO data. Below you can see consumption model for DIAS and EO platforms.
We rely on an open- source technology
We have built and now operate several EO platforms based on a hybrid cloud in an Infrastructure-as-a-service model. Our services consist of open-source components, such as Open-Stack for cloud orchestration, Ceph for storage management, Prometheus and Graphana for monitoring, K8s for processing, and others.
EO platform as a service
Taking as an example CREODIAS, WEkEO, CODE-DE and other platforms built and operated by CloudFerro, we have created the main building blocks and architecture of entirely European cloud platforms based on open-source software.
We have managed to create a new service - an EO platform as a service.
Federation and multi- platform approach
Federation and a multi-platform approach allows users and operators to gain benefits from the synergy between different platforms, such as a shared multi PB data repository available from various platforms and elasticity cloud components for additional processing needs. At the same time, it generates a technical challenge that we need to address. We have met challenges focusing on three main topics:
- Interoperability of the environments: how to support easy migration between different clouds and provide direct access to the repositories available from different computing clouds, with multi-cloud processing enabled and dynamically adaptable usage and payment models.
- Data access and dissemination: the multi-petabyte, scalable EO data repositories should provide fast, instantaneous access to many heterogeneous users and algorithms with different needs. At the same time, the data needs to be discoverable and ready for analysis in a context of constantly evolving data offers (new collections, commercial data, products), and users should be supported in data dissemination.
- Cloud efficiency: how to optimize data access, storage, and processing costs and energy usage both for individual users and for the entire ecosystem.
Why is it so important to use EO platform
Using the EO Platform is connected with numerous benefits. The most significant are:
