Computing Cloud as a perfect environment for Big EO data repositories

Copernicus as a flagship European space programme 

Copernicus is the European Union's Earth Observation (EO) programme managed by the European Commission. The main goal of the programme is to observe our planet and its environment to benefit all European citizens. The programme has revolutionized the European space industry by making access to sattelite data is easier than never before. The enormous amount of EO data available as a public good for anyone at no cost.Huge amount of archival data and the continuous growth of large EO data sets made it necessary not only to create a cloud environment, but also to develop the usability and exploit the potential of EO data - the DIAS platforms (Data and Information Access Services). For this reason, EO platforms were launched, thanks to which we have gained a powerful tool for EO data access and scalable processing.  

Computing Cloud as a perfect environment for Big EO data repositories - sentinel 3 olci sentinel 3 true color on 2022 07 06 1.jpeg 1000x242 q85 crop subsampling 2 upscale

DIAS and EO platforms 

EO platforms are an essential element to enhance the EO data usability, making it possible to change data into information. The platforms provide users with a computing power that enables fast and efficient processing and allow users to access, process and analyse Copernicus products in the cloud directly connected to an EO archive. The availability of data, storage, and processing in a single place makes it easier to develop and distribute scalable digital services based on the EO data. Below you can see consumption model for DIAS and EO platforms. 

Computing Cloud as a perfect environment for Big EO data repositories - zrzut ekranu 2022 07 7 o 130702.png 1000x354 q85 crop subsampling 2 upscale

We rely on an open- source technology

We have built and now operate several EO platforms based on a hybrid cloud in an Infrastructure-as-a-service model. Our services consist of open-source components, such as Open-Stack for cloud orchestration, Ceph for storage management, Prometheus and Graphana for monitoring,  K8s for processing, and others.

EO platform as a service 

Taking as an example CREODIASWEkEO, CODE-DE and other platforms built and operated by CloudFerro, we have created the main building blocks and architecture of entirely European cloud platforms based on open-source software.

We have managed to create a new service - an EO platform as a service.

Computing Cloud as a perfect environment for Big EO data repositories - zrzut ekranu 2022 07 7 o 115600.jpeg 1000x568 q85 crop subsampling 2 upscale

Federation and multi- platform approach

Federation and a multi-platform approach allows users and operators to gain benefits from the synergy between different platforms, such as a shared multi PB data repository available from various platforms and elasticity cloud components for additional processing needs. At the same time, it generates a technical challenge that we need to address. We have met challenges focusing on three main topics: 

  • Interoperability of the environments: how to support easy migration between different clouds and provide direct access to the repositories available from different computing clouds, with multi-cloud processing enabled and dynamically adaptable usage and payment models. 
  • Data access and dissemination: the multi-petabyte, scalable EO data repositories should provide fast, instantaneous access to many heterogeneous users and algorithms with different needs. At the same time, the data needs to be discoverable and ready for analysis in a context of constantly evolving data offers (new collections, commercial data, products), and users should be supported in data dissemination. 
  • Cloud efficiency: how to optimize data access, storage, and processing costs and energy usage both for individual users and for the entire ecosystem. 

Why is it so important to use EO platform

Using the EO Platform is connected with numerous benefits. The most significant are:

Computing Cloud as a perfect environment for Big EO data repositories - zrzut ekranu 2022 07 7 o 130857.png 1000x502 q85 crop subsampling 2 upscale