Case studies - EO IPT

The challenge

In 2014, the European Space Agency (ESA), one of the major stakeholders in the global space industry, has recognized the need for developing a new model of distributing Earth Observation data produced by an ever-growing amount of Sentinel satellites and auxiliary sources. The data distribution concept of that time foresaw data discovery followed by download and computing within the user’s own infrastructure. Such a solution offered no feasibility, once the volume of data to be processed was a level of magnitude larger than reasonably available bandwidths and computing resources for a wide array of users.

ESA addressed the issue of big data transfer and processing in its Earth Observation Ground Segment Evolution strategy. The first evolutionary testbed launched in 2014 was the Thematic Exploitation Platforms, where different technical and organizational approaches have been tested to bring processing and data closer together at the example of different thematic areas.

Following up aforementioned activities call for an innovation testbed, that would explore the maturity levels of cloud industry to deploy and manage big data platform for exploitation of Earth Observation data, was announced. CloudFerro, as a major stakeholder in a consortium, was awarded the contract, starting EO IPT project.

The solution

EO IPT became an answer to the questions of big data industry maturity for enabling remote sensing-enabled value chains and feasibility of the newly suggested paradigm for Earth Observation data discovery, transfer, and processing. CloudFerro has delivered a platform amassing more than 5 PB (5000 TB) of remote sensing data, sourcing from Sentinel and Landsat satellites. That data was enabled for processing with computing resources of more than 1500 virtual cores, 7 TB of RAM, and 500 TB of dedicated disk space.

Successful deployment, development, and maintenance of the platform while fulfilling goals presented in the call demanded providing efficient, petabyte-scale storage for Earth Observation data. Storage that would allow scaling during any period of operation in order to accommodate satellite imagery. Research results have determined Ceph – distributed, scalable, highly-available storage cluster optimized for read-write access – to be the most suitable solution. To enhance user’s ability to take advantage of a large amount of data in one’s value chain, storage resources had been complemented with dedicated cloud processing infrastructure (Virtual Machines, HDD and SSD storage, firewall, virtual networking). The infrastructure had been provided with a fast and reliable internet connection.

In order to improve the efficiency and working comfort of Earth Observation data users, processing use cases had been concentrated within the platform’s environment as well as adequate tools for imagery analysis, such as Sentinel Toolbox.

OpenStack modules of EO IPT infrastructure

In order to face requirements, set by ESA, for providing and generically operating cloud infrastructure, CloudFerro has decided to use OpenStack technology. OpenStack is an open-source cloud operating system for controlling large pools of computing, storage, and networking resources.

State of the art infrastructure created for EO IPT consisted of numerous OpenStack modules, including the following ones:

Customer benefits

Implementation of the Earth Observation Innovation Testbed resulted with:

• Boost to development of the EO service industry, by changing the economy of providing value-added EO services by third parties. The unique breadth of directly accessible EO data offered by the platform enabled many innovative applications.
• Demonstration of the service and business model based on integrating and hosting a wide array of ESA and ESA-related projects and environments, including the Foodsecurity Thematic Exploitation Platform.
• Deployed, tested, and proofed against real-market circumstances pay-per-use business model, something never tried on this scale in the remote sensing field.
• Making a vital array of open and free Earth Observation data available on big data-enabled cloud infrastructure, designed and optimized to hold processing and exploitation platform applications.
• Research on possibilities of integrating other commercial data, third-party services, and algorithms with big data, Earth Observation focused commercial processing platform.
• Active promotion of the platform to multiple stakeholders and value-adders from outside of the European Space Agency.
• Raised awareness throughout a variety of potential users, such as governmental and international institutions, scientific bodies, and commercial entities, towards exploitation of Earth Observation data within their own value chains.

Summary

EO IPT has been a milestone in researching and developing a feasible strategy for handling, transferring, and processing Earth Observation products and their derivates in a big data, cloud environment. Its successful deployment and operation have aided ESA and the Copernicus program in paving way for future remote sensing data exploitation. During the platform’s activity, all stakeholders have had the ability to gain experience and foresight into the current and future needs of the industry.