Python PyTroll
Python PyTroll
PyTroll
An Open-Source, Community-Driven Python Framework to
Process Earth Observation Satellite Data
Martin R aspaud, David Hoese, Adam Dybbroe, Panu Lahtinen, Abhay Devasthale,
Mikhail Itkin, Ulrich Hamann, Lars Ørum R asmussen, Esben Stigård Nielsen, Thomas Leppelt,
Alexander Maul, Christian Kliche, and Hrobjartur Thorsteinsson
Over the last few decades, there has been a rapid of satellite data and ever-evolving user requirements.
increase in both the amount and diversity of Earth The users are more diversified than ever and include
Observation (EO) satellite data. Much of these data forecasters, decision-makers, and other end users.
are assimilated to improve numerical weather predic- To achieve a durable satellite processing system
tion (NWP) (Rabier 2005), while a significant portion supporting highly specialized nowcasting products,
are tailored for nowcasting and early warning. Today’s we argue the need for an open and resilient software
society and global economy is more vulnerable than platform that is able to both quickly adapt to new
yesterday’s to weather and environmental disasters data sources and respond efficiently to new and often
(e.g., floods, forest fires, desert dust, volcanic erup- unexpected user needs. Furthermore, open-source
tions). Consequently, there is a growing public de- development stimulates international collaboration,
mand for early and accurate forecasts and warnings which is a necessity given that there are often rather
to be able to react quickly and minimize damage to limited resources on the national level to operate and
the economy and society. maintain a 24/7 production system under the very
Real-time 24/7 satellite processing systems like demanding conditions outlined above.
those being operated at the national weather services This has been the motivation for the develop-
are being challenged by a steady increase in the amount ment of the Python-based open software framework,
PyTroll (http://pytroll.org), originally started in 2009
as a collaboration between the Danish Meteorologi-
AFFILIATIONS: R aspaud, Dybbroe, and D evasthale — Swedish
Meteorological and Hydrological Institute, Norrköping, Sweden; cal Institute (DMI) and the Swedish Meteorological
Hoese —Space Science and Engineering Center (SSEC), University and Hydrological Institute (SMHI).1 Python was the
of Wisconsin—Madison, Madison, Wisconsin; L ahtinen —Finnish programming language of choice for several reasons.
Meteorological Institute (FMI), Helsinki, Finland; Itkin —Norwe- Python is a widely used, free, and open-source inter-
gian Polar Institute (NPI), Tromsø, Norway; Hamann —Federal active programming language. It allows for a flexible
Office of Meteorology and Climatology (MeteoSwiss), Locarno- bottom-up and fast software development cycle, and
Monti, Switzerland; R asmussen —Danish Meteorological Institute
both SMHI and DMI have had a long tradition of using
(DMI), Copenhagen, Denmark; Nielsen —Think Big Analytics, Co-
penhagen, Denmark; Leppelt and Maul—Deutscher Wetterdienst
Python in operational remote sensing. It is now more
(DWD), Offenbach am Main, Germany; Kliche —EBP Deutschland common to see Python being used as an end-to-end
GmbH, Berlin, Germany; Thorsteinsson —Videntifier Technolo- solution with today’s comprehensive standard library
gies ehf, Reykjavik, Iceland and mature support for optimized numerical and
CORRESPONDING AUTHOR: Martin Raspaud, scientific computations (Lin 2012). With tools like
martin.raspaud@smhi.se Cython, Numexpr, and PyPy, a system built purely
DOI:10.1175/BAMS-D-17-0277.1 upon the Python ecosystem often competes well in
A supplement to this article is available online (10.1175/BAMS-D-17-0277.2).
performance with solutions based on compiled lan-
guages, while preserving its flexibility and ease of use.
©2018 American Meteorological Society
For information regarding reuse of this content and general copy- 1
The historical evolution of the PyTroll code base can be seen
right information, consult the AMS Copyright Policy.
in this video: https://youtu.be/tZcKpB6yPWo.
resampling it, and displaying it on-screen. A true- Spinning Enhanced Visible and Infrared Imager
color image is obtained by combining a red, a green, (SEVIRI) channels by wavelength.
and a blue visible wavelength band, for example, Once data have been loaded, calibrated, geore-
with the Visible and Infrared Imaging Radiometer ferenced, and atmospherically corrected, the data
Suite (VIIRS) instrument on board the Suomi NPP are resampled from the native satellite projection
and JPSS satellites, that would be, respectively, the to a custom grid. This is illustrated in Python code
M4 (or I1), M3, and M2 bands. The right column of in Fig. 1 with newscn = scn.resample(‘eurol’). All
Fig. 1 illustrates another use case with loading two SatPy resampling functionality comes from the
produces cloud parameters as netCDF files. Some as well. For example, the PyGAC package, developed
image products require both the level 1 radiances and in the framework of ESA’s Cloud Climate Change
the level 2 cloud parameters. In that case, the gatherer Initiative (ESA Cloud CCI, www.esa-cloud-cci.org/),
will wait until all input data are available before mes- allows reading, decoding, and calibration of 4-km
saging the flow processor. Global Area Coverage (GAC) data from the AVHRR
Trollflow is used in operations at the Finnish Me- instruments on board a series of NOAA and Metop
teorological Institute for MSG (0° service, RSS and satellites (Devasthale et al. 2017). These 34 years of
IODC), Himawari-8 and GOES-15 geostationary sat- data, available from 1978 to present, have been re-
ellites, and AVHRR/3 (both Metop and NOAA, direct cently processed using PyGAC in the frameworks of
broadcast and EARS) and Suomi-NPP/VIIRS (EARS ESA Cloud CCI and EUMETSAT’s Satellite Applica-
and direct broadcast) polar satellites. As an example, tion Facility for Climate Monitoring (CMSAF, www
the MSG 0° service is processed for five different area .cmsaf.eu) projects (Schulz et al. 2009; Hollmann et al.
definitions for the European area, and altogether 2013). These data were the basis for the subsequent re-
38 different composites are created. Typically, the trievals of the Thematic Climate Data Records (TCDR)
processing for this set takes 43 s for a daylight scene, of global cloud properties (Karlsson et al. 2017; Sten-
and less than 25 s for nighttime scenes when there are gel et al. 2017). As a demonstration, Fig. 4 shows the
less valid composites. The machine used in this case climatological-mean reflectance derived from AVHRR
is a Red Hat Enterprise Linux virtual machine with channel 1 (0.65 µm) using PyGAC for the 34-yr period
4 GB RAM and one CPU core (Intel Xeon) running from 1982 to 2015. In addition to 4-km GAC data, Py-
at 2.40GHz. GAC is capable of reading, decoding, and calibrating
1-km Local Area Coverage (LAC) data from AVHRR
PYTROLL FOR FACILITATING THE DERI- for climate applications. For example, Pareeth et al.
VATION OF CLIMATE DATA RECORDS. (2016) used PyGAC to derive geometrically corrected
While the operational needs triggered the original historical time series of brightness temperatures based
development of PyTroll packages, it offers possibilities on 28 years of 1-km LAC data (1986–2014) over the
to process EO data to cater climate research needs, alpine lakes of northern Italy.
Users interested in investigating individual GAC channel access data unit (CADU) file format in addi-
and/or LAC orbits can use SatPy for easy reading, tion to high-resolution picture transmission (HRPT)
manipulation, and visualization, using the same format, which will allow sharing data from all cur-
method as presented in Fig. 1. rent polar EO satellites. Recent improvements have
been made to SatPy to more effectively handle large
FUTURE WORK. By February 2017, a handful data arrays both on single workstations and data clus-
of the European NMSs had signed a Memorandum ters, which is becoming increasingly important with
of Understanding (http://pytroll.github.io/pytroll the recent launches of Himawari-8 and Himawari-9
_mou_20170222.pdf) showing their long-term and GOES-16, and with new satellite missions on the
commitment to develop and maintain the Py- horizon, such as Meteosat Third Generation (MTG)
Troll software for operational use. All PyTroll and Metop Second Generation (EPS-SG) polar satel-
packages are available on GitHub (https://github. lites. To handle the very high data rates from these
co m /py t ro ll) for susta inabi lit y, which inte- more recent and future missions in a real-time batch
grates with various free cloud services that al- processing system, work toward distributed process-
lows continuous automatic code testing (Travis ing is being pursued (e.g., with the Dask library).
-ci.com and Appveyor); promotes good code documen-
tation (ReadTheDocs.io), code standards (Landscape
.io and Codacy), and test coverage (Coveralls.io); FOR FURTHER READING
and thus secures an overall healthy and robust code
base. Thus far, PyTroll has been mainly focusing on Bentley, J. L., 1975: Multidimensional binary search trees
imager data, but activities are ongoing to better inte- used for associative searching. Commun. ACM, 18,
grate other data sources, like the GOES-16 Lightning 509, https://doi.org/10.1145/361002.361007.
Mapper data and Hyperspectral sounder data like Cohen, B., 2008: The BitTorrent protocol specification.
the Infrared Atmospheric Sounding Interferometer www.bittorrent.org/beps/bep_0003.html.
(IASI) on Metop and Cross-Track Infrared Sounder Devasthale, A., and Coauthors, 2017: PyGAC: An
(CrIS) on Suomi-NPP and JPSS platforms. Work open-source, community-driven Python interface
is progressing to enhance Trollcast to support the to preprocess nearly 40-year AVHRR global area
bookstore.ametsoc.org