In this paper, which is an expository account of a lossless compression techniques that have been developed over the course of a sequence of papers and talks, we have sought to identify and bring out the key features of our approach to the efficient compression of hyperspectral satellite data. In particular we provide the motivation for using our approach, which combines the advantages of a clustering with linear modeling. We will also present a number of visualizations which help clarify why our approach is particularly effective on this dataset. At each stage, our algorithm achieves an efficient grouping of the data points around a relatively small number of lines in a very large dimensional data space. The parametrization of these lines is very efficient, which leads to efficient descriptions of data points. Our method, which we are continuing to refine and tune, has to date yielded compression ratios that compare favorably with what is currently achievable by other approaches. A data sample consisting of an entire day's worth of global AQUA-EOS AIRS Level 1A counts (mean 12.9 bit-depth) data was used to evaluate the compression algorithm. The algorithm was able to achieve a lossless compression ratio on the order of 3.7 to 1.

Contemporary and future ultraspectral sounders represent a significant technical advancement for environmental and meteorological prediction and monitoring. Given their large volume of spectral observations, the use of robust data compression techniques will be beneficial to data transmission and storage. In this paper, we propose a novel Adaptive Vector Quantization (VQ)-based Linear Prediction (AVQLP) method for ultraspectral data compression. The method is compared with several state-of-the-art methods such as CALIC, JPEG-LS and JPEG2000. The compression experiments show that our AVQLP method is the first to surpass the 4 to 1 lossless compression barrier for a selected set of AIRS ultraspectral sounder test data.

The Karhunen-Loeve transform (KLT) is the optimal unitaty transform that yields the maximum coding gain. The prediction-based lower triangular transform (PLT) features the same decorrelation and coding gain properties as the KLT, but with lower complexity. Unlike KLT, PLT has the perfect reconstruction property which allows its use for lossless compression. In this paper we apply PLT to lossless compression of the ultraspectral sounder data. The experiment on the standard ultraspectral test dataset of 10 AIRS digital count granules shows that the PLT compression outperforms JPEG2000, SPIHT, JPEG-LS, and CCSDS IDC 5/3.

The next-generation GOES-R (Geostationary Operational Environmental Satellite) HES (Hyperspectral Environmental Suite) Sounder will be either a grating or interferometer design. The HES will be able to provide hourly atmospheric soundings with spatial resolutions of 5 ~ 10 km with higher accuracy than the current geostationary sounder. A number of GOES-R products will be made from the HES data, this information will help both in forecasting and numerical model initializations. Extensive research has been done with lossless data compression with data from a grating-type ultraspectral instrument. NAST-I aircraft data is chosen for testing data from interferometers until IASI (Infrared Atmospheric Sounding Interferometer) and CrIS (Cross-track Infrared Sounder) are available. Preliminary work at CIMSS with lossless data compression of Michelson Interferometer data achieves compression ratios (CR) above 5.

This paper presents a follow-up to last year's SPIE meeting where we presented a residual encoding method to control maximum absolute error (MAE) based on JPEG2000 Part 2 standard and which was applied to hyperspectral data. In this paper, we evaluate an improved version of the approach on the ultraspectral sounder satellite data made available by NOAA. The data set used consists of a subset of 1,501 bands out of the 2,378 total and where each band is an image of size 90 by 135 pixels. Each pixel or data value is a digital count integer that requires 12 - 14 bits to represent. We present compression performance using a transform in the band (z- or cross-component) direction. We use either the Karhunen-Loeve transform or the discrete wavelet transform with a non-uniform bit-rate allocation to take advantage of the energy compaction. One of the main features of this compression scheme is that residuals (original minus the decompressed values) are also coded in order to control the MAE; therefore, lossless compression can also be accomplished by using a desired MAE of 0.5. In all cases, the quantized residuals are losslessly encoded using the embedded block coding with optimized truncation (EBCOT) bit-plane encoding method that is part of JPEG2000 Part 1. Finally, our recent algorithm for automatically choosing the best (smallest total) combination of the two contributing bit rates is also extended to the 3-dimensional case. The two rates are: (1) the Open Loop rate for the lossy compression using JPEG2000 Part 2 by itself and (2) the EBCOT rate that results from the coding of the quantized residuals. The basis for the approach is the modeling of the residuals using generalized Gaussian random variables. Results for lossless and near-lossless compression will be presented using both an exhaustive search and the automatic search method for finding the minimum overall bit rate.

Presented here is a study comparing various inter-slice bit-allocation strategies when applying JPEG2000 part 2 to ultraspectral sounder satellite data. Two families of algorithms are compared: the first is strictly variance based for which traditional approaches and some variations are used; the second uses rate distortion curves (RDCs) to optimally allocate slice bit-rates in the MSE sense, analogous to the way the PCRD algorithm uses RDCs of code-blocks in the baseline JPEG2000 method. The rate distortion curves are either experimentally gathered or analytically modeled using techniques discussed in previous papers. An additional approach to gather RDCs is considered using the slope-length information computed by a JPEG2000 encoder. The study is done using six different granules of ultraspectral sounder data made available by NOAA. Every subset contains 1,501 spectral bands, each consisting of 90 by 135 pixels, which are decorrelated using either a Karhunen-Loeve transform (KLT) or the discrete wavelet transform (DWT). The most complex RDC-based optimal approach is used to establish a lower distortion bound, in addition to characterizing the typical distortion performance achievable when compressing spectrally decorrelated sounder data with JPEG2000. The bound is used as the basis of comparison against the other methods studied. Moreover, the software tool CompressMD developed by our group is used to provide all data handling, algorithm implementations and collection of results.

AIRS was launched on EOS Aqua on May 4, 2002, together with AMSU-A and HSB, to form a next generation polar orbiting infrared and microwave atmospheric sounding system. The primary products of AIRS/AMSU-A are twice daily global fields of atmospheric temperature-humidity profiles, ozone profiles, sea/land surface skin temperature, and cloud related parameters including OLR. The sounding goals of AIRS are to produce 1 km tropospheric layer mean temperatures with an rms error of 1K, and layer precipitable water with an rms error of 20 percent, in cases with up to 80 percent effective cloud cover. The basic theory used to analyze AIRS/AMSU/HSB data in the presence of clouds, called the at-launch algorithm, and a post-launch algorithm which differed only in the minor details from the at-launch algorithm, have been described previously. The post-launch algorithm, referred to as AIRS Version 4, has been used by the Goddard DAAC to analyze and distribute AIRS retrieval products. In this paper we show two candidates for the AIRS Version 5 algorithm which will be used by the Goddard DAAC starting late in 2006. The methodology used in each is otherwise identical, but one version uses only AIRS observations in the generation of cloud cleared radiances, while the other uses both AIRS and AMSU-A observations as done previously. Improvements made to the retrieval algorithm since Version 4 are described as well as results comparing retrieval accuracy and spatial coverage of each candidate for Version 5 with each other and with those obtained using Version 4.

Errors due to wireless transmission can have an arbitrarily large impact on a compressed file. A single bit error appearing in the compressed file can propagate during a decompression procedure and destroy the entire granule. Such a loss is unacceptable since this data is critical for a range of applications, including weather prediction and emergency response planning. The impact of a bit error in the compressed granule is very sensitive to the error's location in the file. There is a natural hierarchy of compressed data in terms of impact on the final retrieval products. For the considered compression scheme, errors in some parts of the data yield no noticeable degradation in the final products. We formulate a priority scheme for the compressed data and present an error correction approach based on minimizing impact on the retrieval products. Forward error correction codes (e.g., turbo, LDPC) allow the tradeoff between error correction strength and file inflation (bandwidth expansion). We propose segmenting the compressed data based on its priority and applying different-strength FEC codes to different segments. In this paper we demonstrate that this approach can achieve negligible product degradation while maintaining an overall 3-to-1 compression ratio on the final file. We apply this to AIRS sounder data to demonstrate viability for the sounder on the next-generation GOES-R platform.

Most source coding techniques generate bitstream where different regions have unequal influences on data reconstruction. An uncorrected error in a more influential region can cause more error propagation in the reconstructed data. Given a limited bandwidth, unequal error protection (UEP) via channel coding with different code rates for different regions of bitstream may yield much less error contamination than equal error protection (EEP). We propose an optimal UEP scheme that minimizes error contamination after channel and source decoding. We use JPEG2000 for source coding and turbo product code (TPC) for channel coding as an example to demonstrate this technique with ultraspectral sounder data. Wavelet compression yields unequal significance in different wavelet resolutions. In the proposed UEP scheme, the statistics of erroneous pixels after TPC and JPEG2000 decoding are used to determine the optimal channel code rates for each wavelet resolution. The proposed UEP scheme significantly reduces the number of pixel errors when compared to its EEP counterpart. In practice, with a predefined set of implementation parameters (available channel codes, desired code rate, noise level, etc.), the optimal code rate allocation for UEP needs to be determined only once and can be done offline.

The GOES-R series, with a first launch planned around 2012, is currently under development by NOAA. It will be the first in a series of new geostationary environmental satellites to provide greater capabilities needed for weather, atmosphere, climate, and ocean monitoring. An extremely high volume of multi to hyper/ultra spectral remote sensing data is generated from both the onboard sensors and the rebroadcast ground station, with rates in the range of 25 to 140 Mbps. However, the bandwidths allocated to GOES-R are limited, therefore, data compression is necessary. When data are compressed, the information contained in some segments, such as the frame header and sync mark, are much more important than that in other segments. In order to provide unequal protection to different segments of the streaming data, this paper presents a hierarchical modulation that utilizes a 16QAM constellation to transmit three bits per channel use by applying a rate 1/2 forward error correcting code to the least significant bit (LSB). When those bits that require high protection are placed at the LSB positions, the robustness can be significantly enhanced by several orders of magnitude. Depending on the performance requirements of the LSB and the other two bits, the constellation can be optimized so that a minimum amount of transmitted power is needed. The peak-to-average power ratio (PAR) of a 16QAM constellation requires a power back-off when a nonlinear power amplifier, e.g., traveling wave tube or solid-state power amplifier, is used. The power back-off is needed in order to avoid signal distortion. In this paper, the bit error rate (BER) performances of this hierarchical modulation, 8PSK, and rate 3/4 coded 16QAM, which have the same bandwidth efficiency, are compared, taking into account the required power back-off. It is shown that the hierarchical modulation outperforms 8PSK. When the unequal protection is considered, the paper also shows an advantage of using this hierarchical modulation over the rate 3/4 coded 16QAM.

Previous study shows 3D Wavelet Transform with Reversible Variable-Length Coding (3DWT-RVLC) has much better error resilience than JPEG2000 Part 2 on 1-bit random error remaining after channel decoding. Errors in satellite channels might have burst characteristics. Low-density parity-check (LDPC) codes are known to have excellent error correction capability near the Shannon limit performance. In this study, we investigate the burst error correction performance of LDPC codes via the new Digital Video Broadcasting - Second Generation (DVB-S2) standard for ultraspectral sounder data compressed by 3DWT-RVLC. We also study the error contamination after 3DWT-RVLC source decoding. Statistics show that 3DWT-RVLC produces significantly fewer erroneous pixels than JPEG2000 Part 2 for ultraspectral sounder data compression.

Entropy coding techniques aim to achieve the entropy of the source data by assigning variable-length codewords to symbols with the code lengths linked to the corresponding symbol probabilities. Entropy coders (e.g. Huffman coding, arithmetic coding), in one form or the other, are commonly used as the last stage in various compression schemes. While these variable-length coders provide better compression than fixed-length coders, they are vulnerable to transmission errors. Even a single bit error in the transmission process can cause havoc in the subsequent decoded stream. To cope with it, this research proposes a marker-based sentinel mechanism in entropy coding for error detection and recovery. We use arithmetic coding as an example to demonstrate this error-resilient technique for entropy coding. Experimental results on ultraspectral sounder data indicate that the marker-based error-resilient arithmetic coder provides remarkable robustness to correct transmission errors without significantly compromising the compression gains.

The Aerospace Corporation has developed an end-to-end testbed to demonstrate a wide range of modern modulation and coding alternatives for future broadcast by the GOES-R Global Rebroadcast (GRB) system. In particular, this paper describes the development of a compact, low cost, flexible GRB digital receiver that was designed, implemented, fabricated, and tested as part of the development. This receiver demonstrates a 10-fold increase in data rate compared to the rate achievable by the current GOES generation, without a major impact on either cost or size. The digital receiver is integrated on a single PCI card with an FPGA device, and analog-to-digital converters. It supports a wide range of modulations (including 8-PSK and 16-QAM) and turbo coding. With appropriate FPGA firmware and software changes, it can also be configured to receive the current (legacy) GOES signals. The receiver has been validated by sending large image files over a high-fidelity satellite channel emulator, including a space-qualified power amplifier and a white noise source. The receiver is a key component of a future GOES-R weather receiver system (also called user terminal) that includes the antenna, low-noise amplifier, downconverter, filters, digital receiver, and receiver system software. This work describes this receiver proof of concept and its application to providing a very credible estimate of the impact of using modern modulation and coding techniques in the future GOES-R system.

The hardware implementation of a low complexity Low Density Parity Check (LDPC) decoder is described. The design of the LDPC decoder optimized on minimizing the amount of hardware resources necessary for implementation. In addition to implementation details, design tradeoffs considered in the development of the LDPC decoder are discussed. The intended application of the LDPC decoder is a nonlinear satellite communications channel. The nonlinearities and communications signal perturbations include Additive White Gaussian Noise (AWGN), phase noise, phase imbalance, and a model satellite high power amplifier nonlinearity. The LDPC decoder performance is then characterized in the satellite channel.

The National Oceanic and Atmospheric Administration's (NOAA) National Weather Service (NWS) uses a commercial Satellite Broadcast Network (SBN) to distribute weather data to the NWS AWIPS workstations and National Centers and to NWS Family of Service Users. Advances in science and technology from NOAA's observing systems, such as remote sensing satellites and NEXRAD radars, and advances in Numeric Weather Prediction have greatly increased the volume of data to be transmitted via the SBN. The NOAA-NWS SBN Evolution Program did a trade study resulting in the selection of Europe's DVB-S communication protocol as the basis for enabling a significant increase in the SBN capacity. The Digital Video Broadcast (DVB) group, started to develop digital TV for Europe through satellite broadcasting, has become the current standard for defining technology for satellite broadcasting of digital data for much of the world. NOAA-NWS implemented the DVB-S with inexpensive, Commercial Off The Shelf receiving equipment. The modernized NOAA-NWS SBN meets current performance goals and provides the basis for continued future expansion with no increase in current communication costs. This paper discusses aspects of the NOAA-NWS decision and the migration to the DVB-S standard for its commercial satellite broadcasts of observations and Numerical Weather Prediction data.

Satellite communication and archiving systems are now designed according to an outdated Shannon information theory where all data is transmitted in meaningless bit streams. Video bit rates, for example, are determined by screen size, color resolution, and scanning rates. The video "content" is irrelevant so that totally random images require the same bit rates as blank images. An alternative system design, based on the newer Autosophy information theory, is now evolving, which transmits data "contend" or "meaning" in a universally compatible 64bit format. This would allow mixing all multimedia transmissions in the Internet's packet stream. The new systems design uses self-assembling data structures, which grow like data crystals or data trees in electronic memories, for both communication and archiving. The advantages for satellite communication and archiving may include: very high lossless image and video compression, unbreakable encryption, resistance to transmission errors, universally compatible data formats, self-organizing error-proof mass memories, immunity to the Internet's Quality of Service problems, and error-proof secure communication protocols. Legacy data transmission formats can be converted by simple software patches or integrated chipsets to be forwarded through any media - satellites, radio, Internet, cable - without needing to be reformatted. This may result in orders of magnitude improvements for all communication and archiving systems.

The next generation of environmental monitoring satellites will provide several orders of magnitude increase in the data volume of science data collected using advanced imager and sounder instruments. In the cases of geosynchronous satellites, continuous access to this data is needed by data centers and users across the world for weather forecasting and environmental monitoring. The public Internet has sometimes been mentioned as an economical medium to distribute real time data. The purpose of this paper is to dispel some of the unreasonable claims for cost efficiency and performance that are expressed in discussions of using the Internet to distribute real-time environmental data. The paper will use the Geosynchronous Operational Environmental Satellites (GOES) series as a case in point to describe alternative terrestrial-bases telecommunications services for environmental data distribution, and identify examples where the internet-based technologies are practical and advantageous.

The impact to NOAA's Wallops Command and Data Acquisition Station (WCDAS) of the additional capabilities of the GOES-R satellite series is explored. A short history of the WCDAS is presented along with changes and upgrades to the communications and computing systems seen as needed to support this new series of satellites. Also a comparison is made between existing services provided by WCDAS via the current GOES satellites and the future performance enhancements expected to be provided via GOES-R. For some parameters the capabilities of the GOES-R satellite are not currently known in sufficient detail to provide a sound basis for an upgrade path. In such cases, an outline of the expected limits is presented. The views presented here are those of the authors with significant years of experience in this field. They do not necessarily reflect the views of the NOAA GOES-R program Office and may not reflect the final GOES-R system architecture developed by the GOES-R contractor.

The Comprehensive Large Array-data Stewardship System (CLASS), a National Oceanic and Atmospheric Administration (NOAA) IT enterprise solution supporting NOAA's data archive and science data stewardship missions by providing the IT portion of an archive for environmental data. CLASS requirements are defined by NOAA's Archive Requirements Working Group (ARWG), which serves as a clearinghouse for requirement planning and ensures that science and other user requirements are clearly defined with respect to NOAA's archive. Currently, CLASS is part of a major 10-year growth program to add new data sets and functionality to support a broader user base. NOAA needs to define the types of data to be archived, metadata managing standards, and the data search, display, and delivery services that CLASS will provide to users. These requirements are captured in documents that include MOAs, ICDs and OAIS-compliant Data Submission Agreements that drive the software and hardware architecture changes needed to handle the expected future increases in user and data volumes. The paper will present CLASS' approach for other major data campaigns such as Metop, NPP, NPOESS and EOS, historical data and its plans for going forward with the GOES-R data campaign.

As new instruments are developed, it is becoming clear that our ability to generate data is rapidly outstripping our ability to transmit this data. The Advanced Baseline Imager (ABI), that is currently being developed as the future imager on the Geostationary Environmental Satellite (GOES-R) series, will offer more spectral bands, higher spatial resolution, and faster imaging than the current GOES imager. As a result of the instrument development, enormous amounts of data must be transmitted from the platform to the ground, redistributed globally through band-limited channels, as well as archived. This makes efficient compression critical. According to Shannon's Noiseless Coding Theorem, an a upper bound on the compression ratio can be computed by estimating the entropy of the data. Since the data is essentially a stream, we must determine a partition of the data into samples that capture the important correlations. We use a spatial window partition so that as the window size is increased the estimated entropy stabilizes. As part of our analysis we show that we can estimate the entropy despite the high-dimensionality of the data. We achieve this by using nearest neighbor based estimates. We complement these a posteriori estimates with a priori estimates based on an analysis of sensor noise. Using this noise analysis we propose an upper bound on the compression achievable. We apply our analysis to an ABI proxy in order estimate bounds for compression on the upcoming GOES-R imager.

A low-complexity, adaptive predictive technique for lossless compression of hyperspectral imagery is described. This technique is designed to be suitable for implementation in hardware such as a field programmable gate array (FPGA); such an implementation could be used for high-speed compression of hyperspectral imagery onboard a spacecraft. The predictive step of the technique makes use of the sign algorithm, which is a relative of the least mean square (LMS) algorithm from the field of low-complexity adaptive filtering. The compressed data stream consists of prediction residuals encoded using a method similar to that of the JPEG-LS lossless image compression standard. Compression results are presented for several datasets including some raw Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) datasets and raw Atmospheric Infrared Sounder (AIRS) datasets. The compression effectiveness obtained with the technique is competitive with that of the best of previously described techniques with similar complexity.

Inspired by previous work on the modelling of wavelet coefficients, and on the observed differences between distributions of wavelet coefficients belonging to different landscapes, we present a lossless compressor of multi-spectral images based on the prediction of wavelet coefficients, conditioned to the landscape. This compressor operates blockwise. The wavelet transform is applied to each block, and detail coefficients from the two finest scales are predicted by means of a linear combination of other coefficients, which may belong to the same band as the predicted coefficient, or to a previously coded band. The weights for the lineal combination are estimated on-line: for each detail subband, the compressor is trained on all the detail coefficients belonging to the same class. In addition, a different band ordering is considered for each block. Differences in prediction are coded with a conditional entropy coder. Preliminary results reveal that we obtain more accurate predictions.

MODIS (MODerate-resolution Imaging Spectroradiometer) and other satellite data have been staged (http://cimss.ssec.wisc.edu/goes/abi/bitdepthcompression/) for use in ABI (Advanced Baseline Imager) data compression studies. The 16-channel ABI is the next generation imager on the GOES-R (Geostationary Operational Environmental Satellite) series. Most ABI bands can be simulated in this manner from MODIS observations. The advantage of using actual satellite observations is that the small-scale features are more realistic than those simulated from numerical models. High spatial resolution MODIS data have been spatially and radiometrically reformatted and posted for community use. This includes visible (VIS), near-infrared (IR) and IR bands. These images include weather/environmental phenomena, such as: fire and smoke, mountain waves, dust storms and clouds. There are several steps in these ABI simulations: select the original MODIS images for these various cases, select bands with similar central wavenumbers, de-stripe the IR bands, average to the ABI spatial resolution, subset over the area of interest, and correct for planned image bit depth. Sample METEOSAT-8 SEVIRI (Spinning Enhanced Visible and Infrared Imager) data, from EUMETSAT (EUropean organization for the exploitation of METeorological SATellites), have also been posted. There are 11 spectral bands for 3 sequential times for those interested in compression of full disk images. Finally, unaltered GOES-12 Imager Full Disk images have been staged. There are five spectral bands for both a "night" and "day" case. There has been no post-processing on these non-MODIS images. Each of these datasets has advantages and disadvantages in so far as they represent what will be obtained from the ABI.