LXX
Reading time: 6 minutes
Quick read: 2 minutes
Introduction:
When utilizing ATL03 data from ICESat-2, processes such as noise reduction and classification are generally required, and the quality of the final results largely depends on the effectiveness of the processing algorithms. In fact, the ATL08 data product from ICESat-2 has already undergone further signal photon classification and noise filtering through official algorithms applied to the preliminarily noise-removed ATL03 data, which simplifies the cumbersome processing of ATL03 data to some extent. However, despite the ATL08 data product providing category and height information for each photon, it does not offer corresponding location information for the photon points, which limits its practical applications.
Therefore, this article shares the implementation of ATL03 and ATL08 data association using Python to obtain the corresponding location information for each photon, supporting subsequent research on forest height inversion, biomass estimation, and more. (This is based on previously published code, with some errors fixed.)
1
Parsing ATL03 and ATL08 Data
Supplement:
When the laser beam emitted by the ATLAS sensor reaches the ground, the photons containing information about the ground objects are reflected back to the laser receiver. The number of photons depends on the emitted energy, solar and atmospheric conditions, and surface reflectivity. Since the typical surface reflectivity is about 0.3, which is much greater than the reflectivity of vegetation at 0.1, the number of ground photons will significantly exceed that of vegetation photons.
ICESat-2/ATLAS provides single-photon laser data, including 4 levels, totaling 21 types. Among them, there are two data products related to the ground DEM: ATL03 and ATL08. ATL03 is a Level 2 product, providing global geolocated photon data, which includes the longitude, latitude, measurement time, and absolute elevation relative to the WGS84 (Geodetic System 1984) ellipsoid for each photon; ATL08 is a Level 3 product, providing land and vegetation height data, which includes classification information for each photon. The photon information provided by the Level 2 product will be used for the production of all Level 3 products, such as ATL06 land ice elevation product, ATL07 sea ice elevation product, and ATL08 land and vegetation height product. The data is stored in HDF format, with three pairs of six-beam data, forming six ground track information (Ground Tracks, GT) on the ground, arranged from left to right and labeled as GT1L-GT3R.
1.1
ATL03
ATL03 mainly provides global geolocated photon point cloud data, with each ATL03 data including 6 strips: gt1l, gt1r, gt2l, gt2r, gt3l, and gt3r. The data for each strip is segmented along the track at approximately 20 m intervals, recording the longitude, latitude, and elevation information for each segmented photon point, while also providing information such as along-track distance, cross-track distance, segment length, and the number of photons. Additionally, ATL03 data utilizes noise reduction algorithms to classify the original photon points as noise points, low-confidence signal points, medium-confidence signal points, and high-confidence signal points.
1.2
ATL08
ATL08 is a land vegetation height product, obtained by further noise reduction of the ATL03 product to acquire signal points, which are then classified into ground photon points, canopy top photon points, and canopy photon points, extracting ground elevation and vegetation height information corresponding to every 100 m segment along the track. The ground elevation information includes best-fit ground elevation, maximum ground elevation, average ground elevation, median ground elevation, and minimum ground elevation. Forest height includes maximum forest height, average forest height, median forest height, minimum forest height, and forest height percentiles (25%, 50%, 60%, 70%, 75%, 80%, 85%, 90%, 95%, 98% forest height). Additionally, the ATL08 product provides signal-to-noise ratio, slope, cloud cover, ground elevation uncertainty, and forest height uncertainty information corresponding to every 100 m segment.
ATL03 segments are divided every 20m along the track, while ATL08 segments are divided every 100m along the track. The 100m segment is composed of a total of 5 segments of 20m, with the 20m segments connected by segment id in both ATL03 and ATL08 data.
ATL08 data performs further signal photon classification and noise filtering on the preliminarily noise-removed ATL03 data, with main operational steps including ground curve fitting and vegetation surface curve fitting. Photons used for ground curve fitting are labeled as ground photons (Ground), photons used for vegetation surface curve fitting are labeled as canopy top photons (Top of canopy), photons between the two curves are labeled as canopy (Canopy), and the remaining photons are labeled as noise (Noise). The following figure illustrates the matching of ATL03 and ATL08 data.
Illustration of ATL03 and ATL08 Data Matching
2
Associating ATL03 and ATL08 Data
The ATL08 data product provides photon elevation information and photon classification information; the ATL03 data product provides photon elevation information and photon location information. Although the ATL08 data product provides category and height information for each photon, it does not provide corresponding location information. Therefore, it is necessary to calibrate the ATL08 data product with classification labels onto the ATL03 data product to obtain an ATL03 data product that includes classification information, height information, and location information.
2.1
Rules for Associating ATL03 and ATL08 Data
1. ATL03 Data Product, each photon has two indexing methods.
Photon sequence number, numbered sequentially according to the GPS time of the photons, referred to as photon sequence number (Group:/gtx/heights); photon segment number, divided into segments every 20m along the track, with each segment numbered sequentially (Group:/geolocation/segment_id), and the starting photon sequence number for each segment recorded (Group:/geolocation/ph_index_beg).
2. ATL08 Data Product, each photon has one indexing method.
Photon segment number, divided into segments every 100m along the track, with each segment numbered sequentially (Group:/gtx/signal_photons/ph_segment_id), and the relative sequence number for each photon in each segment recorded (Group:/gtx/signal_photons/classed_pc_indx).
The parameters of ICESat-2/ATLAS data are shown in the figure below.
3. Traverse each photon in the ATL08 data product to associate ATL08 with ATL03 data product.
First, match the photon segment number (ph_segment_id=segment_id) to find the same segments in both datasets; then match each photon one by one, where the photon sequence number in ATL03 data equals the starting photon sequence number in ATL03 data plus the relative photon sequence number in ATL08 data (classed_pc_indx+ph_index_beg-1). This allows each photon in ATL08 to be matched to the ATL03 data product, as illustrated in the following figure.
Rules for Associating ATL03 and ATL08 Data
2.2
Implementation of ATL03 and ATL08 Data Association Code
(Below is the key code; for the complete code, see the end of the article)
(1) First, import the required libraries
import h5py
import numpy as np
from tqdm import tqdm
import time
import pandas as pd
import matplotlib.pyplot as plt
from tqdm import tqdm(2) Create the ATLDataLoader class to load ATL03 and ATL08 data and perform association; and initialize the method
class ATLDataLoader: def __init__(self, atl03Path, atl08Path, gtx): # Initialization self.atl03Path = atl03Path self.atl08Path = atl08Path self.gtx = gtx self.load()(3) Load ATL03 and ATL08 data, extract associated information based on the above association rules
def load(self):
'''
Steps:
1) Open and read the ATL03 file, extracting longitude, latitude, segment starting index, segment ID, height, and signal confidence data;
2) Open and read the ATL08 file, extracting photon classification index, classification flag, segment ID, and height data;
3) Use the ismember method to match the segments of ATL03 and ATL08 to determine the correspondence between the same segments;
4) Based on the matching results, determine the new mapping relationship, and associate the classification and height information of ATL08 with ATL03 based on the mapping relationship;
5) Create a DataFrame to store the processed data.
'''
# Read ATL03 segment data f = h5py.File(self.atl03Path, 'r') atl03_lat = np.array(f[self.gtx + '/heights/lat_ph'][:]) atl03_lon = np.array(f[self.gtx + '/heights/lon_ph'][:]) atl03_ph_index_beg = np.array(f[self.gtx + '/geolocation/ph_index_beg'][:]) atl03_segment_id = np.array(f[self.gtx + '/geolocation/segment_id'][:]) atl03_heights = np.array(f[self.gtx + '/heights/h_ph'][:]) atl03_conf = np.array(f[self.gtx + '/heights/signal_conf_ph'][:]) f.close()
# Read ATL08 segment data f = h5py.File(self.atl08Path, 'r') atl08_classed_pc_indx = np.array(f[self.gtx + '/signal_photons/classed_pc_indx'][:]) atl08_classed_pc_flag = np.array(f[self.gtx + '/signal_photons/classed_pc_flag'][:]) atl08_segment_id = np.array(f[self.gtx + '/signal_photons/ph_segment_id'][:]) atl08_ph_h = np.array(f[self.gtx + '/signal_photons/ph_h'][:]) f.close()
# Match the photon segment number (ph_segment_id=segment_id) to find the same segments in both datasets atl03SegsIn08TF, atl03SegsIn08Inds = self.ismember(atl08_segment_id, atl03_segment_id)
# Get the indices and values of ATL08 classification atl08classed_inds = atl08_classed_pc_indx[atl03SegsIn08TF] atl08classed_vals = atl08_classed_pc_flag[atl03SegsIn08TF] atl08_hrel = atl08_ph_h[atl03SegsIn08TF]
......
# Add ATL08 classification information allph_classed[newMapping] = atl08classed_vals allph_hrel[newMapping] = atl08_hrel(4) Open the storage locations for ATL03 and ATL08 data, run the code, and save the associated data to CSV.
atl03Path = "D:/other/icesat/text/0308/03/processed_ATL03_20191225224448_13750506_006_01.h5" atl08Path = "D:/other/icesat/text/0308/08/processed_ATL08_20191225224448_13750506_006_01.h5" gtx = "gt1l" loader = ATLDataLoader(atl03Path, atl08Path, gtx) # Save results to CSV loader.save_to_csv_with_progress("output.csv")The above code only processes the association for one track for testing; to process multiple strips, the above code can be modified as follows:
# Batch processing: Loop through six laser beams
if __name__ == "__main__": import os # Test file paths atl03Path = "D:/other/icesat/text/0308/03/processed_ATL03_20191225224448_13750506_006_01.h5" atl08Path = "D:/other/icesat/text/0308/08/processed_ATL08_20191225224448_13750506_006_01.h5" # Loop through six laser beams for beam in ['gt1l', 'gt1r', 'gt2l', 'gt2r', 'gt3l', 'gt3r']: print("Processing beam:", beam) # Create ATLDataLoader object loader = ATLDataLoader(atl03Path, atl08Path, beam) # Save results to CSV file csvFile = atl03Path.replace(".h5", '_' + beam + ".csv").replace(".hdf", '_' + beam + ".csv") loader.df.to_csv(csvFile, index=False) print("Results saved to:", csvFile)(5) Visualize the associated CSV results.
# Plotting # Color reference from ATL08 official colors class_dict = { -1: {'color': (194/255,197/255,204/255), 'name': 'Unclassified'}, 0: {'color': (194/255,197/255,204/255), 'name': 'Noise'}, 1: {'color': (210/255,184/255,38/255), 'name': 'Ground'}, 2: {'color': (69/255,128/255,26/255), 'name': 'Canopy'}, 3: {'color': (133/255, 243/255, 52/255), 'name': 'Top of canopy'} } csv_path = 'output.csv' visualizer = DataVisualizer(csv_path, class_dict) visualizer.plot()The color selection refers to the official ICESat-2 color scheme; if other colors are needed, simply modify the RGB values.
3
Results
gt1l Association Results
gt2l Association Results
gt3l Association Results
4
References
[1] Zhu Xiaoxiao. Research on Forest Height Inversion Based on ICESat-2 and GEDI Data with 30m Resolution in China [J]. [2024-04-12].
[2] Fu Ying. Joint Satellite-Based Single Photon Laser and Remote Sensing Image for Regional Forest Canopy Height Inversion.
end
Apologies for any delays in responses during backend communications due to regular work, but I will definitely reply!
Please be polite; I am not obligated to do anything for you…
Finally, I wish everyone happiness every day!
