Skyborn GridFill: How Do We Recover Missing Data?

You Must Have Encountered This Problem

Imagine you are looking at a satellite cloud image and find that a certain area is obscured by thick clouds, making it impossible to see the ground temperature; or your thermometer has malfunctioned for a while, resulting in blank data records. What should we do about these “missing” data?

This is the problem that Skyborn’s GridFill aims to solve—how to intelligently “guess” what the missing data should look like.

The “Magic” Principle of GridFill

The Essence of the Problem: A Complex Fill-in-the-Blank Question

GridFill is essentially facing a “fill-in-the-blank” question. Just like when you solve a math problem, if you know the values of a function at certain points, can you deduce the values at other points?

The answer from GridFill is: Yes, and there are mathematical laws to guarantee it!

The Core Idea: The “Laziness” Principle in Physics

In physics, nature always tends to choose the “least effort” state. For example:

The surface of water always tends to be calm
Heat always flows from high temperature to low temperature
The electric field is always the most “smooth” where there are no charges

GridFill utilizes this principle: Assume that the data distribution is the most “smooth” in areas where there is no data.

Mathematical Formula: Looks Complex, But It’s Quite Simple

The Core Equation: Poisson’s Equation

The core of GridFill is to solve this equation:

Don’t be intimidated by this formula! In layman’s terms, this equation means:

❝

In the areas with missing data, we need to find a “most smooth” solution, where the “curvature” in the east-west and north-south directions adds up to zero.

Understanding Through Metaphor

Imagine you have a rubber membrane, with its edges fixed at known data points. When you let this membrane relax to its most smooth state, the shape of the membrane is the result we are looking for.

Rubber membrane = The data field we want to fill
Fixed edges = Known observational data
Most smooth state = The solution to Poisson’s equation

Algorithm Implementation: Step by Step Approaching the Truth

Iterative Method: Like Polishing a Piece of Art

GridFill does not calculate the answer all at once; instead, it improves gradually like a sculptor refining a piece:

This formula means:

Check how “unsmooth” the current guess is (residual)
Make a small adjustment towards a smoother direction (relax)
Repeat this process until it is sufficiently smooth (iteration)

Residual: Measuring the Degree of “Unsmoothness”

This formula checks the relationship between a point and its four neighboring points:

If this point is exactly the average of the four neighbors, residual = 0 (very smooth)
If it is far off, the residual is large (not smooth, needs adjustment)

Convergence Condition: When to Stop?

When the “unsmoothness” of all points is less than the set standard ε, we consider the answer to be good enough.

Relationship with NCL: Algorithm Twins

It is worth mentioning that the algorithm used in Skyborn’s GridFill is the same as the NCL (NCAR Command Language)poisson_grid_fill function. This means:

Algorithm Consistency: Both are based on the same finite difference method for solving Poisson’s equation
Reproducible Results: Under the same parameters, the results of GridFill and NCL are theoretically identical
Compatibility Assurance: NCL users can seamlessly transition to the Python environment
Standardized Implementation: Follows widely recognized algorithm standards in the meteorological community

The differences are:

Programming Language: GridFill is based on Python/Cython
Interface Design: GridFill provides a modern xarray interface
New Features: GridFill adds enhanced features such as linear zonal initialization
Integrated Ecosystem: Better integration into the Python scientific computing ecosystem

Practical Applications: The Power of GridFill in Reality

1. The “Cloud Killer” for Satellite Data

Problem: When satellites observe the Earth, clouds obscure large areas of the ocean.

GridFill’s Solution:

# Original data: Sea surface temperature with cloud cover
sst_with_clouds = load_satellite_data()

# GridFill magic: Fill in the cloud-covered areas
sst_filled = fill(sst_with_clouds, eps=1e-4)

# Result: A complete sea surface temperature map!

2. “Deep Sea Exploration” in Oceanography

Problem: The ocean is vast, and there are few measuring ships, leading to some areas being uncovered and resulting in missing observational data.

GridFill’s Solution: Utilize physical laws to infer a complete ocean picture from sparse observation points.

Input Parameters

eps (Precision): The Degree of Perfection Pursued

1e-3: Fast but rough, suitable for preliminary analysis
1e-4: Balanced point, the choice for most situations
1e-6: Striving for perfection, increased computational demand

relax (Relaxation Factor): The Size of Improvement Steps

Too large (>0.7): Steps are too large, may “overshoot” the optimal solution
Too small (<0.4): Slow convergence, like a snail crawling
Just right (0.5-0.6): The best choice for steady progress

initzonal (Zonal Initialization): Giving the Algorithm a Starting Point

Instead of guessing from 0, first calculate the average value for each latitude as a starting point. It’s like filling in a “possibly correct” answer for uncertain questions in an exam.

initzonal_linear (Linear Zonal Initialization)

Traditional Method: Fill each latitude with the same average valueNew Method: Use linear interpolation to connect known points within each latitude

This is like painting, where instead of using the same color to fill everything, you use gradient colors for a more natural transition.

initial_value (Custom Initial Value)

Problem: Sometimes 0 is not a good starting pointSolution: Set a more reasonable initial guess

For example, sea surface temperatures typically range from 0-30°C, starting from 15°C is more reasonable than starting from 0°C.

Usage Recommendations: Avoid These Pitfalls

Related pitfalls can be found in the official documentation of Skyborn (https://skyborn.readthedocs.io/en/latest/).

When Does GridFill Perform Best?

Data has physical continuity: Physical quantities such as temperature, pressure, wind speed, etc.
Missing areas are not too large: Less than 20% of total data, and the area of missing data should be as small as possible
Boundary conditions are clear: There is enough observational data around

When Should Caution Be Exercised?

Data is highly discontinuous: For example, zonal winds (where there is a sudden drop in speed at the jet stream, see Skyborn’s official documentationhttps://skyborn.readthedocs.io/en/latest/).
Missing areas are huge: More than 50% of data is missing
No physical constraints: Purely random data

Usage Method

Installation of the Skyborn library is required

Environment Requirements: Python 3.9-3.13 (the latest version supportsPython 3.13)

Platform: Linux, Windows, MacOS (supports both Intel CPU and Apple Silicon chips)

pip install -U skyborn

pip install -U --index-url https://pypi.org/simple/ skyborn

# Pseudocode
import xarray as xr
import numpy as np
from skyborn.gridfill.xarray import fill, validate_grid_coverage

# 1. First check data quality
validation = validate_grid_coverage(data, min_coverage=0.2)

# 2. Choose appropriate parameters for filling
filled_data = fill(
    data, 
    eps=1e-4,                    # Convergence tolerance, balancing precision and speed
    relax=0.6,                   # Relaxation parameter, stable convergence speed  
    itermax=300,                 # Maximum number of iterations
    initzonal=True,              # Zonal initialization, effective for atmospheric data
    initzonal_linear=True,       # Linear zonal interpolation, better initialization
    cyclic=True,                 # Global data uses cyclic boundaries
    initial_value=0.0,           # Custom initial value
    verbose=True                 # Monitor the convergence process
)
    
# 3. Save the results
filled_data.to_netcdf('data_filled.nc')

For more information about Gridfill, please visit: Skyborn Documentation———https://skyborn.readthedocs.io/en/latest/

The Conclusion: The “Resurrection” of Data

GridFill is not magic, but it does have near-magical capabilities—bringing “missing” data back to “life”. It is based on rigorous mathematical theory and utilizes the principles of continuity in the physical world to reconstruct information lost due to technical limitations or natural barriers.

Let the missing data no longer be missing, and let the incomplete world become complete again—this is the mission of GridFill.

Skyborn Documentation

https://skyborn.readthedocs.io/en/latest/.