Using the IMI Clustering Options
Why use the clustering options?
The main computational cost of the IMI is running the perturbation simulations necessary to construct the jacobian. This requires running a (GEOS-Chem) Jacobian simulation for each state vector element. The default state vector that is generated with the IMI has state vector elements in native resolution, meaning each element corresponds with a GEOS-Chem grid cell (.25 degree or .5 degree resolution). However, if your state vector has a sufficiently large number of elements this can limit the feasibility of running the IMI – either due to prohibitively high AWS costs or compute time. Clustering your state vector elements reduces the number of state vector elements by aggregating elements together.
Using the IMI clustering config options
To enable the IMI clustering options in the imi config file set
ReducedDimensionStateVector: true
. This enables the clustering component of the IMI.
Once enabled the IMI uses your specified NumberOfElements
to aggregate native resolution state vector elements
within your domain of interest using the specified ClusteringMethod
. Additionally, there are two optional
clustering options MaxClusterSize
and ``ClusteringThreshold``which, respectively, control the maximum number
of elements per cluster and the size distribution of clusters. eg:
ReducedDimensionStateVector: true
ClusteringMethod: "kmeans"
NumberOfElements: 39
MaxClusterSize: 6 # Optional
ClusteringThreshold: 0.4 # Optional
This automatically generates a state vector with 39 elements (including buffer elements) in the
domain of interest. This is done by attempting to ensure all clusters have information content
(estimated_DOFS) above a set ClusteringThreshold
. If no threshold is set, the algorithm
will use the default threshold of total_estimated_DOFs / NumberOfElements
. Increasing the
ClusteringThreshold
allows reduction of the size distribution of state vector elements. If the
user specifies a MaxClusterSize
the algorithm will attempt to ensure that the approximate maximum
cluster size is approximately the specified value. Note: due to the inner workings of kmeans the
actual MaxClusterSize may be slightly higher or lower than the specified value. If the user does not
specify a MaxClusterSize
the algorithm uses 64
as the default value.
The information content of a cluster is generated by aggregating the estimated averaging kernel sensitivity
values for the native resolution grid until a cluster reaches the provided ClusteringThreshold
.
The algorithm starts with the smallest clusters possible and iteratively aggregates clusters until they
reach the threshold.
The ClusteringMethod
specifies which clustering method to use for state vector reduction. Currently
kmeans
or mini-batch-kmeans
are valid options. mini-batch-kmeans
is very similar to kmeans
,
but can be less accurate. It is best used for very large state vectors to speed up state vector reduction.
Note: The IMI preserves the original state vector file as NativeStateVector.nc in your run directory.
Incorporating point source information
If you have prior information of specific locations that you would like to maintain high resolution
(eg. point source detections) you can ensure the clustering algorithm preserves these locations by
using the ForcedNativeResolutionElements
config variable. This variable takes a list of lat/lon
locations using either yaml list or a path to a csv file.
For instance, if the user suspects a location to be an emission hotspot they can specify the
lat/lon coordinates as in the examples below and the clustering algorithm will ensure that the
native resolution element is preserved during the aggregation. In order for the IMI to
preserve the element, you must have enough NumberOfElements
specified to accomodate the
number of gridcells you would like to force to be native resolution.
Additionally, the PointSourceDatasets
config variable can be used to automatically scrape emission
hotspots from external point source datasets. Currently, the only supported dataset is the "SRON"
weekly plumes dataset.
yaml list example:
PointSourceDatasets: ["SRON"]
ForcedNativeResolutionElements:
- [31.5, -104]
- [32.5, -103.5]
csv file example:
PointSourceDatasets: ["SRON"]
ForcedNativeResolutionElements: "/path/to/point_source_locations.csv"
The csv file should have a header row with the column names lat
and lon
using lowercase letters.
The csv file can have additional columns, but they will be ignored.
Dynamic Kalman Filter clustering
When running the IMI in Kalman Filter mode, users can dynamically adjust clusters at each Kalman iteration
to best reflect the available information content by setting the DynamicKFClustering
variable to
true
. See the Kalman Filter IMI documentation for more details.
IMI clustering scheme
The IMI clustering algorithm uses a similar k-means based method as described in Nesser et al., 2021 to maintain native resolution in areas with high information content (high prior emissions, high observation density), while aggregating cells with low information content.
Reducing computational cost while maintaining inversion quality
While clustering is an effective method for alleviating computational constraints for
running inversions at high resolution for large regions, it can introduce aggregation error
and degrade the quality of your inversion
(Turner and Jacob., 2014 ).
Therefore, it is important to weigh the computational benefits of reducing your state vector
against the inversion quality loss. This can be done by iteratively tuning the NumberOfElements
and running the IMI preview.IMI preview to assess
the estimated DOFS. Ideally, you should find a middle groud where the estimated DOFS and
computation cost is at a acceptable level before proceeding with the inversion.