Using the IMI Clustering Options

Why use the clustering options?

The main computational cost of the IMI is running the perturbation simulations necessary to construct the jacobian. This requires running a (GEOS-Chem) Jacobian simulation for each state vector element. The default state vector that is generated with the IMI has state vector elements in native resolution, meaning each element corresponds with a GEOS-Chem grid cell (.25 degree or .5 degree resolution). However, if your state vector has a sufficiently large number of elements this can limit the feasibility of running the IMI – either due to prohibitively high AWS costs or compute time. Clustering your state vector elements reduces the number of state vector elements by aggregating elements together.

Using the IMI clustering config options

To enable the IMI clustering options in the imi config file set ReducedDimensionStateVector: true. This enables the clustering component of the IMI. Once enabled the IMI uses your specified NumberOfElements to aggregate native resolution state vector elements within your domain of interest using the specified ClusteringMethod. Additionally, there are two optional clustering options MaxClusterSize and ``ClusteringThreshold``which, respectively, control the maximum number of elements per cluster and the size distribution of clusters. eg:

ReducedDimensionStateVector: true
ClusteringMethod: "kmeans"
NumberOfElements: 39
MaxClusterSize: 6 # Optional
ClusteringThreshold: 0.4 # Optional

This automatically generates a state vector with 39 elements (including buffer elements) in the domain of interest. This is done by attempting to ensure all clusters have information content (estimated_DOFS) above a set ClusteringThreshold. If no threshold is set, the algorithm will use the default threshold of total_estimated_DOFs / NumberOfElements. Increasing the ClusteringThreshold allows reduction of the size distribution of state vector elements. If the user specifies a MaxClusterSize the algorithm will attempt to ensure that the approximate maximum cluster size is approximately the specified value. Note: due to the inner workings of kmeans the actual MaxClusterSize may be slightly higher or lower than the specified value. If the user does not specify a MaxClusterSize the algorithm uses 64 as the default value.

The information content of a cluster is generated by aggregating the estimated averaging kernel sensitivity values for the native resolution grid until a cluster reaches the provided ClusteringThreshold. The algorithm starts with the smallest clusters possible and iteratively aggregates clusters until they reach the threshold.

The ClusteringMethod specifies which clustering method to use for state vector reduction. Currently kmeans or mini-batch-kmeans are valid options. mini-batch-kmeans is very similar to kmeans, but can be less accurate. It is best used for very large state vectors to speed up state vector reduction.

Note: The IMI preserves the original state vector file as NativeStateVector.nc in your run directory.

Incorporating point source information

If you have prior information of specific locations that you would like to maintain high resolution (eg. point source detections) you can ensure the clustering algorithm preserves these locations by using the ForcedNativeResolutionElements config variable. This variable takes a list of lat/lon locations using either yaml list or a path to a csv file.

For instance, if the user suspects a location to be an emission hotspot they can specify the lat/lon coordinates as in the examples below and the clustering algorithm will ensure that the native resolution element is preserved during the aggregation. In order for the IMI to preserve the element, you must have enough NumberOfElements specified to accomodate the number of gridcells you would like to force to be native resolution.

Additionally, the PointSourceDatasets config variable can be used to automatically scrape emission hotspots from external point source datasets. Currently, the only supported dataset is the "SRON" weekly plumes dataset.

yaml list example:

PointSourceDatasets: ["SRON"]
ForcedNativeResolutionElements:
  - [31.5, -104]
  - [32.5, -103.5]

csv file example:

PointSourceDatasets: ["SRON"]
ForcedNativeResolutionElements: "/path/to/point_source_locations.csv"

The csv file should have a header row with the column names lat and lon using lowercase letters. The csv file can have additional columns, but they will be ignored.

Dynamic Kalman Filter clustering

When running the IMI in Kalman Filter mode, users can dynamically adjust clusters at each Kalman iteration to best reflect the available information content by setting the DynamicKFClustering variable to true. See the Kalman Filter IMI documentation for more details.

IMI clustering scheme

The IMI clustering algorithm uses a similar k-means based method as described in Nesser et al., 2021 to maintain native resolution in areas with high information content (high prior emissions, high observation density), while aggregating cells with low information content.

Reducing computational cost while maintaining inversion quality

While clustering is an effective method for alleviating computational constraints for running inversions at high resolution for large regions, it can introduce aggregation error and degrade the quality of your inversion (Turner and Jacob., 2014 ). Therefore, it is important to weigh the computational benefits of reducing your state vector against the inversion quality loss. This can be done by iteratively tuning the NumberOfElements and running the IMI preview.IMI preview to assess the estimated DOFS. Ideally, you should find a middle groud where the estimated DOFS and computation cost is at a acceptable level before proceeding with the inversion.