1. IntroductionΒΆ

The proposed CF aggregation rules allow for the aggregation of CF fields across multiple dimensions. These rules are based solely on the fields’ metadata, therefore allowing aggregation to be reliably automated. This raises the possibility of large amounts of aggregations being created and therefore storing these collections is clearly desirable.

This document describes how the data arrays of such aggregations could be stored in the memory of an application and proposes a convention – the NCA (netCDF aggregate) convention – for their efficient file storage.

These ideas have been implemented in cf-python. In particular, its Large Amounts of Massive Arrays (LAMA) functionality stores arrays in this fashion.

The key element of the NCA files is that they are CF-compliant netCDF files, albeit ones which require extra processing to realise their aggregated data arrays.

This style of aggregation is distinct from NetCDF Markup Language (NcML) aggregation, which has long-standing use in the community, in that it allows simultaneous aggregation across arbitrarily positioned dimensions, can aggregate arbitrary parts of arrays and can aggregate arrays stored in arbitrary formats (in-memory, netCDF, PP, etc.).

Previous topic

On the storage of arbitrarily aggregated data arrays (v0.2)

Next topic

2. A conceptual framework for the in-memory storage of aggregated arrays

This Page