The 1st TUNER Meeting, June 15, Univ. of Saskatchewan, Saskatoon}

The first meeting of the emerging SPARC activity "Toward Unified Error Reporting (TUNER)" was held at the University of Saskatchewan, Saskatoon, CA, on June 15 2017. The aim of this project is to harmonize the reporting of uncertainties of satellite data of atmospheric temperature and composition.

In order to get an inventory of the retrieval methods and error estimation schemes used in the satellite community, a questionnaire had been distributed. The responses were presented by Thomas von Clarmann and discussed in Saskatoon. Responses were obtained for 12 limb and 1 nadir mission. Limb missions include limb emission, limb scattering, and occultation. Measurements in the following frequency ranges were represented: MW, FIR, IR, NIR, vis, UV. All retrievals are based on a matrix formalism with or without regularization, the latter being either optimal estimation or Tikhonov-type. Some groups provide their data on the native retrieval grid, while others interpolate their data to a regular grid after the retrieval. In the latter case care has to be taken to also transform the diagnostic data onto the new grid. Good agreement was found with respect to the schemes how noise is propagated onto the results but the estimation of parameter errors needs much more discussion. Since parameter errors depend largely on the instrument and the retrieval strategy chosen, harmonization of related error reporting is not a trivial task. All participating groups seem to be well aware of possible forward model errors which might affect their results but quantification of these is often difficult. Some groups prefer to provide total error estimates to the data users while others find it more adequate to provide information on the error components and to leave their combination to the data users. Averaging kernels are provided by all groups who perform constrained retrievals. No final agreement has been reached about the altitude resolution of non-constrained retrievals. Validation papers are available for most of the participating instruments. Within TUNER no validation studies will be made, but it will be heavily drawn upon existing validation studies. These are considered particularly useful to judge which error estimation schemes are adequate. In order not to duplicate work ongoing in other projects, it was decided not to assess instrument drifts within TUNER.

The next point on the agenda was deductive error analysis, which is understood to be the  propagation of ingoing uncertainties through the retrieval system. Several talks were given.
Natalya Kramarova and Paatrick Sheese presented results of error estimation work for OMPS and ACE-FTS, respectively. Both these studies included detailed analyses of the leading error sources. Thomas von Clarmann discussed the problem that covariances between the atmospheric state and averaging kernels can turn the application of mean averaging kernels to mean profile inaccurate and suggested to use a mean covariance term for correction. Stefan Bender reviewed machine learning methods and raised the question if these, due to their mathematical structure similar to that of retrieval and error estimation, might be useful within TUNER.

Under the header of inductive error analysis, which is understood to be error analysis based on the observations, and which is thus closely related to validation, two presentations were given. The first, by Arne Babenhauserheide, Quentin Errera and Thomas von Clarmann, presented by the latter, tackled the problem of natural variability. This, along with less than perfect collocations of measurements seems often to be used as a "universal excuse" whenever discrepancies between two data sets are encountered in validation studies. High-resolved temperature and mixing ratio fields calculated with model "BASCOE" are used to statistically quantify the effect of spatial and temporal mismatch between observations. In a following presentation Thomas von Clarmann showed how by comparison of three or more datasets their precision estimates can be assessed such that it becomes clear which instrument group over- or underestimates their random uncertainties. 

A problem has been identified with respect to the question how user-driven TUNER shall be. On the one hand, the data users shall be provided with the error estimates and other diagnostic data they need. On the other hand, data often do not know how relevant certain diagnostics (e.g. averaging kernels or error covariances) are and would thus never request them. The following solution has been identified. Instead of asking the data users which diagnostics they want, they shall be asked which application of the satelite data they have in mind. The data providers will then decide which diagnostics will be necessary.

Finally, it has been conveyed that TUNER has been selected as an International Team by the International Space Science Intitute in Berne, where two project meetings will be held, and it was decided to propose a special issue on TUNER to the journal Atmospheric Measurement Techniques".