WS Test program results

Ross Bannister, September 2004

1. Introduction

In Var., background error covariances are represented as a set of control variable transforms. Any approach to modelling the transforms in this way leads to approximations which come to light in analyses of the implied covariances. A good background error covariance model should reproduce the position and scale dependencies of forecast errors. The background error covariance model used currently in Var. is unable to achieve this task well, but it is supposed that the scope of the model to capture both position and scale dependencies simultaneously is improved by using a waveband (or wavelet-like) approach, called the "waveband summation" (WS) method.

In [VWP19], a number of WS models are discussed, and are summarized at the start of section 6 in [Bannister]. In the rest of [Bannister], the formulation of one particular waveband model - the simplest - is studied in detail. This includes the transform definition, its pseudo-inverse (it has no exact inverse) and possible procedures for determining the many coefficients in the transforms, which can be estimated by running pairs of forecasts (or NMC runs).

In preparation for a publication of this work, this document presents results from a simple cut-down and off-line version of the WS transform run at Reading. Investigations have been performed into the ability of the WS transform to capture the position and scale dependencies of forecast errors and the accuracy of the pseudo-inverse transform. The results are compared to other representations of background errors - the exact approach (the explicit B-matrix for the cut-down system) and the current Met Office spatial transforms (sections 4.2 and 4.3 of [Bannister]).

The structure of this document is as follows. In section 2 we review the version of the WS transform used, and in section 3 we describe the experiments used to diagnose the properties of this model. Section 4 presents the results and section 5 describes them.

References
  • [VWP19] Andrews P.L.F., Lorenc A.C., On introducing a multi-scale waveband-summation covariance model into the Met Office Var system, Var. Working Paper 19 (2002).
  • [Bannister] Bannister R.N., On control variable transforms in the Met Office 3d and 4d Var., and a description of the proposed waveband summation transformation, gzipped postscript or pdf (2004).


2. The WS transform, the data and the methodology

In the following lists, equation, figure and section numbers refer to [Bannister], unless indicated otherwise.

The transform used
  • The WS U-transform used is shown as Eq. (72), but with no parameter transform. This transform is described in section 6.
  • Some multi-band transforms proposed in [VWP19] use multiple control vectors, the one used here has only one.
  • In the vertical transform used in WS, the modes have been rotated back to model levels (section 4.2.2).
  • Identity inner products have been used throughout for simplicity.
  • We use an undersized last bandpass function (section 6.2.2).
  • No spectral correction function (gamma in [VWP19]) has been used in the calibration.
  • We have the option to use triangular or top-hat band-pass functions and can control separately those used in the U and approximate T-transforms.
The data used
  • ECMWF ERA-40 forecast differences have been used to calibrate the transforms (ie to determine the vertical EOFs, their variances and the horizontal spectra).
  • ECMWF fields are at a high resolution (480 x 240 x 60). For our problem to be practical, this is reduced (80 x 40 x 30).
  • The ECMWF convention of labelling the vertical levels from the top of the model is reversed to match the MetO convention.
The methodology
  • We are not doing assimilation - but we are looking at diagnostics that show some of the covariances implied by the WS transform.
  • We also evaluate how invertible the WS transform is.
  • The problem is demanding on memory and so a three-dimensional problem of reasonable resolution cannot be dealt with on a workstation.
  • A two-dimensional version is implemented - latitude vs. height. This still allows study of the variation of errors with horizontal and vertical position and scale.
  • For the purposes of calibrating the statistics, averages are made over longitude instead of averages over realizations. Each longitude slice of a 3d field is considered to be a realization.
  • The statistics are calibrated for one forecast difference field (actually in January). We are comparing different methods of representing forecast errors and this is expected to be adequate for a first test (this same approach is used for each method).
  • There is no parameter transform - this study is univariate, looking only at spatial and scale dependencies of errors.
  • Simple fourier transforms in latitude are used to convert between horizontal position and horizontal scale.
  • For comparison, the same methodology is used to diagnose all of the transforms considered (the full explicit B-matrix representation, the current Met Office version and the WS transform).


3. The experiments



What representations of B are used?
  • An full, exact representation of B (called 'Exact'). This is actually the explicit B matrix.
  • The Met Office's current transforms (called 'MetO').
  • The WS transform described above (called 'WS').
What diagnostics are examined for each representation?
  • Implied variances of streamfunction and temperature as a function of latitude and model level.
  • Implied correlations of streamfunction and temperature with a given model level (around 500 hPa) as a function of horizontal wavenumber and model level.
There are many configurations of the WS transform. What aspects have been altered in the course of the experiments?
  • The number of bands can be changed.
    • We consider the 'optimal' case (where the typical lengthscale of each band is equal to the band width). For our system, the optimal case has four bands (see Fig. below).
    • Extreme cases of one band and ten bands.
  • The shape of the bands can be either triangular (Fig. 10) or top-hat (Fig. 11). We try out:
    • triangular band shapes in the T and U-transforms,
    • top-hat band shapes in the T and U-transforms, and
    • top-hat band shapes in the T-transform and and triangular band shapes in the U-transform.
    • Note: the T-transform is used during calibration and the U-transform is used for the implied variances/covariances diagnostics. Both are used for the inverse transform tests (below).
The triangular bands for the optimal case of four bands.

The WS transform has no exact inverse. What tests have been done to assess the validity of the approximate inverse (section 6.3)?
  • We operate systematically with UT and UT on delta functions to see if the delta-function is recovered.
  • We plot the resulting field of UT acting on a delta function at a particular latitude/height, and the resulting field of TU acting on a delta function at a particular wavenumber/height.
  • We plot effective submatrices of UT and TU (the complete matrix is too complicated to plot).
  • Note: the approximate inverse, T, has been designed to work as TU, not as UT.
Difficulties
  • During calibration, some of the vertical covariance matrices have very small eigenvalues. Numerical error has caused some of these to become negative. This creates a problem when they are square-rooted. The associated modes are ignored in the transforms, but they will contribute to the transforms being non-invertible.
List of experiments
ID B-model Quantity No. bands T bandshape U bandshape Ref.
-- ------- -------- --------- ----------- ----------- ----
A Exact Streamfn n/a n/a n/a Runs_for_doc1/Run1/
B Exact T n/a n/a n/a Runs_for_doc1/Run1T/
-- ------- -------- --------- ----------- ----------- ----
C MetO Streamfn n/a n/a n/a Runs_for_doc1/Run2/
D MetO T n/a n/a n/a Runs_for_doc1/Run2T/
-- ------- -------- --------- ----------- ----------- ----
E(*)WS Streamfn 4 Top-hat Triangular Runs_for_doc1/Run3/
F(*)WS T 4 Top-hat Triangular Runs_for_doc1/Run3T/
G WS Streamfn 4 Top-hat Top-hat Runs_for_doc1/Run6/
H WS T 4 Top-hat Top-hat Runs_for_doc1/Run6T/
I WS Streamfn 4 Triangular Triangular Runs_for_doc1/Run7/
J WS T 4 Triangular Triangular Runs_for_doc1/Run7T/
K WS Streamfn 1 Top-hat Triangular Runs_for_doc1/Run8/
L WS T 1 Top-hat Triangular Runs_for_doc1/Run8T/
M WS Streamfn 10 Top-hat Triangular Runs_for_doc1/Run9/
N WS T 10 Top-hat Triangular Runs_for_doc1/Run9T/

(*) The configuration expected to be 'best' is used for experiments E and F. The 'optimal' number of bands is four (J=3) using top-hat functions for calibration (T-transform) and triangular functions for the implied covariance diagnostics (U-transform).

4.1 The results

The following documents show the results. Each of the first two links below are divided into two. The upper parts show the exact results (experiments A and B) - these are what we are trying to reproduce with the WS scheme - and the lower parts show the remaining experiments. The results are discussed briefly below.

Implied diagnostics results for streamfunction.

Implied diagnostics results for temperature.


Inverse transform tests for streamfunction.

Inverse transform tests for temperature.



Discussion of the diagnostics for streamfunction

The exact diagnostics for streamfunction (Figs. A.1 and A.2) The 'exact' diagnostics show a strong variance of streamfunction of 0.225 units in the southern hemisphere centred around 50 degrees south and level 28 (Fig. A.1). Mirrored in the northern hemisphere is a much weaker variance structure, but is more spread out in latitude and joins the weak peak of variance close to the tropical tropopause. There is a lot of variance in the tropical and northern stratosphere. There is some variation of vertical scale with horizontal scale (Fig. A.2). Substantial correlations extend up to level 42 which peaks at a wavenumber roughly kmax/4. There is some negative correlations with the top and bottom of the model at roughly 2/3 kmax.

The MetO diagnostics for streamfunction (Figs. C.1 and C.2) The current MetO scheme does a reasonable job at reproducing these. The variances have peaks in the right places with the right strengths (Fig. C.1), but the horizontal scale variation of the vertical scale is too weak. This strong position dependence and weak, not non-zero, scale dependence is what is expected and is a known property of the MetO transforms. Note that the scale dependence is non-zero because the MetO scheme is not exactly separable.

The WS diagnostics for streamfunction, 4 bands, top-hat(T), triangular(U) (Figs. E.1 and E.2) The WS transform with four bands, which has been calibrated using top-hat band-pass functions, but used in the diagnostics with triangular functions does not perform well. The variances (Fig. E.1) peak in the right places but they are far too weak and are spread out. The scale variation (Fig. E.2) is still very weak. These are surprising results. It was expected that there would be some degradation of spatial scale with WS, but it was not expected to be so bad. It was expected that the scale dependence would be superior to the MetO scheme, but it is found to be comparable.

The WS diagnostics for streamfunction, 4 bands, top-hat(T), top-hat(U) (Figs. G.1 and G.2) Using the same number of bands, but top-hat band-pass functions throughout (both calibration and diagnostics stages) does not help (Figs. G.1 and G.2). The band boundaries are visible in places in Fig. G.2.

The WS diagnostics for streamfunction, 4 bands, triangular(T), triangular(U) (Figs. I.1 and I.2) Using the same number of bands, but triangular band-pass functions throughout (Figs. I.1 and I.2) gives worse results, especially for the variances.

The WS diagnostics for streamfunction, 1 bands, top-hat(T), triangular(U) (Figs. K.1 and K.2) We revert back to the use of top-hat functions when calibrating and triangular functions when computing the diagnostics, but we change the number of bands. Using just one band (Figs. K.1 and K.2) gives excellent results for the spatial variation of variances. These are almost identical to the exact results. There is virtually no variation of vertical scale with horizontal scale though in the vertical correlations. This is expected for this limit. Note that in this limit, the results are not expected to be the same as the MetO scheme as here we have rotated the vertical modes from a vertical 'EOF' index, as used at the MetO scheme, back to height. There is thus here no means of associating vertical and horizontal scales.

The WS diagnostics for streamfunction, 10 bands, top-hat(T), triangular(U) (Figs. M.1 and M.2) The case with many (ten) bands (Figs. M.1 and M.2) gives a very similar picture to that in the case with just four bands. This is surprising as it is expected to have further degraded position dependencies and improved scale dependencies.

Please see the note below about vertical eigenvalues of vertical covariances matrices for streamfunction.

Discussion of the diagnostics for temperature

The exact diagnostics for temperature (Figs. B.1 and B.2) The 'exact' diagnostics show a strong variation of variance of temperature with latitude (Fig. B.1). there are double-peaked structures in each hemisphere at around 50 degrees, but the variance in the northern hemisphere extends to the pole. The strongest variances are in the southern hemisphere. the lower peak has a maximum of 25 units at level 18 and the upper peak has a maximum of about 45 units at level 41. There is some variance in the stratosphere. There is a distinct variation of vertical correlation scale with horizontal scale (Fig. B.2). The zero correlation line varies from level 28 at 0 wavenumber to level 24 at kmax with strong negative correlations above this line.

The MetO diagnostics for temperature (Figs. D.1 and D.2) The current MetO scheme does a reasonable job at reproducing these. The variances (Fig. C.1) have peaks in the right places, including the double peaks, but the variances are too weak. The bands of positive and negative correlations (Fig. D.2) are present, but much of the variation of the correlation pattern is lost. In particular, the zero correlation line is flat.

HERE The WS diagnostics for streamfunction, 4 bands, top-hat(T), triangular(U) (Figs. E.1 and E.2) The WS transform with four bands, which has been calibrated using top-hat band-pass functions, but used in the diagnostics with triangular functions does not perform well. The variances (Fig. E.1) peak in the right places but they are far too weak and are spread out. The scale variation (Fig. E.2) is still very weak. These are surprising results. It was expected that there would be some degradation of spatial scale with WS, but it was not expected to be so bad. It was expected that the scale dependence would be superior to the MetO scheme, but it is found to be comparable.

The WS diagnostics for streamfunction, 4 bands, top-hat(T), top-hat(U) (Figs. G.1 and G.2) Using the same number of bands, but top-hat band-pass functions throughout (both calibration and diagnostics stages) does not help (Figs. G.1 and G.2). The band boundaries are visible in places in Fig. G.2.

The WS diagnostics for streamfunction, 4 bands, triangular(T), triangular(U) (Figs. I.1 and I.2) Using the same number of bands, but triangular band-pass functions throughout (Figs. I.1 and I.2) gives worse results, especially for the variances.

The WS diagnostics for streamfunction, 1 bands, top-hat(T), triangular(U) (Figs. K.1 and K.2) We revert back to the use of top-hat functions when calibrating and triangular functions when computing the diagnostics, but we change the number of bands. Using just one band (Figs. K.1 and K.2) gives excellent results for the spatial variation of variances. These are almost identical to the exact results. There is virtually no variation of vertical scale with horizontal scale though in the vertical correlations. This is expected for this limit. Note that in this limit, the results are not expected to be the same as the MetO scheme as here we have rotated the vertical modes from a vertical 'EOF' index, as used at the MetO scheme, back to height. There is thus here no means of associating vertical and horizontal scales.

The WS diagnostics for streamfunction, 10 bands, top-hat(T), triangular(U) (Figs. M.1 and M.2) The case with many (ten) bands (Figs. M.1 and M.2) gives a very similar picture to that in the case with just four bands. This is surprising as it is expected to have further degraded position dependencies and improved scale dependencies.

Please see the note below about vertical eigenvalues of vertical covariances matrices for streamfunction.

Discussion of the inverse tests for streamfunction



Discussion of the inverse tests for temperature