Using Prior Knowledge in Data Assimilation

"Sky and Water I", M.C. Escher

Ross Bannister Data Assimilation Research Centre Univ. of Reading Oxford / RAL Spring School 2004

Data Assimilation

The objective of data assimilation

1. Notation
2. Maximum likelihood
   • Why prior information is necessary
3. Properties of background error covariances
4. Measuring background error covariances
5. Preconditioning in variational assimilation
6. Some current topics
7. Summary
8. References

A. Notation for vectors and matrices

Observation space

Model space

Variances and covariances

One variable Gauss

Two or more variables

B. Maximum likelihood

No prior information

With prior information (background)

Use Bayes' Theorem:

Bayes

Optimal Interpolation equation:

Benefits of using prior information

• Makes the problem well-posed, even when only few observations are present
• Fills data voids with 'good quality' information
• Provides a starting point for the minimization in Var.
  • First set of initial conditions for 4d-Var.
  • Realistic first-guess linearization state for non-linear observation operators in 3d/4d-Var.
  • Expect to be close to correct minimum of

Otherwise known as ...

• A-priori
• First guess
• Forecast

Where does the background state come from?

• Forecast from analysis at earlier time (normally 6 or 12 hours)
  • Propagation of observational information downstream consistent with the laws of physics
• Climatology
• Persistence

Drawbacks

• Need to determine its error characteristics (background error covariance matrix)
• Important to determine background errors well
• Possibility of unknown systematic errors in forecast (e.g. biases)

The data assimilation cycle

3d-Var. cost function and gradient

:

C. The background error covariance matrix

Mathematical properties of B

• represents the error characteristics of in a Gaussian fashion
• Symmetric,
• elements
• Orthogonal eigenvectors ,
• Positive definite,
• Non-sparse (multivariate)
• Huge conditioning number,

• Cannot look at complete matrix in one go
• Examine covariance 'structures'
  • Example covariance structure function response found from acting with on a prepared state:

Physical properties of B - 2

'spreads-out' information in the analysis like a convolution. Use OI expression:




Can see this with a single observation of a model variable at a grid position:

Physical properties of B - 3

• is usually a forecast and so the equations of motion will influence strongly the covariance patterns.

Example: Geostrophic error covariances Geostrophic balance: Pressure-pressure covariances assumption:

• Covariances are synoptically dependent, but have climatological characteristics
• Covariances reflect local length scales
• Horizontal length scales in balanced quantities depends upon the Rossby radius of deformation,
  
• Horizontal length scales are:
  • long in tropics and in the stratosphere
  • short in the extra-tropics and in the troposphere

from Ingleby (2001)

Physical properties of B - 5

Vertical length scales are associated with horizontal length scales

from Ingleby (2001)

Variances reflect the conditions locally

from Ingleby (2001)

D. Measuring B

Practical methods of measuring B

1. Computation of
   • No. of matrix elements makes calculation (and even storage) prohibitive
   • Do not know the truth
   • Do not know the probability distribution function well enough to average over
2. Computation of
   • Computation prohibitive (No. of matrix elements)
   • Do not know well enough

The 'N.M.C.' method 24 hr time separation between two forecasts (valid at the same time) Overestimates errors present in b/g (6 hr forecast) Especially over large error at long horiz. length scales

The 'ensemble' method

There are still too many elements for calculation of - see later

• Construct a model of with a manageable number of parameters
• Use physical intuition and exploit symmetries
• Capture climatological features of only
• Transform to uncorrelated control variables in Var. and precondition at the same time

• The Hessian of the cost function in -space is,
  
• The Hessian describes the orientation and elongation of the contours of
• A high conditioning number () leading to highly elongated contours
• Assume that most of the bad conditioning is due to the background errors
• Perform a change of variables () that reduces the conditioning number (precondition )

Design of transformation - 1

Model space, Cost function (let ) Hessian Very large conditioning number Background errors are strongly correlated Background variances have range of scales Inefficient convergence in Var.

Control space, Cost function (let ) By design let Hessian Reduced conditioning number Background errors are uncorrelated Background variances are all unity Workable convergence efficiency in Var.

SOLVE VAR. PROBLEM IN -SPACE. TRANSFORM TO -SPACE WHEN CONVERGED

• The information concerning has been transferred into the transformation,
• Design of the transformation, is a model of
• is the background error covariance matrix implied by the model of

The Met Office use the following form of (described more easily in terms of its inverse):

Operator	Name	Form	Input vector	Output vector
	As above	As above	model variables, ,	,
	Parameter trans.	See later	"	uncorr. parameters,
	Vertical trans.		uncorr. parameters,	uncorr. parameters,
	Horiz. trans.		uncorr. parameters,	uncorr. parameters,

Key: longitude, latitude, height, horiz. wavenumbers, vertical mode index
vertical modes, their variances, horizontal modes, their variances

The remaining problem is to determine: the uncorrelated parameters, , , ,

• Choose parameters that are likely to be mutually uncorrelated
• Balanced / unbalanced partitioning reflects normal mode structure of Eqs. of motion
• Problem is univariate in terms of parameters
• Still autocorrelation (treat with spatial decorrelation transforms)

Example orthogonalization

'balanced' dynamical parameter (Rossby modes) , 'unbalanced' parameters (gravity modes) 'unbalanced' moisture parameter LBE: linear balance equation (to derive 'balanced' pressure) -Eqn.: omega equation (to derive 'balanced' divergence)

• Do not confuse orthogonality with non-correlation
• is not always a balanced parameter (not balanced at low Burger Nos.)

Calibration of vertical transform

• Determine vertical modes and variances by NMC method
• Treat each parameter separately (e.g. )

In practice, find global average matrix

Vertically (nearly) uncorrelated modes for this parameter are eigenvectors (EOFs), , of

From Ingleby (2001)

Calibration of horizontal transform

• Determine horizontal variances by NMC method
• Assume errors are homogeneous and isotropic in horizontal (on vertical-mode surfaces) for all parameters
  • Spherical harmonics are the uncorrelated horizontal modes,
  • Variance is a function of total wavenumber, , only
• Treat each parameter and each vertical mode separately (e.g. )

Compute horizontal variances for each parameter and vertical mode NMC tends to predict length scales that are too long (statistics need to be massaged)

Implied covariances

	Zonal wind standard deviation	Streamfunction vertical correlation
Measured
Implied

• Decorrelation of the parameters
  • Truly 'balanced' and 'unbalanced' partitioning of variables
  • Treatment of additional fields (trace gases, physical parameters, etc.)
• More accurate model of variation of lengthscales
  • Wavelet-based transforms
• Synoptic (flow) dependence
  • Kalman filter
  • Geostrophic coordinate transform
  • Non-linear balance equation
  • Effect on due to observations
• Biases in the background
• Non-Gaussian and non-linear errors

• Prior knowledge () makes the DA problem well posed
• The probability distribution function of describes its uncertainty and
  • is usually assumed to have a Gaussian form ()
  • spreads observational information in space in the DA algorithm
  • ensures that the analysis increments are largely balanced
  • is represented via control variable transforms
• must be measured carefully
• is one of the most important elements of a DA system

• Kalnay E., Atmospheric Modelling, Data Assimilation and Predictability, Cambridge University Press (2003).
• Daley R., Atmospheric Data Analysis, Cambridge University Press (1996).
• Schlatter, Variational assimilation of meteorological observations in the lower atmosphere: a tutorial on how it works, J. Atmos. Sol. Terr. Phys. 62, pp. 1057-1070 (2000).

Measuring background error covariances

• Hollingsworth A., Lonnberg P., The statistical structure of short-range forecast errors as determined from radiosonde data. Part I: The wind field, Tellus 38A, pp. 111-136 (1986). [Measurement by innovation method - pre NMC]
• Lonnberg P., Hollingsworth A., The statistical structure of short-range forecast errors as determined from radiosonde data. Part II: The covariance of height and wind errors, Tellus 38A, pp. 137-161 (1986). [Measurement by innovation method - pre NMC]
• Parrish D., Derber J.C., The National Meteorological Center's spectral statistical interpolation analysis system, Mon. Wea. Rev. 120, pp. 1747-1763 (1992). [NMC method].
• Fisher M., Background error covariance modelling, in ECMWF seminar proceedings on recent developments in data assimilation for atmosphere and ocean (2003). [Ensemble method].
• Ingleby N.B., The statistical structure of forecast errors and its representation in the Met Office global 3d variational data assimilation scheme, Q.J.R. Meteor. Soc. 127, pp. 209-231 (2001).

Control variable transforms

• Lorenc A.C., Ballard S.P., Bell R.S., Ingleby N.B., Andrews P.L.F., Barker D.M., Bray J.R., Clayton A.M., Dalby T., Li D., Payne T.J., Saunders F.W., The Met Office global 3d variational data assimilation scheme, Q.J.R.Meteor.Soc. 126, pp. 2991-3012 (2000).
• Derber J., Bouttier F., A reformulation of the background error covariance in the ECMWF global data assimilation system, Tellus 51A, pp. 195-221 (1999).
• Bannister R.N., On control variable transforms in the Met Office 3d and 4d Var., and a description of the proposed waveband summation transformation, "http://www.met.rdg.ac.uk/~ross/DataAssim.html" (2003/2004).