Crystals User Guide

Chapter 8: Refinement

8.1: Aims of Refinement

8.2: Factors Influencing Refinement

8.3: Restraints and Constraints

8.4: Notes on Weights

8.5: A Strategy

8.6: A Practical Scheme

8.7: Structure Factor Control List - LIST 23

8.8: Defining the Matrix - LIST 12

8.9: Simple examples

8.10: EXAMPLES

8.11: Advanced examples

8.12: EXAMPLES

8.13: Matrix blocking schemes

8.14: Special positions and floating origins

8.15: Tips for least squares

The refinement of crystal structures is complex and rarely a well behaved process, and only occasionally can it be performed reliably without some human supervision. In CRYSTALS we have tried to provide a structure that will reduce the amount of supervision necessary under normal circumstances, and still be very flexible for use in difficult or unusual cases.

Before discussing the facilities available in CRYSTALS, some of the aims and factors relevent to refinement need considering.

The overall aim is to find an explanation of some observable phenomena in terms of a model which is intellectually acceptable. Inevitably, the thresholds for acceptability will change from observer to observer, and from time to time. However, certain norms exist for the models, and it is deviations from these norms that increase our understanding of the physical process. The steps in arriving at a suitable model are:

1. Deciding what the observations are, and what errors are associated with them.

Postulating a model that will enable the observations to be simulated.

Deciding if the differences between the actual and the simulated observations are acceptably small. If they are, then we understand the phenomena, otherwise we must try to modify the model.

Step 1 poses the problem of resolving the 'observation' from the 'model'. In X-ray crystallography, are the observations 'I' or 'F'? If an absorption correction is applied, is the correction part of the observation, or of the model? The answer to these questions really depends on what additional questions the model must answer.

The refinement process cannot be made to yield more information than that present in the data (though the data may include items other than the diffraction observations). Deficiencies in the data may include:

Generally restraints, also called 'soft constraints', are used to test the hypothesis that the diffraction data will yield parameters not incompatible with those taken from some other source. For example, the restraint may request the refinement to verify that the diffraction data is not incompatible with a certain bond having a length of 1.39(1)A. The user should realise that just because a restraint is satisfied, he has no guarantee that no other hypothesis would fit the data just as well.

Restraints may be used to help speed up the convergence of a refinement. If the restraints are imposed with large weights (i.e. small e.s.d.s) in the initial stages, they will force the refinement rapidly towards the preconceived structure, possibly helping to keep it from false minima. As convergence approaches, the weights can be reduced, or the restraints removed altogether.

The reflection weights for least squares must reflect errors in BOTH Fo and the model since we are using (Fo-Fc) as the argument in the LS procedure. The most serious approximations in the model are usually made in:

 1. Absorption corrections.
 2. The form of the temperature factors.
 3. Atomic scattering factors.
 4. Extinction corrections.

Errors in the diffraction data should ideally follow Poisson statistics. However, it has been our experience that except in the case of painstaking work on very high quality crystals, other errors are present. These generally seem to be approximately correlated with the reflection intensity, and can be represented by an expression like:

 sigma(F) = a*sigma(I) + b*I + c*I**2

Since values for the constants a, b and c are arrived at by inspired guesses, we now prefer to represent sigma(F) by a smooth function in Fo. The function chosen is a Chebychev polynomial with the minimum number of coefficients needed to make w(Fo-Fc)**2 constant as a function of Fo. This procedure only yields weights appropriate for the current model, and so is inappropriate for intermediate stages of the analysis. Futher, introduction of a model dependent weighting scheme too early in a structure analysis may lead to important features being concealed. Until all the parameters that need to be refined have been identified, unit weights or the scheme due to Hughes are used. A valid weighting of the reflections makes w(Fo-Fc)**2 constant for all rational samplings of the data. Changes of weighting scheme usually have the most dramatic effects on thermal and extinction parameters, which should be re-refined whenever the scheme is modified.

The user should not be unduly surprised to find his normal R factor increasing after the application of weights (he is not trying to minimise 'R'), and will generally find Rw higher than R (Rw uses delta squared). If Rw is much higher than R, then the model may be very inadequate in some aspects, or there may be a few 'rogue' reflectins in the data. These should be found, considered, and possibly rejected.

Currently, the only effective way for completing the refinement of a crystal structure at atomic resolution is by the method of least-squares in which we minimise w(Fo-Fc)**2 summed over the diffraction data. In restrained refinement we also simultaneously minimise w(Pt-Pc) ( where Pt and Pc are the theoretical and calculated values for some structural parameter) summed over the required parameters. In both cases 'w' is the weight appropriate for the term in the summation. Fc and Pc are usually non-linear functions of the atomic parameters. The functions are thus expanded as a Taylor series, and the second and higher order terms neglected. The zero'th order terms (Fc and Pc) depend upon the current model, and the neglected terms only really become insignificant as the process approaches convergence. e.g.

 Fc(i) + [x(k).d(Fc(i))/d(p(k))] = Fo(i) 
 Pc(i) + [x(k).d(Pc(i))/d(p(k))] = Po(i) 

where the terms k in [] are summed over the parameters being varied and we solve for the shifts x(k) in parameter p(k). The terms Fc(i) and Pc(i) are effectively observations for the current model, and can be moved to the right hand side.

Because of the approximations made, the process need not be convergent, but this becomes more likely the better the starting model. Fc is more sensitive to reasonable errors in some types of parameter (e.g. x,y,z) than in others (e.g. U's). At the outset of refinement therefore the least sensitive parameters can be given reasonable values (e.g. 0.05 for U[iso] of a carbon atom) which is not refined until the more volatile parameters have stabilised. In this example, the refinement is heavily constrained with the constraint that U[iso] = 0.05. Once the model for the geometry has begun to stabilise, the thermal model can be refined, though here again it is generally wise to constrain the temperature factors to be isotropic initially. Because all the parameters are correlated, it makes no sense to refine these intermediate models to convergence. An r.m.s.(shift/e.s.d.) of about 1 to 3 is adequate before the model can be relaxed (though any individual anomalies should be investigated).

As convergence approaches, the gradient terms (d(F)/d(p)) change only slowly compared with Fc, and it is thus appropriate in large structures only to recompute Fo-Fc (the Right Hand Side) every cycle, and re-use the matrix of derivatives for alternate cycles, thus saving considerable machine time.

Throughout this section, the importance of the starting model has been emphasised. The user should now see why Fourier refinement and Regularisation were discussed in detail earlier. Non-linear least-squares is a powerful method for acheiving a satisfactory result only when it is given a good starting model. While it can give some indications as to redundant parameters (by refining them to absurd values or attributing them large e.s.d.s), it cannot introduce new parameters. The final test of a 'good' structure is not a low 'R' factor or minimisation function - it is more likely to be an intuitive assesment based on the relationship between the current model and similar structures, and a critical examination of difference Fourier syntheses.

The following list is a flow diagram for a typical crystal structure refinement assuming that there are no problems with disorder, pseudo-symmetry, twinning etc.

 1.  Get a trial structure (Patterson or direct methods)
 2.  Calculate structure factors and a Scale Factor.
 3.  Do an Fo map and Fourier refinement.
 4.  Regularise if you can. Use REGULARIZE or MOLAX
 5.  If you have found new atoms, go to 2.
 6.  Refine the positional parameters.
 7.  Refine isotropic temperature factors.
 8.  Locate or compute hydrogen atoms.
 9.  Refine positions again.
 10. Refine aniso temperature factors.
 11. Locate or compute any missing hydrogen atoms.
 12. Enter the final stages of refinement
  a. If a small structure, FULL matrix.
  b. If a medium structure, LARGE BLOCK approximation.
     If refinement is stable and correlation coefficients  
     are small, then use DIAGONAL approximation
  c. If a large structure, DIAGONAL approximation or 
     CASCADE refinement.
 13. Look at the agreement (variance) analysis (\ANALYSE).
     If <w(Fo-Fc)**2> not constant, change weights to SCHEME 1 
     with a parameter P(1) = F(min<w delsq>).
 14. Do 1 cycle of refinement of temperature factors plus 
     a CALCULATION.
 15. Look at the agreement analysis again, and decide if you 
    need a Chebychev weighting scheme.
 16. Refine to convergence, re-using matrix on alternate cycles.
 17. If there are unusual features (distances, angles, planarity, 
     Uaniso) use REGULARIZE, MOLAX, ANISO to correct the feature, 
     and use restrained refinement to test the hypothesis that 
     the 'corrected' model is compatible with the X-ray data.

The overall conditions governing the calculation of structure factors and monitoring the refinement process are generally kept constant throughout a refinement, and so have been collected together into the Structure Factor Control List, LIST 23, which should be input with the other initial data. The directive MODIFY consists of a series of switches for causing the application (or not) of data contained in other lists, thus removing the need to re-input those lists as circumstances change. Note that, for example, though LAYER scale factors or an EXTINCTION parameter may be present in LIST 5, they will only be included in the calculation of Fc if the switches are set in LIST 23.

The MINIMISE directive controls whether or not the restraints are to be added into the minimisation function (i.e. a LIST 16 may be present on the DISK but will not be actioned unless LIST 23 requests it), and whether F or F**2 is to be used for the diffraction data. Note that if you request F**2, you will probably need to concider changing the weighting scheme.

The directives ALLCYCLES and INTERCYCLE control the convergence-divergence parameters.

In PART 5 we introduced the definitions for parameter names and showed how these were associated with parameter values via LIST 5. For least - squares we need to set out a table showing the relationship between least-squares parameter names, least-squares parameter shifts, and the physical parameters. This table is held as LIST 22, which the interested user may care to print (for a small matrix!). For those cases where there is one least-squares parameter per physical parameter and the refinement is full matrix, this table is relatively trivial. For refinements in which physical parameters are EQUIVALENCed or RIDING, or where the parameters are divided into several matrix blocks, it would be unreasonably complex for the user to have to construct this table for himself. The information which CRYSTALS needs to construct this table is presented in a symbolic form as a LIST 12.

LIST 12 consists of directives that define matrix blocks (FULL, BLOCK, DIAGONAL) and directives that modify the contents of the current block (RIDE, EQUIVALENCE, FIX, WEIGHT, PLUS). The directive CONTINUE merely extends the directive begun on the previous line.

For many simple structures a simple full, large-block or atom-block diagonal matrix refinement may be sufficient. Parameters implicitly included by a preceding FULL, BLOCK or DIAGONAL card may be removed from the refinement by the FIX directive.

 \LIST 12            This defines a single matrix block          
 FULL parameters     containing all the specified parameters 
 END                 PLUS the overall scale factor.
 
 \LIST 12            This defines two independant matrix blcks,
 BLOCK parameters(1) one containing parameters(1), the other
 BLOCK parameters(2) containing parameters(2) and (3), i.e. the
 PLUS  parameters(3) PLUS directive is acting rather like CONTINUE.
 END                 The overall scale factor is not implied, and
                    must be specified explicitly in the appropriate
                    block if required. Note that the same parameter
                    must not occur more than once.

 \LIST 12            Refine the positions and anisotropic 
 FULL X'S U'S        temperature factors of all atoms in a single
 END                 matrix block with the overall scale factor.
 
 \LIST 12            As above, only now don't refine the y 
 FULL X'S U'S        coordinate of the lead atom (polar axis? 
 FIX PB(1,Y)         see also restraints).
 END
 
 \LIST 12              Refine the positions for all the carbon 
 BLOCK C(1,X,Y,Z) UNTIL LAST                         one block
 BLOCK C(1,U'S) UNTIL C(10) SCALE          and the anisotropic 
 PLUS  H(11,U[ISO]) UNTIL LAST           t.f.s for the carbons 
 CONT UNTIL LAST      together with an isotropic t.f.s for the
 END            hydrogen atom and the scale factor in another. 
 
 Note that PLUS could be replaced by CONTINUE.

In some situations a simple one-to-one relationship between structural and least squares parameters is not suitable. For example, the x and y coordinates of an atom on the special position X,X,1/6 must move synchronously, that is, be represented by a single least squares parameter. Similarly, an atom on X,-X,11/12 has the x and y coordinates linked, but here the shift to y is in the opposite sense of that for x.

Sometimes the relationship between parameters is not due to space group symmetry, but to some other physical requirement. In the example given above (Restraints), where either Na or K can occupy the same site, we may have site occupancy disorder such that the total occupancy for the ions is fixed, but the ratio is to be determined. In this case, the shifts in the occupation factors for the two ions are equivalenced to a single least squares parameter, but made to move in opposite senses.

Linking of physical parameters like this provides a powerfull way for dealing with certain types of instability in the refinement, and for reducing the cost by reducing the number of parameters refined. For example, the hydrogen atoms on a carbon can be shifted synchronously with it, thus preserving the local geometry (but watch second neighbour relationships). This procedure is sometimes known as a 'riding' refinement. Another use for riding parameters is to give a poorly defined residue a single overall anisotropic temperature factor (not as good as proper TLS of course, but perhaps more realistic than individual atomic temperature factors).

For EQUIVALENCED LINKED or RIDING parameters, the partial derivative for each parameter is computed, multiplied by the WEIGHT if requested, and added into the normal equations. After solution of the equations, the resultant shifts are multiplied by the same weights and then applied to the corresponding structural parameters.

This example is also given in the MANUAL. The structure contains C,N and O in the space group P6122, with N at x,-x,11/12, O at x,x,1/6 and C isotropic. The refinement will be by full matrix. The following 3 LIST 12s all lead to the same matrix. The example in the manual has only 9 directives.

10
 \LIST 12                \LIST 12                  \LIST 12
 FULL C(1,X'S)           FULL X'S                  FULL X'S U'S
                                                  FIX C(1,U'S)
 PLUS C(1,U[ISO])        PLUS C(1,U[ISO])          PLUS C(1,U[ISO])
 PLUS N(1,U[33],U[12])   PLUS N(1,U[33],U[12])
 EQUIV N(1,X) N(1,Y)     EQUIV N(1,X) N(1,Y)       EQUIV N(1,X) N(1,Y)
 WEIGHT -1. N(1,Y)       WEIGHT -1. N(1,Y)         WEIGHT -1. N(1,Y)
                        FIX N(1,Z)                FIX N(1,Z)
 EQUIV N(1,U[11],U[22])  EQUIV N(1,U[11],U[22])    EQUIV N(1,U[11],U[22])
 EQUIV N(1,U[23],U[13])  EQUIV N(1,U[23],U[13])    EQUIV N(1,U[23],U[13])
 PLUS O(1,U[33],U[12])   PLUS O(1,U[33],U[12])
 EQUIV O(1,X) O(1,Y)     EQUIV O(1,X) O(1,Y)       EQUIV O(1,X) O(1,Y)
                        FIX O(1,Z)                FIX O(1,Z)
 EQUIV O(1,U[11],U[22])  EQUIV O(1,U[11],U[22])    EQUIV O(1,U[11],U[22])
 EQUIV O(1,U[23],U[13])  EQUIV O(1,U[23],U[13])    EQUIV O(1,U[23],U[13])
 WEIGHT -1. O(1,U[13])   WEIGHT -1. O(1,U[13])     WEIGHT -1. O(1,U[13])
 END                     END                       END 

The next example is for H(11) H(12) and H(13) riding upon C(1). Note that the hydrogen atoms are all given the same temperature factor shifts. They would also probably start with the same temperature factor values, though for the isotropic refinement of a phenyl group (for example) the investigator may have some a priori reason for starting the atoms off with slightly differing values. We are not concerned here with the rest of the structure.

 \LIST 12
 FULL (parameters)                    The x, y and z coordinates for 
 RIDE C(1,X'S) H(11,X'S) UNTIL H(13)  the four atoms are handled by
 EQUIV H(11,U[ISO]) UNTIL H(13)       3 ls parameters. The 3 U(iso)s
 END                                  are handled by 1 parameter.

There are no fixed rules for deciding how or even whether a given structure should be refined by full or block approximations to the matrix. The choice will change from institution to institution and from time to time.

Broadly, full matrix methods are the most certain to converge. They are also likely to require less cycles of refinement than a block matrix method. However, the time per cycle will always be more than for a block matrix method, and there will be increased storage requirements. Thus, small structures should be refined using the full matrix, and large ones using a block approximation. On computing machinery where time is paid for whether the machine is idle or not (perhaps the users own machine) it makes sense to use the largest approximations to the full matrix compatible with other users of the machine and its efficiency of operation. At some stage, for large structures, a substantial number of the off-diagonal elements of the matrix will be negligable, and the cost of accumulating and paging these may not be justified. Such large structures need a special strategy.

For medium large structures, it is probably convenient to break the matrix into a small number of largish blocks. Experience has shown what parameters can be expected to be correlated: these MUST be included in the same matrix block.

 - bonded atoms
 - non-orthogonal coordinates
 - scale factors, extinction parameters and 
   temperature factors.
 - molecules or fragments related by a pseudo
   symmetry operator.

 \LIST 12
 BLOCK ATOM(1,X'S) UNTIL ATOM(1n/2)
 BLOCK ATOM(1n/2+1,X'S) UNTIL ATOM(n)
 END
 
 \LIST 12
 BLOCK ATOM(1,X'S) UNTIL ATOM(1n/4)
 PLUS  ATOM(3n/4+1,X'S) UNTIL LAST
 BLOCK ATOM(1n/4+1,X'S) UNTIL ATOM(3n/4)
 END

Atoms on special positions can be handled either through constraint or through restraints. The latter method can be applied automatically, and is to be prefered if LIST 12 is already complex because of GROUPed, LINKed RIDING or COMBINed parameters.