# CORRECT.LP

## Statistics of reflections

Near the top of CORRECT.LP we find:

``` 531781 REFLECTIONS ON FILE "INTEGRATE.HKL"
0 CORRUPTED REFLECTION RECORDS (IGNORED)
0 REFLECTIONS INCOMPLETE OR OUTSIDE IMAGE RANGE       1 ...    1799
81 REFLECTIONS OUTSIDE ACCEPTED RESOLUTION RANGES
OR TOO CLOSE TO ROTATION AXIS (IGNORED)
531700 REFLECTIONS ACCEPTED
```

## Statistics of observations

XDS, like SCALA and d*TREK, gives statistics about unaveraged and averaged quantities, but in different tables. The unaveraged values are in a table that is fine-grained in terms of resolution, at the beginning of CORRECT.LP. The Sigma values in that table are corrected to match the RMS scatter.

The table that has information about the averaged data (suitably weighted) is repeated several times. It is less fine-grained in resolution (9 shells, and overall). [if a user wants this table in fine-grained form, s/he can use XSCALE].

The way the tables are printed is the same for both types of tables: at first the definitions of the quantities in the table are given, and then the table itself is printed.

Specifically, the heading of the table which talks about the unaveraged data looks like this:

``` I/Sigma  = mean intensity/Sigma of a reflection in shell
Chi2    = goodness of fit between sample variances of
symmetry-related intensities and their errors
(Chi2 = 1 for perfect agreement)
R-FACTOR
observed = (SUM(ABS(I(h,i)-I(h))))/(SUM(I(h,i)))
expected = expected R-FACTOR derived from Sigma(I)

NUMBER  = number of reflections in resolution shell
used for calculation of R-FACTOR
ACCEPTED = number of accepted reflections
REJECTED = number of rejected reflections (MISFITS),
recognized by comparison with symmetry-related
reflections.
```

and then the table itself is:

```RESOLUTION RANGE  I/Sigma  Chi2  R-FACTOR  R-FACTOR  NUMBER ACCEPTED REJECTED
observed  expected

48.268  17.853     9.63   0.97      5.06      6.10     865     868      44
17.853  13.079    10.02   0.97      5.22      6.14    1301    1305      81
13.079  10.812     9.83   1.10      5.56      5.94    1374    1388      99
10.812   9.423     9.88   1.09      5.32      6.03    1820    1825     108
9.423   8.460     9.56   1.07      6.03      6.21    2087    2101     167

.... (many resolution shells deleted for brevity)
```

## Statistics of unique reflections

Later tables talk about the averaged intensities:

```R-FACTOR
observed = (SUM(ABS(I(h,i)-I(h))))/(SUM(I(h,i)))
expected = expected R-FACTOR derived from Sigma(I)

COMPARED = number of reflections used for calculating R-FACTOR
I/SIGMA  = mean of intensity/Sigma(I) of unique reflections
(after merging symmetry-related observations)
Sigma(I) = standard deviation of reflection intensity I
estimated from sample statistics

R-meas   = redundancy independent R-factor (intensities)
Diederichs & Karplus (1997), Nature Struct. Biol. 4, 269-275.

CC(1/2)  = percentage of correlation between intensities from
random half-datasets. Correlation significant at
the 0.1% level is marked by an asterisk.
Karplus & Diederichs (2012), Science 336, 1030-33
Anomal   = percentage of correlation between random half-sets
Corr      of anomalous intensity differences. Correlation
significant at the 0.1% level is marked.
SigAno   = mean anomalous difference in units of its estimated
standard deviation (|F(+)-F(-)|/Sigma). F(+), F(-)
are structure factor estimates obtained from the
merged intensity observations in each parity class.
Nano    = Number of unique reflections used to calculate
Anomal_Corr & SigAno. At least two observations
for each (+ and -) parity are required.
```

and the table itself is

```      NOTE:      Friedel pairs are treated as different reflections.

SUBSET OF INTENSITY DATA WITH SIGNAL/NOISE >= -3.0 AS FUNCTION OF RESOLUTION
RESOLUTION     NUMBER OF REFLECTIONS    COMPLETENESS R-FACTOR  R-FACTOR COMPARED I/SIGMA   R-meas  CC(1/2)  Anomal  SigAno   Nano
LIMIT     OBSERVED  UNIQUE  POSSIBLE     OF DATA   observed  expected                                      Corr

5.72       23750    7284      7488       97.3%       6.6%      6.6%    23666   14.59     7.9%    99.3*    33*   1.043    3033
4.06       41574   12997     13384       97.1%      10.0%      8.3%    41476   11.40    12.1%    98.3*    45*   1.341    5775
3.32       56679   16961     17336       97.8%      16.8%     15.4%    56494    6.49    20.1%    97.9*    31*   1.079    7697
2.88       67173   20272     20497       98.9%      38.4%     39.0%    66875    2.91    45.9%    93.1*    19*   0.840    9333
2.57       79365   23100     23197       99.6%      77.6%     85.3%    79063    1.46    92.1%    75.3*     5    0.701   10761
2.35       86431   25554     25631       99.7%     128.9%    146.7%    86014    0.86   153.2%    54.7*     3    0.633   11894
2.18       83863   27529     27946       98.5%     197.0%    230.0%    81669    0.49   237.7%    31.6*    -1    0.575   11422
2.04       51338   23815     29966       79.5%     286.2%    343.0%    43478    0.26   361.1%    15.1*     0    0.526    5523
1.92       25803   15877     31898       49.8%     483.3%    577.5%    17026    0.12   635.3%     3.8      2    0.519    1856
total      515976  173389    197343       87.9%      27.8%     29.3%   495761    2.89    33.5%    98.2*    19*   0.781   67294

NUMBER OF REFLECTIONS IN SELECTED SUBSET OF IMAGES  531700
NUMBER OF REJECTED MISFITS                           15698
NUMBER OF SYSTEMATIC ABSENT REFLECTIONS                  0
NUMBER OF ACCEPTED OBSERVATIONS                     516002
NUMBER OF UNIQUE ACCEPTED REFLECTIONS               173398
```

Why is there a discrepancy between "total 515976 173389" versus "NUMBER OF ACCEPTED OBSERVATIONS 516002", and "NUMBER OF UNIQUE ACCEPTED REFLECTIONS 173398" ?? The reason is that the (higher) numbers below the table include observations (and unique reflections) with I < -3*sigma(I), whereas the numbers in the table refer only to those reflections which should be used downstream (for phasing and refinement). Indeed, XDSCONV filters out those unique reflections which have I<-3*sigma(I).

It should also be noted that the alien rejection controlled by REJECT_ALIEN= (default 20) will be performed after making this table. So the number of reflections which you will get from XDSCONV is not the same as reported here. If you want to see the statistics of reflections which will be converted by XDSCONV (thus will be used for further process), you should prepare REMOVE.HKL to explicitly specify the reflections which will be thrown away and run CORRECT step again.

At the bottom of CORRECT.LP we find:

```NUMBER OF UNIQUE ALIEN REFLECTIONS WITH A Z-SCORE ABOVE LIMIT       162
(ALIENS ABOVE LIMIT (REJECT_ALIEN=      20.0) ARE MARKED INVALID)

NUMBER OF REFLECTION RECORDS ON OUTPUT FILE "XDS_ASCII.HKL"      531700
NUMBER OF ACCEPTED OBSERVATIONS (INCLUDING SYSTEMATIC ABSENCES)  515712
NUMBER OF REJECTED MISFITS & ALIENS (marked by -1*SIGMA(IOBS))    15988
```

The file XDS_ASCII.HKL actually has all 531700 reflections.