Difference between revisions of "Xdscc12"

From XDSwiki
(Anomalous Fcalc as reference (under development))
m
 
(10 intermediate revisions by 2 users not shown)
Line 1: Line 1:
XDSCC12 is a program for generating delta-CC<sub>1/2</sub> and delta-CC<sub>1/2</sub>ano values for XDS_ASCII.HKL (written by XDS), or for XSCALE.HKL containing several files of type XDS_ASCII.HKL after scaling in XSCALE (with MERGE=FALSE).  
+
XDSCC12 is a program for generating [[CC1/2|delta-CC<sub>1/2</sub>]] and delta-CC<sub>1/2-ano</sub> values for XDS_ASCII.HKL (written by [[XDS]]), or for XSCALE.HKL (written by [[XSCALE]]) containing data from several files of type XDS_ASCII.HKL after scaling (with MERGE=FALSE).  
  
 
It implements the method described in Assmann, Brehm and Diederichs (2016) Identification of rogue datasets in serial crystallography. J. Appl. Cryst. 49, 1021 [http://journals.iucr.org/j/issues/2016/03/00/zw5005/zw5005.pdf], and it does this not only for the individual datasets in XSCALE.HKL, but also for individual frames, or groups of frames, of a single dataset collected with the rotation method and processed by [[XDS]].
 
It implements the method described in Assmann, Brehm and Diederichs (2016) Identification of rogue datasets in serial crystallography. J. Appl. Cryst. 49, 1021 [http://journals.iucr.org/j/issues/2016/03/00/zw5005/zw5005.pdf], and it does this not only for the individual datasets in XSCALE.HKL, but also for individual frames, or groups of frames, of a single dataset collected with the rotation method and processed by [[XDS]].
  
The program can be downloaded for [ftp://turn5.biologie.uni-konstanz.de/pub/xdscc12.rhel6.64 Linux 64bit] or [ftp://turn5.biologie.uni-konstanz.de/pub/xdscc12-mac Mac].
+
The program can be downloaded for [https://{{SERVERNAME}}/pub/linux_bin/xdscc12 Linux] or [https://{{SERVERNAME}}/pub/mac_bin/xdscc12 Mac].
  
 
Usage (this text can be obtained with <code>xdscc12 -h</code>):
 
Usage (this text can be obtained with <code>xdscc12 -h</code>):
 
<pre>
 
<pre>
xdscc12 KD 2018-9-6. Academic use only; no redistribution. -h option shows options.
+
xdscc12 KD 2019-04-30. Academic use only; no redistribution. -h option shows options.
 
Please cite Assmann, G., Brehm, W., Diederichs, K. (2016) J.Appl.Cryst. 49, 1021-1028
 
Please cite Assmann, G., Brehm, W., Diederichs, K. (2016) J.Appl.Cryst. 49, 1021-1028
usage: xdscc12 [-dmin <lowres>] [-dmax <highres>] [-nbin <nbin>] [-mode <1 or 2>] [-<abcdeftwz>] [-r <ref>] FILE_NAME
+
running 'xdscc12 -h' on 20190502 at 16:11:46 +0200
dmax (default 999A), dmin (default 1A) and nbin (default 10) have the usual meanings.
+
usage: xdscc12 [-dmin <lowres>] [-dmax <highres>] [-nbin <nbin>] [-mode <1 or 2>] [-<abcdefstwz>] [-r <ref>] FILE_NAME
 +
dmin (default 999A), dmax (default 1A) and nbin (default 10) have the usual meanings.
 
mode can be 1 (equal volumes of resolution shells) or 2 (increasing volumes; default).
 
mode can be 1 (equal volumes of resolution shells) or 2 (increasing volumes; default).
  -t: total oscillation (degree) to batch fine-sliced frames into
+
  -r: next parameter: ASCII reference file with lines: h,k,l,Fcalc or h,k,l,Fcalc+,Fcalc-
  -r: next parameter: ASCII reference file with lines: h,k,l,Fcalc
+
      this allows calculation of CC of isomorphous signal with reference
 +
  -s: read two columns from reference: Fcalc(+), Fcalc(-).
 +
      this allows calculation of CC of anomalous signal with that of reference
 +
  -t: total oscillation (degree) to batch fine-sliced frames into
 
  FILE_NAME can be XDS or XSCALE reflection file
 
  FILE_NAME can be XDS or XSCALE reflection file
  other options can be combined (e.g. -def), and switch the following off:
+
  other options can be combined (e.g. -def), and switch the following OFF:
  -a: individual isomorphous summary values
+
  -a: individual isomorphous summary values
  -b: individual (Fisher-transformed) delta-CC1/2 values
+
  -b: individual (Fisher-transformed) delta-CC1/2 values
  -c: individual delta-CC1/2 reflection numbers
+
  -c: individual delta-CC1/2 reflection numbers
  -d: individual anomalous summary values
+
  -d: individual anomalous summary values
  -e: individual (Fisher-transformed) delta-CC1/2ano values
+
  -e: individual (Fisher-transformed) delta-CC1/2ano values
  -f: individual delta-CC1/2ano reflection numbers
+
  -f: individual delta-CC1/2ano reflection numbers
  -w: weighting of intensities with their sigmas
+
  -w: weighting of intensities with their sigmas
  -z: no Fisher transformation of delta-CC1/2 values
+
  -z: Fisher transformation of delta-CC1/2 values
 
</pre>
 
</pre>
  
The program output is terse but supposed to be self-explanatory. The isomorphous delta-CC<sub>1/2</sub> of a batch of frames (width chosen with the -t option) relative to all data is most easily visualized via [[XDSGUI]] (Statistics tab); the anomalous delta-CC<sub>1/2</sub> may be plotted with e.g. gnuplot after grepping the relevant lines from the output.
+
The program output in the terminal window is terse but supposed to be self-explanatory; it can (and most often should) be saved or re-directed to a file.
 +
xdscc12 ... > xdscc12.log  #  or xdscc12 ... | tee xdscc12.log
 +
All statistics (tables) produced by xdscc12 may be visualized with e.g. gnuplot, after grepping the relevant lines from the output.
 +
If xdscc12 is used with a XDS_ASCII.HKL reflection file (from XDS), the isomorphous delta-CC<sub>1/2</sub> of a batch of frames (width chosen with the -t option) relative to all data is most easily visualized via [[XDSGUI]] (Statistics tab). Negative numbers indicate a worsening of the overall signal.
  
For multiple datasets, the output lines show the contribution of each dataset toward the total CC<sub>1/2</sub>. Negative numbers indicate a worsening of the overall signal.
+
If xdscc12 is used with a XSCALE.HKL generated from multiple datasets, the output lines show the contribution of each dataset toward the total CC<sub>1/2</sub>. In this case, the program writes a file called XSCALE.INP.rename_me which shows statistics of delta-CC<sub>1/2</sub> and delta-CC<sub>1/2-ano</sub> values, and has a sorted enumeration of the INPUT_FILEs - the first of these provides the best data set, and the last one is the worst one. This XSCALE.INP.rename_me can then be edited (i.e. for deleting a few data sets with very negative delta-CC<sub>1/2</sub>), and renamed to XSCALE.INP.
  
Statistics are given (in resolution shells) for the isomorphous and the anomalous signal.
+
Statistics are given (in resolution shells) for the isomorphous and the anomalous signal. In case of [[SSX]] data (which have few reflections per data set, compared to complete data sets), we typically use nbin of 1 in xdscc12.
  
Important: to identify outliers in [[XSCALE]]d data, you should use the -w option. Otherwise, a and b are adjusted such that the sigmas are very high, which reduces the delta-CC<sub>1/2</sub> signal.
+
To find out about the influence of the ''a'' and ''b'' parameters of the XDS/XSCALE-adjusted error model, you may try the -w option; this assigns the same sigma to all reflections. Likewise, the [https://en.wikipedia.org/wiki/Fisher_transformation Fisher transformation], which serves to make changes in CC<sub>1/2</sub> comparable across resolution ranges, may be switched off for testing purposes, with the -z option.
  
A complete description of how to process serial crystallography data with XDS/XSCALE is given in [[SSX]]. A program that implements the method of [https://doi.org/10.1107/S1399004713025431 Brehm and Diederichs (2014)] and theory of [https://doi.org/10.1107/S2059798317000699 Diederichs (2017)] is [[xscale_isocluster]].
+
== Correlation against a reference data set (-r <reference> option) ==
 
+
The correlation of the experimental data set against the user-supplied reference data is shown in the lines starting with r.
== Preparing a reference data set ==
+
To prepare a reference data set if the refinement was done with phenix.refine, one could use e.g.
If the refinement was done with phenix.refine, one could use
 
 
<pre>
 
<pre>
 
mtz2various hklin 2bn3_refine_001.mtz hklout temp.hkl <<eof
 
mtz2various hklin 2bn3_refine_001.mtz hklout temp.hkl <<eof
Line 54: Line 60:
 
</pre>
 
</pre>
  
== Fcalc with anomalous signal as reference ==
+
=== Reference data with anomalous signal (additional -s option) ===
 +
The correlation of the anomalous difference of the experimental data set against the anomalous signal of the user-supplied reference data is shown in the lines starting with s.
 
A simple way to obtain Fcalc(+) and Fcalc(-) is to run <code>phenix.refine</code> with options (in case of S as anomalous scatterer)
 
A simple way to obtain Fcalc(+) and Fcalc(-) is to run <code>phenix.refine</code> with options (in case of S as anomalous scatterer)
  refinement.input.xray_data.labels="F(+),SIGF(+),F(-),SIGF(-),merged"  
+
  refinement.input.xray_data.labels="F(+),SIGF(+),F(-),SIGF(-),merged"  refinement.refine.anomalous_scatterers.group.selection="element S" strategy=individual_sites+individual_adp+group_anomalous+occupancies
  refinement.refine.anomalous_scatterers.group.selection="element S"
 
strategy=individual_sites+individual_adp+group_anomalous+occupancies
 
 
and then
 
and then
 
<pre>
 
<pre>
Line 68: Line 73:
 
eof
 
eof
 
</pre>
 
</pre>
in which case <code>sftools</code> outputs only the acentric reflections - only those have anomalous differences. <code>XDSCC12</code> then has to be run with the <code>-s -r anom-reference.hkl</code> option. This allows a direct comparison of CC Iobs-Icalc, with Icalc either as (Fcalc(+) + Fcalc(-))/2 (these are the output lines starting with r) or with Iobs(+) being compared with Fcalc(+)<sup>2</sup>, and Iobs(-) being compared with Fcalc(-)<sup>2</sup> (output lines starting with s). Preliminary insight: the r and s lines do not differ significantly.
+
in which case <code>sftools</code> outputs only the acentric reflections - only those have anomalous differences. <code>XDSCC12</code> then has to be run with the <code>-s -r anom-reference.hkl</code> option.
 +
 
 +
== See also ==
 +
 
 +
A complete description of how to process serial crystallography data with XDS/XSCALE is given in [[SSX]].
 +
 
 +
A program that implements the method of [https://doi.org/10.1107/S1399004713025431 Brehm and Diederichs (2014)] and theory of [https://doi.org/10.1107/S2059798317000699 Diederichs (2017)] is [[xscale_isocluster]].
 +
 
 +
To remove bad frames from a XDS_ASCII.HKL file, you can use the program [[exclude_data_range_from_XDS_ASCII.HKL]], or re-INTEGRATE with the keyword [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_parameters.html#EXCLUDE_DATA_RANGE= EXCLUDE_DATA_RANGE] in [[XDS.INP]].

Latest revision as of 16:10, 30 May 2020

XDSCC12 is a program for generating delta-CC1/2 and delta-CC1/2-ano values for XDS_ASCII.HKL (written by XDS), or for XSCALE.HKL (written by XSCALE) containing data from several files of type XDS_ASCII.HKL after scaling (with MERGE=FALSE).

It implements the method described in Assmann, Brehm and Diederichs (2016) Identification of rogue datasets in serial crystallography. J. Appl. Cryst. 49, 1021 [1], and it does this not only for the individual datasets in XSCALE.HKL, but also for individual frames, or groups of frames, of a single dataset collected with the rotation method and processed by XDS.

The program can be downloaded for Linux or Mac.

Usage (this text can be obtained with xdscc12 -h):

xdscc12 KD 2019-04-30. Academic use only; no redistribution. -h option shows options.
Please cite Assmann, G., Brehm, W., Diederichs, K. (2016) J.Appl.Cryst. 49, 1021-1028
running 'xdscc12 -h' on 20190502 at 16:11:46 +0200
usage: xdscc12 [-dmin <lowres>] [-dmax <highres>] [-nbin <nbin>] [-mode <1 or 2>] [-<abcdefstwz>] [-r <ref>] FILE_NAME
dmin (default 999A), dmax (default 1A) and nbin (default 10) have the usual meanings.
mode can be 1 (equal volumes of resolution shells) or 2 (increasing volumes; default).
  -r: next parameter: ASCII reference file with lines: h,k,l,Fcalc or h,k,l,Fcalc+,Fcalc-
      this allows calculation of CC of isomorphous signal with reference
  -s: read two columns from reference: Fcalc(+), Fcalc(-). 
      this allows calculation of CC of anomalous signal with that of reference
  -t: total oscillation (degree) to batch fine-sliced frames into
 FILE_NAME can be XDS or XSCALE reflection file
 other options can be combined (e.g. -def), and switch the following OFF:
  -a: individual isomorphous summary values
  -b: individual (Fisher-transformed) delta-CC1/2 values
  -c: individual delta-CC1/2 reflection numbers
  -d: individual anomalous summary values
  -e: individual (Fisher-transformed) delta-CC1/2ano values
  -f: individual delta-CC1/2ano reflection numbers
  -w: weighting of intensities with their sigmas
  -z: Fisher transformation of delta-CC1/2 values

The program output in the terminal window is terse but supposed to be self-explanatory; it can (and most often should) be saved or re-directed to a file.

xdscc12 ... > xdscc12.log  #  or xdscc12 ... | tee xdscc12.log

All statistics (tables) produced by xdscc12 may be visualized with e.g. gnuplot, after grepping the relevant lines from the output. If xdscc12 is used with a XDS_ASCII.HKL reflection file (from XDS), the isomorphous delta-CC1/2 of a batch of frames (width chosen with the -t option) relative to all data is most easily visualized via XDSGUI (Statistics tab). Negative numbers indicate a worsening of the overall signal.

If xdscc12 is used with a XSCALE.HKL generated from multiple datasets, the output lines show the contribution of each dataset toward the total CC1/2. In this case, the program writes a file called XSCALE.INP.rename_me which shows statistics of delta-CC1/2 and delta-CC1/2-ano values, and has a sorted enumeration of the INPUT_FILEs - the first of these provides the best data set, and the last one is the worst one. This XSCALE.INP.rename_me can then be edited (i.e. for deleting a few data sets with very negative delta-CC1/2), and renamed to XSCALE.INP.

Statistics are given (in resolution shells) for the isomorphous and the anomalous signal. In case of SSX data (which have few reflections per data set, compared to complete data sets), we typically use nbin of 1 in xdscc12.

To find out about the influence of the a and b parameters of the XDS/XSCALE-adjusted error model, you may try the -w option; this assigns the same sigma to all reflections. Likewise, the Fisher transformation, which serves to make changes in CC1/2 comparable across resolution ranges, may be switched off for testing purposes, with the -z option.

Correlation against a reference data set (-r <reference> option)

The correlation of the experimental data set against the user-supplied reference data is shown in the lines starting with r. To prepare a reference data set if the refinement was done with phenix.refine, one could use e.g.

mtz2various hklin 2bn3_refine_001.mtz hklout temp.hkl <<eof
OUTPUT USER *
LABIN FC=F-model PHIC=PHIF-model
END
eof

- the column corresponding to PHIC will not be used by xdscc12. Alternatively,

sftools
read mymodel_001.mtz
write temp.hkl format(3i5,f10.3) col F-model
y
quit

Reference data with anomalous signal (additional -s option)

The correlation of the anomalous difference of the experimental data set against the anomalous signal of the user-supplied reference data is shown in the lines starting with s. A simple way to obtain Fcalc(+) and Fcalc(-) is to run phenix.refine with options (in case of S as anomalous scatterer)

refinement.input.xray_data.labels="F(+),SIGF(+),F(-),SIGF(-),merged"  refinement.refine.anomalous_scatterers.group.selection="element S" strategy=individual_sites+individual_adp+group_anomalous+occupancies

and then

sftools <<eof
read mymodel_001.mtz
write anom-reference.hkl format(3i5,2f10.3) col "F-model(+)" "F-model(-)"
y
quit
eof

in which case sftools outputs only the acentric reflections - only those have anomalous differences. XDSCC12 then has to be run with the -s -r anom-reference.hkl option.

See also

A complete description of how to process serial crystallography data with XDS/XSCALE is given in SSX.

A program that implements the method of Brehm and Diederichs (2014) and theory of Diederichs (2017) is xscale_isocluster.

To remove bad frames from a XDS_ASCII.HKL file, you can use the program exclude_data_range_from_XDS_ASCII.HKL, or re-INTEGRATE with the keyword EXCLUDE_DATA_RANGE in XDS.INP.