Changes

From XDSwiki
Jump to navigationJump to search
10,513 bytes added ,  07:41, 16 October 2021
m
→‎The script: improve wording
Line 1: Line 1: −
This script generates XDS.INP based on a list of frame names supplied on the commandline. It currently works for MarCCD, ADSC and Pilatus detectors; since this is just a bash script, extension to other detectors is very easy.
+
This script generates XDS.INP based on a list of frame names supplied on the commandline. It currently works for MarCCD, ADSC, Pilatus, Eiger, some Rigaku and one Bruker detector(s); since this is just a bash script, extension to other detectors should be very easy.
 
  −
On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required (I think). These are a part of the (larger, but also free) [http://developer.apple.com/tools/xcode Xcode] package.
      
== Usage ==
 
== Usage ==
    
Usage is just (don't forget the quotation marks!):
 
Usage is just (don't forget the quotation marks!):
  generate_XDS.INP "frms/mydata_1_???.img"
+
  generate_XDS.INP "/home/myname/frms/mydata_1_???.img"
 
XDS [http://strucbio.biologie.uni-konstanz.de/~dikay/XDS_html_doc/html_doc/xds_parameters.html#NAME_TEMPLATE_OF_DATA_FRAMES= supports] bzip2-ed frames. Thus, when specifying the frame name parameter of the script, you should leave out any .bz2 extension.
 
XDS [http://strucbio.biologie.uni-konstanz.de/~dikay/XDS_html_doc/html_doc/xds_parameters.html#NAME_TEMPLATE_OF_DATA_FRAMES= supports] bzip2-ed frames. Thus, when specifying the frame name parameter of the script, you should leave out any .bz2 extension.
 +
 +
For improved interaction with [[XDSGUI]], it is advantageous to provide an ''absolute'' filename for the data files - one that starts with a slash ("/").
    
== The script ==
 
== The script ==
    
<pre>
 
<pre>
 +
 
#!/bin/bash                                                                   
 
#!/bin/bash                                                                   
 
# purpose: generate XDS.INP                                                   
 
# purpose: generate XDS.INP                                                   
 
#                                                                             
 
#                                                                             
# tested with some datasets from ALS, SSRL, SLS, ESRF and BESSY; only MARCCD, ADSC/SMV, PILATUS detectors;  
+
# tested with some datasets from ALS, SSRL, SLS, ESRF, BESSY, SPring-8 and PF; only MAR, ADSC/SMV, PILATUS, Eiger, RAXIS (in-house), Bruker (PHOTON II) detectors;  
 
# for other detectors, values marked with XXX must be manually filled in.                                   
 
# for other detectors, values marked with XXX must be manually filled in.                                   
 
#                                                                                                           
 
#                                                                                                           
Line 47: Line 48:  
# revision 0.30 . Keitaro 3/2013 - for ADSC: write all possible beam center conventions in XDS.INP as comments
 
# revision 0.30 . Keitaro 3/2013 - for ADSC: write all possible beam center conventions in XDS.INP as comments
 
# revision 0.31 . Keitaro 3/2013 - add comment for reversed phi for SPring-8
 
# revision 0.31 . Keitaro 3/2013 - add comment for reversed phi for SPring-8
REVISION="0.31 (06-Mar-2013)"                                                                              
+
# revision 0.32 . Keitaro 3/2013 - add RAXIS support. only tested with RAXIS IV++ and VII.
 +
# revision 0.33 . Keitaro 5/2013 - automatically set ROTATION_AXIS=-1 0 0 for SPring-8 BL32XU/41XU/44XU beamlines based on detector serial numbers.
 +
# revision 0.34 . Keitaro 5/2013 - recognize ADSC detectors in Photon Factory and choose correct beam center convention based on detector serial numbers.
 +
# revision 0.35 . KD 6/2013 - reduce 7000 to 6000 for shadow detection; insert comment about *_RESOLUTION_RANGE lines
 +
# revision 0.36 . KD 6/2013 - insert NUMBER_OF_PROFILE_GRID* lines for Pilatus (suggested by C.Vonrhein)
 +
# revision 0.37 . Keitaro 10/2013 - fix for MX225HS detector on SPring-8 BL32XU (Ignore case when matching marccd in header); see also rev-0.39
 +
# revision 0.38 . KD 2/2014 - change defaults for REFINE(IDXREF) and REFINE(INTEGRATE) such that more stable results are obtained for difficult datasets
 +
# revision 0.39 . Keitaro 4/2014 - automatically set ROTATION_AXIS=-1 0 0 for MX225HS at SPring-8 BL32XU.
 +
# revision 0.40 . Jan Gebauer /KD 4/2014 - simple implementation of MAR345 detector
 +
# revision 0.41 . recognize header starting with R-AXIS instead of RAXIS
 +
# revision 0.42 . Keitaro 5/2014 automatically set ROTATION_AXIS=-1 0 0 for Q315 at SPring-8 BL38B1.
 +
# revision 0.43 . Keitaro 5/2014 add experimental support of dTREK format (raxis_smv)
 +
# revision 0.44 . Keitaro 5/2014 automatically set ROTATION_AXIS=-1 0 0 for PILATUS3 at SPring-8 BL41XU.
 +
# revision 0.45 . KD cope with blanks in filenames
 +
# revision 0.46 . Keitaro 6/2014 automatically set ROTATION_AXIS=-1 0 0 for Mar225 at SPring-8 BL26B2.
 +
# revision 0.47 . Keitaro 7/2014 more generic dTREK format support (Saturn and RAXIS)
 +
# revision 0.48 . Kip Guja 11/2014 add detector serial number for ALS 5.0.2 to beam center convention 1
 +
# revision 0.49 . Nobuhisa 2/2015 add  detector serial number for AichiSR BL2S1 to beam center convention 1
 +
# revision 0.50 . KD 03/2015 workaround for Mar-1 change of parameter name "DISTANCE" to "POSITION" in REFINE(*) keywords
 +
# revision 0.51 . Keitaro 03/2015 add .gz and .xz support and remove limitation - frame numbers can start with any.
 +
# revision 0.52 . Keitaro 05/2015 fix ADSC beam center convention for SPring-8 and DET_SN acquisition for PILATUS (didn't work on Mac)
 +
# revision 0.53 . KD add ADSC beam center convention for APS Argonne but only as commented line in XDS.INP (not detector serial no)
 +
# revision 0.54 . KD add ADSC S/N 911 for APS Argonne, and fix spurious output arising from THETADISTANCE (?!)
 +
# revision 0.55 . KD add ADSC S/N 446 for APS, and check w/ 12 datasets from data.sbgrid.org. No rule found: S/N 916 @ APS 24_ID_E !
 +
# revision 0.56 . Keitaro 12/2015 show error message when user's input didn't match any files
 +
# revision 0.57 . KD 12/2015 start to take care of vertical ROTATION_AXIS at Diamond I24 - for now only introduce comment
 +
# revision 0.58 . Keitaro 01/2016 fix for dTREK image: take 'rotation axis' information from header
 +
# revision 0.59 . KD 04/04/2016 check for ADSC detector _after_ dtrek detector, to correct wrong choice for https://zenodo.org/record/45756
 +
# revision 0.60 . KD 04/04/2016 ADSC detector SN=458 at APS 19-ID has reverse phi (https://zenodo.org/record/45756)
 +
# revision 0.61 . Keitaro 10/04/2016 Add Eiger hdf5 support (may be incomplete; UNTRUSTED_RECTANGLE=s not set) NEED h5dump.
 +
# revision 0.62 . Keitaro 11/04/2016 Can give foo_master.h5 instead of foo_??????.h5.
 +
# revision 0.63 . Keitaro 13/04/2016 Set UNTRUSTED_RECTANGLE=s for EIGER 9M and 16M (KD).
 +
# revision 0.64 . KD 16/06/2016 reverse phi @APS 19ID (reported by Wolfram Tempel)
 +
# revision 0.65 . Keitaro 07/09/2016 Fix for "too many arguments" problem in ls
 +
# revision 0.66 . KD 15/09/2016 add Bruker PHOTON II with .cbf frames
 +
# revision 0.67 . KD 02/10/2016 add BM30A (ADSC SERIAL 924) reverse_phi
 +
# revision 0.68 . KD 24/10/2016 add -H option (follow symlinks) to "find" command (thanks to Jan Gebauer!)
 +
# revision 0.69 . KD 04/11/2016 add CMOS-1 MBC Detector at ALS 4.2.2
 +
# revision 0.70beta . KD 08/12/2016 ROTATION_AXIS=0 -1 0 at Diamond I24; depending on CBF header
 +
# revision 0.70 . KD 12/01/2017 remove error message if h5dump does not find /entry/sample/transformations/omega/vector
 +
# revision 0.71 . KD 27/02/2017 implement rule for S/N 916 @ APS 24_ID_E
 +
# revision 0.72 . KD 8/03/2017 fix nframes lookup in Eiger master file
 +
# revision 0.73 . KD 18/05/2017 for Andrey Nascimento: add Pilatus 2M S/N 24-0109 with ROTATION_AXIS=-1 0 0
 +
# revision 0.74 . Keitaro 02/08/2017 Add PILATUS3 6M, S/N 60-0127 at CHESS F1 with  ROTATION_AXIS=-1 0 0
 +
# revision 0.75 . KD 30/08/2017 reversed ORGX and ORGY for marCCD @ BM14 (Indian beamline @ ESRF)
 +
# revision 0.76 . KD 4/09/2017 include POSITION into REFINE(IDXREF) because latest XDS is more robust. Add comments to keywords.
 +
# revision 0.77 . KD 19/12/2017 obtain QX QY from CBF header.
 +
# revision 0.78 . KD 21/12/2017 if possible and sensible, provide LIB= line with hardcoded /usr/local/lib64/dectris-neggia.so .
 +
# revision 0.79 . KD 16/01/2018 read OVERLOAD from Pilatus miniCBF header instead of fixing at 1048576
 +
# revision 0.80 . KD 13/02/2018 remove DISTANCE keyword from REFINE() list; remove POSITION from REFINE(IDXREF)
 +
# revision 0.81 . KD 21/02/2018 when encountering CBF files from Eiger (ESRF), treat as Pilatus detector
 +
# revision 0.82 . KD 01/03/2018 STARTING_ANGLE for MarCCD/Pilatus/PHOTON, enabling to use dials.rs_mapper with spot2pdb.
 +
# revision 0.83 . KD 25/06/2018 for ADSC detector #458 at APS BM19, revert the definition of ROTATION_AXIS=-1 0 0. See "Beamline notes" in this wiki.
 +
# revision 0.84 . KD 10/10/2018 implement Pilatus detector with d*TREK header
 +
# revision 0.85 . Jie Nan 09/01/2019 STARTING_ANGLE for Eiger
 +
# revision 0.86 . Keitaro 03/05/2019 Add PILATUS3 6M, S/N 60-0123 at SSRF BL18U-1 with ROTATION_AXIS=-1 0 0
 +
# revision 0.87 . KD 12/10/2019 Add PILATUS XXX, S/N XX-XXX at SSRF BL19U1 and MarCCD detector #43 at BL17B1 with ROTATION_AXIS=-1 0 0
 +
# revision 0.88 . KD 16/10/2019 fixes for SSRF, add "-maxdepth 1" to "find -H ..."
 +
# revision 0.89 . KD 21/10/2019 add ADSC S/N 905 at ALS 8.2.1, S/N 928 at Australian Synchrotron MX2 beamline; final SSRF fixes
 +
# revision 0.90 . KD 25/10/2019 add OLDMAR detector type. Tested w/ SBGrid data set 6. Anomalous signal may have wrong hand!
 +
# revision 0.91 . KD 16/01/2020 Allow negative starting angle for Eiger (found -33 at SLS !).
 +
# revision 0.92 . KD 27/02/2020 read *_master.h5 from Diamond Light Source
 +
# revision 0.93 . KD 13/03/2020 print out 2theta for MarCCD (DETECTOR_*_AXIS can be derived from this)
 +
# revision 0.94 . KD 16/03/2020 bugfix for Bruker-cbf to make bc accept e.g. 3.1e-005 by using awk printf "%.5f" instead of awk print
 +
# revision 0.95 . KD 29/07/2020 fix DLS Eiger HDF5 variant OSCILLATION_RANGE, STARTING_ANGLE. Attention: DLS Eiger variant needs h5dump 1.10 for OVERLOAD!
 +
# revision 0.96 . KD 03/10/2020 make script echo detector serial number if inverted ROTATION_AXIS is detected.
 +
# revision 0.97 . KD 19/10/2020 add UNTRUSTED_RECTANGLEs for Eiger2 (which has a few pixels less than Eiger (thanks to Andreas Förster)
 +
# revision 0.98 . Thomas Hauß (HZB) 06/11/2020 switch Python print syntax to Python3
 +
# revision 0.99 . Aaron Finke (CHESS) 07/11/2020 add ROTATION_AXIS=-1 0 0 for EIGER2 16M detector at MX beamline ID7B2 (CHESS)
 +
# revision 1.00 . Gleb Bourenkov 10/11/2020 add ROTATION_AXIS=0 -1 0 for beamline P14 (EMBL Hamburg) detectors Eiger 16M S/N E-32-0107; previously Eiger 4M S/N E-08-0107 and PILATUS 6M-F, S/N 60-0115-F
 +
# revision 1.01 . KD 07/12/2020 ROTATION_AXIS=0 -1 0 for Pilatus3 2M, S/N 24-0118 at ID23-2 (http://www.globalphasing.com/autoproc/wiki/index.cgi?BeamlineSettings)
 +
# revision 1.02 . KD 11/01/2021 recognize mar555 detector as mar345 (thanks to Thomas Hauß, HZB)
 +
# revision 1.03 . Feng YU 18/7/2021 fix/expand information about Shanghai Synchrotron Radiation Facility (SSRF)
 +
# revision 1.04 . Zhipu Luo 13/08/2021 extract ORGX,ORGY for electron diffraction from SMV header if the wavelength value starts with 0.0
 +
# revision 1.05 . KD specialcase extraction of number of images for BNL detectors E-32-0101 and E-18-0104
 +
# revision 1.06 . KD fix URLs in output. Availability of Apple M1 processor dectris-neggia-Apple-arm64.so . LC_ALL=C .
 +
REVISION="1.06 (15-OCT-2021)"
 +
 
 
#                                                                                                             
 
#                                                                                                             
# usage: e.g. generate_XDS.INP "frms/mydata_1_???.img"                                                       
+
# usage: e.g. generate_XDS.INP "/file/system/frms/mydata_1_???.img"                                                       
 
# make sure to have the two quotation marks !                                                                 
 
# make sure to have the two quotation marks !                                                                 
 
# the ? are wildcards for the frame numbers.                                                                 
 
# the ? are wildcards for the frame numbers.                                                                 
#                                                                                                           
  −
# limitations:                                                                                             
  −
# - frame numbers are assumed to start with 1 and run consecutively                                         
   
#                                                                                                             
 
#                                                                                                             
 
# known problems:                                                                                             
 
# known problems:                                                                                             
 
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
 
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
 
# - the same might be a problem for MAR headers, too (not sure about this)  
 
# - the same might be a problem for MAR headers, too (not sure about this)  
# - on Mac OS X, Xcode (from http://developer.apple.com/tools/xcode) might be needed - not sure about this                             
+
# - on Mac OS X, the Xcode command line tools (from https://developer.apple.com/download/more/) are needed.                           
 
#                                                                                                                 
 
#                                                                                                                 
 
# notes for debugging of the script:                                                                             
 
# notes for debugging of the script:                                                                             
# - add the -v option to the first line, to see where an error occurs                                             
+
# - add the -x option to the first line, to see where an error occurs                                             
 
# - comment out the removal of tmp1 and tmp2 in the last line                                                     
 
# - comment out the removal of tmp1 and tmp2 in the last line                                                     
 
#                                                                                                                 
 
#                                                                                                                 
Line 69: Line 144:  
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP                                 
 
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP                                 
 
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then                                           
 
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then                                           
   echo usage: generate_XDS.INP \"frms/mydata_1_???.img\"  \(_with_ the quotation marks!\)                       
+
   echo usage: generate_XDS.INP \"/file/system/frms/mydata_1_???.img\"  \(_with_ the quotation marks!\)                       
 
   echo if the frames are compressed with bzip2, leave out the .bz2 extension!                                     
 
   echo if the frames are compressed with bzip2, leave out the .bz2 extension!                                     
 
   exit                                                                                                           
 
   exit                                                                                                           
 
fi                                                                                                               
 
fi                                                                                                               
 +
# make sure the locale does not interfere with e.g. awk calculations:
 +
LC_ALL="C";export LC_ALL
 
#                                                                                                                 
 
#                                                                                                                 
 
# defaults:                                                                                                       
 
# defaults:                                                                                                       
#                                                                                                              
+
#      
 +
# conversion radian / degrees:
 +
DEGTOR=57.2957795
 +
                                                                                                       
 
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"                                                         
 
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"                                                         
 +
REVERSE_PHI="no"
 
ORGX=XXX                                                                                                         
 
ORGX=XXX                                                                                                         
 
ORGY=XXX                                                                                                         
 
ORGY=XXX                                                                                                         
Line 86: Line 167:  
NX=XXX                                                                                                           
 
NX=XXX                                                                                                           
 
NY=XXX                                                                                                           
 
NY=XXX                                                                                                           
SENSOR_THICKNESS=0                                                                                              
+
SENSOR_THICKNESS=0  
 +
TRUSTED_REGION="0.0 1.2 ! partially use corners of detector (0 1.4143: use all pixels)"                                                                                         
 +
# default MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT
 +
MNOPIAS=3
 +
# default DIRECTION_OF_DETECTOR_X-AXIS
 +
DIRECTION_OF_DETECTOR_X_AXIS="1 0 0"
 +
# default FRACTION_OF_POLARIZATION
 +
pol_frac=0.98
 +
STARTING_ANGLE=0
 +
dtrek_det=""
 +
SEPMIN=7.0
 +
CLUSTER_RADIUS=3.5
 +
REFINE_CORRECT="CELL BEAM ORIENTATION AXIS POSITION ! Default is: refine everything"
 +
 
 +
dname=`echo "$1" | xargs dirname`
 +
test "${dname}" == "" && dname="."
 +
bname=`echo "$1" | xargs basename`
 
# see how we are called:                                                                                         
 
# see how we are called:                                                                                         
NAME_TEMPLATE_OF_DATA_FRAMES="$1"
+
NAME_TEMPLATE_OF_DATA_FRAMES="${dname}/${bname}"
 
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
 
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
 
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II  
 
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II  
/bin/ls -C1 $1 $1.bz2 2>/dev/null | egrep -v "_00000.cbf|_000.img" > tmp1 || exit 1
+
# cope with blanks in directory / file name
 +
IFS=$'\n'
 +
find -H $dname -maxdepth 1 -name "$bname" -or -name "${bname}.bz2" -or -name "${bname}.gz" -or -name "${bname}.xz" | egrep -v "_00000.cbf|_000.img" | sort > tmp1
 +
if [ ! -s tmp1 ]
 +
then
 +
echo "Error! No files matched: $1"
 +
rm -f tmp1
 +
exit 1
 +
fi
 +
unset IFS
 +
# we can continue - the frames are found
   −
# we can continue - the frames are found
+
if echo $NAME_TEMPLATE_OF_DATA_FRAMES | grep '_master.h5$' > /dev/null; then
 +
NAME_TEMPLATE_OF_DATA_FRAMES=`echo "$NAME_TEMPLATE_OF_DATA_FRAMES" | sed -e 's/_master.h5$/_??????.h5/'`
 +
else
 +
# Find the first '?' position and the number of '?' to determine DATA_RANGE=.
 +
pos1=`echo "$NAME_TEMPLATE_OF_DATA_FRAMES" | awk '{print index($0, "?")}'`
 +
pos2=`echo "$NAME_TEMPLATE_OF_DATA_FRAMES" | sed -e "s/[^\?]//g" | awk '{print length+'$pos1' - 1}'`
 +
data_first=`cut -b $pos1-$pos2 tmp1 | head -n1 | bc`
 +
data_last=`cut -b $pos1-$pos2 tmp1 | tail -n1 | bc`
 +
DATA_RANGE="$data_first $data_last"
 +
echo DATA_RANGE=$DATA_RANGE
 +
 +
# set SPOT_RANGE to first half of DATA_RANGE
 +
data_num=`wc -l tmp1 | awk '{print $1}'`
 +
data_half=`echo "scale=0; $data_num/2" | bc -l`                       
 +
data_half=`echo "if ($data_half<=1) 1;if ($data_half>1) $data_half" | bc -l`
 +
spot_last=`echo "scale=0; $data_first+$data_half-1" | bc -l`
 +
SPOT_RANGE="$data_first $spot_last"
 +
fi
   −
# set upper limit of DATA_RANGE to number of frames (see "limitations" above)
+
FIRSTFRAME=`head -1 tmp1`
DATA_RANGE=`wc -l tmp1 | awk '{print $1}'`                                 
+
echo $FIRSTFRAME | grep "\.h5$" && is_h5=1 || is_h5=0
   −
# set upper limit of SPOT_RANGE to half of DATA_RANGE, but not less than 1
+
# find out detector type
SPOT_RANGE=`echo "scale=0; $DATA_RANGE/2" | bc -l`                       
+
DET=XXX             
SPOT_RANGE=`echo "if ($SPOT_RANGE<1) 1;if ($SPOT_RANGE>1) $SPOT_RANGE" | bc -l`
+
IFS=$'\n'   
 +
echo $FIRSTFRAME | grep -q '\.bz2$' && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
 +
# for mac/linux compatibility. zcat foo.gz doesn't work on mac.
 +
echo $FIRSTFRAME | grep -q '\.gz$' && zcat < $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
 +
echo $FIRSTFRAME | grep -q '\.xz$' && xzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
   −
echo DATA_RANGE=1 $DATA_RANGE
+
unset IFS
 +
if [ "$is_h5" == 0 ]; then
 +
strings $FIRSTFRAME > tmp2 
 +
# TODO: whenever FIRSTFRAME is used below, it should be copied to tmp2 (using IFS as above), and tmp2 should be used instead
 +
# this was done for "mccd", but still has to be done for the "raxis" detector types
 +
# the reason is that FIRSTFRAME may contain a blank, which makes some commands fail                                               
 +
egrep -qi 'marccd|Corrected' tmp2 && DET=mccd                               
 +
grep -q PILATUS tmp2            && DET=pilatus
 +
grep -iq Eiger tmp2              && DET=pilatus                           
 +
head -n1 tmp2 | grep -q "^RAXIS" && DET=raxis
 +
head -n1 tmp2 | grep -q "^R-AXIS" && DET=raxis
 +
grep -q "^SOURCE_WAVELENGTH= *1" tmp2 && DET=dtrek
 +
grep -q BEAM_CENTER_X tmp2      && DET=adsc                               
 +
grep -q mar345 tmp2 && DET=MAR345
 +
# rev. 1.02: in one case, mar555 data could be processed pretending it is mar345, so:
 +
grep -q mar555 tmp2 && DET=MAR345
 +
grep -q BRUKER tmp2 && grep -q CBF tmp2  && DET=Bruker-cbf 
 +
grep -q CMOS1 tmp2 && DET=adsc-CMOS1 
 +
grep -q MARCONTROL tmp2 && DET=OLDMAR 
 +
grep -q "WAVELENGTH=0.0" tmp2 && DET=experimental-ED                                   
 +
else
 +
h5dump -d "/entry/instrument/detector/description" $FIRSTFRAME | grep -i Eiger > /dev/null && DET=eiger
 +
fi
   −
# find out detector type
+
# identify other detector types in the same way  
DET=XXX               
  −
FIRSTFRAME=`head -1 tmp1`
  −
echo $FIRSTFRAME | grep -q bz2 && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
  −
strings $FIRSTFRAME > tmp2                                                 
  −
egrep -q 'marccd|Corrected' tmp2 && DET=mccd                               
  −
grep -q PILATUS tmp2            && DET=pilatus                             
  −
grep -q BEAM_CENTER_X tmp2      && DET=adsc                               
  −
# identify other detector types in the same way (MAR IP would be straightforward)
      
# parse ASCII header of first frame
 
# parse ASCII header of first frame
Line 123: Line 265:  
elif [ "$DET" == "mccd" ]; then  
 
elif [ "$DET" == "mccd" ]; then  
 
   echo Data from a MarCCD detector
 
   echo Data from a MarCCD detector
 +
# http://www.sb.fsu.edu/~xray/Manuals/marCCD165header.html has header information
 
                                    
 
                                    
 
   DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
 
   DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
 
   SENSOR_THICKNESS=0.01                                           
 
   SENSOR_THICKNESS=0.01                                           
 
   # use first frame of dataset to obtain parameters               
 
   # use first frame of dataset to obtain parameters               
 +
 +
  # Check detector serial number and recognize beamline for reversed-phi setting.
 +
  # Known detectors for reversed-phi in SPring-8: 24: BL26B2 Mar225, 31: BL32XU MX225HE, 38: BL44XU MX225HE, 42: BL44XU MX300HE, 40: BL41XU MX225HE, 106: BL32XU MX225HS
 +
  # same for SSRF: BL17B1 rayonix MX300. As on 2019-10-13, this also needs doubling of ORGX and ORGY. But the beamline staff (Wenming) wants to fix the header.
 +
  REVERSEPHI_SNs="
 +
24
 +
31
 +
38
 +
40
 +
42
 +
43
 +
106
 +
"
 +
  # get detector serial number and check if it is included in the list
 +
  DET_SN=`grep "Detector Serial Number =" tmp2 | sed "s/Detector Serial Number = //"`
 +
  if echo "${DET_SN}${REVERSEPHI_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
 +
    REVERSE_PHI="yes"
 +
    echo inverted ROTATION_AXIS since Detector Serial Number is "${DET_SN}"
 +
  fi
    
   # offsets are documented; values can be found in mccd_xdsparams.pl script
 
   # offsets are documented; values can be found in mccd_xdsparams.pl script
 +
  IFS=$'\n'
 +
  cp $FIRSTFRAME tmp2
 +
  unset IFS
 
   let SKIP=1024+80                                                         
 
   let SKIP=1024+80                                                         
   NX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
+
   NX=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
 
   let SKIP=$SKIP+4                                                                         
 
   let SKIP=$SKIP+4                                                                         
   NY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
+
   NY=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
    
   let SKIP=1720
 
   let SKIP=1720
   DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
+
   DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
 
   DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`                                     
 
   DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`                                     
                                                                                                         
+
 
 +
# Mar 12, 2020 KD
 +
  let SKIP=1724
 +
  TWOTHETA=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
 +
  TWOTHETA=`echo "scale=3; $TWOTHETA/1000" | bc -l`
 +
  echo 2THETA= $TWOTHETA
 +
 
 +
  let SKIP=1024+256+128+256+44
 +
  STARTING_ANGLE=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
 +
  STARTING_ANGLE=`echo "scale=2; $STARTING_ANGLE/1000" | bc -l `
 +
  echo STARTING_ANGLE= $STARTING_ANGLE 
 +
                                                                                                     
 
   let SKIP=1024+256+128+256+4                                                                             
 
   let SKIP=1024+256+128+256+4                                                                             
   ORGX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')             
+
   ORGX=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')             
 
   ORGX=`echo "scale=2; $ORGX/1000" | bc -l `                                                               
 
   ORGX=`echo "scale=2; $ORGX/1000" | bc -l `                                                               
 
   let SKIP=$SKIP+4                                                                                         
 
   let SKIP=$SKIP+4                                                                                         
   ORGY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')             
+
   ORGY=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')             
 
   ORGY=`echo "scale=2; $ORGY/1000" | bc -l `                                                               
 
   ORGY=`echo "scale=2; $ORGY/1000" | bc -l `                                                               
 +
# fixed Aug 30, 2017 after IUCr2017 @ Hyderabad
 +
  if [ "$DET_SN" == "4" ]; then
 +
    TEMP=$ORGY
 +
    ORGY=$ORGX
 +
    ORGX=$TEMP
 +
    echo reversed ORGX and ORGY for marCCD @ ESRF BM14
 +
  fi
    
   let SKIP=1024+736
 
   let SKIP=1024+736
   OSCILLATION_RANGE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')  
+
   OSCILLATION_RANGE=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')  
 
   OSCILLATION_RANGE=`echo "scale=3; $OSCILLATION_RANGE/1000" | bc -l`   
 
   OSCILLATION_RANGE=`echo "scale=3; $OSCILLATION_RANGE/1000" | bc -l`   
 
                                                                                                  
 
                                                                                                  
 
   let SKIP=1024+256+128+256+128+4                                                                 
 
   let SKIP=1024+256+128+256+128+4                                                                 
   QX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')       
+
   QX=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')       
 
   QX=`echo "scale=10; $QX/1000000" |bc -l `                                                       
 
   QX=`echo "scale=10; $QX/1000000" |bc -l `                                                       
 
   let SKIP=$SKIP+4                                                                               
 
   let SKIP=$SKIP+4                                                                               
   QY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')       
+
   QY=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')       
 
   QY=`echo "scale=10; $QY/1000000" |bc -l `                                                       
 
   QY=`echo "scale=10; $QY/1000000" |bc -l `                                                       
    
   let SKIP=1024+256+128+256+128+128+12
 
   let SKIP=1024+256+128+256+128+128+12
   X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
+
   X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 tmp2 | head -1 | awk '{print $2}')
 
   X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`                                     
 
   X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`                                     
   Line 184: Line 367:  
       # find NX, QX, ORGX and ORGY:
 
       # find NX, QX, ORGX and ORGY:
 
       NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
 
       NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
 +
      NY=`grep SIZE2 tmp2 | tail -1 | sed s/SIZE2=//`                                           
 
       QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
 
       QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
# FIXME - next 2 lines should be done properly, from header
  −
      NY=$NX                                             
   
       QY=$QX                                               
 
       QY=$QX                                               
 
       BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
 
       BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
 
       BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
 
       BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -  
+
   
 
       COMMENT_ORGXY="
 
       COMMENT_ORGXY="
 
! Following are possible beam center interpretations for ADSC detectors"
 
! Following are possible beam center interpretations for ADSC detectors"
# at ESRF, PF, and ... (pls fill in!) the following should be used:         
+
# at ESRF, PF, ALS 5.0.2, AS MX2 and ... (pls fill in!) the following should be used:         
       ORGX=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`              
+
       ORGX1=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`
       ORGY=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`              
+
       ORGY1=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`
       echo - at ESRF,PF BLs use: ORGX=$ORGX ORGY=$ORGY                    
+
       echo - at ESRF, PF, ALS 8.2.1, APS Argonne BLs use: ORGX=$ORGX1 ORGY=$ORGY1                    
 
       COMMENT_ORGXY="${COMMENT_ORGXY}
 
       COMMENT_ORGXY="${COMMENT_ORGXY}
! ORGX= $ORGX ORGY= $ORGY ! For ESRF,PF,..."
+
! ORGX= $ORGX1 ORGY= $ORGY1 ! For ESRF, PF, APS, AS MX2 ..."
 
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
 
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
       ORGX=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `                                                      
+
       ORGX2=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `
       ORGY=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `                                                          
+
       ORGY2=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
       echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX ORGY=$ORGY                                                          
+
       echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX2 ORGY=$ORGY2                                                          
 
       COMMENT_ORGXY="${COMMENT_ORGXY}
 
       COMMENT_ORGXY="${COMMENT_ORGXY}
! ORGX= $ORGX ORGY= $ORGY ! For ALS 5.0.3,.."
+
! ORGX= $ORGX2 ORGY= $ORGY2 ! For ALS 5.0.3,.."
 
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...  
 
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...  
       ORGX=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `                                                          
+
# this alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
       ORGY=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `                                                      
+
       ORGX3=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
       echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX ORGY=$ORGY - this is written to XDS.INP                            
+
       ORGY3=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `
 +
       echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX3 ORGY=$ORGY3 - this is written to XDS.INP if beamline is not detected
 +
      COMMENT_ORGXY="${COMMENT_ORGXY}
 +
! ORGX= $ORGX3 ORGY= $ORGY3 ! For ALS 8.2.2,.."
 +
# this 4th alternative convention should be used at the following beamlines (pls complete the list): SPring-8, ...
 +
      ORGX4=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `
 +
      ORGY4=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `
 +
      echo - at e.g. SPring-8 use: ORGX=$ORGX4 ORGY=$ORGY4
 
       COMMENT_ORGXY="${COMMENT_ORGXY}
 
       COMMENT_ORGXY="${COMMENT_ORGXY}
! ORGX= $ORGX ORGY= $ORGY ! For ALS 8.2.2,.."
+
! ORGX= $ORGX4 ORGY= $ORGY4 ! For SPring-8,.."
# the latter alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
+
 
 +
      # Decision of beam center convention based on detector serial numbers.
 +
      DET_SN=`grep DETECTOR_SN tmp2 | sed -e "s/DETECTOR_SN=//"`
 +
      echo Detector serial number is $DET_SN
 +
      # For convention 1; Known PF detectors = 449: NW12A Q210, 472: NE3A Q270, 474: BL17A Q270, 912: BL5A Q315, 923: ALS BL5.0.2 Q315, 933: AichiSR BL2S1 Q315, 916: APS 24 IDE, 928: AS MX2
 +
      ORG1_SNs="
 +
449
 +
472
 +
474
 +
912
 +
923
 +
933
 +
911
 +
446
 +
916
 +
905
 +
928
 +
"
 +
      ORG4_SNs="
 +
915
 +
"
 +
      if echo "${DET_SN}${ORG1_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
 +
      ORGX=$ORGX1
 +
      ORGY=$ORGY1
 +
      echo the following was chosen based on detector serial number:
 +
      elif echo "${DET_SN}${ORG4_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
 +
      ORGX=$ORGX4
 +
      ORGY=$ORGY4
 +
      echo the following was chosen based on detector serial number:
 +
      else
 +
      ORGX=$ORGX3
 +
      ORGY=$ORGY3
 +
      echo the following default was chosen because the detector serial number was not special-cased:
 +
      fi
 +
 
 +
      # Check detector serial number and recognize beamline for reversed-phi setting.
 +
      # Known detectors for reversed-phi in SPring-8: 915: BL38B1 Q315; APS 19-ID: 458; BM30A: 924
 +
      # 928 is at Australian Beamline MX2
 +
      # revision 0.83 of this script removes 458 from the list!
 +
      REVERSEPHI_SNs="
 +
915
 +
924
 +
928
 +
"
 +
      if echo "${DET_SN}${REVERSEPHI_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
 +
        REVERSE_PHI="yes"
 +
        echo inverted ROTATION_AXIS since detector serial number is ${DET_SN}
 +
      fi
 +
 
 
       # find DETECTOR_DISTANCE and OSCILLATION_RANGE:                                                                                 
 
       # find DETECTOR_DISTANCE and OSCILLATION_RANGE:                                                                                 
       DETECTOR_DISTANCE=`grep DISTANCE tmp2 | sed s/DISTANCE=//`                                                                     
+
       DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=//`                                                                     
       OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`                                                                  
+
       OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`      
 +
                                                           
 +
elif [ "$DET" == "adsc-CMOS1" ]; then
 +
  DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
 +
  echo Data from CMOS1 MBC detector.
 +
  SENSOR_THICKNESS=0.01                                                       
 +
  sed s/\;// tmp2 > tmp1                                                     
 +
  mv tmp1 tmp2                                                               
 +
  X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`
 +
  NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
 +
  QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
 +
  NY=`grep SIZE2 tmp2 | tail -1 | sed s/SIZE2=//`                                           
 +
  QY=$QX                                             
 +
  ORGX=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
 +
  ORGY=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
 +
  REVERSE_PHI="yes"
 +
  DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=/-/`                                                                   
 +
  OSCILLATION_RANGE=`grep OMEGA_DELTA tmp2 | sed s/OMEGA_DELTA=//` 
 +
  DIRECTION_OF_DETECTOR_X_AXIS="-1 0 0"
 +
                                                               
 +
elif [ "$DET" == "experimental-ED" ]; then
 +
  DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000  ! OVERLOAD is really unknown "
 +
  echo --- Electron diffraction data in SMV format!
 +
  SENSOR_THICKNESS=0.01                                                       
 +
  sed s/\;// tmp2 > tmp1                                                     
 +
  mv tmp1 tmp2                                                               
 +
  X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`
 +
  NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
 +
  QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
 +
  NY=`grep SIZE2 tmp2 | tail -1 | sed s/SIZE2=//`                                           
 +
  QY=$QX 
 +
  BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
 +
  BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
 +
  QXY=`echo "scale=1; $QX*($BEAM_CENTER_X+$BEAM_CENTER_Y)" | bc -l`
 +
      if [ "$QXY" "<" "$BEAM_CENTER_X" ] ; then
 +
      ORGX=`echo "scale=2; $BEAM_CENTER_Y/$QX" | bc -l`
 +
      ORGY=`echo "scale=2; $BEAM_CENTER_X/$QX" | bc -l`
 +
      else
 +
      ORGX=`echo "scale=2; $BEAM_CENTER_X/1" | bc -l`
 +
      ORGY=`echo "scale=2; $BEAM_CENTER_Y/1" | bc -l`
 +
      fi
 +
  DETECTOR_DISTANCE=`grep ^DISTANCE tmp2 | sed s/DISTANCE=//`                                                                   
 +
  OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`
 +
  STARTING_ANGLE=`grep OSC_START tmp2 | sed s/OSC_START=//`
 +
  echo --- For TIMEPIX detector, please fix ROTATION_AXIS yourself!
 +
  DIRECTION_OF_DETECTOR_X_AXIS="1 0 0"
 +
  REFINE_CORRECT="ORIENTATION CELL AXIS BEAM ! for ED, no POSITION when CELL is refined"
    
elif [ "$DET" == "pilatus" ]; then
 
elif [ "$DET" == "pilatus" ]; then
  DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576  !PILATUS"
  −
  QX=0.172 QY=0.172                                                       
   
   echo Data from a Pilatus detector                                         
 
   echo Data from a Pilatus detector                                         
 
   sed s/#// tmp2 > tmp1                                                     
 
   sed s/#// tmp2 > tmp1                                                     
 
   mv tmp1 tmp2                                                               
 
   mv tmp1 tmp2                                                               
 +
  OVERLOAD=1048574
 +
  SEPMIN=4
 +
  CLUSTER_RADIUS=2
 +
  grep -q Count_cutoff tmp2 && OVERLOAD=`awk '/Count_cutoff/{print $2}' tmp2`
 +
  DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= $OVERLOAD  !PILATUS"
 +
  QX=0.172 QY=0.172                                                       
 +
# the default above guards against missing Pixel_size line in CBF header
 +
  grep -q Pixel_size tmp2 && QX=`awk '/Pixel_size/{print 1000*$2}' tmp2`
 +
  grep -q Pixel_size tmp2 && QY=`awk '/Pixel_size/{print 1000*$5}' tmp2`                                                   
    
       # find SENSOR_THICKNESS:
 
       # find SENSOR_THICKNESS:
Line 236: Line 526:  
       ORGY=`grep Beam_xy tmp2 | sed -e s/\(// -e s/\)// -e s/\,// | awk '{print $3}'`
 
       ORGY=`grep Beam_xy tmp2 | sed -e s/\(// -e s/\)// -e s/\,// | awk '{print $3}'`
   −
       # find DETECTOR_DISTANCE and OSCILLATION_RANGE:
+
       # find DETECTOR_DISTANCE, OSCILLATION_RANGE, and STARTING_ANGLE:
       DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
+
       DETECTOR_DISTANCE=`awk '/Detector_distance/{print $2}' tmp2`
 
       DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`
 
       DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`
   −
       OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`
+
       OSCILLATION_RANGE=`awk '/Angle_increment/{print $2}' tmp2`
 +
 
 +
      STARTING_ANGLE=`awk '/Start_angle/{print $2}' tmp2`
 +
      echo STARTING_ANGLE= $STARTING_ANGLE 
 +
 
 +
      # get detector serial number and check if it is included in the list
 +
      # Known detectors for reversed-phi in SPring-8: BL41XU PILATUS3 6M 60-0125
 +
      # Known detectors for reversed-phi in APS: 19ID PILATUS3 6M 60-0132
 +
      # Known detectors for reversed-phi at MX2 beamline (Brazilian Synchrotron National Laboratory - LNLS)
 +
      # Known detectors for reversed-phi at CHESS F1 PILATUS3 6M, S/N 60-0127
 +
      # Known detectors for reversed-phi at SSRF BL18U1 (S/N 60-0123) and BL19U1 (S/N XX-XXX) (!; 2019-10-19: staff will fix this)
 +
      DET_SN=`grep "Detector:" tmp2 | sed "s/^.*Detector: *//"`
 +
      REVERSEPHI_SNs="
 +
PILATUS3 6M, S/N 60-0125
 +
PILATUS3 6M, S/N 60-0132
 +
PILATUS 2M, S/N 24-0109
 +
PILATUS3 6M, S/N 60-0127
 +
PILATUS3 6M, S/N 60-0123
 +
"
 +
      if echo "${DET_SN}${REVERSEPHI_SNs}" | sort | uniq -d | grep [0-9] > /dev/null; then
 +
        REVERSE_PHI="yes"
 +
        echo inverted ROTATION_AXIS since detector serial number is ${DET_SN}
 +
      fi
 +
      if [ "$DET_SN" == "PILATUS XXX, S/N XX-XXX" ] ; then
 +
        REVERSE_PHI="yes"
 +
        echo inverted rotation axis at SSRF BL19U1
 +
      fi
 +
     
 +
# Diamond I24:
 +
      if [ "$DET_SN" == "PILATUS3 6M, S/N 60-0119" ] ; then
 +
        if grep -q "Oscillation_axis X.CW +SLOW" tmp2 ; then
 +
          rotation_axis="0 -1 0"
 +
          echo ROTATION_AXIS="0 -1 0" at Diamond I24
 +
        fi
 +
      fi
 +
# PETRA P14: raw data from Eiger are stored as CBF files so this is treated as Pilatus
 +
      if [ "$DET_SN" == "Dectris Eiger 16M, E-32-0107" -o "$DET_SN" == "Dectris Eiger 4M, E-08-0107" -o "$DET_SN" == "PILATUS 6M-F, S/N 60-0115-F" ] ; then
 +
        rotation_axis="0 -1 0"
 +
        echo ROTATION_AXIS="0 -1 0" at PETRA P14
 +
      fi
 +
# ESRF ID23-2:
 +
      if [ "$DET_SN" == "PILATUS3 2M, S/N 24-0118, ESRF ID23" ] ; then
 +
        rotation_axis="0 -1 0"
 +
        echo ROTATION_AXIS="0 -1 0" at ESRF ID23-2
 +
      fi
 +
 
 +
elif [ "$DET" == "eiger" ]; then
 +
nframes=`h5dump -d "/entry/instrument/detector/detectorSpecific/nimages" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
 +
DATA_RANGE="1 $nframes"
 +
SPOT_RANGE="1 `echo "scale=0; if (${nframes}<2) 1; if (${nframes}>1) ${nframes}/2"|bc -l`"
 +
 +
# find out if HDF5 from Diamond (DLS=1) or Dectris (DLS=0)
 +
  DLS=0
 +
  OVERLOAD=`h5dump -d "/entry/instrument/detector/detectorSpecific/countrate_correction_count_cutoff" $FIRSTFRAME 2>/dev/null` || DLS=1
 +
  if [ "$DLS" == 1 ]; then
 +
    echo Eiger HDF5 from Diamond
 +
# unfortunately h5dump 1.10 is required to get this right for the DLS .h5 files. This version is available at DLS but maybe not elsewhere
 +
    OVERLOAD=`h5dump -d "/entry/instrument/detector/saturation_value" $FIRSTFRAME | awk '/\(0\):/{print $2}'`
 +
  # v0.95: fix the next 2 lines by taking care of negative values with the \- , and stop after first "(0)"
 +
    OSCILLATION_RANGE=`h5dump -d "/entry/data/omega" $FIRSTFRAME | awk '/\(0\): [\-0-9]/{print $3-$2;exit}'`
 +
    STARTING_ANGLE=`h5dump    -d "/entry/data/omega" $FIRSTFRAME | awk '/\(0\): [\-0-9]/{print $2;exit}' | sed -e "s/,//"`
 +
    echo OSCILLATION_RANGE=$OSCILLATION_RANGE STARTING_ANGLE=$STARTING_ANGLE
 +
  # rotation_axis=`h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME 2>/dev/null | grep "(0):" | sed -e "s/^.*://; s/,//g"`
 +
  # the above gives -1 0 0 for DLS data instead of the correct 1 0 0, so commented out for now
 +
  else
 +
    echo Eiger HDF5 from Dectris
 +
    OVERLOAD=`h5dump -d "/entry/instrument/detector/detectorSpecific/countrate_correction_count_cutoff" $FIRSTFRAME | awk '/\(0\):/{print $2}'`   
 +
    OSCILLATION_RANGE=`h5dump -d "/entry/sample/goniometer/omega_range_average" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
 +
  # STARTING_ANGLE:  the \- was introduced in version 0.91 to allow negative values :
 +
    STARTING_ANGLE=`h5dump -d "/entry/sample/goniometer/omega_start" $FIRSTFRAME | awk '/\(0\): [\-0-9]/{print $2}'`
 +
  # /entry/sample/goniometer/omega_start is missing in some eiger2 detectors (e.g. Eiger2 9M with fw version release-2020.2.1 and SIMPLON API 1.8) (Feng YU 2021-07-18)
 +
    if [ "$STARTING_ANGLE" == "" ]; then
 +
  echo "/entry/sample/goniometer/omega_start not found, try /entry/sample/goniometer/omega"
 +
  STARTING_ANGLE=`h5dump -d /entry/sample/goniometer/omega $FIRSTFRAME | grep "(0):" | head -n 1 | awk '{print $2}' | sed -e "s/,//g"`
 +
    fi
 +
    echo "STARTING_ANGLE=$STARTING_ANGLE"
 +
  # If rotation vector set (NeXus)
 +
    rotation_axis=`h5dump -a "/entry/sample/transformations/omega/vector" $FIRSTFRAME 2>/dev/null | grep "(0):" | sed -e "s/^.*://; s/,//g"`
 +
  # EIGER2 16M CHESS ID7B2 has S/N E-32-0123 (A. Finke 2020-11-07) v0.99
 +
    SN=`h5dump -d "/entry/instrument/detector/detector_number" $FIRSTFRAME | awk '/\(0\): /{print $2}' | sed s/\"//g`
 +
    if [ "$SN" == "E-32-0123" ]; then
 +
      rotation_axis="-1 0 0"
 +
      echo CHESS ID7B2 with inverted rotation axis
 +
    fi
 +
  # SSRF BL17U1 and SSRF BL10U2 (Feng YU 2021-07-18)
 +
  # Eiger X 16M (S/N E-32-0111) was installed at SSRF BL17U1 from Oct 2017 to Feb 2021. After Feb 2021, it was moved back to SSRF BL10U2.
 +
# 2021-02-01 00:00:00 CST time stamp is 1612108800
 +
    if [ "$SN" == "E-32-0111" ]; then
 +
      collection_time=`h5dump -d "/entry/instrument/detector/detectorSpecific/data_collection_date" $FIRSTFRAME | grep "(0):" | awk '{print $2}' | sed -e "s/\.\(.*\)/CST/g; s/\"//g"`
 +
      if [ `uname -s` == "Darwin" ]; then
 +
        collection_timestamp=`date -j -f "%Y-%m-%dT%H:%M:%S%Z" $collection_time +%s`
 +
      elif [ `uname -s` == "Linux" ]; then
 +
        collection_timestamp=`date -d $collection_time +%s`
 +
      else
 +
        collection_timestamp=0
 +
      fi
 +
      if [ $collection_timestamp -eq 0 ]; then
 +
        is_rotation_axis_set=1
 +
        rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME 2>/dev/null || is_rotation_axis_set=0`
 +
        if [ "$is_rotation_axis_set" == "1" ]; then
 +
          rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME | grep "(0):" | sed -e "s/^.*://; s/,//g"`
 +
          echo "The rotation axis of SSRF BL10U2 (Eiger X 16M) is $rotation_axis"
 +
        else
 +
          rotation_axis="-1 0 0    ! Cannot determine rotation axis. SSRF BL17U1: -1 0 0; SSRF BL02U1: 0 -1 0"
 +
        fi
 +
      elif [ $collection_timestamp -ge 0 ] && [ $collection_timestamp -le 1612108800 ]; then
 +
        # SSRF BL17U1
 +
        rotation_axis="-1 0 0"
 +
        echo "SSRF BL17U1 (Eiger X 16M) with inverted rotation axis"
 +
      else
 +
        # SSRF BL10U2
 +
        is_rotation_axis_set=1
 +
        rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME 2>/dev/null || is_rotation_axis_set=0`
 +
        if [ "$is_rotation_axis_set" == "1" ]; then
 +
          rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME | grep "(0):" | sed -e "s/^.*://; s/,//g"`
 +
          echo "The rotation axis of SSRF BL10U2 (Eiger X 16M) is $rotation_axis"
 +
        else
 +
          rotation_axis="0 -1 0"
 +
          echo "SSRF BL10U2 (Eiger X 16M) with vertical rotation axis"
 +
        fi
 +
      fi
 +
    fi
 +
  # SSRF BL02U1 (Feng YU 2021-07-18)
 +
  # EIGER2 S 9M SSRF BL02U1 has S/N E-18-0121
 +
    if [ "$SN" == "E-18-0121" ]; then
 +
      is_rotation_axis_set=1
 +
      rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME 2>/dev/null || is_rotation_axis_set=0`
 +
      if [ "$is_rotation_axis_set" == "1" ]; then
 +
        rotation_axis=`h5dump -d "/SSRF/RotationAxis" $FIRSTFRAME | grep "(0):" | sed -e "s/^.*://; s/,//g"`
 +
        echo "The rotation axis of SSRF BL02U1 (Eiger2 S 9M) is $rotation_axis"
 +
      else
 +
        rotation_axis="1 0 0"
 +
        echo "SSRF BL02U1 (Eiger2 S 9M) with horizontal rotation axis"
 +
      fi
 +
    fi
 +
# revision 1.05 specialcase nframes for Eiger detectors at BNL
 +
    if [ "$SN" == "E-18-0121" -o "$SN" == "E-32-0101" ]; then
 +
      echo specialcase nframes for Eiger detectors at BNL:
 +
      nframes=`h5dump -A -g "/entry/data" $FIRSTFRAME | grep "DATASPACE  SIMPLE" | sed -e "s/,.*//" | awk '{a+=$5}END{print a}'`
 +
      DATA_RANGE="1 $nframes"
 +
      SPOT_RANGE="1 `echo "scale=0; if (${nframes}<2) 1; if (${nframes}>1) ${nframes}/2"|bc -l`"
 +
    fi
 +
  fi
 +
  echo DATA_RANGE=$DATA_RANGE
 +
  DETECTOR="EIGER MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= $OVERLOAD"
 +
  QX=`h5dump -d "/entry/instrument/detector/x_pixel_size" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2*1000}'`
 +
  QY=`h5dump -d "/entry/instrument/detector/y_pixel_size" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2*1000}'`
 +
 
 +
  echo OVERLOAD=$OVERLOAD
 +
  SENSOR_THICKNESS=`h5dump -d "/entry/instrument/detector/sensor_thickness" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2*1000}'`
 +
  X_RAY_WAVELENGTH=`h5dump -d "/entry/instrument/beam/incident_wavelength" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
    +
  NX=`h5dump -d "/entry/instrument/detector/detectorSpecific/x_pixels_in_detector" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
 +
  NY=`h5dump -d "/entry/instrument/detector/detectorSpecific/y_pixels_in_detector" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
 +
 +
  # find ORGX and ORGY:
 +
  ORGX=`h5dump -d "/entry/instrument/detector/beam_center_x" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
 +
  ORGY=`h5dump -d "/entry/instrument/detector/beam_center_y" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2}'`
 +
 +
  # find DETECTOR_DISTANCE :
 +
  DETECTOR_DISTANCE=`h5dump -d "/entry/instrument/detector/detector_distance" $FIRSTFRAME | awk '/\(0\): [0-9]/{print $2*1000}'` 
 +
 +
  SEPMIN=4
 +
  CLUSTER_RADIUS=2
 +
 +
elif [ "$DET" == "raxis" ]; then
 +
  echo Data from a RAXIS detector
 +
 +
  DETECTOR="RAXIS MINIMUM_VALID_PIXEL_VALUE=0  OVERLOAD=2000000"
 +
  #let SKIP=768
 +
  #NX=$(od -t x -j $SKIP -N 4 $FIRSTFRAME |awk 'NR==1{print toupper($2)}'|perl -nle '@array= $_ =~/.{2}/g; print "ibase=16;obase=A;".join("",reverse @array)'|bc)
 +
!  NX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(768);print "%.4d"%struct.unpack(">i",f.read(4))')
 +
!  NY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(772);print "%.4d"%struct.unpack(">i",f.read(4))')
 +
  NX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(768);print("%.4d"%struct.unpack(">i",f.read(4)))')
 +
  NY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(772);print("%.4d"%struct.unpack(">i",f.read(4)))')
 +
 +
!  DETECTOR_DISTANCE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(344);print "-%.4f"%struct.unpack(">f",f.read(4))')
 +
  DETECTOR_DISTANCE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(344);print("-%.4f"%struct.unpack(">f",f.read(4)))')
 +
 +
!  ORGX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(540);print "%.4f"%struct.unpack(">f",f.read(4))')
 +
!  ORGY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(544);print "%.4f"%struct.unpack(">f",f.read(4))')
 +
  ORGX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(540);print("%.4f"%struct.unpack(">f",f.read(4)))')
 +
  ORGY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(544);print("%.4f"%struct.unpack(">f",f.read(4)))')
 +
 +
!  OSCILLATION_RANGE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(524);phis,phie=struct.unpack(">ff",f.read(8));print "%.4f"%(phie-phis)')
 +
  OSCILLATION_RANGE=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(524);phis,phie=struct.unpack(">ff",f.read(8));print("%.4f"%(phie-phis))')
 +
 +
!  QX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(776);print "%.6f"%struct.unpack(">f",f.read(4))')
 +
!  QY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(780);print "%.6f"%struct.unpack(">f",f.read(4))')
 +
  QX=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(776);print("%.6f"%struct.unpack(">f",f.read(4)))')
 +
  QY=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(780);print("%.6f"%struct.unpack(">f",f.read(4)))')
 +
 +
!  X_RAY_WAVELENGTH=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(292);print "%.6f"%struct.unpack(">f",f.read(4))')
 +
  X_RAY_WAVELENGTH=$(python -c 'import struct; f=open("'$FIRSTFRAME'","rb");f.seek(292);print("%.6f"%struct.unpack(">f",f.read(4)))')
 +
 +
elif [ "$DET" == "dtrek" ]; then
 +
  echo "Data from a RAXIS or Saturn or Pilatus detector with dTREK format"
 +
 +
  sed s/\;// tmp2 > tmp1
 +
  mv tmp1 tmp2
 +
 +
  dname=`grep "DETECTOR_NAMES=" tmp2 | sed -e "s/.*=//"`
 +
  flip=1
 +
  if [ "$dname" == "CCD_" ]; then
 +
    DETECTOR="SATURN MINIMUM_VALID_PIXEL_VALUE=1"
 +
    dtrek_det="saturn"
 +
 +
    # Find rotation axis. Warning: currently not support flipping (det(tmpmat)<0)
 +
    # I'm not sure this method is really valid - but at least mosflm seems to read this to determine rotation axis.
 +
    tmpmat=(`grep ${dname}SPATIAL_DISTORTION_VECTORS= tmp2 | tail -1 | sed -e "s/.*=//"`)
 +
    rotx=`echo "scale=6; -1.0*${tmpmat[0]}" | bc -l`
 +
    roty=`echo "scale=6; -1.0*${tmpmat[1]}" | bc -l`
 +
    rotation_axis="$rotx $roty 0"
 +
  if [ `echo "(${tmpmat[0]}*${tmpmat[3]}-(${tmpmat[1]}*${tmpmat[2]}))/1"|bc` -lt 0 ]; then
 +
    echo ""
 +
    echo "WARNING!! not-supported SPATIAL_DISTORTION_VECTORS header detected."
 +
    echo "Please report this to XDSwiki author."
 +
    echo ""
 +
  fi
 +
  elif [ "$dname" == "PILT_" ]; then
 +
    DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0"
 +
    dtrek_det="pilatus"
 +
    SEPMIN=3
 +
    CLUSTER_RADIUS=1.5
 +
    rotation_axis="0 1 0"  # TODO: read from header
 +
    flip=-1
 +
    SENSOR_THICKNESS=0.45  # TODO: read from header
 +
# 2theta
 +
    TWOTHETA=`awk '/PILT_GONIO_VALUES=/{print $2}' tmp2`
 +
    echo TWOTHETA=$TWOTHETA
 +
    R1=`echo "scale=7;  c($TWOTHETA/$DEGTOR)" | bc -l`
 +
    R3=`echo "scale=7; -s($TWOTHETA/$DEGTOR)" | bc -l`
 +
    DIRECTION_OF_DETECTOR_X_AXIS="$R1 0 $R3"
 +
  elif [ "$dname" == "RX_" ]; then
 +
    DETECTOR="RAXIS MINIMUM_VALID_PIXEL_VALUE=0"
 +
    dtrek_det="raxis"
 +
  else
 +
    DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX"
 +
    dtrek_det="unknown"
 +
  fi
 +
  # find OVERLOAD
 +
  SATURATED_VALUE=`grep SATURATED_VALUE tmp2 | head -1 | sed s/SATURATED_VALUE=//`
 +
  DETECTOR="${DETECTOR}  OVERLOAD=${SATURATED_VALUE}"
 +
 +
  # find X_RAY_WAVELENGTH:
 +
  X_RAY_WAVELENGTH=(`grep SOURCE_WAVELENGTH tmp2 | head -1 | sed s/SOURCE_WAVELENGTH=//`)
 +
  X_RAY_WAVELENGTH=${X_RAY_WAVELENGTH[1]}
 +
 +
  # find NX,NY,QX,QY
 +
  # NX,NY should be read from *_DETECTOR_DIMENSIONS?
 +
  NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
 +
  NY=`grep SIZE2 tmp2 | tail -1 | sed s/SIZE2=//`
 +
  DET_SIZE=(`grep ${dname}DETECTOR_SIZE tmp2 | tail -1 | sed s/.*_DETECTOR_SIZE=//`)
 +
  QX=`echo "scale=6; ${DET_SIZE[0]} / $NX" | bc -l`
 +
  QY=`echo "scale=6; ${DET_SIZE[1]} / $NY" | bc -l`
 +
 +
  # find ORGX, ORGY
 +
  SPATIAL_DISTORTION_INFO=(`grep ${dname}SPATIAL_DISTORTION_INFO tmp2 | tail -1 | sed s/.*_SPATIAL_DISTORTION_INFO=//`)
 +
  ORGX=${SPATIAL_DISTORTION_INFO[0]}
 +
  ORGY=${SPATIAL_DISTORTION_INFO[1]}
 +
 +
  # find DETECTOR_DISTANCE
 +
  GONIO_NAMES=(`grep ${dname}GONIO_NAMES= tmp2 | tail -1 | sed s/.*_GONIO_NAMES=//`)
 +
  GONIO_VALUES=(`grep ${dname}GONIO_VALUES= tmp2 | tail -1 | sed s/.*_GONIO_VALUES=//`)
 +
#  GONIO_UNITS=(`grep ${dname}GONIO_UNITS= tmp2 | tail -1 | sed s/.*_GONIO_UNITS=//`)
 +
  for i in `seq 1 ${#GONIO_NAMES[*]}`
 +
  do
 +
  idx=$((i-1))
 +
  if [ "${GONIO_NAMES[$idx]}" == "Distance" ]; then
 +
    DETECTOR_DISTANCE="${GONIO_VALUES[$idx]}"
 +
    # TODO: Check unit!! - ${GONIO_UNITS[$idx]}
 +
  fi
 +
  if [ $flip -gt 0 ]; then
 +
    DETECTOR_DISTANCE="-${GONIO_VALUES[$idx]}"
 +
    # TODO: Check unit!! - ${GONIO_UNITS[$idx]}
 +
    echo "using distance <0"
 +
  fi
 +
  done
 +
 +
  # find OSCILLATION_RANGE
 +
  ROTATION=(`grep "^ROTATION=" tmp2 | tail -1 | sed s/ROTATION=//`)
 +
  OSCILLATION_RANGE=${ROTATION[2]}
 +
 +
elif [ "$DET" == "MAR345" ]; then
 +
 
 +
echo  "Data from a Mar345 image plate detector"
 +
DETECTOR="MAR345  MINIMUM_VALID_PIXEL_VALUE=0  OVERLOAD=130000"
 +
NX=`awk '/FORMAT/{print $2}' tmp2`
 +
# next line is rev 1.02 (previously NY= $NX):
 +
NY=`awk -v NX=$NX '/FORMAT/{print $4/NX}' tmp2`
 +
QX=`awk '/PIXEL/{print $3/1000.}' tmp2`
 +
QY=`awk '/PIXEL/{print $5/1000.}' tmp2`
 +
if grep -q 'CENTER' tmp2; then
 +
  echo Beam center found.
 +
  ORGX=`grep 'CENTER' tmp2 | awk '{print $3}'`
 +
  ORGY=`grep 'CENTER' tmp2 | awk '{print $5}'`
 +
else
 +
  echo No beam center was found. Setting beam center to the middle of the detector.
 +
  ORGX=`echo $NX / 2 | bc`
 +
  ORGY=`echo $NY / 2 | bc`
 +
fi
 +
DETECTOR_DISTANCE=`grep 'DISTANCE' tmp2 | awk '{print $2}'`             
 +
X_RAY_WAVELENGTH=`grep 'WAVELENGTH' tmp2 | awk '{print $2}'`
 +
OSCILLATION_RANGE=`grep 'PHI' tmp2 | awk '{print $5-$3}'`   
 +
TRUSTED_REGION="0 0.99"   
 +
 
 +
elif [ "$DET" == "OLDMAR" ]; then
 +
 
 +
echo  "Data from old type MAR image plate detector"
 +
DETECTOR="MAR  MINIMUM_VALID_PIXEL_VALUE=0  OVERLOAD=130000"
 +
NX=`awk 'NR==2 {print $2}' tmp2`
 +
NY=$NX
 +
QX=`awk 'NR==2 {print $15}' tmp2`
 +
QY=$QX
 +
ORGX=`awk 'NR==2 {print $19}' tmp2`
 +
ORGY=`awk 'NR==2 {print $20}' tmp2`
 +
DETECTOR_DISTANCE=`awk 'NR==2 {print $22}' tmp2`             
 +
X_RAY_WAVELENGTH=`awk 'NR==2 {print $21}' tmp2`
 +
OSCILLATION_RANGE=`awk 'NR==2 {print $24-$23}' tmp2`   
 +
TRUSTED_REGION="0 0.99" 
 +
rotation_axis="0 1 0" 
 +
echo unsure if sign of anomalous signal is correct - please verify or try both hands!
 +
     
 +
elif [ "$DET" == "Bruker-cbf" ]; then
 +
 
 +
echo  "Data from a Bruker-cbf detector"
 +
# MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT:
 +
MNOPIAS=6
 +
# use complete detector including corners:
 +
TRUSTED_REGION="0 1.42"       
 +
# polarization
 +
pol_frac=`awk '/polarizn_source_ratio/{print $2}' tmp2`
 +
pol_frac=`echo "${pol_frac}+0.5" | bc -l`
 +
OVERLOAD=`awk '/_array_intensities.overload/{print $2}' tmp2`
 +
DETECTOR="BRUKER MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD=${OVERLOAD}"
 +
NX=`awk '/X-Binary-Size-Fastest-Dimension/{print $2}' tmp2`
 +
NY=`awk '/X-Binary-Size-Second-Dimension/{print $2}' tmp2`
 +
 +
QX=`awk '/ELEMENT_X ELEMENT_X/{print $4}' tmp2`
 +
QY=`awk '/ELEMENT_Y ELEMENT_Y/{print $4}' tmp2`
 +
 +
# ORGX/Y-offsets relative to center of detector:
 +
ORGX=`grep " 0 0 ? ? ?" tmp2 | awk '/ H /{print $2}'`
 +
ORGY=`grep " 0 0 ? ? ?" tmp2 | awk '/ V /{print $2}'`
 +
# total ORGX/Y
 +
ORGX=`echo "scale=2; ${NX}/2+($ORGX/$QX)" | bc -l`
 +
ORGY=`echo "scale=2; ${NY}/2-($ORGY/$QY)" | bc -l`
 +
 +
DETECTOR_DISTANCE=`grep "0 0 ? ? ?" tmp2 | awk '/DX /{print $2}'`
 +
 +
X_RAY_WAVELENGTH=`awk '/diffrn_radiation_wavelength.wavelength/{print $2}' tmp2`
 +
# fix 16.3.20: instead of print, use printf "%.5f", because bc does not accept e.g. 3.1e-005
 +
OMEGA=`awk '/OMEGA \? \? \?/{printf "%.5f",$5}' tmp2`
 +
DELTAOMEGA=`awk '/OMEGA \? \? \?/{printf "%.5f",$6}' tmp2`
 +
PHI=`awk '/PHI \? \? \?/{printf "%.5f",$5}' tmp2`
 +
DELTAPHI=`awk '/PHI \? \? \?/{printf "%.5f",$6}' tmp2`
 +
KAPPA=`awk '/CHI \? \? \?/{printf "%.5f",$5}' tmp2`
 +
# echo OMEGA DELTAOMEGA PHI DELTAPHI KAPPA= $OMEGA $DELTAOMEGA $PHI $DELTAPHI $KAPPA
 +
 +
# test whether the absolute value of deltaphi is > absolute value of deltaomega
 +
if (( $(echo "${DELTAPHI}^2 > ${DELTAOMEGA}^2"|bc -l) )); then
 +
    echo PHI scan
 +
    R3=`echo "scale=7; s($KAPPA/$DEGTOR)*s($OMEGA/$DEGTOR)" | bc -l`
 +
    R1=`echo "scale=7; s($KAPPA/$DEGTOR)*c($OMEGA/$DEGTOR)" | bc -l`
 +
    R2=`echo "scale=7; c($KAPPA/$DEGTOR)          " | bc -l`
 +
    rotation_axis="$R1 $R2 $R3"
 +
    OSCILLATION_RANGE=${DELTAPHI}
 +
    STARTING_ANGLE=${PHI}
 +
# here we could check if DELTAPHI is <0, and if so, negate it and rotation_axis
 +
else
 +
    echo OMEGA scan
 +
    rotation_axis="0 -1 0"
 +
    OSCILLATION_RANGE=${DELTAOMEGA}
 +
    STARTING_ANGLE=${OMEGA}
 +
# here we could check if DELTAOMEGA is <0, and if so, negate it and rotation_axis
 +
fi
 +
echo STARTING_ANGLE= $STARTING_ANGLE        ! only read by IDXREF               
 +
 +
# 2theta
 +
  TWOTHETA=`awk '/TWOTHETA \? \? \?/{print $5}' tmp2`
 +
  echo TWOTHETA=$TWOTHETA
 +
  R1=`echo "scale=7; c($TWOTHETA/$DEGTOR)" | bc -l`
 +
  R3=`echo "scale=7; s($TWOTHETA/$DEGTOR)" | bc -l`
 +
  DIRECTION_OF_DETECTOR_X_AXIS="$R1 0 $R3"
 +
# end of Bruker-cbf section
 
else
 
else
 
   echo should never come here
 
   echo should never come here
Line 248: Line 921:     
echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
 
echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE                  
+
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE ! only read by XYCORR, IDXREF                   
echo OSCILLATION_RANGE= $OSCILLATION_RANGE                  
+
echo OSCILLATION_RANGE= $OSCILLATION_RANGE ! only read by IDXREF
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH                    
+
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH   ! only read by IDXREF                 
    
# now we know everything that is required to generate XDS.INP
 
# now we know everything that is required to generate XDS.INP
Line 257: Line 930:  
! written by generate_XDS.INP version $REVISION
 
! written by generate_XDS.INP version $REVISION
 
JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT
 
JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT
ORGX= $ORGX ORGY= $ORGY  ! check these values with adxv !\
+
ORGX= $ORGX ORGY= $ORGY  ! values from frame header; only read by XYCORR, IDXREF   
 
$COMMENT_ORGXY
 
$COMMENT_ORGXY
DETECTOR_DISTANCE= $DETECTOR_DISTANCE                  
+
DETECTOR_DISTANCE= $DETECTOR_DISTANCE !read by XYCORR, IDXREF. Negative if detector normal points to crystal.             
OSCILLATION_RANGE= $OSCILLATION_RANGE                  
+
OSCILLATION_RANGE= $OSCILLATION_RANGE  
 +
STARTING_ANGLE= $STARTING_ANGLE               
 
X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH                       
 
X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH                       
 
NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES
 
NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES
 
! REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing   
 
! REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing   
DATA_RANGE=1 $DATA_RANGE                                                     
+
DATA_RANGE=$DATA_RANGE                                                     
SPOT_RANGE=1 $SPOT_RANGE                                                     
+
SPOT_RANGE=$SPOT_RANGE                                                     
 
! BACKGROUND_RANGE=1 10 ! rather use defaults (first 5 degree of rotation)   
 
! BACKGROUND_RANGE=1 10 ! rather use defaults (first 5 degree of rotation)   
   Line 271: Line 945:  
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known
 
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known
 
INCLUDE_RESOLUTION_RANGE=50 0  ! after CORRECT, insert high resol limit; re-run CORRECT
 
INCLUDE_RESOLUTION_RANGE=50 0  ! after CORRECT, insert high resol limit; re-run CORRECT
 +
! IDXREF now obeys INCLUDE_RESOLUTION_RANGE and EXCLUDE_RESOLUTION_RANGE to exclude ice-rings
    
FRIEDEL'S_LAW=FALSE    ! This acts only on the CORRECT step
 
FRIEDEL'S_LAW=FALSE    ! This acts only on the CORRECT step
Line 287: Line 962:  
!
 
!
 
! parameters with changes wrt default values:
 
! parameters with changes wrt default values:
TRUSTED_REGION=0.00 1.2  ! partially use corners of detectors; 1.41421=full use
+
TRUSTED_REGION=$TRUSTED_REGION
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=7000. 30000. ! often 8000 is ok
+
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=6000. 30000. ! often 7000 or 8000 is ok
 
STRONG_PIXEL=4          ! COLSPOT: only use strong reflections (default is 3)
 
STRONG_PIXEL=4          ! COLSPOT: only use strong reflections (default is 3)
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3 ! default of 6 is sometimes too high
+
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=$MNOPIAS ! default of 6 is sometimes too high
! close spots: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 6 and 3, e.g. to 4 and 2
+
! close spots/long cell axis: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 7 and 3.5
! for bad or low resolution data remove the "!" in the following line:
+
SEPMIN=$SEPMIN  CLUSTER_RADIUS=$CLUSTER_RADIUS ! 4 and 2 for Pixel Array Detectors
! REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
+
! since XDS 01-MAR-2015, POSITION supersedes DISTANCE.
REFINE(INTEGRATE)=CELL BEAM ORIENTATION ! AXIS DISTANCE
+
! nowadays headers are usually correct so refine POSITION in INTEGRATE but not IDXREF if low to medium resolution
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE ! Default is: refine everything
+
! however, if the spots from COLSPOT extend to 2A then POSITION could, and if 1.5A POSITION should be refined
 
+
REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! add POSITION if high resolution, or DETECTOR_DISTANCE inaccurate
 +
REFINE(INTEGRATE)= POSITION BEAM ORIENTATION ! AXIS CELL . If 1.5A or higher it is ok to refine CELL (unless electron diffraction)
 +
REFINE(CORRECT)= $REFINE_CORRECT
 
! parameters specifically for this detector and beamline:
 
! parameters specifically for this detector and beamline:
 
DETECTOR= $DETECTOR
 
DETECTOR= $DETECTOR
 
SENSOR_THICKNESS= $SENSOR_THICKNESS
 
SENSOR_THICKNESS= $SENSOR_THICKNESS
 
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
 
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon)
+
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon).
 +
! Better: read the article http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/SILICON .
 
NX= $NX NY= $NY  QX= $QX  QY= $QY ! to make CORRECT happy if frames are unavailable
 
NX= $NX NY= $NY  QX= $QX  QY= $QY ! to make CORRECT happy if frames are unavailable
 +
 +
eof
 +
 +
if [ "$DET" == "eiger" ] && [ "$is_h5" == 1 ]; then
 +
  if [ "$DLS" == 0 ] ; then
 +
    if [ -e /usr/local/lib64/dectris-neggia.so ]; then
 +
      echo LIB=/usr/local/lib64/dectris-neggia.so >> XDS.INP
 +
      echo LIB= line was written to XDS.INP . For Apple ARM64 processors, you must modify the name.
 +
    else
 +
      echo !LIB=/usr/local/lib64/dectris-neggia.so >> XDS.INP
 +
      echo /usr/local/lib64/dectris-neggia.so was not found - specify location manually!
 +
    fi
 +
  else
 +
    if [ -e /usr/local/lib64/durin-plugin.so ]; then
 +
      echo LIB=/usr/local/lib64/durin-plugin.so >> XDS.INP
 +
      echo LIB= line was written to XDS.INP
 +
    else
 +
      echo !LIB=/usr/local/lib64/durin-plugin.so >> XDS.INP
 +
      echo /usr/local/lib64/durin-plugin.so was not found - specify location manually!
 +
    fi
 +
  fi
 +
fi
 +
 +
if [ "$DET" == "raxis" -o "$dtrek_det" == "raxis" ]; then
 +
cat >> XDS.INP << eof
 
DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
 
DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
 +
DIRECTION_OF_DETECTOR_Y-AXIS=0 -1 0
 +
INCIDENT_BEAM_DIRECTION=0 0 1  ! only read by IDXREF
 +
ROTATION_AXIS=0 1 0            ! only read by IDXREF
 +
!FRACTION_OF_POLARIZATION=0.98  ! uncomment if synchrotron; only used by CORRECT
 +
POLARIZATION_PLANE_NORMAL=1 0 0  ! only used by CORRECT
 +
eof
 +
else
 +
if [ "$rotation_axis" != "" ]; then
 +
  echo "ROTATION_AXIS= $rotation_axis  ! only read by IDXREF" >> XDS.INP
 +
elif [ "$REVERSE_PHI" == "no" ]; then
 +
  echo 'ROTATION_AXIS=1 0 0  ! Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?), ESRF BM30A, SPring-8, SSRF need -1 0 0. Diamond ID24 needs 0 -1 0' >> XDS.INP
 +
else
 +
  echo 'ROTATION_AXIS=-1 0 0  ! if this is wrong, please contact author.' >> XDS.INP
 +
fi
 +
if [ "$dtrek_det" == "saturn" ]; then
 +
cat >> XDS.INP << eof
 +
DIRECTION_OF_DETECTOR_X-AXIS=-1 0 0
 +
DIRECTION_OF_DETECTOR_Y-AXIS= 0 1 0
 +
eof
 +
else
 +
cat >> XDS.INP << eof
 +
DIRECTION_OF_DETECTOR_X-AXIS=$DIRECTION_OF_DETECTOR_X_AXIS
 
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
 
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
INCIDENT_BEAM_DIRECTION=0 0 1
+
eof
ROTATION_AXIS=1 0 0  ! at e.g. Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?), ESRF BM30A, SPring-8 this needs to be -1 0 0
+
fi
FRACTION_OF_POLARIZATION=0.98   ! better value is provided by beamline staff!
+
cat >> XDS.INP << eof
POLARIZATION_PLANE_NORMAL=0 1 0
+
INCIDENT_BEAM_DIRECTION=0 0 1         ! only read by IDXREF
 +
FRACTION_OF_POLARIZATION=${pol_frac}   ! better value is provided by beamline staff!
 +
POLARIZATION_PLANE_NORMAL=0 1 0       ! only used by CORRECT
 +
eof
 +
fi
 +
cat >> XDS.INP << eof
 
!used by DEFPIX and CORRECT to exclude ice-reflections / ice rings - uncomment if necessary
 
!used by DEFPIX and CORRECT to exclude ice-reflections / ice rings - uncomment if necessary
 +
!fine-grained list is in Thorn et al http://journals.iucr.org/d/issues/2017/09/00/hi5647/index.html
 
!EXCLUDE_RESOLUTION_RANGE= 3.93 3.87 !ice-ring at 3.897 Angstrom
 
!EXCLUDE_RESOLUTION_RANGE= 3.93 3.87 !ice-ring at 3.897 Angstrom
 
!EXCLUDE_RESOLUTION_RANGE= 3.70 3.64 !ice-ring at 3.669 Angstrom
 
!EXCLUDE_RESOLUTION_RANGE= 3.70 3.64 !ice-ring at 3.669 Angstrom
Line 320: Line 1,051:  
!EXCLUDE_RESOLUTION_RANGE= 1.913 1.853 !ice-ring at 1.883 Angstrom - weak
 
!EXCLUDE_RESOLUTION_RANGE= 1.913 1.853 !ice-ring at 1.883 Angstrom - weak
 
!EXCLUDE_RESOLUTION_RANGE= 1.751 1.691 !ice-ring at 1.721 Angstrom - weak
 
!EXCLUDE_RESOLUTION_RANGE= 1.751 1.691 !ice-ring at 1.721 Angstrom - weak
 +
! additional ice-ring resolution ranges: 1.524 1.519, 1.473 1.470, 1.444 1.440, 1.372 1.368, 1.367 1.363,
 +
! 1.299 1.296, 1.275 1.274, 1.261 1.259, 1.224 1.222, 1.171 1.168, 1.124 1.122 (compiled by GlobalPhasing)
 +
 +
eof
 +
if [ "$DET" == "Bruker-cbf" ]; then
 +
  echo "DELPHI=15 ! refine less often than the default of 5" >> XDS.INP
 +
elif [ "$DET" == "adsc-CMOS1" ]; then
 +
  echo UNTRUSTED_RECTANGLE= 0 1468  2451 2631  >> XDS.INP
 +
elif [ "$DET" == "pilatus" -o "$DET" == "eiger" ]; then
 +
cat >> XDS.INP << eof
 +
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=13 ! Default is 9 - Increasing may improve data
 +
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA=13      ! accuracy, particularly if finely-sliced on phi,
 +
!                                                  and does not seem to have any downsides.
 +
eof
 +
  if [ "$NX" == "1028"  -a "$NY" == "1062" ]; then
 +
# Eiger2 1M ; v0.97 numbers from Andreas Förster
 +
    cat >> XDS.INP << eof
 +
UNTRUSTED_RECTANGLE=    0 1029    512  551
 
eof
 
eof
if [ "$DET" == "pilatus" ]; then
+
  elif [ "$NX" == "1475" ]; then
  if [ $NX == "1475" ]; then
   
     if ! grep -q Flat_field tmp2 ; then
 
     if ! grep -q Flat_field tmp2 ; then
 
     cat >> XDS.INP << eof
 
     cat >> XDS.INP << eof
Line 355: Line 1,103:  
eof
 
eof
 
     fi
 
     fi
   elif [ $NX == "2463" ]; then
+
   elif [ "$NX" == "2068"  -a "$NY" == "2162" ]; then
 +
# Eiger2 4M ; v0.97 numbers from Andreas Förster
 +
    cat >> XDS.INP << eof
 +
!EXCLUSION OF VERTICAL DEAD AREAS OF THE EIGER2 4M DETECTOR
 +
UNTRUSTED_RECTANGLE= 1028 1041      0 2163
 +
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE EIGER 4M DETECTOR
 +
UNTRUSTED_RECTANGLE=    0 2069    512  551
 +
UNTRUSTED_RECTANGLE=    0 2069  1062 1101
 +
UNTRUSTED_RECTANGLE=    0 2069  1612 1651
 +
eof
 +
  elif [ "$NX" == "2463" ]; then
 
# Pilatus 6M
 
# Pilatus 6M
 
# FIXME: here we could test if a Flat_field correction was applied like we do for 2M
 
# FIXME: here we could test if a Flat_field correction was applied like we do for 2M
Line 374: Line 1,132:  
UNTRUSTED_RECTANGLE=  0 2464  2103 2121
 
UNTRUSTED_RECTANGLE=  0 2464  2103 2121
 
UNTRUSTED_RECTANGLE=  0 2464  2315 2333
 
UNTRUSTED_RECTANGLE=  0 2464  2315 2333
 +
eof
 +
  elif [ "$NX" == "3110"  -a "$NY" == "3269" ]; then
 +
# Eiger 9M
 +
    cat >> XDS.INP << eof
 +
!EXCLUSION OF VERTICAL DEAD AREAS OF THE EIGER 9M DETECTOR
 +
UNTRUSTED_RECTANGLE= 1029 1042 0 3269
 +
UNTRUSTED_RECTANGLE= 2069 2082 0 3269
 +
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE EIGER 9M DETECTOR
 +
UNTRUSTED_RECTANGLE= 0 3110  513  553
 +
UNTRUSTED_RECTANGLE= 0 3110 1064 1104
 +
UNTRUSTED_RECTANGLE= 0 3110 1615 1655
 +
UNTRUSTED_RECTANGLE= 0 3110 2166 2206
 +
UNTRUSTED_RECTANGLE= 0 3110 2717 2757
 +
eof
 +
  elif [ "$NX" == "3108"  -a "$NY" == "3262" ]; then
 +
# Eiger2 9M ; v0.97 numbers from Andreas Förster
 +
    cat >> XDS.INP << eof
 +
!EXCLUSION OF VERTICAL DEAD AREAS OF THE EIGER2 9M DETECTOR
 +
UNTRUSTED_RECTANGLE= 1028 1041      0 3262
 +
UNTRUSTED_RECTANGLE= 2068 2081      0 3263
 +
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE EIGER2 9M DETECTOR
 +
UNTRUSTED_RECTANGLE=    0 3109    512  551
 +
UNTRUSTED_RECTANGLE=    0 3109  1062 1101
 +
UNTRUSTED_RECTANGLE=    0 3109  1612 1651
 +
UNTRUSTED_RECTANGLE=    0 3109  2162 2201
 +
UNTRUSTED_RECTANGLE=    0 3109  2712 2751
 +
eof
 +
  elif [ "$NX" == "4150" -a "$NY" == "4371" ]; then
 +
# Eiger 16M
 +
    cat >> XDS.INP << eof
 +
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE EIGER 16M DETECTOR + ONE PIXEL ON EACH SIDE
 +
UNTRUSTED_RECTANGLE=    0 4150    513  553
 +
UNTRUSTED_RECTANGLE=    0 4150  1064 1104
 +
UNTRUSTED_RECTANGLE=    0 4150  1615 1655
 +
UNTRUSTED_RECTANGLE=    0 4150  2166 2206
 +
UNTRUSTED_RECTANGLE=    0 4150  2717 2757
 +
UNTRUSTED_RECTANGLE=    0 4150  3268 3308
 +
UNTRUSTED_RECTANGLE=    0 4150  3819 3859
 +
!EXCLUSION OF VERTICAL DEAD AREAS OF THE EIGER 16M DETECTOR + ONE PIXEL ON EACH SIDE
 +
UNTRUSTED_RECTANGLE= 1029 1042      0 4371
 +
UNTRUSTED_RECTANGLE= 2069 2082      0 4371
 +
UNTRUSTED_RECTANGLE= 3109 3122      0 4371
 +
eof
 +
  elif [ "$NX" == "4148" -a "$NY" == "4362" ]; then
 +
# Eiger2 16M ; v0.97 numbers from Andreas Förster
 +
    cat >> XDS.INP << eof
 +
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE EIGER2 16M DETECTOR
 +
UNTRUSTED_RECTANGLE= 1028 1041      0 4363
 +
UNTRUSTED_RECTANGLE= 2068 2081      0 4363
 +
UNTRUSTED_RECTANGLE= 3108 3121      0 4363
 +
!EXCLUSION OF VERTICAL DEAD AREAS OF THE EIGER2 16M DETECTOR
 +
UNTRUSTED_RECTANGLE=    0 4149    512  551
 +
UNTRUSTED_RECTANGLE=    0 4149  1062 1101
 +
UNTRUSTED_RECTANGLE=    0 4149  1612 1651
 +
UNTRUSTED_RECTANGLE=    0 4149  2162 2201
 +
UNTRUSTED_RECTANGLE=    0 4149  2712 2751
 +
UNTRUSTED_RECTANGLE=    0 4149  3262 3301
 +
UNTRUSTED_RECTANGLE=    0 4149  3812 3851
 
eof
 
eof
 
   fi
 
   fi
 
fi
 
fi
 
echo XDS.INP is ready for use. The file has only the most important keywords.
 
echo XDS.INP is ready for use. The file has only the most important keywords.
echo Full documentation, including complete detector templates, is at
+
echo Full documentation, including complete detector templates, at xds.mr.mpg.de .
echo http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
+
echo More documentation in strucbio.biologie.uni-konstanz.de/xdswiki/index.php .
echo After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
+
echo After running xds, inspect at least the agreement of predicted and observed  
echo BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!
+
echo spots in FRAME.cbf!
 
rm -f tmp1 tmp2
 
rm -f tmp1 tmp2
 +
 
</pre>
 
</pre>
   Line 395: Line 1,212:  
By using your own file, you can easily update to the latest revision, or even change the script, without having to bother the system administrator.
 
By using your own file, you can easily update to the latest revision, or even change the script, without having to bother the system administrator.
   −
== Generating generate_XDS.INP from this webpage ==
+
See also [[Generate_XDS.INP#Dependencies]] below, and the [[Installation]] article.
 +
 
 +
== Copying generate_XDS.INP from XDSwiki webserver ==
 +
On Linux:
 +
wget https://strucbio.biologie.uni-konstanz.de/pub/linux_bin/generate_XDS.INP
 +
chmod a+x generate_XDS.INP
 +
On Mac:
 +
curl -o generate_XDS.INP https://strucbio.biologie.uni-konstanz.de/pub/linux_bin/generate_XDS.INP
 +
chmod a+x generate_XDS.INP
 +
See also [[Installation]].
 +
 
 +
== Obtaining generate_XDS.INP from this webpage ==
    
Instead of cutting-and-pasting the lines of the script, you (or the system administrator) could just cut-and-paste the following four lines
 
Instead of cutting-and-pasting the lines of the script, you (or the system administrator) could just cut-and-paste the following four lines
Line 401: Line 1,229:  
  wget http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP -O - | \
 
  wget http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP -O - | \
 
   sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \
 
   sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \
   awk '/^#/,/rm -f tmp1 tmp2/' > generate_XDS.INP
+
   sed '/# end of generate_XDS.INP/,$d' | awk '/^#/,/rm -f tmp1 tmp2/' > generate_XDS.INP
 +
chmod +x generate_XDS.INP
 +
</pre>
 +
to copy the script from this website into an executable file generate_XDS.INP in your current directory. On a Mac (which does not seem to have wget), one could try
 +
<pre>
 +
curl -L -o - http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP | \
 +
  sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \
 +
  sed '/# end of generate_XDS.INP/,$d' | awk '/^#/,/rm -f tmp1 tmp2/' > generate_XDS.INP
 
  chmod +x generate_XDS.INP
 
  chmod +x generate_XDS.INP
 
</pre>
 
</pre>
to copy the script from this website into an executable file generate_XDS.INP in your current directory.
+
If you do use cut-and-paste from the webpage, be aware of the following problem report: On the Mac, after loading frames, by clicking “generate XDS.INP”, the program gives some strange symbol “” in XDS.INP. And the more you click “save” button, the more “” appear. This looks like e.g. <br>SPACE_GROUP_NUMBER=0  ! 0 if unknown <br>UNIT_CELL_CONSTANTS= 70 80 90 90 90 90  <br> ''The problem is due to the “Rich text” format in TextEdit when saving "generate_XDS.INP". It is solved by re-downloading the script, and changing format to Plain - everything should work then.''
   −
==Variant script (xds_generate_all)==
+
== Calling generate_XDS.INP from a Python script ==
This script includes some minor modifications of the Generate_XDS.INP script - for instance, it tells the user the energy, as well as the wavelength of the X-rays, and gives the user the beam center in mm as well as pixels - values that may be useful when switching between HKL2000, MOSFLM and XDS.
     −
It also generates two executable shell scripts in the same directory as XDS.INP.
+
It is recommended to use the [http://docs.python.org/2/library/subprocess.html subprocess.Popen()] module instead of [http://docs.python.org/2/library/os.html os.system()]:
 
+
<pre>
The first of these scripts, xds_graph.sh, extracts various statistics from INTEGRATE.LP and CORRECT.LP, pipes them out to a logfile, INTEGRATE_STATS.LP, and plots them using loggraph. This script does not have some of the more useful features of XDSSTAT, such as calculation of per frame Rmeas values, but it does give a quick overview of various data quality parameters by resolution and image number. This script will only work with recent versions of XDS (after calculation of CC(1/2) - not sure exactly which version).
+
subprocess.Popen(["generate_XDS.INP",imagepath],stdout=outputfile)
 
+
</pre>
The second of these scripts, xds_to_ccp4.sh, will take XDS_ASCII.HKL and generate a CCP4-format MTZ file, containing anomalous data (as F(+)/F(-) and DANO/SigDANO) and a test set of 5% of reflections for calculation of the free R-factor. It will also generate an unmerged SHELX format HKL file, for input into SHELXC/D/E (e.g. via the HKL2MAP GUI).
+
where imagepath is a string containing the path to an image and outputfile is either a chosen variable for an output file or subprocess.PIPE if you're not interested in the output of the script.
 
  −
Both these shell scripts can by installed system wide by transferring them to a directory in your path, for example the directory containing your XDS binaries.
     −
Here is the script:
+
The module os.system() internally uses /bin/sh to execute the command and overrides #!/bin/bash at the beginning of the script. While this is not a problem on most operating systems, /bin/sh points to dash instead of bash on Ubuntu, which leads to a program crash with the error message
 
<pre>
 
<pre>
#!/bin/bash                                                                 
+
sh: 1: Syntax error: Bad fd number
# purpose: xds_generate_all                                                 
+
</pre>
#                                                                           
  −
# tested with some datasets from ALS, SSRL, SLS, ESRF and BESSY; only MARCCD, ADSC/SMV, PILATUS detectors;
  −
# for other detectors, values marked with XXX must be manually filled in.                                 
  −
#                                                                                                         
  −
# revision 0.03 . Kay Diederichs 2/2010                                                                   
  −
# revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC           
  −
# revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY 
  −
# revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on
  −
# revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes       
  −
# revision 0.08 . KD 6/2010 - fixes for Pilatus 6M                                                         
  −
# revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od" 
  −
# revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters   
  −
# revision 0.11 . KD 7/2010 - for MarCCD: look for distance info at different byte position               
  −
# revision 0.12 . KD 7/2010 - fix for negative PHISTART                                                   
  −
# revision 0.13 . KD 8/2010 - store correct NX NY QX QY in XDS.INP                                         
  −
# revision 0.14 . KD 1/2011 - SENSOR_THICKNESS for Pilatus; MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3           
  −
# revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis
  −
# revision 0.16 . KD 3/2011 - SENSOR_THICKNESS=0.01 for ADSC and MarCCD. Add comment about SILICON=         
  −
# revision 0.17 . KD 3/2011 - make it work for .bz2 frames; improve screen output                           
  −
# revision 0.18 . KD 4/2011 - faster by doing "strings" only once; revert "images/${1##/*/}" "correction"   
  −
# revision 0.19 . KD 6/2011 - bugfix for 0.18                                                               
  −
# revision 0.20 . KD 7/2011 - redirect stderr of /bin/ls to /dev/null                                       
  −
# revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)
  −
# revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header
  −
# revision 0.23 . KD 1/2012 - add UNTRUSTED_QUADRILATERAL, remove MINIMUM_ZETA (0.05 is default now)
  −
# revision 0.24 . KD 3/2012 - remove revision 0.10 since XDS now takes much longer paths
  −
# revision 0.25 . KD 3/2012 - remove revision 0.22 for PSI Pilatus 2M; see http://www.globalphasing.com/autoproc/wiki/index.cgi?TroubleShootingKnownIssues
  −
# revision 0.26 Oliver Clarke 10/12 - Added generation of shell scripts for conversion to MTZ and visualisation of statistics. Other minor alterations to XDS.INP.
  −
# revision 0.27 Oliver Clarke 10/12 - xds_graph.sh will now run xdsstat, if available, and plot Rmeas and Rd per image.
  −
# revision 0.28 Oliver Clarke 10/12 - xds_graph was not plotting h00, 0k0, 00l reflections when a reference dataset was present. Fixed.
  −
# revision 0.29 KD 11/2012 - EXCLUDE_RESOLUTION_RANGE lines and generic Pilatus Flat_field test
  −
# revision 0.30 Keitaro 11/2012 - for MarCCD: read oscillation range from the position 1024+736 (fix for omega rotation) XXX for STARTING_ANGLE=, needs more work to support both phi and omega rotation
  −
REVISION="0.30 (28-Nov-2012)"                                                                               
  −
#                                                                                                           
  −
# usage: e.g. xds_generate_all "frms/mydata_1_???.img"                                                     
  −
# make sure to have the two quotation marks !                                                               
  −
# the ? are wildcards for the frame numbers.                                                               
  −
#                                                                                                           
  −
# limitations:                                                                                              
  −
# - frame numbers are assumed to start with 1 and run consecutively                                         
  −
#                                                                                                           
  −
# known problems:                                                                                           
  −
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
  −
# - the same might be a problem for MAR headers, too (not sure about this)                                     
  −
#                                                                                                               
  −
# notes for debugging of the script:                                                                            
  −
# - add the -v option to the first line, to see where an error occurs                                           
  −
# - comment out the removal of tmp1 and tmp2 in the last line                                                   
  −
#                                                                                                               
  −
# ====== Start of script ======                                                                                 
  −
echo generate_XDS.INP version $REVISION . Obtain the latest version from                                       
  −
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP                               
  −
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then                                         
  −
  echo usage: xds_generate_all \"frms/mydata_1_???.img\"  \(_with_ the quotation marks!\)                     
  −
  echo if the frames are compressed with bzip2, leave out the .bz2 extension!                                   
  −
  exit                                                                                                         
  −
fi                                                                                                             
  −
#                                                                                                               
  −
# defaults:                                                                                                     
  −
#                                                                                                               
  −
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"                                                       
  −
ORGX=XXX                                                                                                       
  −
ORGY=XXX
  −
ORGXMM=XXX
  −
ORGYMM=XXX
  −
DETECTOR_DISTANCE=XXX                                                                                           
  −
OSCILLATION_RANGE=XXX                                                                                           
  −
X_RAY_WAVELENGTH=XXX
  −
X_RAY_ENERGY=XXX
  −
STARTING_ANGLE=XXX
  −
QX=XXX                                                                                                         
  −
QY=XXX                                                                                                         
  −
NX=XXX                                                                                                         
  −
NY=XXX
  −
NXMM=XXX
  −
NYMM=XXX
  −
SENSOR_THICKNESS=0                                                                                             
  −
# see how we are called:                                                                                       
  −
NAME_TEMPLATE_OF_DATA_FRAMES="$1"
  −
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
  −
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II
  −
/bin/ls -C1 $1 $1.bz2 2>/dev/null | egrep -v "_00000.cbf|_000.img" > tmp1 || exit 1
  −
 
  −
# we can continue - the frames are found
  −
 
  −
# set upper limit of DATA_RANGE to number of frames (see "limitations" above)
  −
DATA_RANGE=`wc -l tmp1 | awk '{print $1}'`                                 
  −
 
  −
# set upper limit of SPOT_RANGE to half of DATA_RANGE, but not less than 1
  −
SPOT_RANGE=`echo "scale=0; $DATA_RANGE/2" | bc -l`                       
  −
SPOT_RANGE=`echo "if ($SPOT_RANGE<1) 1;if ($SPOT_RANGE>1) $SPOT_RANGE" | bc -l`
  −
 
  −
echo DATA_RANGE=1 $DATA_RANGE
  −
 
  −
# find out detector type
  −
DET=XXX               
  −
FIRSTFRAME=`head -1 tmp1`
  −
echo $FIRSTFRAME | grep -q bz2 && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
  −
strings $FIRSTFRAME > tmp2                                                 
  −
egrep -q 'marccd|Corrected' tmp2 && DET=mccd                               
  −
grep -q PILATUS tmp2            && DET=pilatus                             
  −
grep -q BEAM_CENTER_X tmp2      && DET=adsc                               
  −
# identify other detector types in the same way (MAR IP would be straightforward)
  −
 
  −
# parse ASCII header of first frame
  −
 
  −
if [ "$DET" == "XXX" ]; then
  −
  echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
  −
  DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"                           
  −
 
  −
# find parameters of first frame
  −
elif [ "$DET" == "mccd" ]; then
  −
  echo Data from a MarCCD detector
  −
                                 
  −
  DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
  −
  SENSOR_THICKNESS=0.01                                         
  −
  # use first frame of dataset to obtain parameters             
  −
 
  −
  # offsets are documented; values can be find in mccd_xdsparams.pl script
  −
  let SKIP=1024+80                                                       
  −
  NX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
  −
  let SKIP=$SKIP+4                                                                       
  −
  NY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
  −
 
  −
  let SKIP=1720
  −
  DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
  −
  DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`                                   
  −
                                                                                                         
  −
  let SKIP=1024+256+128+256+4                                                                           
  −
  ORGX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')           
  −
  ORGX=`echo "scale=2; $ORGX/1000" | bc -l `                                                             
  −
  let SKIP=$SKIP+4                                                                                       
  −
  ORGY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')           
  −
  ORGY=`echo "scale=2; $ORGY/1000" | bc -l `                                                             
  −
 
  −
  let SKIP=1024+256+128+256+44
  −
  PHISTART=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
  −
  STARTING_ANGLE=`echo "scale=3; ($PHISTART/1000)" | bc -l`
  −
  let SKIP=1024+736
  −
  OSCILLATION_RANGE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
  −
  OSCILLATION_RANGE=`echo "scale=3; $OSCILLATION_RANGE/1000" | bc -l` 
  −
                                                                                             
  −
  let SKIP=1024+256+128+256+128+4                                                               
  −
  QX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')     
  −
  QX=`echo "scale=10; $QX/1000000" |bc -l `                                                     
  −
  let SKIP=$SKIP+4                                                                             
  −
  QY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')     
  −
  QY=`echo "scale=10; $QY/1000000" |bc -l `                                                     
  −
 
  −
  let SKIP=1024+256+128+256+128+128+12
  −
  X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
  −
  X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`                                   
  −
  X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
  −
# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:
  −
  NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `                             
  −
  ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `                             
  −
  if [ $ORGXINT -lt $NXBYFOUR ]; then
  −
    ORGXMM=$ORGX
  −
    ORGYMM=$ORGY
  −
    NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
  −
    NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`   
  −
    ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`                           
  −
    ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`                           
  −
    echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
  −
  else                                                                               
  −
    ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
  −
    ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
  −
    NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
  −
    NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`         
  −
    echo MARCCD detector: header ORGX, ORGY seem to be in pixel units               
  −
  fi                                                                                 
  −
 
  −
elif [ "$DET" == "adsc" ]; then
  −
 
  −
  DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
  −
  echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:
  −
  SENSOR_THICKNESS=0.01                                                       
  −
  sed s/\;// tmp2 > tmp1                                                     
  −
  mv tmp1 tmp2                                                               
  −
 
  −
      # find X_RAY_WAVELENGTH:
  −
      X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`
  −
 
  −
      # find NX, QX, ORGX and ORGY:
  −
      X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
  −
      STARTING_ANGLE=`grep PHI tmp2 | sed s/PHI=//`
  −
      NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
  −
      QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
  −
# FIXME - next 2 lines should be done properly, from header
  −
      NY=$NX                                             
  −
      QY=$QX
  −
      NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
  −
      NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`               
  −
      BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
  −
      BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
  −
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives - 
  −
# at ESRF and ... (pls fill in!) the following should be used:       
  −
      ORGX=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`             
  −
      ORGY=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`             
  −
      echo - at ESRF BLs use: ORGX=$ORGX ORGY=$ORGY                   
  −
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
  −
      ORGX=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `                                                     
  −
      ORGY=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `                                                         
  −
      echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX ORGY=$ORGY                                                         
  −
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ...
  −
      ORGX=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `                                                           
  −
      ORGY=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `
  −
      ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
  −
      ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`                                                     
  −
      echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX ORGY=$ORGY - this is written to XDS.INP                             
  −
# the latter alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
  −
      # find DETECTOR_DISTANCE and OSCILLATION_RANGE:                                                                               
  −
      DETECTOR_DISTANCE=`grep DISTANCE tmp2 | sed s/DISTANCE=//`                                                                   
  −
      OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`                                                                 
  −
 
  −
elif [ "$DET" == "pilatus" ]; then
  −
  DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576  !PILATUS"
  −
  QX=0.172 QY=0.172                                                       
  −
  echo Data from a Pilatus detector                                       
  −
  sed s/#// tmp2 > tmp1                                                   
  −
  mv tmp1 tmp2                                                             
  −
 
  −
      # find SENSOR_THICKNESS:
  −
      SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Silicon sensor, thickness'// | awk '{print $1*1000}'`
  −
      # find X_RAY_WAVELENGTH:                                                                             
  −
      X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`           
  −
 
  −
      X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
  −
      # find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M
  −
      NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`
  −
      NY=`grep X-Binary-Size-Second-Dimension tmp2 | awk '{print $2}'`
  −
      NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
  −
      NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`               
  −
      # find ORGX and ORGY:
  −
      ORGX=`grep Beam_xy tmp2 | sed -e s/\(// -e s/\)// -e s/\,// | awk '{print $2}'`
  −
      ORGY=`grep Beam_xy tmp2 | sed -e s/\(// -e s/\)// -e s/\,// | awk '{print $3}'`
  −
      ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
  −
      ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
  −
      # find DETECTOR_DISTANCE and OSCILLATION_RANGE:
  −
      DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
  −
      DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`
  −
 
  −
      OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`
  −
 
  −
else
  −
  echo should never come here
  −
  exit 1                   
  −
fi                         
  −
 
  −
echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
  −
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE                 
  −
echo OSCILLATION_RANGE= $OSCILLATION_RANGE                 
  −
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH                   
  −
# now we know everything that is required to generate XDS.INP
  −
 
  −
cat > XDS.INP << eof
  −
!****************************************************************************************
  −
! Written by generate_XDS.INP version $REVISION
  −
!
  −
! A few notes on usage and modification of this input file:
  −
!
  −
! If XDS fails after IDXREF with an error indicating that
  −
! an insufficient percentage of reflections were indexed,
  −
! but you are confident that the indexing solution is
  −
! correct, rerun XDS with the JOB keyword changed to
  −
! "DEFPIX INTEGRATE CORRECT".
  −
!
  −
! After the first round of integration (in a triclinic cell)
  −
! it is often helpful to reintegrate using the globally
  −
! refined cell parameters and the correct space group.
  −
! You can do this by renaming GXPARM.XDS to XPARM.XDS
  −
! - e.g. by the command "cp GXPARM.XDS XPARM.XDS" -
  −
! and rerunning DEFPIX, INTEGRATE and CORRECT.
  −
!
  −
! If you are trying to determine the optimal starting point for
  −
! data collection starting from a single image, first complete
  −
! a run of XDS using default parameters. Then, rerun XDS using
  −
! the correct cell and spacegroup as described above
  −
! ("cp GXPARM.XDS XPARM.XDS"), but on the second run of XDS, give
  −
! the JOB keyword as "JOB=DEFPIX XPLAN".
  −
!
  −
! Be warned that the starting spindle angles given by XPLAN are
  −
! relative to the first image given - XDS assumes that the first image
  −
! given has a phi of 0 unless explicitly told otherwise. If this is not
  −
! the case (e.g. you are indexing from a snapshot taken at phi=90,
  −
! you will need to add the actual phi (as reported in the image header)
  −
! to the values given by XPLAN in order for these values to make sense.
  −
!****************************************************************************************
  −
!
  −
!****************************************************************************************
  −
!General parameters:
  −
!****************************************************************************************
  −
 
  −
JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT
  −
 
  −
ORGX= $ORGX ORGY= $ORGY  ! check these values with adxv!
  −
!In mm, ORGX=$ORGXMM and ORGY=$ORGYMM
  −
 
  −
DETECTOR_DISTANCE= $DETECTOR_DISTANCE                   
  −
OSCILLATION_RANGE= $OSCILLATION_RANGE                   
  −
 
  −
X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
  −
!X-ray energy is $X_RAY_ENERGY eV (calculated from wavelength)                   
  −
 
  −
NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES
  −
 
  −
!REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing.
  −
!XDS will *only* use the reference dataset if the space group and cell
  −
!parameters are specified below. 
  −
 
  −
DATA_RANGE=1 $DATA_RANGE                                                   
  −
SPOT_RANGE=1 $SPOT_RANGE
  −
 
  −
!BACKGROUND_RANGE=1 10 !Default is first five degrees.
  −
 
  −
SPACE_GROUP_NUMBER=0                  ! 0 if unknown
  −
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known
  −
 
  −
INCLUDE_RESOLUTION_RANGE=50 0  ! After CORRECT, insert high resol limit; re-run CORRECT
  −
!                                Ice rings can be excluded from scaling with the keyword
  −
!                                EXCLUDE_RESOLUTION_RANGE, e.g. EXCLUDE_RESOLUTION_RANGE=2.28 2.22
  −
 
  −
!SECONDS=60 !Uncomment this line to tell XDS to wait 1min for the next image before aborting.
  −
 
  −
!****************************************************************************************
  −
!Parameters important for processing anomalous data:
  −
!****************************************************************************************
  −
 
  −
FRIEDEL'S_LAW=FALSE    ! This acts only on the CORRECT step. Even if the anomalous signal
  −
!                        in your dataset is very small, it still may be useful for
  −
!                        calculating an anomalous difference map.
  −
 
  −
STRICT_ABSORPTION_CORRECTION=FALSE !Change this to TRUE if the anomalous signal
  −
!                                  is strong: in that case, in CORRECT.LP the three
  −
!                                  "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values
  −
!                                  will be significantly> 1, e.g. 1.5.
  −
 
  −
WFAC1=1.0 !This parameter controls rejection of misfits during scaling. Sometimes,
  −
!          strong anomalous pairs may be rejected as misfits, in which case increasing
  −
!          WFAC1 (e.g. to 1.5) may improve anomalous signal.
  −
 
  −
!****************************************************************************************
  −
!Regions of the detector to be excluded during indexing and integration:
  −
!****************************************************************************************
  −
! exclude (mask) untrusted areas of detector, e.g. beamstop shadow :
  −
!UNTRUSTED_RECTANGLE= !1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat
  −
!UNTRUSTED_ELLIPSE= !2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed
  −
!UNTRUSTED_QUADRILATERAL= !x1 y1 x2 y2 x3 y3 x4 y4 ! see documentation
  −
 
  −
TRUSTED_REGION=0.00 1.2  ! Partially use corners of detectors; 1.41421=full use, 1.0=edge
  −
!                          It is important to note that TRUSTED_REGION is the only resolution
  −
!                          cutoff obeyed during indexing and integration;
  −
!                          Both INCLUDE_RESOLUTION_RANGE and EXCLUDE_RESOLUTION_RANGE are
  −
!                          ignored during IDXREF and INTEGRATE. So this parameter can be
  −
!                          important when indexing or integration steps fail.
  −
 
  −
!****************************************************************************************
  −
!Other parameters:
  −
!****************************************************************************************
  −
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=7000. 30000. ! Often 8000 is ok.
  −
STRONG_PIXEL=4          ! COLSPOT: only use strong reflections (default is 3; 6 may be better for strong data)
  −
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3 ! default of 6 is sometimes too high
  −
MAXIMUM_ERROR_OF_SPOT_POSITION=3 ! Increasing may help for poor quality data
  −
 
  −
MINPK=75 !Increase (e.g. MINPK=98) to improve data accuracy at the expense of completeness.
     −
DELPHI=5 !Size of integration wedge. Increasing to 10 or 20 may be helpful in some cases.
+
== Dependencies ==
 
+
The script makes use of many GNU commands, like <code>ls, grep, egrep, awk, cut, cat, echo, wc, bc, head, sed, tail, cp, od, python</code>. Some of them (like <code>od</code> and <code>python</code>) are only used in case of specific detectors (MarCCD and RAXIS, respectively).
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=13 ! Default is 9 - Increasing may improve data
+
The script will only work if all the required commands are available. They reside in either the <code>coreutils</code> RPM, or specific RPMs (<code>gawk, sed, bc, grep, python</code> ...). Please note that to get the <code>strings</code> command on some Linux distributions (e.g. FC23) you need to install the <code>binutils</code> RPM package.
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA=13      ! accuracy, particularly if finely-sliced on phi,
+
For Eiger data processing, the <code>h5dump</code> program must be installed. This is part of <code>hdf5-tools</code> (Ubuntu) or <code>hdf5</code> (RHEL). The .h5 files collected at Diamond Light Source require a very new version of h5dump (namely h5dump 1.10) to extract the OVERLOAD parameter from the .h5 file; this version is available by default in Ubuntu 20.04 and RHEL/CentOS 8.
!                                                  and does not seem to have any downsides.  
     −
MINIMUM_ZETA=0.05 ! Controls how close to the blind region (about phi) reflections should be
+
On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required (open a terminal and type <code>xcode-select --install</code>). These are also part of the (larger, but also free) [http://developer.apple.com/tools/xcode Xcode] package. This package comes with a license that has to be accepted by the user when running a Command Line Tool (e.g. <code>strings</code>) for the first time.
!                  integrated. 0.05 is the default; increasing MINIMUM_ZETA to 0.15 *may*
  −
!                  improve data quality by removing unreliable reflections near the phi
  −
!                  axis, but will reduce completeness in low symmetry space groups.
     −
SEPMIN=6        ! Reduce if spots are close together (due to a long axis). Consider
+
One way to check for missing programs is
CLUSTER_RADIUS=3 ! increasing if crystal is split and unit cell dimensions are relatively short.
+
#!/bin/bash
 +
for i in  ls grep egrep awk cut cat echo wc bc head sed tail cp od python strings h5dump ; do
 +
    if [ ! -x /bin/$i ] && [ ! -x /usr/bin/$i ]; then
 +
      echo $i not found
 +
    fi
 +
done
   −
! For bad or low resolution data remove the "!" in the following line (default is ALL):
+
A command that should go a long way in providing all these tools for RedHat-derived distros is (as root)
! REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
+
yum -y install coreutils binutils gawk sed bc grep python hdf5
REFINE(INTEGRATE)=CELL BEAM ORIENTATION ! AXIS DISTANCE (If integration is unstable, comment out this line.
+
and for Ubuntu this would be (untested!)
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE ! Default is: refine everything
+
sudo apt-get install coreutils binutils gawk sed bc grep python hdf5-tools
   −
ROTATION_AXIS=1 0 0  !At e.g. Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?), BM30A this needs to be -1 0 0
+
See also [[Installation]].
   −
!****************************************************************************************
+
== Limitations ==
! Parameters specifically for this detector and beamline (shouldn't need changing):
  −
!****************************************************************************************
  −
DETECTOR= $DETECTOR
  −
SENSOR_THICKNESS= $SENSOR_THICKNESS
  −
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
  −
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon)
     −
NX= $NX NY= $NY  QX= $QX  QY= $QY ! to make CORRECT happy if frames are unavailable
+
* The script tries to interpret the header of the frames, so is currently limited to Dectris (Pilatus, Eiger), ADSC (Quantum), Rigaku (several types), MAR (CCD and image plate) detectors, and one Bruker detector. Other detectors need some values to be manually filled into XDS.INP - the relevant places are marked with XXX. These are detector properties (type, pixel size and number, min and max counts in a pixel), and experimental parameters like ROTATION_AXIS, OSCILLATION_RANGE, X-RAY_WAVELENGTH, DETECTOR_DISTANCE, and XORG, YORG. For fine-tuning of detector parameters, see the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates].
!In mm, NX=$NXMM and NY=$NYMM
+
* The authors have made a "best effort" to provide a XDS.INP that results in the correct sign of the anomalous signal. In the case of one detector type (internally called Rigaku SMV) this requires reversal of one detector axis, and a negative DETECTOR_DISTANCE, as is found in some of the [http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_prepare.html detector-specific templates]. '''For an unusual or unknown detector setup, the correct sign of the anomalous signal needs to be established and verified e.g. with a good dataset from a test crystal that has an anomalous signal.''' The authors do not take any responsibility for problems arising from incorrect sign of the anomalous signal, nor - obviously! - for any other mischief arising in or from data processing.
 +
* At some beamlines, the ROTATION_AXIS should be -1 0 0 ("backwards") instead of the usual 1 0 0 ("horizontal"), or even 0 1 0 ("vertical") like at one of the PETRA Hamburg BLs. The frame headers do not have this information, so the default chosen by [[generate_XDS.INP]] may be wrong and need manual correction. Pls also see the article [[Beamline notes]].
 +
* Sometimes, the x- and y- values of the primary beam position recorded in the header should be used for ORGY and ORGX (i.e reversed) instead of as ORGX and ORGY. For ADSC, this has been implemented in the script for a number of beamlines.
 +
* there are apparently several flavours of HDF5 files produced at Diamond Light Source. They differ e.g. in the naming of the header items. This means that items like NX, NY, DETECTOR_DISTANCE and number of images cannot be found by the [[generate_XDS.INP]] script. Example: The data at /dls/i04-1/data/2021/mx28114-9/processing/Lenye_Diamini/ThiL/ThiL found during the CCP4 2021 online Cape Town workshop. A workaround is to use e.g. xia2 pipeline=3dii to process these files, and - if needed - extract those items from its output files, e.g. from DEFAULT/NATIVE/SWEEP1/index/XDS.INP .
   −
DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
+
== See also ==
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
  −
INCIDENT_BEAM_DIRECTION=0 0 1
  −
FRACTION_OF_POLARIZATION=0.98  ! better value is provided by beamline staff!
  −
POLARIZATION_PLANE_NORMAL=0 1 0
  −
!used by DEFPIX and CORRECT to exclude ice-reflections / ice rings - uncomment if necessary
  −
!EXCLUDE_RESOLUTION_RANGE= 3.93 3.87 !ice-ring at 3.897 Angstrom
  −
!EXCLUDE_RESOLUTION_RANGE= 3.70 3.64 !ice-ring at 3.669 Angstrom
  −
!EXCLUDE_RESOLUTION_RANGE= 3.47 3.41 !ice-ring at 3.441 Angstrom
  −
!EXCLUDE_RESOLUTION_RANGE= 2.70 2.64 !ice-ring at 2.671 Angstrom
  −
!EXCLUDE_RESOLUTION_RANGE= 2.28 2.22 !ice-ring at 2.249 Angstrom
  −
!EXCLUDE_RESOLUTION_RANGE= 2.102 2.042 !ice-ring at 2.072 Angstrom - strong
  −
!EXCLUDE_RESOLUTION_RANGE= 1.978 1.918 !ice-ring at 1.948 Angstrom - weak
  −
!EXCLUDE_RESOLUTION_RANGE= 1.948 1.888 !ice-ring at 1.918 Angstrom - strong
  −
!EXCLUDE_RESOLUTION_RANGE= 1.913 1.853 !ice-ring at 1.883 Angstrom - weak
  −
!EXCLUDE_RESOLUTION_RANGE= 1.751 1.691 !ice-ring at 1.721 Angstrom - weak
  −
eof
  −
if [ "$DET" == "pilatus" ]; then
  −
  if [ $NX == "1475" ]; then
  −
    if ! grep -q FF_p2m0109_E12398_T6199_vrf_m0p20.tif tmp2 ; then
  −
    cat >> XDS.INP << eof
  −
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
  −
UNTRUSTED_RECTANGLE= 486  496    0 1680
  −
UNTRUSTED_RECTANGLE= 980  990    0 1680
  −
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
  −
UNTRUSTED_RECTANGLE=  0 1476  194  214
  −
UNTRUSTED_RECTANGLE=  0 1476  406  426
  −
UNTRUSTED_RECTANGLE=  0 1476  618  638
  −
UNTRUSTED_RECTANGLE=  0 1476  830  850
  −
UNTRUSTED_RECTANGLE=  0 1476  1042 1062
  −
UNTRUSTED_RECTANGLE=  0 1476  1254 1274
  −
UNTRUSTED_RECTANGLE=  0 1476  1466 1486
  −
eof
  −
    fi
  −
#  elif [ $NX == "2463" ]; then
  −
#    cat >> XDS.INP << eof
  −
#eof
  −
  fi
  −
fi
  −
touch xds_to_ccp4.sh
  −
echo "#!/bin/bash
  −
#Removes old log file if present
  −
rm xds_to_ccp4.log
  −
#Makes new log file
  −
touch xds_to_ccp4.log
  −
#Pipe stdout and stderr to both logfile and tty
  −
tail -f -n1 xds_to_ccp4.log &
  −
exec > xds_to_ccp4.log 2>&1
  −
echo \"
  −
 
  −
 
  −
If this message is all you see,
  −
something has probably gone wrong.
  −
Are you sure XDS_ASCII.HKL is present
  −
in the current directory?
  −
 
  −
 
  −
\"
  −
#Make first MTZ file (with DANO/SIGDANO)
  −
echo \"INPUT_FILE=XDS_ASCII.HKL
  −
OUTPUT_FILE=temp.hkl CCP4
  −
MERGE=TRUE
  −
GENERATE_FRACTION_OF_TEST_REFLECTIONS=0.05
  −
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
  −
xdsconv
  −
f2mtz HKLOUT temp.mtz<F2MTZ.INP
  −
 
  −
#Make second MTZ file (with F(+)/F(-))
  −
echo \"INPUT_FILE=XDS_ASCII.HKL
  −
OUTPUT_FILE=temp2.hkl CCP4_F
  −
MERGE=TRUE
  −
GENERATE_FRACTION_OF_TEST_REFLECTIONS=0.05
  −
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
  −
xdsconv
  −
f2mtz HKLOUT temp2.mtz<F2MTZ.INP
  −
 
  −
#Make SHELX format unmerged hkl file
  −
echo \"INPUT_FILE=XDS_ASCII.HKL
  −
OUTPUT_FILE=xds_shelx.hkl SHELX
  −
MERGE=FALSE
  −
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
  −
xdsconv
  −
 
  −
#cad the two previously generated MTZ files together
  −
cad HKLIN1 temp.mtz HKLIN2 temp2.mtz HKLOUT xds_ccp4_merged.mtz<<EOF
  −
LABIN  FILE 1 E1=FP      E2=SIGFP      E3=DANO    E4=SIGDANO    E5=FreeRflag
  −
LABIN  FILE 2 E1=F(+)    E2=SIGF(+)    E3=F(-)    E4=SIGF(-)
  −
END
  −
EOF
  −
 
  −
#Remove temporary files
  −
rm temp.mtz
  −
rm temp2.mtz
  −
rm temp.hkl
  −
rm temp2.hkl
  −
rm F2MTZ.INP
  −
 
  −
echo \"
  −
 
  −
********************************XDS_TO_CCP4**********************************
  −
 
  −
This script takes XDS_ASCII.HKL and uses XDSCONV and F2MTZ to
  −
generate a merged CCP4-format MTZ file named
  −
\\\"xds_ccp4_merged.mtz\\\" with a test (free) set
  −
labeled \\\"FreeRflag\\\" constituting 5% of reflections.
  −
 
  −
The space group of the output MTZ file will be the same as
  −
that present in XDS_ASCII.HKL. It is probably worth checking
  −
that this is the correct space group - either manually inspect
  −
the systematic absences in CORRECT.LP, or run XDS_ASCII.HKL
  −
through the CCP4 program POINTLESS (\\\"pointless xdsin XDS_ASCII.HKL\\\")
  −
 
  −
Anomalous data is kept, if present, and output as both
  −
DANO/SIGDANO and F(+)/F(-).
  −
 
  −
This script also generates an unmerged SHELX hkl file,
  −
suitable for input into SHELXC/D/E, named \\\"xds_shelx.hkl\\\".
  −
 
  −
A log of all processes is present in xds_to_ccp4.log.
  −
 
  −
Parts of this script have been shamelessly copied from
  −
the XDS Wiki, at http://www.strucbio.biologie.uni-konstanz.de/xdswiki/
  −
 
  −
******************************************************************************
  −
\"" > xds_to_ccp4.sh
  −
touch xds_graph.sh
  −
echo "#!/bin/bash
  −
#Makes new log file
  −
touch INTEGRATE_STATS.LP
  −
#Pipe stdout and stderr to both logfile and tty
  −
tail -f -n1 INTEGRATE_STATS.LP &
  −
exec > INTEGRATE_STATS.LP 2>&1
  −
if command -v xdsstat >/dev/null; then
  −
echo \"\" | xdsstat > XDSSTAT.LP
  −
echo \"\" | xdsstat 100 6 > XDSSTAT.LP_low
  −
echo \"
  −
\\\$TABLE:Rmeas vs image (to 6A):
  −
\\\$SCATTER:Rmeas vs image for low resolution reflections:A:1,2: \\$\\$
  −
IMAGE RMEAS
  −
\\$\\$ \\$\\$
  −
\"
  −
grep ' L\$' XDSSTAT.LP_low | awk '{print \$1,\$9}'
  −
echo \" \\$\\$ \"
  −
echo \"
  −
\\\$TABLE:Rmeas vs image (all):
  −
\\\$SCATTER:Rmeas vs image (all reflections):A:1,2: \\$\\$
  −
IMAGE RMEAS
  −
\\$\\$ \\$\\$
  −
\"
  −
grep ' L$' XDSSTAT.LP | awk '{print \$1,\$9}'
  −
echo \" \\$\\$ \"
  −
echo \"
  −
\\\$TABLE:Rd vs image (to 6A):
  −
\\\$SCATTER:Rd (from XDSSTAT) for low resolution reflections:A:1,2: \\$\\$
  −
IMAGE RD
  −
\\$\\$ \\$\\$
  −
\"
  −
grep 'DIFFERENCE' XDSSTAT.LP_low | awk '{print \$1,\$7}'
  −
echo \" \\$\\$ \"
  −
echo \"
  −
\\\$TABLE:Rd vs image (all):
  −
\\\$SCATTER:Rd (from XDSSTAT) for all reflections:A:1,2: \\$\\$
  −
IMAGE RD
  −
\\$\\$ \\$\\$
  −
\"
  −
grep 'DIFFERENCE' XDSSTAT.LP | awk '{print \$1,\$7}'
  −
echo \" \\$\\$ \"
  −
fi
  −
 
  −
echo \"
  −
\\\$TABLE: XDS Integration Statistics :
  −
\\\$SCATTER:Scale factors by image:A:1,3: :Number of overloads by image:A:1,5: :Number of strong reflections by image:A:1,7: :Number of rejects by image:A:1,8: :Mosaicity by image:A:1,10: \\$\\$
  −
IMAGE IER SCALE NBKG NOVL NEWALD NSTRONG NREJ SIGMAB SIGMAR
  −
\\$\\$ \\$\\$
  −
\"
  −
egrep \"[0-9]\.[0-9][0-9][0-9][0-9]  [0-9]\.[0-9][0-9][0-9][0-9]\" INTEGRATE.LP
  −
echo \" \\$\\$ \"
  −
echo \"
  −
\\\$TABLE: XDS Scaling Statistics by Resolution (X-axis is resolution in Angstroems) :
  −
\\\$GRAPHS:Chi-Squared by resolution:A:2,4: :Observed and expected R-factors by resolution (as %):A:2,5,6: :Rejects by resolution:A:2,9: :Rejects by resolution as a percentage of total reflections:A:2,10: \\$\\$
  −
LOW_RES Resolution(Angstroems)  I/Sigma  Chi^2  R-FACTOR(OBSERVED)  R-FACTOR(EXPECTED)  NUMBER_OF_REFLECTIONS ACCEPTED REJECTS REJECT_PERCENT
  −
\\$\\$ \\$\\$
  −
\"
  −
awk '/RESOLUTION RANGE  I\/Sigma  Chi\^2  R\-FACTOR  R\-FACTOR  NUMBER ACCEPTED REJECTED/, /\-\-\-\-\-\-\-\-\-/' CORRECT.LP | egrep -v \"[a-z,A-Z,] | \-\-|\-99\.9\" | egrep \"[0-9]\" | awk '{print \$1,\$2,\$3,\$4,\$5,\$6,\$7,\$8,\$9,(\$9/(\$8+\$9))*100}'
  −
echo \" \\$\\$\"
  −
echo \"
  −
\\\$TABLE: Overall data quality by Resolution (X-axis is resolution in Angstroems) :
  −
\\\$GRAPHS:Completeness vs resolution :A:1,6: :Redundancy vs resolution :A:1,4: :I/Sigma by resolution:A:1,10: :Rmeas by resolution:A:1,11: :Anomalous correlation by resolution:A:1,13: :Significance of the anomalous signal by resolution:A:1,14:  \\$\\$
  −
Resolution(Angstroems) OBSERVED_REFLECTIONS UNIQUE_REFLECTIONS REDUNDANCY POSSIBLE_REFLECTIONS COMPLETENESS R-FACTOR(OBSERVED) R-FACTOR(EXPECTED) COMPARED I/SIGMA R-meas CC(1/2) Anomalous_Correlation SigAno Nano
  −
\\$\\$ \\$\\$
  −
\"
  −
egrep -B25 \" WILSON STATISTICS \" CORRECT.LP | egrep -v \"[a-z,A-Z]|\*\*\*\" | egrep \"  \" | awk '{ gsub(/[%*]/,\" \"); print }' | awk '{print \$1,\$2,\$3,\$2/\$3,\$4,\$5,\$6,\$7,\$8,\$9,\$10,\$11,\$12,\$13,\$14}'
  −
echo \" \\$\\$ \"
  −
 
  −
echo \"
  −
\\\$TABLE: I/Sigma of h00 reflections (for identification of systematic absences):
  −
\\\$GRAPHS:I/Sigma vs h :A:1,5: \\$\\$
  −
h k l RES I/SIGMA NUMBER
  −
\\$\\$ \\$\\$
  −
\"
  −
awk '/REFLECTIONS OF TYPE H/,/COMP/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$1 > 0)'
  −
echo \"\\$\\$\"
  −
 
  −
echo \"
  −
\\\$TABLE: I/Sigma of 0k0 reflections (for identification of systematic absences):
  −
\\\$GRAPHS:I/Sigma vs k :A:2,5: \\$\\$
  −
h k l RES I/SIGMA NUMBER
  −
\\$\\$ \\$\\$
  −
\"
  −
awk '/REFLECTIONS OF TYPE H/,/COMP/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$2 > 0)'
  −
echo \"\\$\\$\"
  −
 
  −
echo \"
  −
\\\$TABLE: I/Sigma of 00l reflections (for identification of systematic absences):
  −
\\\$GRAPHS:I/Sigma vs l :A:3,5: \\$\\$
  −
h k l RES I/SIGMA NUMBER
  −
\\$\\$ \\$\\$
  −
\"
  −
awk '/REFLECTIONS OF TYPE H/,/COMP/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$3 > 0)'
  −
echo \"\\$\\$\"
  −
 
  −
echo \"
  −
\\\$TABLE:Beam center (ORGX/ORGY) plotted vs wedge (DELPHI):
  −
\\\$GRAPHS:ORGX and ORGY vs batch (DELPHI) :A:1,2,3: \\$\\$
  −
N ORGX ORGY
  −
\\$\\$ \\$\\$
  −
\"
  −
egrep \"DETECTOR ORIGIN\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
  −
echo \"\\$\\$\"
  −
 
  −
echo \"
  −
\\\$TABLE:Unit cell parameters plotted vs wedge:
  −
\\\$GRAPHS:Unit cell lengths a, b and c plotted vs wedge:A:1,2,3,4: :Unit cell angles alpha, beta and gamma plotted vs wedge:A:1,5,6,7: \\$\\$
  −
N a b c ALPHA BETA GAMMA
  −
\\$\\$ \\$\\$
  −
\"
  −
egrep \"UNIT CELL PARAMETERS\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
  −
echo \"\\$\\$\"
  −
 
  −
echo \"
  −
\\\$TABLE:Standard deviation of spot position (pixels) plotted vs wedge:
  −
\\\$GRAPHS:Standard deviation of spot position plotted vs wedge:A:1,2: \\$\\$
  −
N SD
  −
\\$\\$ \\$\\$
  −
\"
  −
egrep \"SPOT    POSITION\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
  −
echo \"\\$\\$\"
  −
 
  −
echo \"
  −
\\\$TABLE:Detector distance (mm) plotted vs wedge (DELPHI):
  −
\\\$GRAPHS:Detector distance plotted vs batch:A:1,2: \\$\\$
  −
N DIST
  −
\\$\\$ \\$\\$
  −
\"
  −
egrep \"DETECTOR DISTANCE \\(mm\\)\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
  −
 
  −
echo \"\\$\\$\"
  −
 
  −
loggraph INTEGRATE_STATS.LP">xds_graph.sh
  −
chmod +x xds_to_ccp4.sh
  −
chmod +x xds_graph.sh
  −
echo "XDS.INP is ready for use. The file has only the most important keywords.
  −
    Full documentation, including complete detector templates, is at
  −
    http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
  −
    After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
  −
    BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!
  −
 
  −
    Two shell scripts have been generated that may be of use.
  −
 
  −
    The first, xds_graph.sh, will plot various statistics after an XDS run.
  −
    After both INTEGRATE and CORRECT have finished, run it by typing \"./xds_graph.sh\"
  −
    without the quotation marks. This script uses loggraph to plot data, so you will
  −
    need CCP4 to see the graphical output. All the raw data is piped out to a log file,
  −
    INTEGRATE_STATS.LP.
  −
 
  −
    If XDSSTAT is present in your path, xds_graph will also plot some of the statistics it
  −
    calculates - specifically the Rmeas and Rd per image, for all reflections and only
  −
    low resolution reflections.
  −
 
  −
    The second script, xds_to_ccp4.sh, will take XDS_ASCII.HKL and generate both a CCP4
  −
    format MTZ file with anomalous data retained (as F(+)/F(-) and DANO/SigDANO)
  −
    and an unmerged SHELX format hkl file for input to SHELXC/D/E. Run it after CORRECT
  −
    by typing \"./xds_to_ccp4.sh\" at the prompt."
  −
 
  −
rm -f tmp1 tmp2
  −
</pre>
     −
Use this script in the same manner as Generate_XDS.INP. The two shell scripts created should be executable; If they are not, then make them so (chmod +x ./xxx.sh).
+
* GlobalPhasing's list of beamlines and their settings is at http://www.globalphasing.com/autoproc/wiki/index.cgi?BeamlineSettings .
 +
* [[Beamline notes]] to collect information about specific settings
 +
* [[spot2pdb]] for visualizing reciprocal space
2,461

edits

Cookies help us deliver our services. By using our services, you agree to our use of cookies.

Navigation menu