Generate XDS.INP

From XDSwiki
Jump to navigationJump to search

This script generates XDS.INP based on a list of frame names supplied on the commandline. It currently works for MarCCD, ADSC and Pilatus detectors; since this is just a bash script, extension to other detectors is very easy.

On Mac OS X, installation of the "Command Line Tools" (from http://developer.apple.com/downloads; requires Apple ID) is required (I think). These are a part of the (larger, but also free) Xcode package.

Usage

Usage is just (don't forget the quotation marks!):

generate_XDS.INP "frms/mydata_1_???.img"

XDS supports bzip2-ed frames. Thus, when specifying the frame name parameter of the script, you should leave out any .bz2 extension.

The script

#!/bin/bash                                                                   
# purpose: generate XDS.INP                                                   
#                                                                             
# tested with some datasets from ALS, SSRL, SLS, ESRF and BESSY; only MARCCD, ADSC/SMV, PILATUS detectors; 
# for other detectors, values marked with XXX must be manually filled in.                                  
#                                                                                                          
# revision 0.03 . Kay Diederichs 2/2010                                                                    
# revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC             
# revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY   
# revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on 
# revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes        
# revision 0.08 . KD 6/2010 - fixes for Pilatus 6M                                                          
# revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od"   
# revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters    
# revision 0.11 . KD 7/2010 - for MarCCD: look for distance info at different byte position                 
# revision 0.12 . KD 7/2010 - fix for negative PHISTART                                                     
# revision 0.13 . KD 8/2010 - store correct NX NY QX QY in XDS.INP                                          
# revision 0.14 . KD 1/2011 - SENSOR_THICKNESS for Pilatus; MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3            
# revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis
# revision 0.16 . KD 3/2011 - SENSOR_THICKNESS=0.01 for ADSC and MarCCD. Add comment about SILICON=          
# revision 0.17 . KD 3/2011 - make it work for .bz2 frames; improve screen output                            
# revision 0.18 . KD 4/2011 - faster by doing "strings" only once; revert "images/${1##/*/}" "correction"    
# revision 0.19 . KD 6/2011 - bugfix for 0.18                                                                
# revision 0.20 . KD 7/2011 - redirect stderr of /bin/ls to /dev/null                                        
# revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)
# revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header
# revision 0.23 . KD 1/2012 - add UNTRUSTED_QUADRILATERAL, remove MINIMUM_ZETA (0.05 is default now)
# revision 0.24 . KD 3/2012 - remove revision 0.10 since XDS now takes much longer paths
# revision 0.25 . KD 3/2012 - remove revision 0.22 for PSI Pilatus 2M; see http://www.globalphasing.com/autoproc/wiki/index.cgi?TroubleShootingKnownIssues
# revision 0.26 . KD 7/2012 - Mac-compatibility: replace od flags --skip-bytes= and --read-bytes= with -j and -N (thanks to Oliver Clarke for working this out!)
REVISION="0.26 (02-Jul-2012)"                                                                                
#                                                                                                            
# usage: e.g. generate_XDS.INP "frms/mydata_1_???.img"                                                       
# make sure to have the two quotation marks !                                                                
# the ? are wildcards for the frame numbers.                                                                 
#                                                                                                            
# limitations:                                                                                               
# - frame numbers are assumed to start with 1 and run consecutively                                          
#                                                                                                            
# known problems:                                                                                            
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - the same might be a problem for MAR headers, too (not sure about this) 
# - on Mac OS X, Xcode (from http://developer.apple.com/tools/xcode) might be needed - not sure about this                               
#                                                                                                                
# notes for debugging of the script:                                                                             
# - add the -v option to the first line, to see where an error occurs                                            
# - comment out the removal of tmp1 and tmp2 in the last line                                                    
#                                                                                                                
# ====== Start of script ======                                                                                  
echo generate_XDS.INP version $REVISION . Obtain the latest version from                                         
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP                                 
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then                                           
  echo usage: generate_XDS.INP \"frms/mydata_1_???.img\"   \(_with_ the quotation marks!\)                       
  echo if the frames are compressed with bzip2, leave out the .bz2 extension!                                    
  exit                                                                                                           
fi                                                                                                               
#                                                                                                                
# defaults:                                                                                                      
#                                                                                                                
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"                                                        
ORGX=XXX                                                                                                         
ORGY=XXX                                                                                                         
DETECTOR_DISTANCE=XXX                                                                                            
OSCILLATION_RANGE=XXX                                                                                            
X_RAY_WAVELENGTH=XXX                                                                                             
QX=XXX                                                                                                           
QY=XXX                                                                                                           
NX=XXX                                                                                                           
NY=XXX                                                                                                           
SENSOR_THICKNESS=0                                                                                               
# see how we are called:                                                                                         
NAME_TEMPLATE_OF_DATA_FRAMES="$1"
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II 
/bin/ls -C1 $1 $1.bz2 2>/dev/null | egrep -v "_00000.cbf|_000.img" > tmp1 || exit 1

# we can continue - the frames are found

# set upper limit of DATA_RANGE to number of frames (see "limitations" above)
DATA_RANGE=`wc -l tmp1 | awk '{print $1}'`                                   

# set upper limit of SPOT_RANGE to half of DATA_RANGE, but not less than 1
SPOT_RANGE=`echo "scale=0; $DATA_RANGE/2" | bc -l`                        
SPOT_RANGE=`echo "if ($SPOT_RANGE<1) 1;if ($SPOT_RANGE>1) $SPOT_RANGE" | bc -l`

echo DATA_RANGE=1 $DATA_RANGE

# find out detector type
DET=XXX                 
FIRSTFRAME=`head -1 tmp1`
echo $FIRSTFRAME | grep -q bz2 && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
strings $FIRSTFRAME > tmp2                                                   
egrep -q 'marccd|Corrected' tmp2 && DET=mccd                                 
grep -q PILATUS tmp2             && DET=pilatus                              
grep -q BEAM_CENTER_X tmp2       && DET=adsc                                 
# identify other detector types in the same way (MAR IP would be straightforward)

# parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then
  echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
  DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"                            

# find parameters of first frame
elif [ "$DET" == "mccd" ]; then 
  echo Data from a MarCCD detector
                                  
  DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
  SENSOR_THICKNESS=0.01                                           
  # use first frame of dataset to obtain parameters               

  # offsets are documented; values can be found in mccd_xdsparams.pl script
  let SKIP=1024+80                                                        
  NX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
  let SKIP=$SKIP+4                                                                         
  NY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')

  let SKIP=1720
  DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
  DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`                                     
                                                                                                          
  let SKIP=1024+256+128+256+4                                                                             
  ORGX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')             
  ORGX=`echo "scale=2; $ORGX/1000" | bc -l `                                                              
  let SKIP=$SKIP+4                                                                                        
  ORGY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')             
  ORGY=`echo "scale=2; $ORGY/1000" | bc -l `                                                              

  let SKIP=1024+256+128+256+44
  PHISTART=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
  let SKIP=1024+256+128+256+76                                                                   
  PHIEND=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')  
  OSCILLATION_RANGE=`echo "scale=3; ($PHIEND-($PHISTART))/1000" | bc -l`                         
                                                                                                 
  let SKIP=1024+256+128+256+128+4                                                                
  QX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')      
  QX=`echo "scale=10; $QX/1000000" |bc -l `                                                      
  let SKIP=$SKIP+4                                                                               
  QY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')      
  QY=`echo "scale=10; $QY/1000000" |bc -l `                                                      

  let SKIP=1024+256+128+256+128+128+12
  X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
  X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`                                    

# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:
  NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `                               
  ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `                              
  if [ $ORGXINT -lt $NXBYFOUR ]; then                                     
     ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`                             
     ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`                             
     echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
  else                                                                                
     echo MARCCD detector: header ORGX, ORGY seem to be in pixel units                
  fi                                                                                  

elif [ "$DET" == "adsc" ]; then

  DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
  echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:
  SENSOR_THICKNESS=0.01                                                        
  sed s/\;// tmp2 > tmp1                                                       
  mv tmp1 tmp2                                                                 

      # find X_RAY_WAVELENGTH:
      X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`

      # find NX, QX, ORGX and ORGY:
      NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
      QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
# FIXME - next 2 lines should be done properly, from header
      NY=$NX                                               
      QY=$QX                                               
      BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
      BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -  
# at ESRF and ... (pls fill in!) the following should be used:         
      ORGX=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`               
      ORGY=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`               
      echo - at ESRF BLs use: ORGX=$ORGX ORGY=$ORGY                    
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
      ORGX=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `                                                       
      ORGY=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `                                                           
      echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX ORGY=$ORGY                                                          
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ... 
      ORGX=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `                                                            
      ORGY=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l `                                                        
      echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX ORGY=$ORGY - this is written to XDS.INP                              
# the latter alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
      # find DETECTOR_DISTANCE and OSCILLATION_RANGE:                                                                                
      DETECTOR_DISTANCE=`grep DISTANCE tmp2 | sed s/DISTANCE=//`                                                                     
      OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`                                                                   

elif [ "$DET" == "pilatus" ]; then
  DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576  !PILATUS"
  QX=0.172 QY=0.172                                                         
  echo Data from a Pilatus detector                                         
  sed s/#// tmp2 > tmp1                                                     
  mv tmp1 tmp2                                                              

      # find SENSOR_THICKNESS:
      SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Silicon sensor, thickness'// | awk '{print $1*1000}'`
      # find X_RAY_WAVELENGTH:                                                                               
      X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`            

      # find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M
      NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`
      NY=`grep X-Binary-Size-Second-Dimension tmp2 | awk '{print $2}'` 

      # find ORGX and ORGY:
      ORGX=`grep Beam_xy tmp2 | sed -e s/\(// -e s/\)// -e s/\,// | awk '{print $2}'`
      ORGY=`grep Beam_xy tmp2 | sed -e s/\(// -e s/\)// -e s/\,// | awk '{print $3}'`

      # find DETECTOR_DISTANCE and OSCILLATION_RANGE:
      DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
      DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

      OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`

else
  echo should never come here
  exit 1                     
fi                           

echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE                   
echo OSCILLATION_RANGE= $OSCILLATION_RANGE                   
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH                     

# now we know everything that is required to generate XDS.INP

cat > XDS.INP << eof
! written by generate_XDS.INP version $REVISION
JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT
ORGX= $ORGX ORGY= $ORGY  ! check these values with adxv !
DETECTOR_DISTANCE= $DETECTOR_DISTANCE                    
OSCILLATION_RANGE= $OSCILLATION_RANGE                    
X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH                      
NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES
! REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing  
DATA_RANGE=1 $DATA_RANGE                                                     
SPOT_RANGE=1 $SPOT_RANGE                                                     
! BACKGROUND_RANGE=1 10 ! rather use defaults (first 5 degree of rotation)   

SPACE_GROUP_NUMBER=0                   ! 0 if unknown
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known
INCLUDE_RESOLUTION_RANGE=50 0  ! after CORRECT, insert high resol limit; re-run CORRECT

FRIEDEL'S_LAW=FALSE     ! This acts only on the CORRECT step
! If the anom signal turns out to be, or is known to be, very low or absent,
! use FRIEDEL'S_LAW=TRUE instead (or comment out the line); re-run CORRECT

! remove the "!" in the following line:
! STRICT_ABSORPTION_CORRECTION=TRUE
! if the anomalous signal is strong: in that case, in CORRECT.LP the three
! "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values are significantly> 1, e.g. 1.5
!
! exclude (mask) untrusted areas of detector, e.g. beamstop shadow :
! UNTRUSTED_RECTANGLE= 1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat
! UNTRUSTED_ELLIPSE= 2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed
! UNTRUSTED_QUADRILATERAL= x1 y1 x2 y2 x3 y3 x4 y4 ! see documentation
!
! parameters with changes wrt default values:
TRUSTED_REGION=0.00 1.2  ! partially use corners of detectors; 1.41421=full use
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=7000. 30000. ! often 8000 is ok
STRONG_PIXEL=4           ! COLSPOT: only use strong reflections (default is 3)
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3 ! default of 6 is sometimes too high
! close spots: reduce SEPMIN and CLUSTER_RADIUS from their defaults of 6 and 3, e.g. to 4 and 2
! for bad or low resolution data remove the "!" in the following line: 
! REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
REFINE(INTEGRATE)=CELL BEAM ORIENTATION ! AXIS DISTANCE
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE ! Default is: refine everything

! parameters specifically for this detector and beamline:
DETECTOR= $DETECTOR
SENSOR_THICKNESS= $SENSOR_THICKNESS
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon)
NX= $NX NY= $NY  QX= $QX  QY= $QY ! to make CORRECT happy if frames are unavailable
DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
INCIDENT_BEAM_DIRECTION=0 0 1
ROTATION_AXIS=1 0 0  ! at e.g. Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?) this needs to be -1 0 0
FRACTION_OF_POLARIZATION=0.98   ! better value is provided by beamline staff!
POLARIZATION_PLANE_NORMAL=0 1 0
eof
if [ "$DET" == "pilatus" ]; then
  if [ $NX == "1475" ]; then
    if ! grep -q FF_p2m0109_E12398_T6199_vrf_m0p20.tif tmp2 ; then
    cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 486  496     0 1680
UNTRUSTED_RECTANGLE= 980  990     0 1680
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE=   0 1476   194  214
UNTRUSTED_RECTANGLE=   0 1476   406  426
UNTRUSTED_RECTANGLE=   0 1476   618  638
UNTRUSTED_RECTANGLE=   0 1476   830  850
UNTRUSTED_RECTANGLE=   0 1476  1042 1062
UNTRUSTED_RECTANGLE=   0 1476  1254 1274
UNTRUSTED_RECTANGLE=   0 1476  1466 1486
eof
    fi
#  elif [ $NX == "2463" ]; then
#    cat >> XDS.INP << eof
#eof
  fi
fi
echo XDS.INP is ready for use. The file has only the most important keywords.
echo Full documentation, including complete detector templates, is at
echo http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
echo After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
echo BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!
rm -f tmp1 tmp2

System-wide or personal installation

Ask your system adminstrator to cut-and-paste the script into e.g. /usr/local/bin/generate_XDS.INP, and to make it "executable".

But you may also cut-and-paste the script from this webpage into a file in e.g. your home directory; the filename should be generate_XDS.INP. After creating the file, make it executable - e.g. if it's in your $HOME, use:

chmod +x ~/generate_XDS.INP

After that, you can just run it in a similar way as if it were installed in your $PATH:

~/generate_XDS.INP "frms/mydata_1_???.img"

By using your own file, you can easily update to the latest revision, or even change the script, without having to bother the system administrator.

Generating generate_XDS.INP from this webpage

Instead of cutting-and-pasting the lines of the script, you (or the system administrator) could just cut-and-paste the following four lines

 wget http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP -O - | \
   sed -e s/\&nbsp\;/\ /g -e s/\&gt\;/\>/g -e s/\&lt\;/\</g -e s/amp\;//g -e s/\&quot\;/\"/g -e s/\&\#\1\6\0\;/\ /g | \
   awk '/^#/,/rm -f tmp1 tmp2/' > generate_XDS.INP
 chmod +x generate_XDS.INP

to copy the script from this website into an executable file generate_XDS.INP in your current directory.

Variant script (xds_generate_all)

This script includes some minor modifications of the Generate_XDS.INP script - for instance, it tells the user the energy, as well as the wavelength of the X-rays, and gives the user the beam center in mm as well as pixels - useful when switching between HKL2000, MOSFLM and XDS.

It also generates two executable shell scripts in the same directory as XDS.INP.

The first of these scripts, xds_graph.sh, extracts various statistics from INTEGRATE.LP and CORRECT.LP, pipes them out to a logfile, INTEGRATE_STATS.LP, and plots them using loggraph. This script does not have some of the more useful features of XDSSTAT, such as calculation of per frame Rmeas values, but it does give a quick overview of various data quality parameters by resolution and image number. This script will only work with recent versions of XDS (after calculation of CC(1/2) - not sure exactly which version).

The second of these scripts, xds_to_ccp4.sh, will take XDS_ASCII.HKL and generate a CCP4-format MTZ file, containing anomalous data (as F(+)/F(-) and DANO/SigDANO) and a test set of 5% of reflections for calculation of the free R-factor. It will also generate an unmerged SHELX format HKL file, for input into SHELXC/D/E (e.g. via the HKL2MAP GUI).

Here is the script:

#!/bin/bash                                                                  
# purpose: xds_generate_all                                                   
#                                                                             
# tested with some datasets from ALS, SSRL, SLS, ESRF and BESSY; only MARCCD, ADSC/SMV, PILATUS detectors; 
# for other detectors, values marked with XXX must be manually filled in.                                  
#                                                                                                          
# revision 0.03 . Kay Diederichs 2/2010                                                                    
# revision 0.04 . Kay Diederichs 4/2010 - include alternative ORGX, ORGY calculations for ADSC             
# revision 0.05 . Kay Diederichs 5/2010 - grep for "Corrected" in addition to "marccd"; needed for BESSY   
# revision 0.06 . KD 6/2010 - add UNTRUSTED_RECTANGLE and UNTRUSTED_ELLIPSE; use `whereis catmar` and so on 
# revision 0.07 . KD 6/2010 - decide about ORGX/Y info in MAR header being pixels or mm; other fixes        
# revision 0.08 . KD 6/2010 - fixes for Pilatus 6M                                                          
# revision 0.09 . KD 6/2010 - get rid of requirement for mccd_xdsparams.pl and/or catmar; rather use "od"   
# revision 0.10 . Tim Gruene 7/2010 - set link 'images' to image directory if path exceeds 72 characters    
# revision 0.11 . KD 7/2010 - for MarCCD: look for distance info at different byte position                 
# revision 0.12 . KD 7/2010 - fix for negative PHISTART                                                     
# revision 0.13 . KD 8/2010 - store correct NX NY QX QY in XDS.INP                                          
# revision 0.14 . KD 1/2011 - SENSOR_THICKNESS for Pilatus; MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3            
# revision 0.15 . KD 2/2011 - add comment for -ive sign of APS 19-ID and Australian Synchrotron rotation axis
# revision 0.16 . KD 3/2011 - SENSOR_THICKNESS=0.01 for ADSC and MarCCD. Add comment about SILICON=          
# revision 0.17 . KD 3/2011 - make it work for .bz2 frames; improve screen output                            
# revision 0.18 . KD 4/2011 - faster by doing "strings" only once; revert "images/${1##/*/}" "correction"    
# revision 0.19 . KD 6/2011 - bugfix for 0.18                                                                
# revision 0.20 . KD 7/2011 - redirect stderr of /bin/ls to /dev/null                                        
# revision 0.21 . KD 11/2011 - SEPMIN, CLUSTER_RADIUS hints; read NX NY from header (for Pilatus 2M)
# revision 0.22 . KD 12/2011 - Pilatus 2M UNTRUSTED_RECTANGLE lines, SENSOR_THICKNESS from header
# revision 0.23 . KD 1/2012 - add UNTRUSTED_QUADRILATERAL, remove MINIMUM_ZETA (0.05 is default now)
# revision 0.24 . KD 3/2012 - remove revision 0.10 since XDS now takes much longer paths
# revision 0.25 . KD 3/2012 - remove revision 0.22 for PSI Pilatus 2M; see http://www.globalphasing.com/autoproc/wiki/index.cgi?TroubleShootingKnownIssues
# revision 0.26 Oliver Clarke 10/12 - Added generation of shell scripts for conversion to MTZ and visualisation of statistics. Other minor alterations to XDS.INP.
REVISION="0.26 (18-Oct-2012)"                                                                                
#                                                                                                            
# usage: e.g. xds_generate_all "frms/mydata_1_???.img"                                                       
# make sure to have the two quotation marks !                                                                
# the ? are wildcards for the frame numbers.                                                                 
#                                                                                                            
# limitations:                                                                                               
# - frame numbers are assumed to start with 1 and run consecutively                                          
#                                                                                                            
# known problems:                                                                                            
# - for ADSC detectors, there are at least three ways to obtain ORGX and ORGY values from the header (see below);
# - the same might be a problem for MAR headers, too (not sure about this)                                       
#                                                                                                                
# notes for debugging of the script:                                                                             
# - add the -v option to the first line, to see where an error occurs                                            
# - comment out the removal of tmp1 and tmp2 in the last line                                                    
#                                                                                                                
# ====== Start of script ======                                                                                  
echo generate_XDS.INP version $REVISION . Obtain the latest version from                                         
echo http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/generate_XDS.INP                                 
if [ "$1" == "help" ] || [ "$1" == "-help" ] || [ "$1" == "-h" ]; then                                           
  echo usage: xds_generate_all \"frms/mydata_1_???.img\"   \(_with_ the quotation marks!\)                       
  echo if the frames are compressed with bzip2, leave out the .bz2 extension!                                    
  exit                                                                                                           
fi                                                                                                               
#                                                                                                                
# defaults:                                                                                                      
#                                                                                                                
DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"                                                        
ORGX=XXX                                                                                                         
ORGY=XXX
ORGXMM=XXX
ORGYMM=XXX
DETECTOR_DISTANCE=XXX                                                                                            
OSCILLATION_RANGE=XXX                                                                                            
X_RAY_WAVELENGTH=XXX 
X_RAY_ENERGY=XXX
QX=XXX                                                                                                           
QY=XXX                                                                                                           
NX=XXX                                                                                                           
NY=XXX
NXMM=XXX
NYMM=XXX
SENSOR_THICKNESS=0                                                                                               
# see how we are called:                                                                                         
NAME_TEMPLATE_OF_DATA_FRAMES="$1"
# list frames matching the wildcards in NAME_TEMPLATE_OF_DATA_FRAMES
# don't accept the "direct beam" shot at SLS/Pilatus PX-I and PX-II 
/bin/ls -C1 $1 $1.bz2 2>/dev/null | egrep -v "_00000.cbf|_000.img" > tmp1 || exit 1

# we can continue - the frames are found

# set upper limit of DATA_RANGE to number of frames (see "limitations" above)
DATA_RANGE=`wc -l tmp1 | awk '{print $1}'`                                   

# set upper limit of SPOT_RANGE to half of DATA_RANGE, but not less than 1
SPOT_RANGE=`echo "scale=0; $DATA_RANGE/2" | bc -l`                        
SPOT_RANGE=`echo "if ($SPOT_RANGE<1) 1;if ($SPOT_RANGE>1) $SPOT_RANGE" | bc -l`

echo DATA_RANGE=1 $DATA_RANGE

# find out detector type
DET=XXX                 
FIRSTFRAME=`head -1 tmp1`
echo $FIRSTFRAME | grep -q bz2 && bzcat $FIRSTFRAME > tmp1 && FIRSTFRAME=tmp1
strings $FIRSTFRAME > tmp2                                                   
egrep -q 'marccd|Corrected' tmp2 && DET=mccd                                 
grep -q PILATUS tmp2             && DET=pilatus                              
grep -q BEAM_CENTER_X tmp2       && DET=adsc                                 
# identify other detector types in the same way (MAR IP would be straightforward)

# parse ASCII header of first frame

if [ "$DET" == "XXX" ]; then
  echo "this is not a MAR, ADSC/SMV or PILATUS detector - fill in XXX values manually!"
  DETECTOR="XXX MINIMUM_VALID_PIXEL_VALUE=XXX OVERLOAD=XXX"                            

# find parameters of first frame
elif [ "$DET" == "mccd" ]; then 
  echo Data from a MarCCD detector
                                  
  DETECTOR="CCDCHESS MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65500"
  SENSOR_THICKNESS=0.01                                           
  # use first frame of dataset to obtain parameters               

  # offsets are documented; values can be find in mccd_xdsparams.pl script
  let SKIP=1024+80                                                        
  NX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
  let SKIP=$SKIP+4                                                                         
  NY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')

  let SKIP=1720
  DETECTOR_DISTANCE=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
  DETECTOR_DISTANCE=`echo "scale=3; $DETECTOR_DISTANCE/1000" | bc -l`                                     
                                                                                                          
  let SKIP=1024+256+128+256+4                                                                             
  ORGX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')             
  ORGX=`echo "scale=2; $ORGX/1000" | bc -l `                                                              
  let SKIP=$SKIP+4                                                                                        
  ORGY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')             
  ORGY=`echo "scale=2; $ORGY/1000" | bc -l `                                                              

  let SKIP=1024+256+128+256+44
  PHISTART=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
  let SKIP=1024+256+128+256+76                                                                   
  PHIEND=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')  
  OSCILLATION_RANGE=`echo "scale=3; ($PHIEND-($PHISTART))/1000" | bc -l`                         
                                                                                                 
  let SKIP=1024+256+128+256+128+4                                                                
  QX=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')      
  QX=`echo "scale=10; $QX/1000000" |bc -l `                                                      
  let SKIP=$SKIP+4                                                                               
  QY=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')      
  QY=`echo "scale=10; $QY/1000000" |bc -l `                                                      

  let SKIP=1024+256+128+256+128+128+12
  X_RAY_WAVELENGTH=$(od -t dI -j $SKIP -N 4 $FIRSTFRAME | head -1 | awk '{print $2}')
  X_RAY_WAVELENGTH=`echo "scale=5; $X_RAY_WAVELENGTH/100000" | bc -l`                                    
  X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
# at most BLs, ORGX and ORGY are in pixels, but sometimes in mm ... guess:
  NXBYFOUR=`echo "scale=0; $NX/4" | bc -l `                               
  ORGXINT=`echo "scale=0; $ORGX/1" | bc -l `                              
  if [ $ORGXINT -lt $NXBYFOUR ]; then 
     ORGXMM=$ORGX
     ORGYMM=$ORGY
     NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
     NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`     
     ORGX=`echo "scale=1; $ORGX/$QX" | bc -l`                             
     ORGY=`echo "scale=1; $ORGY/$QY" | bc -l`                             
     echo MARCCD detector: header ORGX, ORGY seem to be in mm ... converting to pixels
  else                                                                                
     ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
     ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
     NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
     NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`           
     echo MARCCD detector: header ORGX, ORGY seem to be in pixel units                
  fi                                                                                  

elif [ "$DET" == "adsc" ]; then

  DETECTOR="ADSC MINIMUM_VALID_PIXEL_VALUE= 1 OVERLOAD= 65000"
  echo Data from ADSC detector. Obtaining ORGX, ORGY depends on beamline setup:
  SENSOR_THICKNESS=0.01                                                        
  sed s/\;// tmp2 > tmp1                                                       
  mv tmp1 tmp2                                                                 

      # find X_RAY_WAVELENGTH:
      X_RAY_WAVELENGTH=`grep WAVELENGTH tmp2 | head -1 | sed s/WAVELENGTH=//`

      # find NX, QX, ORGX and ORGY:
      X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
      NX=`grep SIZE1 tmp2 | tail -1 | sed s/SIZE1=//`
      QX=`grep PIXEL_SIZE tmp2 | sed s/PIXEL_SIZE=//`
# FIXME - next 2 lines should be done properly, from header
      NY=$NX                                               
      QY=$QX
      NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
      NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`                
      BEAM_CENTER_X=`grep BEAM_CENTER_X tmp2 | sed s/BEAM_CENTER_X=//`
      BEAM_CENTER_Y=`grep BEAM_CENTER_Y tmp2 | sed s/BEAM_CENTER_Y=//`
# fix 2010-04-26 - tell user about possible ORGX, ORGY alternatives -  
# at ESRF and ... (pls fill in!) the following should be used:         
      ORGX=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l`               
      ORGY=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l`               
      echo - at ESRF BLs use: ORGX=$ORGX ORGY=$ORGY                    
# this 2nd alternative convention should be used at the following beamlines (pls complete the list): ALS 5.0.3, ...
      ORGX=`echo "scale=1; $NX-$BEAM_CENTER_X/$QX" | bc -l `                                                       
      ORGY=`echo "scale=1; $BEAM_CENTER_Y/$QX" | bc -l `                                                           
      echo - at e.g. ALS 5.0.3 use: ORGX=$ORGX ORGY=$ORGY                                                          
# this 3rd alternative convention should be used at the following beamlines (pls complete the list): ALS 8.2.2, ... 
      ORGX=`echo "scale=1; $BEAM_CENTER_X/$QX" | bc -l `                                                            
      ORGY=`echo "scale=1; $NX-$BEAM_CENTER_Y/$QX" | bc -l ` 
      ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
      ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`                                                       
      echo - at e.g. ALS 8.2.2 use: ORGX=$ORGX ORGY=$ORGY - this is written to XDS.INP                              
# the latter alternative is written into the generated XDS.INP ! You have to correct this manually in XDS.INP, or adjust this script.
      # find DETECTOR_DISTANCE and OSCILLATION_RANGE:                                                                                
      DETECTOR_DISTANCE=`grep DISTANCE tmp2 | sed s/DISTANCE=//`                                                                     
      OSCILLATION_RANGE=`grep OSC_RANGE tmp2 | sed s/OSC_RANGE=//`                                                                   

elif [ "$DET" == "pilatus" ]; then
  DETECTOR="PILATUS MINIMUM_VALID_PIXEL_VALUE=0 OVERLOAD= 1048576  !PILATUS"
  QX=0.172 QY=0.172                                                         
  echo Data from a Pilatus detector                                         
  sed s/#// tmp2 > tmp1                                                     
  mv tmp1 tmp2                                                              

      # find SENSOR_THICKNESS:
      SENSOR_THICKNESS=`grep thickness tmp2 | sed -e s/'Silicon sensor, thickness'// | awk '{print $1*1000}'`
      # find X_RAY_WAVELENGTH:                                                                               
      X_RAY_WAVELENGTH=`grep Wavelength tmp2 | sed -e s/Wavelength// -e s/A// | awk '{print $1}'`            

      X_RAY_ENERGY=`echo "12398.5/$X_RAY_WAVELENGTH" | bc -l | awk '{printf "%5.1f", $1}'`
      # find NX and NY; 2463/2527 is 6M, 1475/1679 is 2M
      NX=`grep X-Binary-Size-Fastest-Dimension tmp2 | awk '{print $2}'`
      NY=`grep X-Binary-Size-Second-Dimension tmp2 | awk '{print $2}'` 
      NXMM=`echo "$NX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
      NYMM=`echo "$NY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`                
      # find ORGX and ORGY:
      ORGX=`grep Beam_xy tmp2 | sed -e s/\(// -e s/\)// -e s/\,// | awk '{print $2}'`
      ORGY=`grep Beam_xy tmp2 | sed -e s/\(// -e s/\)// -e s/\,// | awk '{print $3}'`
      ORGXMM=`echo "$ORGX*$QX" | bc -l | awk '{printf "%3.2f", $1}'`
      ORGYMM=`echo "$ORGY*$QY" | bc -l | awk '{printf "%3.2f", $1}'`
      # find DETECTOR_DISTANCE and OSCILLATION_RANGE:
      DETECTOR_DISTANCE=`awk '/distance/{print $2}' tmp2`
      DETECTOR_DISTANCE=`echo "$DETECTOR_DISTANCE*1000" | bc -l`

      OSCILLATION_RANGE=`awk '/Angle/{print $2}' tmp2`

else
  echo should never come here
  exit 1                     
fi                           

echo ORGX= $ORGX ORGY= $ORGY - check these values with adxv !
echo DETECTOR_DISTANCE= $DETECTOR_DISTANCE                   
echo OSCILLATION_RANGE= $OSCILLATION_RANGE                   
echo X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH                     
# now we know everything that is required to generate XDS.INP

cat > XDS.INP << eof
!****************************************************************************************
! Written by generate_XDS.INP version $REVISION
!
! A few notes on usage and modification of this input file:
!
! If XDS fails after IDXREF with an error indicating that
! an insufficient percentage of reflections were indexed,
! but you are confident that the indexing solution is
! correct, rerun XDS with the JOB keyword changed to
! "DEFPIX INTEGRATE CORRECT".
!
! After the first round of integration (in a triclinic cell)
! it is often helpful to reintegrate using the globally 
! refined cell parameters and the correct space group.
! You can do this by renaming GXPARM.XDS to XPARM.XDS
! - e.g. by the command "cp GXPARM.XDS XPARM.XDS" -
! and rerunning DEFPIX, INTEGRATE and CORRECT.
!
! If you are trying to determine the optimal starting point for 
! data collection starting from a single image, first complete
! a run of XDS using default parameters. Then, rerun XDS using
! the correct cell and spacegroup as described above 
! ("cp GXPARM.XDS XPARM.XDS"), but on the second run of XDS, give
! the JOB keyword as "JOB=DEFPIX XPLAN".
!****************************************************************************************
!
!****************************************************************************************
!General parameters:
!****************************************************************************************

JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT

ORGX= $ORGX ORGY= $ORGY  ! check these values with adxv!
!In mm, ORGX=$ORGXMM and ORGY=$ORGYMM

DETECTOR_DISTANCE= $DETECTOR_DISTANCE                    
OSCILLATION_RANGE= $OSCILLATION_RANGE                    

X-RAY_WAVELENGTH= $X_RAY_WAVELENGTH
!X-ray energy is $X_RAY_ENERGY eV (calculated from wavelength)                    

NAME_TEMPLATE_OF_DATA_FRAMES=$NAME_TEMPLATE_OF_DATA_FRAMES

!REFERENCE_DATA_SET=xxx/XDS_ASCII.HKL ! e.g. to ensure consistent indexing.
!XDS will *only* use the reference dataset if the space group and cell
!parameters are specified below.  

DATA_RANGE=1 $DATA_RANGE                                                     
SPOT_RANGE=1 $SPOT_RANGE

!BACKGROUND_RANGE=1 10 !Default is first five degrees. 

SPACE_GROUP_NUMBER=0                   ! 0 if unknown
UNIT_CELL_CONSTANTS= 70 80 90 90 90 90 ! put correct values if known

INCLUDE_RESOLUTION_RANGE=50 0  ! After CORRECT, insert high resol limit; re-run CORRECT
!                                Ice rings can be excluded from scaling with the keyword 
!                                EXCLUDE_RESOLUTION_RANGE, e.g. EXCLUDE_RESOLUTION_RANGE=2.28 2.22

!SECONDS=60 !Uncomment this line to tell XDS to wait 1min for the next image before aborting.

!****************************************************************************************
!Parameters important for processing anomalous data:
!****************************************************************************************

FRIEDEL'S_LAW=FALSE     ! This acts only on the CORRECT step. Even if the anomalous signal
!                         in your dataset is very small, it still may be useful for
!                         calculating an anomalous difference map.

STRICT_ABSORPTION_CORRECTION=FALSE !Change this to TRUE if the anomalous signal 
!                                   is strong: in that case, in CORRECT.LP the three
!                                   "CHI^2-VALUE OF FIT OF CORRECTION FACTORS" values 
!                                   will be significantly> 1, e.g. 1.5.

WFAC1=1.0 !This parameter controls rejection of misfits during scaling. Sometimes,
!          strong anomalous pairs may be rejected as misfits, in which case increasing
!          WFAC1 (e.g. to 1.5) may improve anomalous signal.

!****************************************************************************************
!Regions of the detector to be excluded during indexing and integration:
!****************************************************************************************
! exclude (mask) untrusted areas of detector, e.g. beamstop shadow :
!UNTRUSTED_RECTANGLE= !1800 1950 2100 2150 ! x-min x-max y-min y-max ! repeat
!UNTRUSTED_ELLIPSE= !2034 2070 1850 2240 ! x-min x-max y-min y-max ! if needed
!UNTRUSTED_QUADRILATERAL= !x1 y1 x2 y2 x3 y3 x4 y4 ! see documentation

TRUSTED_REGION=0.00 1.2  ! Partially use corners of detectors; 1.41421=full use, 1.0=edge
!                          It is important to note that TRUSTED_REGION is the only resolution 
!                          cutoff obeyed during indexing and integration; 
!                          Both INCLUDE_RESOLUTION_RANGE and EXCLUDE_RESOLUTION_RANGE are 
!                          ignored during IDXREF and INTEGRATE. So this parameter can be 
!                          important when indexing or integration steps fail.

!****************************************************************************************
!Other parameters:
!****************************************************************************************
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS=7000. 30000. ! Often 8000 is ok.
STRONG_PIXEL=4           ! COLSPOT: only use strong reflections (default is 3; 6 may be better for strong data)
MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=3 ! default of 6 is sometimes too high
MAXIMUM_ERROR_OF_SPOT_POSITION=3 ! Increasing may help for poor quality data

MINPK=75 !Increase (e.g. MINPK=98) to improve data accuracy at the expense of completeness.

DELPHI=5 !Size of integration wedge. Increasing to 10 or 20 may be helpful in some cases.

NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=13 ! Default is 9 - Increasing may improve data 
NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA=13      ! accuracy, particularly if finely-sliced on phi, 
!                                                   and does not seem to have any downsides. 

MINIMUM_ZETA=0.05 ! Controls how close to the blind region (about phi) reflections should be 
!                   integrated. 0.05 is the default; increasing MINIMUM_ZETA to 0.15 *may* 
!                   improve data quality by removing unreliable reflections near the phi 
!                   axis, but will reduce completeness in low symmetry space groups.

SEPMIN=6         ! Reduce if spots are close together (due to a long axis). Consider 
CLUSTER_RADIUS=3 ! increasing if crystal is split and unit cell dimensions are relatively short.

! For bad or low resolution data remove the "!" in the following line (default is ALL): 
! REFINE(IDXREF)=CELL BEAM ORIENTATION AXIS ! DISTANCE
REFINE(INTEGRATE)=CELL BEAM ORIENTATION ! AXIS DISTANCE (If integration is unstable, comment out this line.
! REFINE(CORRECT)=CELL BEAM ORIENTATION AXIS DISTANCE ! Default is: refine everything

ROTATION_AXIS=1 0 0  !At e.g. Australian Synchrotron, SERCAT ID-22 (?), APS 19-ID (?) this needs to be -1 0 0

!****************************************************************************************
! Parameters specifically for this detector and beamline (shouldn't need changing):
!****************************************************************************************
DETECTOR= $DETECTOR
SENSOR_THICKNESS= $SENSOR_THICKNESS
! attention CCD detectors: for very high resolution (better than 1A) make sure to specify SILICON
! as about 32* what CORRECT.LP suggests (absorption of phosphor is much higher than that of silicon)

NX= $NX NY= $NY  QX= $QX  QY= $QY ! to make CORRECT happy if frames are unavailable
!In mm, NX=$NXMM and NY=$NYMM

DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
INCIDENT_BEAM_DIRECTION=0 0 1
FRACTION_OF_POLARIZATION=0.98   ! better value is provided by beamline staff!
POLARIZATION_PLANE_NORMAL=0 1 0
eof
if [ "$DET" == "pilatus" ]; then
  if [ $NX == "1475" ]; then
    if ! grep -q FF_p2m0109_E12398_T6199_vrf_m0p20.tif tmp2 ; then
    cat >> XDS.INP << eof
!EXCLUSION OF VERTICAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE= 486  496     0 1680
UNTRUSTED_RECTANGLE= 980  990     0 1680
!EXCLUSION OF HORIZONTAL DEAD AREAS OF THE PILATUS 2M DETECTOR
UNTRUSTED_RECTANGLE=   0 1476   194  214
UNTRUSTED_RECTANGLE=   0 1476   406  426
UNTRUSTED_RECTANGLE=   0 1476   618  638
UNTRUSTED_RECTANGLE=   0 1476   830  850
UNTRUSTED_RECTANGLE=   0 1476  1042 1062
UNTRUSTED_RECTANGLE=   0 1476  1254 1274
UNTRUSTED_RECTANGLE=   0 1476  1466 1486
eof
    fi
#  elif [ $NX == "2463" ]; then
#    cat >> XDS.INP << eof
#eof
  fi
fi
touch xds_to_ccp4.sh
echo "#!/bin/bash
#Removes old log file if present
rm xds_to_ccp4.log
#Makes new log file
touch xds_to_ccp4.log
#Pipe stdout and stderr to both logfile and tty
tail -f -n1 xds_to_ccp4.log &
exec > xds_to_ccp4.log 2>&1
echo \"


If this message is all you see,
something has probably gone wrong.
Are you sure XDS_ASCII.HKL is present
in the current directory?


\"
#Make first MTZ file (with DANO/SIGDANO)
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=temp.hkl CCP4
MERGE=TRUE
GENERATE_FRACTION_OF_TEST_REFLECTIONS=0.05
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv
f2mtz HKLOUT temp.mtz<F2MTZ.INP

#Make second MTZ file (with F(+)/F(-))
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=temp2.hkl CCP4_F
MERGE=TRUE
GENERATE_FRACTION_OF_TEST_REFLECTIONS=0.05
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv
f2mtz HKLOUT temp2.mtz<F2MTZ.INP

#Make SHELX format unmerged hkl file
echo \"INPUT_FILE=XDS_ASCII.HKL
OUTPUT_FILE=xds_shelx.hkl SHELX
MERGE=FALSE
FRIEDEL'S_LAW=FALSE\" > XDSCONV.INP
xdsconv

#cad the two previously generated MTZ files together
cad HKLIN1 temp.mtz HKLIN2 temp2.mtz HKLOUT xds_ccp4_merged.mtz<<EOF
 LABIN  FILE 1 E1=FP       E2=SIGFP       E3=DANO     E4=SIGDANO     E5=FreeRflag
 LABIN  FILE 2 E1=F(+)     E2=SIGF(+)     E3=F(-)     E4=SIGF(-)
 END
EOF

#Remove temporary files
rm temp.mtz
rm temp2.mtz
rm temp.hkl
rm temp2.hkl
rm F2MTZ.INP

echo \"

********************************XDS_TO_CCP4**********************************

This script takes XDS_ASCII.HKL and uses XDSCONV and F2MTZ to 
generate a merged CCP4-format MTZ file named 
\\\"xds_ccp4_merged.mtz\\\" with a test (free) set
labeled \\\"FreeRflag\\\" constituting 5% of reflections. 

The space group of the output MTZ file will be the same as
that present in XDS_ASCII.HKL. It is probably worth checking
that this is the correct space group - either manually inspect
the systematic absences in CORRECT.LP, or run XDS_ASCII.HKL
through the CCP4 program POINTLESS (\\\"pointless xdsin XDS_ASCII.HKL\\\")

Anomalous data is kept, if present, and output as both 
DANO/SIGDANO and F(+)/F(-).

This script also generates an unmerged SHELX hkl file,
suitable for input into SHELXC/D/E, named \\\"xds_shelx.hkl\\\".

A log of all processes is present in xds_to_ccp4.log.

Parts of this script have been shamelessly copied from
the XDS Wiki, at http://www.strucbio.biologie.uni-konstanz.de/xdswiki/

******************************************************************************
\"" > xds_to_ccp4.sh
touch xds_graph.sh
echo "#!/bin/bash
#Makes new log file
touch INTEGRATE_STATS.LP
#Pipe stdout and stderr to both logfile and tty
tail -f -n1 INTEGRATE_STATS.LP &
exec > INTEGRATE_STATS.LP 2>&1
echo \"
\\\$TABLE: XDS Integration Statistics :
\\\$SCATTER:Scale factors by image:A:1,3: :Number of overloads by image:A:1,5: :Number of strong reflections by image:A:1,7: :Number of rejects by image:A:1,8: :Mosaicity by image:A:1,10: \\$\\$
IMAGE IER SCALE NBKG NOVL NEWALD NSTRONG NREJ SIGMAB SIGMAR
\\$\\$ \\$\\$
\"
egrep \"[0-9]\.[0-9][0-9][0-9][0-9]  [0-9]\.[0-9][0-9][0-9][0-9]\" INTEGRATE.LP
echo \" \\$\\$ \"
echo \"
\\\$TABLE: XDS Scaling Statistics by Resolution (X-axis is resolution in Angstroems) :
\\\$GRAPHS:Chi-Squared by resolution:A:2,4: :Observed and expected R-factors by resolution (as %):A:2,5,6: :Rejects by resolution:A:2,9: :Rejects by resolution as a percentage of total reflections:A:2,10: \\$\\$
LOW_RES Resolution(Angstroems)  I/Sigma  Chi^2  R-FACTOR(OBSERVED)  R-FACTOR(EXPECTED)  NUMBER_OF_REFLECTIONS ACCEPTED REJECTS REJECT_PERCENT
\\$\\$ \\$\\$
\"
awk '/RESOLUTION RANGE  I\/Sigma  Chi\^2  R\-FACTOR  R\-FACTOR  NUMBER ACCEPTED REJECTED/, /\-\-\-\-\-\-\-\-\-/' CORRECT.LP | egrep -v \"[a-z,A-Z,] | \-\-|\-99\.9\" | egrep \"[0-9]\" | awk '{print \$1,\$2,\$3,\$4,\$5,\$6,\$7,\$8,\$9,(\$9/(\$8+\$9))*100}'
echo \" \\$\\$\"
echo \"
\\\$TABLE: Overall data quality by Resolution (X-axis is resolution in Angstroems) :
\\\$GRAPHS:Completeness vs resolution :A:1,6: :Redundancy vs resolution :A:1,4: :I/Sigma by resolution:A:1,10: :Rmeas by resolution:A:1,11: :Anomalous correlation by resolution:A:1,13: :Significance of the anomalous signal by resolution:A:1,14:  \\$\\$
Resolution(Angstroems) OBSERVED_REFLECTIONS UNIQUE_REFLECTIONS REDUNDANCY POSSIBLE_REFLECTIONS COMPLETENESS R-FACTOR(OBSERVED) R-FACTOR(EXPECTED) COMPARED I/SIGMA R-meas CC(1/2) Anomalous_Correlation SigAno Nano
\\$\\$ \\$\\$
\"
egrep -B25 \" WILSON STATISTICS \" CORRECT.LP | egrep -v \"[a-z,A-Z]|\*\*\*\" | egrep \"  \" | awk '{ gsub(/[%*]/,\" \"); print }' | awk '{print \$1,\$2,\$3,\$2/\$3,\$4,\$5,\$6,\$7,\$8,\$9,\$10,\$11,\$12,\$13,\$14}'
echo \" \\$\\$ \"

echo \"
\\\$TABLE: I/Sigma of h00 reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs h :A:1,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMPLETENESS AND QUALITY/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$1 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE: I/Sigma of 0k0 reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs k :A:2,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMPLETENESS AND QUALITY/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$2 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE: I/Sigma of 00l reflections (for identification of systematic absences):
\\\$GRAPHS:I/Sigma vs l :A:3,5: \\$\\$
h k l RES I/SIGMA NUMBER
\\$\\$ \\$\\$
\"
awk '/REFLECTIONS OF TYPE H/,/COMPLETENESS AND QUALITY/' CORRECT.LP | egrep \"[1-9]\" | awk '{print \$1,\$2,\$3,\$4,\$7,\$8}' | awk '(\$3 > 0)'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Beam center (ORGX/ORGY) plotted vs wedge (DELPHI):
\\\$GRAPHS:ORGX and ORGY vs batch (DELPHI) :A:1,2,3: \\$\\$
N ORGX ORGY
\\$\\$ \\$\\$
\"
egrep \"DETECTOR ORIGIN\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Unit cell parameters plotted vs wedge:
\\\$GRAPHS:Unit cell lengths a, b and c plotted vs wedge:A:1,2,3,4: :Unit cell angles alpha, beta and gamma plotted vs wedge:A:1,5,6,7: \\$\\$
N a b c ALPHA BETA GAMMA
\\$\\$ \\$\\$
\"
egrep \"UNIT CELL PARAMETERS\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Standard deviation of spot position (pixels) plotted vs wedge:
\\\$GRAPHS:Standard deviation of spot position plotted vs wedge:A:1,2: \\$\\$
N SD
\\$\\$ \\$\\$
\"
egrep \"SPOT    POSITION\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'
echo \"\\$\\$\"

echo \"
\\\$TABLE:Detector distance (mm) plotted vs wedge (DELPHI):
\\\$GRAPHS:Detector distance plotted vs batch:A:1,2: \\$\\$
N DIST
\\$\\$ \\$\\$
\"
egrep \"DETECTOR DISTANCE \\(mm\\)\" INTEGRATE.LP | grep -n \"\" | awk '{ gsub(/[a-z,A-Z,),(,:]/,\"\"); print }'

echo \"\\$\\$\"

loggraph INTEGRATE_STATS.LP">xds_graph.sh
chmod +x xds_to_ccp4.sh
chmod +x xds_graph.sh
echo "XDS.INP is ready for use. The file has only the most important keywords.
     Full documentation, including complete detector templates, is at
     http://www.mpimf-heidelberg.mpg.de/~kabsch/xds . More documentation in XDSwiki
     After running xds, inspect, using XDS-Viewer, at least the beamstop mask in
     BKGPIX.cbf, and the agreement of predicted and observed spots in FRAME.cbf!

     Two shell scripts have been generated that may be of use. 

     The first, xds_graph.sh, will plot various statistics after an XDS run. 
     After both INTEGRATE and CORRECT have finished, run it by typing \"./xds_graph.sh\" 
     without the quotation marks. This script uses loggraph to plot data, so you will
     need CCP4 to see the graphical output. All the raw data is piped out to a log file,
     INTEGRATE_STATS.LP.

     The second script, xds_to_ccp4.sh, will take XDS_ASCII.HKL and generate both a CCP4
     format MTZ file with anomalous data retained (as F(+)/F(-) and DANO/SigDANO)
     and an unmerged SHELX format hkl file for input to SHELXC/D/E. Run it after CORRECT
     by typing \"./xds_to_ccp4.sh\" at the prompt."

rm -f tmp1 tmp2

Use this script in the same manner as Generate_XDS.INP. The two shell scripts created should be executable; If they are not, then make them so (chmod +x ./xxx.sh).