2,652
edits
Problems: Difference between revisions
Jump to navigation
Jump to search
→Problems with OpenMP
(remove text referring to 32bit binary) |
|||
| (15 intermediate revisions by the same user not shown) | |||
| Line 3: | Line 3: | ||
== XDS crashes == | == XDS crashes == | ||
XDS should never crash (if it terminates with an error message, this does not count as crash). If it does, it is either a bug in the program which should be brought to the attention of Wolfgang Kabsch or Kay Diederichs, and will be fixed, or it is a problem with your computer. | XDS should never crash (if it terminates with an error message, this does not count as crash). If it does, it is either a bug in the program which should be brought to the attention of Wolfgang Kabsch or Kay Diederichs, and will be fixed, or it is a problem with your computer, or (rarely) it is due to improper inputs ([[Problems#INTEGRATE_cell_and.2For_distance_run_away.3B_xds_crashes_or_has_to_be_killed|this]] is the only example I know of). | ||
If it crashes for the second reason, these are the things to try/consider: | If it crashes for the second reason, these are the things to try/consider: | ||
| Line 22: | Line 22: | ||
ulimit -s 102400 | ulimit -s 102400 | ||
The numbers above mean a 10-fold increase over the default, and should be enough. I've found this to be necessary for unusually large frames (32 MB). | The numbers above mean a 10-fold increase over the default, and should be enough. I've found this to be necessary for unusually large frames (32 MB). | ||
xds_par in this case also might need an increase of the environment variable OMP_STACKSIZE (e.g. "setenv OMP_STACKSIZE 128M"). | |||
xds_par in this case also might need an increase of the environment variable OMP_STACKSIZE (e.g. "setenv OMP_STACKSIZE 128M"). | |||
According to http://stackoverflow.com/questions/13264274/why-segmentation-fault-is-happening-in-this-openmp-code one should also check the "virtual address space limit" with ulimit (increase with ulimit -v) . | |||
=== Problems with OpenMP === | === Problems with OpenMP === | ||
If the "xds_par" binary crashed, try "xds". xds_par uses OpenMP for parallelization, which adds complexity. If it works with xds, but not with xds_par, then there is a chance that some environment variable needs to be set/changed. In any case the XDS developers would like to learn about this. | If the "xds_par" binary crashed, try "xds". xds_par uses OpenMP for parallelization, which adds complexity. If it works with xds, but not with xds_par, then there is a chance that some environment variable needs to be set/changed. In any case the XDS developers would like to learn about this. | ||
As of March 2021, Thomas Hauß (HZB) reported an OpenMP-related problem that is due to a bug in a library that the ifort compiler links into xds_par. It is unknown to us which version of the compiler has or does not have this bug. The bug has been seen with xds_par (VERSION Jan 31, 2020 BUILT=20200417) and is a crash with error message containing: | |||
<pre> | |||
OMP: Error #13: Assertion failure at z_Linux_util.cpp(2361). | |||
OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://www.intel.com/software/products/support/. | |||
forrtl: error (76): Abort trap signal | |||
Image PC Routine Line Source | |||
xds_par 00000000005620C4 Unknown Unknown Unknown | |||
libpthread-2.17.s 00002ADC08AAA630 Unknown Unknown Unknown | |||
libc-2.17.so 00002ADC08EF1387 gsignal Unknown Unknown | |||
libc-2.17.so 00002ADC08EF2A78 abort Unknown Unknown | |||
xds_par 000000000063FB83 Unknown Unknown Unknown | |||
xds_par 000000000062BDFF Unknown Unknown Unknown | |||
xds_par 000000000060748C Unknown Unknown Unknown | |||
xds_par 000000000067980E Unknown Unknown Unknown | |||
xds_par 000000000063D737 Unknown Unknown Unknown | |||
xds_par 000000000063EC58 Unknown Unknown Unknown | |||
xds_par 0000000000629A0E Unknown Unknown Unknown | |||
xds_par 0000000000419056 xds_ 21586 MAIN_XDS.f90 | |||
xds_par 0000000000418951 MAIN__ 1 MAIN_XDS.f90 | |||
xds_par 0000000000415862 Unknown Unknown Unknown | |||
libc-2.17.so 00002ADC08EDD555 __libc_start_main Unknown Unknown | |||
xds_par 0000000000415769 Unknown Unknown Unknown | |||
</pre> | |||
The bug is not related to the xds_par source code, but also happens with other software. The workaround is to set the environment variable | |||
bash: export KMP_INIT_AT_FORK=FALSE | |||
tcsh: setenv KMP_INIT_AT_FORK FALSE | |||
=== 64bit binary on a 32bit operating system === | === 64bit binary on a 32bit operating system === | ||
| Line 55: | Line 85: | ||
since otherwise, FRAME.cbf is overwritten by INTEGRATE. | since otherwise, FRAME.cbf is overwritten by INTEGRATE. | ||
To look at '''all''' spot positions found by COLSPOT, you | To look at '''all''' spot positions found by COLSPOT, you can use the COLSPOT tab of [[XDSGUI]]. This nicely shows ice rings, and may also help to find shaded regions on the detector. | ||
This may also help to find shaded regions on the detector. | |||
=== ugly diffraction pattern === | === ugly diffraction pattern === | ||
| Line 110: | Line 138: | ||
THE "DATA_RANGE=" IN FILE "XDS.INP" AND START ALL OVER AGAIN. | THE "DATA_RANGE=" IN FILE "XDS.INP" AND START ALL OVER AGAIN. | ||
This is printed out for you to actually read, and take action accordingly. In most cases you just change the | This is printed out for you to actually read, and take action accordingly. In most cases you just change the JOB= line in XDS.INP to read | ||
JOB= DEFPIX INTEGRATE CORRECT | JOB= DEFPIX INTEGRATE CORRECT | ||
and then continue to run XDS. In other cases you may want to change SPOT.XDS, or other keywords in [[XDS.INP]] (see also below). But in any case this is an important alert that should make you check the correctness of the parameters that describe the data collection (X-RAY_WAVELENGTH, DETECTOR_DISTANCE, ORGX, ORGY, OSCILLATION_RANGE, NAME_TEMPLATE_OF_DATA_FRAMES). | and then continue to run XDS. In other cases you may want to change SPOT.XDS, or other keywords in [[XDS.INP]] (see also below). But in any case this is an important alert that should make you check the correctness of the parameters that describe the data collection (X-RAY_WAVELENGTH, DETECTOR_DISTANCE, ORGX, ORGY, OSCILLATION_RANGE, NAME_TEMPLATE_OF_DATA_FRAMES). | ||
| Line 127: | Line 155: | ||
1 0.0004199-0.0012633-0.0043826 2442. 0.97 -0.04 -0.00 | 1 0.0004199-0.0012633-0.0043826 2442. 0.97 -0.04 -0.00 | ||
2 0.0101757 0.0060701-0.0019630 2175. -0.03 -1.02 0.01 | 2 0.0101757 0.0060701-0.0019630 2175. -0.03 -1.02 0.01 | ||
3 -0.0076118 0.0114167-0.0040603 1965. 0.13 0.04 0.50 | 3 -0.0076118 0.0114167-0.0040603 1965. 0.13 0.04 0.50 <---- half-integer indices!? | ||
4 0.0100552 0.0071646 0.0025337 1944. -1.01 -0.99 0.01 | 4 0.0100552 0.0071646 0.0025337 1944. -1.01 -0.99 0.01 | ||
5 0.0072840-0.0101455 0.0084405 1841. -1.10 -0.01 -0.49 | 5 0.0072840-0.0101455 0.0084405 1841. -1.10 -0.01 -0.49 <---- half-integer indices!? | ||
6 0.0000976 0.0027828 0.0089584 1792. -2.00 0.00 -0.00 | 6 0.0000976 0.0027828 0.0089584 1792. -2.00 0.00 -0.00 | ||
7 0.0103103 0.0043851-0.0063902 1790. 0.98 -1.02 0.00 | 7 0.0103103 0.0043851-0.0063902 1790. 0.98 -1.02 0.00 | ||
8 0.0025742 0.0163995-0.0098507 1760. 0.95 -0.99 0.51 | 8 0.0025742 0.0163995-0.0098507 1760. 0.95 -0.99 0.51 <---- half-integer indices!? | ||
9 0.0068686-0.0089437 0.0128884 1724. -2.08 0.03 -0.49 | 9 0.0068686-0.0089437 0.0128884 1724. -2.08 0.03 -0.49 <---- half-integer indices!? | ||
10 -0.0174443 0.0123504 0.0161324 1694. -4.01 0.99 0.52 | 10 -0.0174443 0.0123504 0.0161324 1694. -4.01 0.99 0.52 <---- half-integer indices!? | ||
11 0.0272195-0.0050005-0.0136142 1678. 2.99 -1.97 -0.50 | 11 0.0272195-0.0050005-0.0136142 1678. 2.99 -1.97 -0.50 <---- half-integer indices!? | ||
... | ... | ||
| Line 164: | Line 192: | ||
The reason for the indexing failure in these cases seems to be the fact that the crystal changed its orientation within the SPOT_RANGE by more than ~ 0.1°. | The reason for the indexing failure in these cases seems to be the fact that the crystal changed its orientation within the SPOT_RANGE by more than ~ 0.1°. | ||
'''The other reason for this failure mode''' - but in this case there is only one SUBTREE with high POPULATION - is when the default <code>SEPMIN= 6.00 CLUSTER_RADIUS= 3</code> are used, but the spots are closer than 6 pixels. This often happens with Pilatus data; the recommendation for this detector is to use <code>SEPMIN= 4.00 CLUSTER_RADIUS= 2</code> in XDS.INP, or even <code>SEPMIN= | '''The other reason for this failure mode''' - but in this case there is only one SUBTREE with high POPULATION - is when the default <code>SEPMIN= 6.00 CLUSTER_RADIUS= 3</code> are used, but the spots are closer than 6 pixels. This often happens with Pilatus data; the recommendation for this detector is to use <code>SEPMIN= 4.00 CLUSTER_RADIUS= 2</code> in XDS.INP, or even <code>SEPMIN= 2.00 CLUSTER_RADIUS= 1</code>. | ||
=== Difference vectors are neither integers nor halfs === | |||
This can happen if SPACE_GROUP_NUMBER is wrong, i.e. the user "forces" a lattice (e.g. body-centered) that does not match the true one (which may be primitive). So the first thing to try is SPACE_GROUP_NUMBER=0. | |||
Sometimes, IDXREF nevertheless finds no good lattice: | |||
# COORDINATES OF VECTOR CLUSTER FREQUENCY CLUSTER INDICES | |||
1 0.0090336-0.0044767 0.0137041 1636. 0.99 0.99 -0.00 | |||
2 -0.0239422 0.0085088 0.0078304 1556. -2.00 1.00 0.00 | |||
3 0.0088799 0.0092726 0.0077661 1555. 0.50 0.50 -0.56 | |||
4 -0.0000164 0.0138382-0.0058215 1486. -0.49 -0.49 -0.56 <---- 0.56 is neither close to 0.5 nor to 0 or 1 | |||
5 -0.0148222 0.0040131 0.0216634 1468. -1.00 2.00 -0.00 | |||
6 0.0098829 0.0070246-0.0042191 1430. 0.50 -0.50 -0.44 <---- same here | |||
7 -0.0030325 0.0110238 0.0056577 1422. -0.50 0.50 -0.44 <---- and here | |||
... | |||
If it is not a case of SEPMIN and CLUSTER_RADIUS being too large (see above), try to increase INDEX_ERROR - in this case, it indexes beautifully with INDEX_ERROR=0.14 . | |||
=== IDXREF produces too long axes === | === IDXREF produces too long axes === | ||
| Line 186: | Line 233: | ||
If this does not help, try to refine even less items, e.g. leave out AXIS. | If this does not help, try to refine even less items, e.g. leave out AXIS. | ||
=== IDXREF prints ERROR IN REFINE !!! RETURN CODE IS IER= 0 === | |||
This is due to either wrong inputs in XDS.INP, or due to bad data, e.g. spots from many crystals whose diffraction patterns overlap. The first possibility applies if the diffraction pattern is clean. The second applies if the diffraction pattern is ugly, and unsuitable for indexing. In that case, maybe try a different SPOT_RANGE. | |||
=== IDXREF.LP does not show the expected lattice === | === IDXREF.LP does not show the expected lattice === | ||
| Line 213: | Line 264: | ||
== Integration failure == | == Integration failure == | ||
=== INTEGRATE stops with error message === | |||
If INTEGRATE stops after e.g. | If INTEGRATE stops after e.g. | ||
****************************************************************************** | ****************************************************************************** | ||
| Line 223: | Line 275: | ||
BEAM_DIVERGENCE= BEAM_DIVERGENCE_E.S.D.= | BEAM_DIVERGENCE= BEAM_DIVERGENCE_E.S.D.= | ||
then you should reduce the upper limit of the DATA_RANGE, to stop before the problematic frames, and re-run INTEGRATE. In this example, you would modify XDS.INP to have | then there are two ways to resolve this: | ||
* you could insert a line DELPHI=10 (or DELPHI=20, for 10 or 20 degrees batches; the default is 5 degree batches) into XDS.INP and re-run INTEGRATE | |||
* as the error message suggests, you should reduce the upper limit of the DATA_RANGE, to stop before the problematic frames, and re-run INTEGRATE. In this example, you would modify XDS.INP to have | |||
DATA_RANGE=1 135 | DATA_RANGE=1 135 | ||
JOB=INTEGRATE CORRECT | JOB=INTEGRATE CORRECT | ||
Save XDS.INP, run XDS and inspect INTEGRATE.LP, to find the lines (e.g.) | Save XDS.INP, run XDS and inspect INTEGRATE.LP, to find the lines (e.g.) | ||
BEAM_DIVERGENCE= 0.478 BEAM_DIVERGENCE_E.S.D.= 0.048 | BEAM_DIVERGENCE= 0.478 BEAM_DIVERGENCE_E.S.D.= 0.048 | ||
REFLECTING_RANGE= 1.100 REFLECTING_RANGE_E.S.D.= 0.157 | REFLECTING_RANGE= 1.100 REFLECTING_RANGE_E.S.D.= 0.157 | ||
Copy them to XDS.INP. Restore the original DATA_RANGE and continue. | Copy them to XDS.INP. Restore the original DATA_RANGE and continue. | ||
If this problem happens with multiple XDS jobs working on the same data set, you might also get a message like | |||
.../bin1_02.tmp | |||
Image PC Routine Line Source | |||
xds_par 0000000000592B91 Unknown Unknown Unknown | |||
xds_par 00000000004BEF21 joinintegrate_ 8995 MAIN_XDS.f90 | |||
xds_par 0000000000410A5B xds_ 21690 MAIN_XDS.f90 | |||
xds_par 000000000040B3A4 MAIN__ 1 MAIN_XDS.f90 | |||
xds_par 00000000004083F6 Unknown Unknown Unknown | |||
libc.so.6 00007FB40222E830 Unknown Unknown Unknown | |||
xds_par 00000000004082E9 Unknown Unknown Unknown | |||
which looks like a real crash of the program, but in this case with a known reason. | |||
Another error mode of INTEGRATE (in processing of small-molecule data) is ... | Another error mode of INTEGRATE (in processing of small-molecule data) is ... | ||
| Line 242: | Line 308: | ||
The error message is misleading in this case: there are too few reflections to build the average profile. The fix is: restart INTEGRATE after inserting e.g. | The error message is misleading in this case: there are too few reflections to build the average profile. The fix is: restart INTEGRATE after inserting e.g. | ||
DELPHI=20 ! default is 5, so try with e.g. 10, 20, 45, 90, 180 | DELPHI=20 ! default is 5, so try with e.g. 10, 20, 45, 90, 180 | ||
and re-run INTEGRATE. | in XDS.INP and re-run INTEGRATE. | ||
=== INTEGRATE cell and/or distance run away; xds crashes or has to be killed === | |||
If during INTEGRATE the cell keeps increasing or the distance decreasing or both, then xds starts to consume large amounts of memory and becomes very slow. This may finally exhaust all available memory, and the job either crashes or has to be killed by the user. | |||
The fix here is to use | |||
REFINE(INTEGRATE)= BEAM POSITION ORIENTATION ! CELL | |||
instead of (what used to be the default until June 2017) | |||
REFINE(INTEGRATE)= BEAM POSITION ORIENTATION CELL | |||
Quite generally, the more conservative setting without CELL refinement is adequate unless your crystals diffract to quite high resolution. A sure sign that you should not be refining CELL is that the refined cell and/or distance values in INTEGRATE.LP fluctuate without obvious physical reason. Moreover, distance ("POSITION") refinement nicely soaks up any cell change resulting from radiation damage. | |||
A compromise (if you suspect that the cell parameters actually change ''differently'') would be | |||
REFINE(INTEGRATE)= BEAM ORIENTATION CELL ! POSITION | |||
This should also avoid the run-away but gives more freedom to the refinement. Of course it requires a refined distance (from IDXREF, or rather from GXPARM.XDS->XPARM.XDS renaming). | |||
== See also == | == See also == | ||
| Line 248: | Line 327: | ||
[[Low dose data]] | [[Low dose data]] | ||
[[Indexing]] | |||
[[IDXREF.LP]] | |||