Eiger: Difference between revisions

Eiger (view source)

Revision as of 22:13, 5 June 2016

423 bytes added , 5 June 2016

no edit summary

Kay

Bureaucrats

2,652

edits

@@ Line 8: / Line 8: @@
 A suitable [[XDS.INP]] may have been written by the data collection (beamline) software. Latest [[generate_XDS.INP]] or the XDS_from_H5.py script (below) can be used if XDS.INP is not available.
-The number of pixels of the Eiger 16M is three times higher than that of the Pilatus 6M, but since the Eiger firmware update in November 2015, the ("bit shufflle LZ4") compression of the .h5 files containing data is better than that of CBF files, which mostly compensates for the increased number of pixels. However, the size of the *master.h5 file from a Eiger 16M experiment at SLS X06SA is more than 300MB, ''no matter how many frames are collected''. It is therefore advisable to compress (by ~75%) the *master.h5 files on-site, before transferring them home using disk or internet. The fastest (parallel) program with the best compression that I found is [http://lbzip2.org lbzip2] (available from the EPEL repository for RHEL clones). It is supposedly fully compatible with bzip2.
+== Compression ==
+The number of pixels of the Eiger 16M is three times higher than that of the Pilatus 6M, but since the Eiger firmware update in November 2015, the ("bit shufflle LZ4") compression of the .h5 files containing data is better than that of CBF files, which mostly compensates for the increased number of pixels.
+The size of the *master.h5 file from a Eiger 16M experiment at SLS X06SA is more than 300MB, ''no matter how many frames are collected''. It is therefore advisable to compress (by ~75%) the *master.h5 files on-site, before transferring them home using disk or internet. The fastest (parallel) program with the best compression that I found is [http://lbzip2.org lbzip2] (available from the EPEL repository for RHEL clones). It is supposedly fully compatible with bzip2.
+Update 2016-06-05 (Toine Schreurs): a HDF5 file may be compressed with [https://www.hdfgroup.org/HDF5/docNewFeatures/FileSpace/h5repack.htm h5repack], e.g. by <cod>h5repack -i <in.h5> -o <out.h5> -f GZIP=6</code>. This should be a good way to reduce the size of master files while keeping them compatible with processing, but needs to be tested. Whether h5repack uses parallel gzip is not clear from the docs.
 == A benchmark ==