https://wiki.uni-konstanz.de/ccp4/api.php?action=feedcontributions&user=Karsten&feedformat=atomCCP4 wiki - User contributions [en]2024-03-29T10:11:56ZUser contributionsMediaWiki 1.39.6https://wiki.uni-konstanz.de/ccp4/index.php?title=CCP4_wiki:Privacy_policy&diff=2809CCP4 wiki:Privacy policy2023-11-06T11:12:23Z<p>Karsten: /* Name and address of the data protection officer */</p>
<hr />
<div><br />
== Name and address of the data protection officer ==<br />
The university’s data protection officer is:<br />
<br />
Irina Weiß<br><br />
DDSK GmbH<br><br />
Email: datenschutzbeauftragter@uni-konstanz.de<br><br />
Website: [https://www.uni-konstanz.de/en/data-protection/ Data protection officer]<br />
<br />
== Name and address of the responsible institution ==<br />
<br />
The responsible institution as per the General Data Protection Regulation (GDPR), other national data protection laws of the member states as well as additional data protection regulations is the:<br />
<br />
University of Konstanz<br><br />
represented by its rector, Professor Dr. Katharina Holzinger<br><br />
Universitaetsstrasse 10<br><br />
78464 Konstanz<br><br />
GERMANY<br><br />
Phone: +49 7531 88-0<br />
<br />
<br />
The '''responsibility for the contents of this wiki''' is with the authors of the individual articles. In particular, '''authors are responsible for not infringing any copyrights, and not compromising the privacy of other people'''.<br />
<br />
If any '''privacy or copyright violations''' are noted, they should be brought to the attention of the '''administrator of this wiki''':<br />
<br />
'''Dr. Karsten Schäfer'''<br><br />
karsten.schaefer at uni-konstanz.de<br><br />
Phone: +49 7531 882900<br />
<br />
Alternatively,<br />
<br />
'''Prof. Dr. Kay Diederichs'''<br><br />
kay.diederichs at uni-konstanz.de<br><br />
Phone: +49 7531 884049<br />
<br />
can be contacted.<br />
<br />
== Providing access to the website and creation of log files ==<br />
<br />
Every time a user accesses a University of Konstanz web page, file or other resource, the following data related to this process is stored in a log file:<br />
<br />
date and time of the request<br />
address and file size (in bytes) of the resource requested and/or accessed<br />
server response (HTTP status code, e.g. “file sent”, “file not found”, etc.)<br />
connection information from the browser and operating system used, if available<br />
referrer website visited prior to the university web page as well as the search terms entered, if available<br />
The last byte of the IP address is removed to anonymise the log files. As a result, it is no longer possible to associate the collected data with a particular person.<br />
<br />
The logged information is used to identify, isolate and fix disruptions or errors in the systems needed to operate the University of Konstanz web pages. These may also include disruptions or errors that lead to the restricted availability of information and communication services or allow unauthorised access to the systems.<br />
<br />
The anonymised information is also used for statistical purposes in order to continually improve the quality of the web services provided.<br />
<br />
<br />
== Using cookies ==<br />
<br />
When a page is visited, the browser will save a so-called “session cookie” to the end device in use. The cookie will be deleted as soon as the browser is closed. The “session cookies” do not contain personal information. It is not intended nor possible to use the session ID to identify a user. The temporary logging of these cookies can be disabled by changing the appropriate settings of the internet browser. This does not affect the access or use of the page, but disabling cookies may increase the server’s response time slightly.<br />
<br />
This does not apply to session cookies saved in connection with a one-time authentication process required to access protected areas of the website. The purpose here is to recognise multiple related requests from the same user, which preserves the user's authentification status while using the website. Blocking these cookies can impair access to and utilisation of the website.<br />
<br />
<br />
== Forms ==<br />
<br />
Description and scope of the processing of data<br />
<br />
Using particular services such as registering for events or filling out contact forms may require collecting, processing, and using personal information. The same applies to personal data transmitted to the email address provided on the respective website (as an alternative to the online form). Information about the legal basis, the purpose of processing and use of the data as well as deletion deadlines is provided on the form pages.<br />
<br />
Information about the data entered by the users themselves, the user's browser as well as the date, time and duration of the entry is saved (spam protection).<br />
<br />
<br />
== Web analysis using awstats==<br />
<br />
We use the open source software tool awstats to analyse our users’ browsing behaviour. The software saves a cookie to the user’s computer (for information on cookies, please see above). When individual university web pages are accessed, the following information is stored:<br />
<br />
(1) the first three bytes of the IP address of the user’s system<br />
<br />
(2) the web page accessed<br />
<br />
(3) the referrer website that the user accessed before visiting the university website<br />
<br />
(4) the sub-pages accessed<br />
<br />
(5) the duration of the user’s visit to the web page<br />
<br />
(6) how often the user accesses the web page<br />
<br />
(7) browser<br />
<br />
(8) browser recognition<br />
<br />
(9) resolution<br />
<br />
(10) the user’s device type (smartphone, tablet, etc.)<br />
<br />
The software runs exclusively on our servers. User data is only saved to our servers. It is not passed on to third parties.<br />
<br />
The software has been configured to mask the last byte of the IP address (e.g.: 192.168.134.xxx). That way, it is impossible to associate the IP address with the device used to access the website.<br />
<br />
You can deactivate or restrict cookies by changing the settings of your internet browser. Saved cookies can be deleted at any time, even automatically, if you wish. If you disable cookies for our website, you may experience difficulties when accessing some if its functions.<br />
<br />
<br />
== Rights of the parties involved ==<br />
<br />
In accordance with § 15 GDPR, you have the right to request information from the University of Konstanz about any data it saves that is related to your person and/or to have incorrect data corrected as per § 16 GDPR.<br />
<br />
You also have the right to demand that your data be deleted (§ 17 GDPR) or that the processing and use thereof be restricted (§ 18 GDPR), as well as to object to the processing and use of your data (§ 21 GDPR).<br />
<br />
You can withdraw your consent regarding the processing and use of your data at any time. The fact that all data processed between the point in time that consent was given and it being withdrawn was processed lawfully remains untouched.<br />
<br />
To better understand and exercise your rights, please contact our data protection officer by emailing datenschutzbeauftragter@uni-konstanz.de.<br />
<br />
You also have the right to file a complaint with the regulating authority if you believe that the processing and use of your personal data is in violation of the law (§ 77 GDPR). The responsible contact person at the regulating authority is the Landesbeauftragter für den Datenschutz und die Informationsfreiheit Baden-Württemberg (state commissioner for data protection and the freedom of information in Baden-Württemberg) (https://www.baden-wuerttemberg.datenschutz.de).</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Main_Page&diff=2808Main Page2023-06-20T09:19:42Z<p>Karsten: </p>
<hr />
<div>This [http://www.ccp4.ac.uk/ CCP4] user community wiki ("CCP4 wiki" in short) is meant to be a collection of crystallographic knowledge as discussed on the [http://www.ccp4.ac.uk/ccp4bb.php CCP4 mailing list] (CCP4BB), and elsewhere. It may contain information about anything relevant to protein crystallographers, whether methods-related ("what is the best program for purpose X?"), problem-oriented ("my crystals melt if I look at them"), or concerning hardware ("what is your opinion on robot X / computer Y?"). <br />
<br />
The topics (side bar) include articles about [[Expression_and_Purification| protein expression and purification]],<br />
[[Crystals| crystal growth, handling and data collection]],[[Crystallography| crystallographic theory and practice]],<br />
and crystal structure related [[Bioinformatics|bioinformatics]] and [[Xtal_computing|computer-related issues]]. There are also announcements for [[Positions|open positions]], [[Courses and Conferences|courses and conferences]] and a [[FAQ|FAQ]].<br />
<br />
Many interesting questions are currently being answered on CCP4BB, but the answers are not easily accessible for others after some time. The intention is that the wiki reflect the breadth of topics on CCP4BB, which will make it a useful resource e.g. for "frequently asked questions", offloading some of the question/answer traffic on CCP4BB to a more permanent mode of storage. A wiki has the additional advantage that mathematical formulas and images may be presented nicely.<br />
<br />
== What is CCP4? ==<br />
[[CCP4]] is a crystallographic program system, and can be downloaded from [http://www.ccp4.ac.uk/download.php]. <br />
It is free for academic use; the license is at [http://www.ccp4.ac.uk/ccp4license.php].<br />
(Legacy) documentation is at [http://legacy.ccp4.ac.uk/docs.php], and a mailing list (CCP4BB) is at [http://www.ccp4.ac.uk/ccp4bb.php] that can also be viewed [http://www.mail-archive.com/ccp4bb@jiscmail.ac.uk/ here] or [https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=ccp4bb here].<br />
<br />
== How to use this wiki ==<br />
* you may read the existing articles with a web browser (does not require an account).<br />
* you are invited to contribute to CCP4 wiki, by creating new articles, and by adding information to, or correcting existing articles. Before you do this for the first time, you need to create an account. <br />
<br />
Due to the amount of spam created, we had to stop self-service account creation - send us a personal email (karsten. schaefer at uni-konstanz.de, or kay.diederichs at uni-konstanz.de) and we'll open account creation for you.<br />
<br />
This wiki is under the same [[Copyright]] (termed "GNU Free Documentation License") as [http://en.wikipedia.org Wikipedia], so any information in it may be transferred to other wikis subject to the [[Copyright]].<br />
<br />
== Getting started ==<br />
Consult the [http://meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software, and see [[Creating an article]].<br />
<br />
Transparent inter-wiki linking is now very easy due to an extension called "[http://www.mediawiki.org/wiki/Extension:Special_page_to_work_with_the_interwiki_table SpecialInterwiki]". These links appear like internal links (see the next sentences!), and can point to subsections of an article, or to figures (untested!). This also means that if a wiki changes its DNS name, only a single inter-wiki link has to be updated. <br />
Currently, the [[ccp4dev:Main_Page|CCP4 developers' wiki]], the [[xds:Main_Page|XDSwiki]] and (as of May 02, 2008) the [[iucr:Main_Page|IUCr Online Dictionary of Crystallography]] are in the [[Special:Interwiki|Inter-Wiki table]]. <br />
<br />
<br />
== Wiki contents ==<br />
<br />
* [[Topics]] - an attempt to list possible items hierarchically<br />
* [[Special:Allpages|All pages]] - list of all pages of this wiki<br />
* [[Courses and Conferences]]<br />
* [[Positions]]<br />
* articles can also be found using the search box in the left column</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Test&diff=2804Test2023-04-24T12:50:37Z<p>Karsten: </p>
<hr />
<div>hallo<br />
<br />
We test tex:<br />
<br />
<math><br />
F_{hkl} = \sum_{i=1}^{N} e^{-2\pi\imath\left( h\frac{x}{a}+k\frac{y}{b}+l\frac{z}{c}\right)}<br />
</math><br />
<br />
[[Image:Acrb.jpg|thumb|test]]<br />
<br />
All crystals should look like this (or better)<br />
[[Image:xtals.jpg||pretty xtals]]<br />
<br />
Problem? - I tried uploading xtals.png but Wiki does not let me upload, quoting that 'wrong file type or extension' - yet it allowed me to load up the same image in jpg format. Png is a nice (some would even say the best) format so it'd be good to be able to use it! -AGE<br />
This is fixed now. --[[User:Kay|Kay]] 10:12, 5 February 2008 (CET)<br />
<br/><br />
Remark: <br />
MIME type of PNG files was not correctly identified. To fix this, we added the lines<br />
# PNG<br />
1 string PNG image/png<br />
to the file /etc/httpd/conf/magic<br />
<br />
Logo prototype #1<br />
[[Image:logo1.png]]<br />
<br />
Logo prototype #2<br />
[[Image:logo2.png]]<br />
<br />
do you like any of these?<br />
<br />
Testing the gnuplot extension (the plot is generated in realtime, not a bitmap)<br />
<gnuplot><br />
set hidden3d<br />
set isosamples 50<br />
splot sin(x)*cos(y)<br />
</gnuplot></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Test&diff=2803Test2023-04-24T12:50:13Z<p>Karsten: </p>
<hr />
<div>[https://www.google.de hallo]<br />
<br />
We test tex:<br />
<br />
<math><br />
F_{hkl} = \sum_{i=1}^{N} e^{-2\pi\imath\left( h\frac{x}{a}+k\frac{y}{b}+l\frac{z}{c}\right)}<br />
</math><br />
<br />
[[Image:Acrb.jpg|thumb|test]]<br />
<br />
All crystals should look like this (or better)<br />
[[Image:xtals.jpg||pretty xtals]]<br />
<br />
Problem? - I tried uploading xtals.png but Wiki does not let me upload, quoting that 'wrong file type or extension' - yet it allowed me to load up the same image in jpg format. Png is a nice (some would even say the best) format so it'd be good to be able to use it! -AGE<br />
This is fixed now. --[[User:Kay|Kay]] 10:12, 5 February 2008 (CET)<br />
<br/><br />
Remark: <br />
MIME type of PNG files was not correctly identified. To fix this, we added the lines<br />
# PNG<br />
1 string PNG image/png<br />
to the file /etc/httpd/conf/magic<br />
<br />
Logo prototype #1<br />
[[Image:logo1.png]]<br />
<br />
Logo prototype #2<br />
[[Image:logo2.png]]<br />
<br />
do you like any of these?<br />
<br />
Testing the gnuplot extension (the plot is generated in realtime, not a bitmap)<br />
<gnuplot><br />
set hidden3d<br />
set isosamples 50<br />
splot sin(x)*cos(y)<br />
</gnuplot></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Coot&diff=2675Coot2019-09-10T07:14:52Z<p>Karsten: </p>
<hr />
<div>[[Image:Coot-with-ATP-vector.png|25%|thumb|right]]<br />
Coot is a graphics program for building, refining and analysing macromolecular models obtained with crystallographic procedures.<br />
<br />
There is a [http://www2.mrc-lmb.cam.ac.uk/Personal/pemsley/coot/ homepage] with extensive [http://www2.mrc-lmb.cam.ac.uk/Personal/pemsley/coot/web/docs/ documentation]. The program may be downloaded for Linux and Windows computers from the [http://www2.mrc-lmb.cam.ac.uk/Personal/pemsley/coot/binaries/pre-release/ primary server]. The license of Coot is GNU GPL. <br />
<br />
=Installing Coot=<br />
==Installing Coot on OS X==<br />
<br />
OS X install packages for nightly builds that work on 10.8.X and 10.9.X are available here: [http://scottlab.ucsc.edu/~wgscott/xtal/wiki/index.php/Stand-Alone_Coot Coot OS X package installers]<br />
<br />
Please refer to the [http://sage.ucsc.edu/xtal/wiki/index.php/Installing_Coot_on_OS_X Installing Coot on OS X] page<br />
<br />
==Installing Coot on Windows==<br />
Please refer to the [http://www.ysbl.york.ac.uk/~lohkamp/coot/wincoot-download.html WinCoot install and download] page.<br />
<br />
==Installing Coot on Linux==<br />
<br />
Installing coot on linux is rather more straightforward than on OS X, because most linux systems are based on gnome and/or kde, and tend to have many of the required components already installed. Most of the other dependencies are also readily available.<br />
<br />
=== Installation from a distributed binary tarball package ===<br />
This is the recommended way for those who do not want to delve into the mysteries of compiling and linking a great but complex piece of software. Read the (somewhat outdated, it seems) [http://www.ysbl.york.ac.uk/%7Eemsley/coot/coot-faq.html Coot FAQ] to find "Additional Notes" for your operating system.<br />
<br />
In short, just go to http://www.ysbl.york.ac.uk/~emsley/software/binaries/nightlies/pre-release/ (a mirror is at ftp://turn5.biologie.uni-konstanz.de/coot/software/binaries/nightlies/pre-release/ ) and pick a suitable binary, e.g.<br />
coot-0.5-pre-1-revision-1003-binary-Linux-i386-fedora-5.tar.gz for a Red Hat Enterprise Linux 5 or CentOS-5 system (Fedora 6 corresponds to RHEL5, thus Fedora 5 binaries are OK). If you prefer a "stable" binary, these are at http://www.ysbl.york.ac.uk/~emsley/software/binaries/stable/.<br />
<br />
Then un-tar it under /usr/local/src (or in your $HOME), and establish a symlink (ln -s) between /usr/local/bin/coot and the bin/coot of the freshly unpacked distribution.<br />
<br />
If you then run coot, and the loader complains that a certain library is missing, just ask<br />
yum whatprovides <thatlibrary><br />
and install the library, again using yum (assuming yum is available in your distribution, otherwise use apt or whatever is there for this purpose).<br />
<br />
==== Example: installing a 64bit nightly CentOS5 binary build on 64bit SL6.1 ====<br />
First of all, SL (Scientific Linux) is a derivative of RHEL, as is CentOS. So all three OSs behave exactly the same.<br />
The binaries with "x86_64" binaries are for 64bit systems; the "i386" binaries are for 32bit systems. Since my notebook is 64bits ("uname -a" reports "x86_64" more than once), I download coot-0.7-pre-1-revision-3999-binary-Linux-x86_64-centos-5-python-gtk2.tar.gz. As root, I "cd /usr/local/src" and un-tar. Next, have to find out which libraries are missing. This can be achieved by (''note the use of LD_LIBRARY_PATH in the second command - do not permanently modify LD_LIBRARY_PATH !''):<br />
[root@localhost]# cd coot-Linux-x86_64-centos-5-gtk2-python<br />
[root@localhost]# LD_LIBRARY_PATH=lib ldd bin/coot-real | grep found <br />
libssl.so.6 => not found<br />
libcrypto.so.6 => not found<br />
libssl.so.6 => not found<br />
libcrypto.so.6 => not found<br />
<br />
So only two libraries are missing! Either they can be installed using yum, or they are already available, but have a higher version.<br />
* First possibility: find out about installable RPM packages (preferred way):<br />
<br />
[root@localhost src]# yum provides libssl.so.6 libcrypto.so.6<br />
Loaded plugins: refresh-packagekit<br />
openssl098e-0.9.8e-17.el6.i686 : A compatibility version of a general<br />
: cryptography and TLS library<br />
Repo : sl<br />
Matched from:<br />
Other : libssl.so.6<br />
... (the package is repeated, and libcrypto.so.6 is also mentioned)<br />
: Now don't just install the openssl098e-0.9.8e-17.el6.i686 and its dependencies - it is a 32bit library (the name ends with ".i686")! Installing it does not solve the problem - we need a 64bit library. Unfortunately "yum provides" does not tell us about the 64bit library (is that a yum bug?). By specifying just the package name (openssl098e.x86_64 would also work, and would avoid any 32bit package)<br />
yum install openssl098e<br />
: we install both libssl.so.6 and libcrypto.so.6 in their 64bit versions - done! <br />
<br />
* Second possibility: find out if the system already has a higher version of the two libraries:<br />
[root@localhost locate libssl.so<br />
/usr/lib64/.libssl.so.1.0.0.hmac<br />
/usr/lib64/.libssl.so.10.hmac<br />
/usr/lib64/libssl.so<br />
/usr/lib64/libssl.so.1.0.0<br />
/usr/lib64/libssl.so.10<br />
<br />
: So the answer is: there is /usr/lib64/libssl.so which is at version 10, which is compatible with the version we need (6). For libcrypto.so the same is true. So just <br />
cd coot-Linux-x86_64-centos-5-gtk2-python/lib/<br />
ln -s /usr/lib64/libssl.so libssl.so.6<br />
ln -s /usr/lib64/libcrypto.so libcrypto.so.6<br />
: The way these symlinks are made they would even work if RHEL upgrades libssl or libcrypto to higher versions. Works for me.<br />
<br />
Final step (this does not need to be repeated for a new coot version): create /usr/local/bin/coot with<br />
#!/bin/csh -f<br />
setenv LANG C<br />
exec /usr/local/src/coot-Linux-x86_64-centos-5-gtk2-python/bin/coot $*<br />
and make it executable with <br />
chmod a+x /usr/local/bin/coot<br />
<br />
=== Installation from source code via autobuild scripts ===<br />
<br />
Installation of coot and all of its dependencies are handled automatically through the autobuild scripts. There are two versions:<br />
* [http://www.ysbl.york.ac.uk/~emsley/build-logs/build-it GTK1] - the old user interface. This script builds coot and all its dependencies.<br />
* [http://www.ysbl.york.ac.uk/~emsley/build-logs/build-it-gtk2-simple GTK2] - the new user interface. This script builds coot and most of the dependencies, excluding GTK2.<br />
<br />
To build Coot, all you should need to do is edit a few settings in the top of the build script, or alternatively specify those settings as environment variables. For example, the following sequence of instructions will build the latest pre-release of the GTK 2 version with python support:<br />
<br />
wget http://www.ysbl.york.ac.uk/~emsley/build-logs/build-it-gtk2-simple<br />
<br />
export AUTOBUILD_INSTALLED=${HOME}/autobuild/coot<br />
export AUTOBUILD_BUILD=${HOME}/autobuild/<br />
export LOGS=$AUTOBUILD_BUILD/logs<br />
export NIGHTLY_DEST_DIR=$AUTOBUILD_BUILD<br />
export STABLE_DEST_DIR=$AUTOBUILD_BUILD<br />
export build_coot_prerelease=1<br />
<br />
bash build-it-gtk2-simple python > build.log<br />
(This script works in bash. For tcsh, replace 'export' with 'setenv' and '=' with ' '.<br />
<br />
In some cases you may need to download additional development packages in order to build all the components.<br />
<br />
=== Installation from source code manually ===<br />
<br />
There are also instructions for [[Custom building Coot from source code]].<br />
<br />
=Running Coot=<br />
==General Topics==<br />
[[Image:Coot-screenshot-small.png]]<br />
<br />
===Controls===<br />
<br />
[[Image:Coot-controls-small.png]]<br />
<br />
<br />
===Stereographic Display ===<br />
<br />
Coot has several options for stereographic display, ranging from cross-eyed and wall-eyed split-screen stereo, to hardware-stereo modes that work with CRT systems and most recently the new Zalman 3-D LCD monitor.<br />
<br />
==== Side-by-Side ====<br />
<br />
Either cross-eyed or wall-eyed split-screen stereo mode can be invoked using the "Stereo" menu item under "Draw", as is shown in the image below:<br />
<br />
[[Image:stereo_menu_screenshot.png]]<br />
<br />
==== Hardware Stereo ====<br />
<br />
Similarly, hardware stereo can be invoked (assuming you have the CRT, correct graphics card, emitter, etc) using the same menu item, by selecting "Hardware Stereo".<br />
<br />
[[Image:a_zalman_zm_m220w__2d_35_pic.jpg|150px|thumb|right|3d lcd]]<br />
<br />
==== Zalman Stereo ====<br />
<br />
The first viable LCD monitor for stereographics display is made by Zalman and costs about $300: [http://www.zalman.co.kr/eng/product/product_read.asp?idx=219 Zalman ZM-M220W]<br />
<br />
The attributes for this monitor have been tested and [http://pymol.org/zalman/ described rather extensively by Warren DeLano] on the PyMOL site. Please read it for important details and suggested purchasing sources.<br />
<br />
The [[coot zalman]] page describes specifically how to get this to work with coot on Mac OS X, but the instructions should be generalizable to linux and Windoze.<br />
<br />
Note that the stereo effect is very sensitive to the vertical position of your eyes relative to the screen: if you don't see stereo, try tilting the screen.<br />
<br />
=== Stereo: left/right (and front/back) interchanged? === <br />
<br />
Establish an additional toolbutton "swap stereo":<br />
<br />
Main Toolbar -> right mouse click-> Manage buttons-> select Swap Stereo<br />
<br />
Or for the script minded:<br />
<br />
switch_stereo_sides()<br />
<br />
This will toggle the stereo images left and right.<br />
<br />
===External Links===<br />
====[http://www.ysbl.york.ac.uk/~emsley/coot/doc/user-manual.html On-line User Manual]====<br />
====[http://www.ysbl.york.ac.uk/~emsley/coot/ Coot's home page]====<br />
====[http://www.mail-archive.com/coot@jiscmail.ac.uk/ Current mailing list archives]====<br />
<br />
====[http://www.ysbl.york.ac.uk/~emsley/coot/mbox/ Mailing list archives: (no longer) current]====<br />
<br />
====[http://www.ysbl.york.ac.uk/~emsley/coot/mbox-2004-2005/ Mailing list archives: 2004-05]====<br />
<br />
==Scheme Scripts==<br />
<br />
Coot can be scripted in scheme (guile) or python - support for each is more or less equal these days. <br />
<br />
Several examples of coot extensions to the language can be seen by examining the 0-coot-state.scm file that coot leaves behind when it finishes.<br />
<br />
===COOPS===<br />
Coops generates a coot script from the output of molprobity, specifically probe, reduce, cluster and clashlist.<br />
<br />
For an explanation of the principals underlying reduce and clashlist see [http://kinemage.biochem.duke.edu/research/aac.php the Dots Page]. Get Molprobity software [http://kinemage.biochem.duke.edu/software/index.php here].<br />
<br />
Use Coot version 0.1 or higher.<br />
<br />
Invoke like this (from the directory in which you run coot):<br />
<br />
<pre>$ coops myfile.pdb</pre><br />
<br />
The use Calculate->Scripting to read in and run coops.scm<br />
<br />
Get COOPS [http://www.ysbl.york.ac.uk/~emsley/software/coops/coops-1.0.tar.gz here].<br />
<br />
===Example Scheme Script 1: Move to Molecule Centres===<br />
This example can be found in the coot scheme sources (the function name is molecule-centres-gui and is in the xxx/share/coot/scheme/coot-gui.scm file). It is a simple function that creates a button box - a button for each coordinates molecule in Coot. It is annotated. Reproduced as [[coot-scheme1]].<br />
<br />
===Example Scheme Script 2: Demo a Few of Coot's Features===<br />
<br />
[[ Composite Example Script | Reading in a pdb file, an MTZ file and manipulating the model ]]<br />
<br />
This is a composite script and demonstrate reading pdb file, an MTZ file, translations, zoom, spin zooms, contour level changing, map masking, real space refinement, water addition and loop fitting.<br />
<br />
The data files used in the example can be obtained [http://www.ysbl.york.ac.uk/~emsley/tutorial/rnasa-1.8-all_refmac1.mtz here] and [http://www.ysbl.york.ac.uk/~emsley/tutorial/tutorial-modern.pdb here]. Put them in the directory where you start coot. Save the script <br />
to your disk, then use Calculate -> Run Script... to activate it.<br />
<br />
===Example Scheme Script 3: Read CNS data===<br />
This [[ CNS data reading script ]] is a Cootenization of the CN2COOT script written by Joel Bard (it is based on his csh [http://www.ysbl.york.ac.uk/~emsley/coot/cn2coot script]) and can be used to compare and contrast scheme programming <br />
and shell script programming (the coot version is longer to some extent because it does extra error checking).<br />
<br />
As well as doing the conversion the resulting mtz/maps are loaded into Coot.<br />
<br />
It is part of Coot as of version 0.1.2.<br />
<br />
===Example Scheme Script 4: Load the Latest Data and PDB files Automatically ===<br />
<br />
To load the most recent files, do this:<br />
<br />
coot --[[script latest-files.scm]]<br />
<br />
which enables the scripting function: (load-latest-files)<br />
<br />
For extra gui goodness (you will need 0.1.2): <br />
<br />
coot --[[script latest-files.scm]] --[[script extensions.scm]]<br />
<br />
===Example Scheme Script 5: Saving a Partial model ===<br />
<br />
Here we create a small function to save part of a molecule and add a <br />
gui interface, it can be used in the usual way (i.e. with --script on the command line, Calculate->Run Script... or <br />
add the script to your ~/.coot file.<br />
<br />
[[save-partial.scm]]<br />
<br />
===Example Scheme Script 6: Creating an interface for the Powermate Dial ===<br />
<br />
The Powermate dial can be used with coot. One could just assign the rotations to +/-y keys and be done with it, but this script gives you a way of having positive and negative rotations in all three cartesian directions. The F1 key is mapped to positive rotation, the F2 key to negative rotation, and the F3 key permits you to toggle through x, y, and z, on successive key presses. I then map F1 and F2 into the ordinary rotations on the powermate (using send key equivalents) and then I map F3 into the single click on the dial, making it easy to toggle through x, y and z. The press-and-rotate options remain available; I map these into scroll up and down, and put them on the slowest response setting, which makes contouring density easier to control than it is from my mouse scroll wheel.<br />
<br />
[http://code.google.com/p/zsh-templates-osx/source/browse/trunk/Library/init/zsh/zshrc.d/local-functions/etc/dotfiles/cootrc_powermate_and_keybindings.scm powermate-coot.scm]<br />
<br />
===Example Scheme Script 7: Applying arbitrary value to "B" factor column ===<br />
<br />
Imagine you have a file of some property (Chemical Shifts, for example) of a residue that you wish to apply to the<br />
atoms of a particular model from a pdb file as pseudo B factors. Here's how to do that in Coot:<br />
<br />
We have a file "cs.tab" like this, the residue number then the chemical shift value (one for each residue in a particular chain):<br />
<br />
1 1.53159<br />
<br />
2 4.35884<br />
<br />
3 4.07123<br />
<br />
4 4.16932<br />
<br />
5 6.69103<br />
<br />
6 7.12071<br />
<br />
7 10.7419<br />
<br />
8 9.57176<br />
<br />
Use apply-cs.scm to apply these values as pseudo temperature factors. Typical usage, where "A" is the chain id, and cs.tab the file of values per residue.<br />
<br />
(apply-cs (read-pdb "test.pdb") "A" "cs.tab")<br />
<br />
--script [[apply-cs.scm]]<br />
<br />
<br />
===Example Script 8: Partial Occupancy Dialog===<br />
<br />
Imagine that you have a structure that has residues with partial occupancy. After refinement, it would be convenient to quickly navigate to all such residues. How can that be done?<br />
<br />
Start coot with command line arguments:<br />
<br />
--script [[partial-occupancy-navigation.scm]]<br />
<br />
This will provide an extra menu item called "Extras", clicking on "Residues with low occupancy..." therein will lead you through the process.<br />
Note that this will often work with SHELXL molecules, because they have atoms with negative (e.g -31) occupancies.<br />
<br />
Note also that you will need a recent version of Coot to use this, as it stands. This will not work on stock Coot version 0.4.x. You can enable this for use with 0.4.x if you update/replace your xxx/share/coot/scheme/coot-gui.scm file from [http://coot.googlecode.com/svn/trunk/scheme/coot-gui.scm here].<br />
<br />
===Example Script 9: A GUI for Chopping Back Sidechains from a Residue Range===<br />
<br />
This is a simple interface to the delete-sidechain-range function, it illustrates how arguments can be transfered<br />
from the GUI to the scripting function. It was written in response to a question from Byron DeLaBarre.<br />
<br />
Unfortunately (prior to 0.5) there was an error in the standard delete-sidechain-range function, which is why we over-ride it.<br />
<br />
--script [[chop-side-chains-gui.scm]]<br />
<br />
===Example 10: How do I bind a key to Toggle the display of NCS ghosts?===<br />
<br />
With this script: [[toggle-ncs-ghosts-script]]<br />
<br />
===Example 11: Paul Emsley's Key Bindings===<br />
<br />
Just so you get an idea of the customization by key bindings here are what Paul uses currently (add to your .coot file).<br />
<br />
[[pauls-key-bindings-for-coot]]<br />
<br />
===Optional Wrappers and (External) Shell Script Enhancements===<br />
<br />
I (wgscott) wrote a [http://xanana.ucsc.edu/Library/init/zsh/local-functions/xtal/coot coot] wrapper shell script that lets you convert xplor/cns maps on the fly (you need to install [http://xray.bmc.uu.se/usf mapman] first) and has a few other enhancements.<br />
<br />
I also made a Coot OS X applet that allows you to drag and drop a cns/xplor or ccp4 mapfile or any other coot-compatable file (mtz or pdb file, for example). Using the File > Get Info dialog, you can program this applet to open all .map and all .mtz files, if you want to, making these files double-clickable.<br />
<br />
[http://xanana.ucsc.edu/xtal/osx_xtal_applets.dmg.gz Download the Applet] (requires a separate working coot installation)<br />
<br />
==Python Scripts==<br />
<br />
===Example 1: Bernhard Lohkamp's Key Bindings===<br />
<br />
Just so you get an idea of the customization by key bindings here are what Bernhard/Paul uses currently (add to your .coot file or put the file in .coot-preferences directory).<br />
<br />
[[bernhards_key_bindings_for_coot.py]]<br />
<br />
===Example 2: More key bindings (inspired by the Coot BB)===<br />
<br />
For (re-)colouring maps blue:<br />
<br />
[[blueify_map_keys.py]]<br />
<br />
To (re-)colour coordinate molecules yellow:<br />
<br />
[[yellowify_molecule_keys.py]]<br />
<br />
===Example 3: NCS Rotamer differences===<br />
<br />
To show NCS where NCS-related side-chains have different rotamers:<br />
<br />
[[ncs_rotamer_differences.py]]<br />
<br />
===Example 4: Morphing GUI===<br />
<br />
GUI to easily access jiggle fit and morphing (currently pre-release Coot required, may be moved into trunk):<br />
<br />
[[morph_residues_gui.py]]<br />
<br />
===Example 5: Ensemble GUI===<br />
<br />
GUI to allow navigation through structural ensembles as obtained e.g. from ensemble refinement:<br />
<br />
[[ensemble_plugin.py]]<br />
<br />
==Python to Scheme and return==<br />
<br />
===Translating between Python and Scheme===<br />
Python scripting is different to (default) scheme scripting which is mainly described in Paul Emsley's documentation (although it's mentioned somewhere, fairly hidden). You have to change the commands in the following way:<br />
<br />
GUILE scripting: (guile-command argument1 argument2 ...)<br />
<br />
PYTHON scripting: python_command(argument1, argument2, ...)<br />
<br />
====Simple rules for Scheme to Python translations====<br />
Here some simple rules how to translate from Scheme to Python. To translate the other way around, i.e. Python to Scheme, just turn the rules around:<br />
<br />
# Replace all '-' with '_' (except in equation when you need arithmetic '-' minus signs)<br />
# Move the brackets around the argument(s)<br />
# Separate multiple arguments by commas rather than spaces<br />
# Replace 'define' with 'def' for functions and with '=' for assignments<br />
# Make sure to use indentation for the function content [Python is indentation sensitive] and a ':' after the function definition.<br />
<br />
Some additional/advanced(?) rules:<br />
# #f -> False<br />
# #t -> True<br />
# (set! variable value) -> variable=value<br />
<br />
====A simple example====<br />
<br />
In Scheme we may have the following script:<br />
<br />
(define mol2-pdbFile "somePDBfile.pdb" )<br />
(define mol2-model (read-pdb mol2-pdbFile))<br />
(define (read-mol-again)<br />
(clear-and-update-model-molecule-from-file mol2-model mol2-pdbFile))<br />
(read-mol-again)<br />
<br />
Which will translate into Python:<br />
<br />
mol2_pdbFile = "somePDBfile.pdb"<br />
mol2_model = read_pdb(mol2_pdbFile)<br />
def read_mol_again():<br />
clear_and_update_model_molecule_from_file(mol2_model, mol2_pdbFile)<br />
read_mol_again()<br />
<br />
===Running a Scheme/Python command from Python/Scheme===<br />
<br />
As of Coot 0.5 (and if you have both scripting languages available) you an use the following commands to run a script or command in the other language:<br />
<br />
(run-python-command "python_command(arg1, arg2, ...)") [from guile/scheme]<br />
<br />
run_scheme_command("(scheme-command arg1 arg2 ...)") [from python]<br />
<br />
=Enhanced Menu Appearance=<br />
<br />
<br />
==As of 0.4, coot works with gtk+2==<br />
<br />
This permits use of themes for a more OSX-like experience, among other things.<br />
<br />
Click on the thumbnail image below to see a full-size screenshot of Coot with a gtk+2 Aqua-like theme.<br />
<br />
[[Image:Thumb-coot-gtk2.png]]<br />
<br />
To get this effect, you need the Glossy_P gtk+2 theme:<br />
<br />
http://art.gnome.org/download/themes/gtk2/571/GTK2-Glossy_P.tar.gz<br />
<br />
Edit a file called ~/.gtkrc-2.0 and put into it the following line:<br />
<br />
include "/usr/share/themes/Glossy\ P/gtk-2.0/gtkrc"<br />
<br />
Alternatively, if you use gnome or xfce4, you can open the theme manager and just make it open the downloaded Glossy_P tarball, and it should add this as a theme.<br />
<br />
=Assorted questions and answers (from the mailinglist)=<br />
<br />
It should be noted that the answers ("A") are from Paul Emsley himself (and were maybe slightly edited).<br />
<br />
==Coot development==<br />
<br />
Q: How can I get involved with Coot development?<br />
<br />
A: Join the [[Coot Janitors]] project. This is a project to get new people involved in improving Coot, by acting as a clearing house for simple tasks which need doing, and providing documentation for doing them.<br />
<br />
<br />
== Get rid of the "fix nomenclature" check ==<br />
Q: Is it possible to deactivate the nomenclature errors check? Sometimes this check is not very useful and it becomes rather annoying when one has several molecules loaded only wants to look at the structures...<br />
<br />
A: Add to your ~/.coot or whatever:<br />
(set-nomenclature-errors-on-read "ignore")<br />
<br />
<br />
==NCS edits== <br />
Q: I am sure this exists somewhere through scripting in COOT, but can I apply NCS edits to only a subset of NCS copies? In other words, can I tell coot which are NCS related chains, and which aren't. I am working on this nightmarish case of asymmetrical homodimers, where the sequences are very similar, but the structures are not, so I need to tell coot which chains are actually related to each other.<br />
<br />
A: Nightmare. If you have a recent [1632 or later for the scheme version, 1646 for the python version] Coot, you can do this:<br />
<br />
(manual-ncs-ghosts imol resno-start resno-end chain-id-list)<br />
<br />
where <br />
* imol is 0 (say)<br />
* resno-start and resno-end is the residue range for the LSQ fitting to return the NCS matrix,<br />
* chain-id-list is the list of chain-ids, starting with the master/reference chain-id and followed by the peer chain-ids that are NCS related, e.g. (list "A" "B" "D")<br />
<br />
The python interface is similar.<br />
<br />
There is also a GUI to activate this feature under Extensions -> NCS.<br />
<br />
==SHELXL==<br />
Description of problematic situation: I am using [[SHELXL]] to refine my 1.2 Å data and I am refining the hydrogen atoms. Subsequent rebuilding in coot is difficult though since hydrogens often does not "follow" when you do side chain rebuilding. For the moment I have quit transfering hydrogens to coot and add the hydrogens every refinement cycle, though it would be good I think if I could see them in coot without bothering about wrong positions. So these are my specific questions:<br />
<br />
Specific Q1: Using "edit chi angles" does not work properly.<br />
<br />
A: This fails because for chi angles Coot uses the Refmac dictionary to know what is connected to what (if it can). The work-around is to rename the refmac dictionary so Coot can't find it - which will force Coot to find bond by distance criteria.<br />
<br />
Specific Q2: Using "real space refine" does not work properly.<br />
<br />
A: Yes this fails. Hydrogens are named differently to SHELX hydrogens. In principal this could be made to work if the dictionary was reworked to use SHELX hydrogen names. This would also fix the chi angles problem too of course.<br />
<br />
Specific Q3: I am unable to open the output pdb file from ShelXL in Coot.<br />
<br />
A: Well, it's hard to know what's the problem without details - the console should say something. But when handling the output of shelxl, I suggest you read the .res file rather than the pdb, then the subsequent .ins file contains lots of "header" information.<br />
<br />
Another answer to questions 1+2 is to rename the hydrogen atoms in the shelxl res-file to match the mmCIF dictionaries used by Coot. This only needs to be done once as shelxl does not modify these names. Except for a few manual editions, the renaming can be done semi-automatically using regular expressions (replacing A->1, B->2, etc).<br />
<br />
Concerning question 3, the Coot -> Extensions -> Module -> SHELXL menu entry works really well now. It reads in all relevant shelxl files and provides a menu highlighting the problematic areas in the model.<br />
<br />
==Image quality on NVidia cards==<br />
Q: improvement of image quality on machines with NVidia cards?<br />
<br />
A: After talking about antialiasing with Stuart McNicholas, I discovered a program called nvidia-settings. I found that if I do:<br />
Antialiasing Settings -> Override Application Setting, and set the slider to 16x<br />
then I start Coot, I see that it starts up in nice antialiasing mode (which is a lot better that the poor mode that Coot has built-in).<br />
I don't know if this works on other/newer systems, but it works for me on my oldish GeForce 6600. (Of course the FPS takes a hit.) <br />
<br />
<br />
==Setting default to show symmetry-related molecules==<br />
Q: How to set the default to display symmetry related molecules?<br />
<br />
A: Add (set-show-symmetry-master 1) to the appropriate file. <br />
<br />
==Startup files==<br />
Q: I still have a ".coot" file in my home folder for a few coot preferences that I couldn't find in the new ".coot-preferences/coot-preferences.scm". There is a warning that I should not add commands to this file. So is a "~/.coot" still the proper place to add default commands for coot? <br />
<br />
A: Coot does not create a ~/.coot file for you, but will read it if it exists. Likewise, ~/.coot.py in which you can write python commands.~/.coot-preferences is a directory in which all .scm files and .py files are executed. coot-preferences.scm and coot_preferences.py there are generated by using the Edit -> Preferences dialog (and thus it overwrites older versions - hence the warning).<br />
<br />
If you want to create a script that will be read by everyone then put those files into a directory defined by environment variable $COOT_SCHEME_EXTRAS_DIR (for *.scm) or $COOT_PYTHON_EXTRAS_DIR (*.py). All *.scm in $COOT_SCHEME_EXTRAS_DIR and *.py files in $COOT_PYTHON_EXTRAS_DIR will be executed at start up.<br />
<br />
So you have a variety of places. Personally I mostly use ~/.coot. <br />
<br />
==Torsion general==<br />
Q: How do I use "torsion general"?<br />
<br />
A: Thanks for pointing out the lack of documentation on this. I'll make a note to add some.<br />
<br />
You need to click on the torsion-general icon, then click 4 atoms that describe the torsion - the first atom will be the base (non moving) part of the atom tree, on clicking the 4th atom a dialog will pop up with a "Reverse" button [1].<br />
<br />
Move this dialog out of the way and then left mouse click and drag in the main<br />
window will rotate the moving/"top" part of the residue round the clicked atoms<br />
2 and 3. When you are happy, click "Accept".<br />
<br />
Window focus may be an issue - depending on your setting, the window manager may<br />
eat one of your clicks as you change focus between the dialog and the main<br />
graphics window (this I find annoying and there are instructions in the FAQ on<br />
how to turn that off for various systems).<br />
<br />
[1] which may not work in 0.6-pre (grumble/sigh/sorry). If it doesn't not work,<br />
the "Reverse" button should invert the moving and "base" part of the residue.<br />
<br />
==Peak heights in maps==<br />
Q: I have some peaks in my map which take water or sodium/magnesium or chlorine atom with out giving out any positive or negative density upon further refinement. Is there any easy way of calculating the peak height / number of electrons at a given position, say a mouse click point in coot? Is there any formula to calculate the number of electrons based on sigma level and peak height, as given in difference map peaks in coot?<br />
<br />
A: First, go to the Coot wiki and pick up the [[Coot#Scheme_Scripts|scheme key bindings]]. <br />
<br />
If you want density information at a given cursor point: point at the blob, press the 'g' key (which recentres on the biggest density under the cursor).<br />
<br />
using the Scheme scripting window:<br />
(apply density-at-point (imol-refinement-map) (rotation-centre))<br />
There is no user access to the peak integration code of coot as yet.<br />
<br />
==Disulfide bond across crystallographic axis==<br />
Q: I have a pair of disulfide bonds which link two monomers in separate asymmetric units. There is a single monomer in the asymmetric unit, and two monomers come together to form disulfides between Cys 26-Cys45, and Cys45-Cys26. When I real-space-refine these residues, they do not form a nice disulfide, and Coot does not seem to recognize them as a disulfide. <br />
<br />
A: For the record, you can't refine symmetry-related disulfides in Coot (as of Nov 3, 2009).<br />
<br />
==Macros in COOT==<br />
<br />
Q: How to use macros in COOT? Do they need to be written in Python or another language that I had not heard of before? Where can I find a low level description of how to write macros with some examples (I know nothing about Python, except that it is fashionable)?<br />
<br />
A: The other language is a form of Lisp, called [http://en.wikipedia.org/wiki/Scheme_(programming_language) Scheme]. You can learn about programming python in many ways of course (not least the [http://docs.python.org/tutorial/ python tutorial], which is what I read first). The coot python extensions are described in the documentation. There is a standard trivial formatting change that has to be made to get the syntax right for python, see "Python Scripting" [[http://www.ysbl.york.ac.uk/~lohkamp/coot/wincoot-faq.html|here]]. There is a growing collection of coot scripts in this Wiki article.<br />
<br />
== building loops ==<br />
<br />
Q: Is there any similar function in COOT as lego_auto_mainchain command in O program?<br />
<br />
A: there are 2 loop fitting tools in Coot<br />
<br />
# C alpha -> Mainchain [http://www2.mrc-lmb.cam.ac.uk/personal/pemsley/coot/web/docs/coot.html#C_002dalpha-_002d_003e-Mainchain],[http://www2.mrc-lmb.cam.ac.uk/personal/pemsley/coot/web/docs/coot.html#Building-Links-and-Loops]<br />
# DB Loop: (No good documentation) [http://www2.mrc-lmb.cam.ac.uk/personal/pemsley/coot/web/docs/coot.html#protein_002ddb_002dloops] Extensions -> Modelling -> DB Loop...<br />
<br />
==LSQ superpositions==<br />
<br />
Q: Do an LSQ superposition using specified residues in multiple chains (superposing one oligomer on another).<br />
<br />
A: Something like this then? <br />
<br />
clear_lsq_matches()<br />
# specs for reference then moving<br />
add_lsq_match(20, 90, "A", 20, 90, "A", 1)<br />
add_lsq_match(20, 90, "B", 20, 90, "B", 1)<br />
add_lsq_match(15, 75, "D", 15, 75, "D", 1)<br />
apply_lsq_matches(1, 2)<br />
<br />
which presumes that the reference molecule is in 1 and the moving molecule 2.<br />
<br />
Q: How to do a LSQ superposition of a homologous structure onto my working structure using ± N residues about the current position, where N is a variable (not essential, could be fixed) and the current position is the last residue that I clicked on.<br />
<br />
A: That is more involved - and more useful because it can be dynamic. Something like the following perhaps (in Scheme, just for amusement (not tested)). You will need to set imol-ref, perhaps by reading in the reference pdb, as demonstrated below. The function is bound to Shift-Y.<br />
<br />
(define dynamic-lsq-range-extent 2) ;; ± 2 residues either side of centre residue<br />
(define imol-ref (read-pdb "reference.pdb"))<br />
<br />
;; convert between the input reference chain id and the chain id of<br />
;; the moving molecule that corresponds to that chain<br />
;;<br />
(define (mov-match-chain ref-chain-id)<br />
ref-chain-id)<br />
<br />
(define (dynamic-lsq-match)<br />
<br />
;; get the current residue and use that to make residue ranges for<br />
;; an LSQ fit<br />
;;<br />
<br />
(using-active-atom<br />
(clear-lsq-matches)<br />
(add-lsq-match (- aa-res-no dynamic-lsq-range-extent)<br />
(+ aa-res-no dynamic-lsq-range-extent)<br />
aa-chain-id<br />
(- aa-res-no dynamic-lsq-range-extent)<br />
(+ aa-res-no dynamic-lsq-range-extent)<br />
(mov-match-chain aa-chain-id)<br />
1)<br />
(apply-lsq-matches aa-imol imol-ref)))<br />
<br />
<br />
(add-key-binding "Dynamic LSQ overlay" "Y" dynamic-lsq-match)<br />
<br />
== reading MTZ file with experimental PHI and FOM using --auto ==<br />
<br />
Q: There is the --auto <filename> commandline option for auto-reading mtz files (mtz file has the default labels FWT, PHWT). Can this be made to work with a SHELXE .phs output file after converting with convert2mtz ? - the resulting MTZ file has labels F PHI FOM.<br />
<br />
A: use: coot --python -c 'make_and_draw_map("sad.mtz", "F", "PHI", "FOM", "/HKL_base/HKL_base/FOM",1, 0)'<br />
<br />
== NCS Rotamer differences ==<br />
<br />
Show me where NCS-related side-chains have different rotamers<br />
<br />
<br />
(define (compare-ncs-rotamer imol chain-A chain-B)<br />
(let ((n-residues (chain-n-residues chain-A imol))<br />
(mismatched-rotamers '()))<br />
(for-each <br />
(lambda (serial-number)<br />
<br />
(let ((res-name-A (resname-from-serial-number imol chain-A serial-number))<br />
(res-no-A (seqnum-from-serial-number imol chain-A serial-number))<br />
(ins-code-A (insertion-code-from-serial-number imol chain-A serial-number))<br />
(res-name-B (resname-from-serial-number imol chain-A serial-number))<br />
(res-no-B (seqnum-from-serial-number imol chain-A serial-number))<br />
(ins-code-B (insertion-code-from-serial-number imol chain-A serial-number)))<br />
(if (not (= res-no-A res-no-B))<br />
(begin<br />
(format #t "sequence number for ~s do not match~%" res-no-A))<br />
(if (not (string=? res-name-A res-name-B))<br />
(begin<br />
(format #t "residue names for ~s do not match~%" res-no-A))<br />
(let ((rot-name-A (get-rotamer-name imol chain-A res-no-A ins-code-A))<br />
(rot-name-B (get-rotamer-name imol chain-B res-no-B ins-code-B)))<br />
(if (not (string=? rot-name-A rot-name-B))<br />
(begin<br />
(set! mismatched-rotamers <br />
(cons (list imol chain-A res-no-A ins-code-A <br />
res-name-A <br />
(if (string=? rot-name-A "") "-" rot-name-A)<br />
(if (string=? rot-name-B "") "-" rot-name-B))<br />
mismatched-rotamers))))<br />
)))))<br />
(range n-residues))<br />
(dialog-box-of-buttons "Mismatched Rotamers"<br />
(cons 300 300)<br />
(map (lambda(rotamer) <br />
(let ((label (string-append " " <br />
(list-ref rotamer 1)<br />
" "<br />
(number->string (list-ref rotamer 2))<br />
(list-ref rotamer 3) <br />
" "<br />
(list-ref rotamer 4) ;; res-name<br />
": "<br />
(list-ref rotamer 5)<br />
" vs. "<br />
(list-ref rotamer 6)))<br />
(thunk (lambda ()<br />
(set-go-to-atom-molecule imol)<br />
(set-go-to-atom-chain-residue-atom-name <br />
(list-ref rotamer 1) <br />
(list-ref rotamer 2) " CA "))))<br />
(list label thunk)))<br />
mismatched-rotamers)<br />
" Close ")))<br />
<br />
<br />
<br />
And one would use this something like:<br />
<br />
;; example usage:<br />
(let ((imol (read-pdb "test.pdb")))<br />
(compare-ncs-rotamer imol "A" "B"))<br />
<br />
== make RSR in coot 0.8.1 behave like in earlier versions ==<br />
<br />
Q: We've noticed a new behavior in real space refinement in coot 0.8.1 whereby dragged atoms are more tightly restrained to their initial positions than in earlier versions. This seems to be described in the release notes by:<br />
<br />
o BUG-FIX: The amount that the other atoms ove with moving the picked atom has been reduced (but is configurable) <br />
<br />
A: Add e.g. this to your ~/.coot.py file:<br />
<br />
set_refinement_drag_elasticity(0.8)<br />
<br />
Q: I'm wondering why this was changed. Does the optimum elasticity change with resolution, map quality, or another experimental limitation? Or does it more of a user preference?<br />
<br />
A: Because of cis-peptides. My worry was that in the previous regime, it was <br />
too easy to introduce cis-peptides when fitting to low resolution maps. <br />
I believe the current default setting is much less likely to do that.<br />
<br />
Q: I've tried various settings of refinement_drag_elasticity and I need to lower it to 0.5 or so before any semblance of earlier behavior appears.<br />
<br />
A: It used to be 0.167, I think.<br />
<br />
== Molprobity not active in COOT ==<br />
<br />
Q: I am using COOT 0.8.1 EL that comes with the CCP4 6.5.010 on my Mac OS X 10.10.2. I wanted to run molprobity but the Validate > Probe clashes button in my pull down menu is not active. Is this function available in this COOT version?<br />
<br />
A: Reduce and probe are separate programs available from the Richardson’s lab at Duke http://kinemage.biochem.duke.edu/. Download and install on your box. Then coot needs to be told in some instances where it can find these executables. I have the following lines in my ~/.coot file in Linux. <br />
<br />
<pre><br />
;; .coot<br />
;; This file is required. As of coot 0.8pre no other mechanism for<br />
;; enabling probe in coot works<br />
;;<br />
;; This is full pathname of molprobity's probe program<br />
(define *probe-command* "/apps/xray/bin/probe")<br />
;; This is full pathname of molprobity's reduce program<br />
(define *reduce-command* "/apps/xray/bin/reduce")<br />
</pre><br />
Untried: if you have Phenix installed: it comes with phenix.probe and phenix.reduce - you could insert the paths to these binaries into the above definitions.<br />
<br />
== some symmetry mates not shown ==<br />
<br />
Q: This structure has been solved and refined using phenix in the hexagonal setting of space group R 3. There is one copy per asymmetric unit in R 3. As you can see from the attached image, coot is rendering some but not all of the symmetry mates.<br />
<br />
A: Turn up the radius a bit and use (set-symmetry-shift-search-size 3) . I would have thought that 2 is big enough, but maybe not in this case.</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Coot&diff=2674Coot2019-09-10T07:08:29Z<p>Karsten: </p>
<hr />
<div>[[Image:Coot-with-ATP-vector.png|30%|thumb|right]]<br />
Coot is a graphics program for building, refining and analysing macromolecular models obtained with crystallographic procedures.<br />
<br />
There is a [http://www2.mrc-lmb.cam.ac.uk/Personal/pemsley/coot/ homepage] with extensive [http://www2.mrc-lmb.cam.ac.uk/Personal/pemsley/coot/web/docs/ documentation]. The program may be downloaded for Linux and Windows computers from the [http://www2.mrc-lmb.cam.ac.uk/Personal/pemsley/coot/binaries/pre-release/ primary server]. The license of Coot is GNU GPL. <br />
<br />
=Installing Coot=<br />
==Installing Coot on OS X==<br />
<br />
OS X install packages for nightly builds that work on 10.8.X and 10.9.X are available here: [http://scottlab.ucsc.edu/~wgscott/xtal/wiki/index.php/Stand-Alone_Coot Coot OS X package installers]<br />
<br />
Please refer to the [http://sage.ucsc.edu/xtal/wiki/index.php/Installing_Coot_on_OS_X Installing Coot on OS X] page<br />
<br />
==Installing Coot on Windows==<br />
Please refer to the [http://www.ysbl.york.ac.uk/~lohkamp/coot/wincoot-download.html WinCoot install and download] page.<br />
<br />
==Installing Coot on Linux==<br />
<br />
Installing coot on linux is rather more straightforward than on OS X, because most linux systems are based on gnome and/or kde, and tend to have many of the required components already installed. Most of the other dependencies are also readily available.<br />
<br />
=== Installation from a distributed binary tarball package ===<br />
This is the recommended way for those who do not want to delve into the mysteries of compiling and linking a great but complex piece of software. Read the (somewhat outdated, it seems) [http://www.ysbl.york.ac.uk/%7Eemsley/coot/coot-faq.html Coot FAQ] to find "Additional Notes" for your operating system.<br />
<br />
In short, just go to http://www.ysbl.york.ac.uk/~emsley/software/binaries/nightlies/pre-release/ (a mirror is at ftp://turn5.biologie.uni-konstanz.de/coot/software/binaries/nightlies/pre-release/ ) and pick a suitable binary, e.g.<br />
coot-0.5-pre-1-revision-1003-binary-Linux-i386-fedora-5.tar.gz for a Red Hat Enterprise Linux 5 or CentOS-5 system (Fedora 6 corresponds to RHEL5, thus Fedora 5 binaries are OK). If you prefer a "stable" binary, these are at http://www.ysbl.york.ac.uk/~emsley/software/binaries/stable/.<br />
<br />
Then un-tar it under /usr/local/src (or in your $HOME), and establish a symlink (ln -s) between /usr/local/bin/coot and the bin/coot of the freshly unpacked distribution.<br />
<br />
If you then run coot, and the loader complains that a certain library is missing, just ask<br />
yum whatprovides <thatlibrary><br />
and install the library, again using yum (assuming yum is available in your distribution, otherwise use apt or whatever is there for this purpose).<br />
<br />
==== Example: installing a 64bit nightly CentOS5 binary build on 64bit SL6.1 ====<br />
First of all, SL (Scientific Linux) is a derivative of RHEL, as is CentOS. So all three OSs behave exactly the same.<br />
The binaries with "x86_64" binaries are for 64bit systems; the "i386" binaries are for 32bit systems. Since my notebook is 64bits ("uname -a" reports "x86_64" more than once), I download coot-0.7-pre-1-revision-3999-binary-Linux-x86_64-centos-5-python-gtk2.tar.gz. As root, I "cd /usr/local/src" and un-tar. Next, have to find out which libraries are missing. This can be achieved by (''note the use of LD_LIBRARY_PATH in the second command - do not permanently modify LD_LIBRARY_PATH !''):<br />
[root@localhost]# cd coot-Linux-x86_64-centos-5-gtk2-python<br />
[root@localhost]# LD_LIBRARY_PATH=lib ldd bin/coot-real | grep found <br />
libssl.so.6 => not found<br />
libcrypto.so.6 => not found<br />
libssl.so.6 => not found<br />
libcrypto.so.6 => not found<br />
<br />
So only two libraries are missing! Either they can be installed using yum, or they are already available, but have a higher version.<br />
* First possibility: find out about installable RPM packages (preferred way):<br />
<br />
[root@localhost src]# yum provides libssl.so.6 libcrypto.so.6<br />
Loaded plugins: refresh-packagekit<br />
openssl098e-0.9.8e-17.el6.i686 : A compatibility version of a general<br />
: cryptography and TLS library<br />
Repo : sl<br />
Matched from:<br />
Other : libssl.so.6<br />
... (the package is repeated, and libcrypto.so.6 is also mentioned)<br />
: Now don't just install the openssl098e-0.9.8e-17.el6.i686 and its dependencies - it is a 32bit library (the name ends with ".i686")! Installing it does not solve the problem - we need a 64bit library. Unfortunately "yum provides" does not tell us about the 64bit library (is that a yum bug?). By specifying just the package name (openssl098e.x86_64 would also work, and would avoid any 32bit package)<br />
yum install openssl098e<br />
: we install both libssl.so.6 and libcrypto.so.6 in their 64bit versions - done! <br />
<br />
* Second possibility: find out if the system already has a higher version of the two libraries:<br />
[root@localhost locate libssl.so<br />
/usr/lib64/.libssl.so.1.0.0.hmac<br />
/usr/lib64/.libssl.so.10.hmac<br />
/usr/lib64/libssl.so<br />
/usr/lib64/libssl.so.1.0.0<br />
/usr/lib64/libssl.so.10<br />
<br />
: So the answer is: there is /usr/lib64/libssl.so which is at version 10, which is compatible with the version we need (6). For libcrypto.so the same is true. So just <br />
cd coot-Linux-x86_64-centos-5-gtk2-python/lib/<br />
ln -s /usr/lib64/libssl.so libssl.so.6<br />
ln -s /usr/lib64/libcrypto.so libcrypto.so.6<br />
: The way these symlinks are made they would even work if RHEL upgrades libssl or libcrypto to higher versions. Works for me.<br />
<br />
Final step (this does not need to be repeated for a new coot version): create /usr/local/bin/coot with<br />
#!/bin/csh -f<br />
setenv LANG C<br />
exec /usr/local/src/coot-Linux-x86_64-centos-5-gtk2-python/bin/coot $*<br />
and make it executable with <br />
chmod a+x /usr/local/bin/coot<br />
<br />
=== Installation from source code via autobuild scripts ===<br />
<br />
Installation of coot and all of its dependencies are handled automatically through the autobuild scripts. There are two versions:<br />
* [http://www.ysbl.york.ac.uk/~emsley/build-logs/build-it GTK1] - the old user interface. This script builds coot and all its dependencies.<br />
* [http://www.ysbl.york.ac.uk/~emsley/build-logs/build-it-gtk2-simple GTK2] - the new user interface. This script builds coot and most of the dependencies, excluding GTK2.<br />
<br />
To build Coot, all you should need to do is edit a few settings in the top of the build script, or alternatively specify those settings as environment variables. For example, the following sequence of instructions will build the latest pre-release of the GTK 2 version with python support:<br />
<br />
wget http://www.ysbl.york.ac.uk/~emsley/build-logs/build-it-gtk2-simple<br />
<br />
export AUTOBUILD_INSTALLED=${HOME}/autobuild/coot<br />
export AUTOBUILD_BUILD=${HOME}/autobuild/<br />
export LOGS=$AUTOBUILD_BUILD/logs<br />
export NIGHTLY_DEST_DIR=$AUTOBUILD_BUILD<br />
export STABLE_DEST_DIR=$AUTOBUILD_BUILD<br />
export build_coot_prerelease=1<br />
<br />
bash build-it-gtk2-simple python > build.log<br />
(This script works in bash. For tcsh, replace 'export' with 'setenv' and '=' with ' '.<br />
<br />
In some cases you may need to download additional development packages in order to build all the components.<br />
<br />
=== Installation from source code manually ===<br />
<br />
There are also instructions for [[Custom building Coot from source code]].<br />
<br />
=Running Coot=<br />
==General Topics==<br />
[[Image:Coot-screenshot-small.png]]<br />
<br />
===Controls===<br />
<br />
[[Image:Coot-controls-small.png]]<br />
<br />
<br />
===Stereographic Display ===<br />
<br />
Coot has several options for stereographic display, ranging from cross-eyed and wall-eyed split-screen stereo, to hardware-stereo modes that work with CRT systems and most recently the new Zalman 3-D LCD monitor.<br />
<br />
==== Side-by-Side ====<br />
<br />
Either cross-eyed or wall-eyed split-screen stereo mode can be invoked using the "Stereo" menu item under "Draw", as is shown in the image below:<br />
<br />
[[Image:stereo_menu_screenshot.png]]<br />
<br />
==== Hardware Stereo ====<br />
<br />
Similarly, hardware stereo can be invoked (assuming you have the CRT, correct graphics card, emitter, etc) using the same menu item, by selecting "Hardware Stereo".<br />
<br />
[[Image:a_zalman_zm_m220w__2d_35_pic.jpg|150px|thumb|right|3d lcd]]<br />
<br />
==== Zalman Stereo ====<br />
<br />
The first viable LCD monitor for stereographics display is made by Zalman and costs about $300: [http://www.zalman.co.kr/eng/product/product_read.asp?idx=219 Zalman ZM-M220W]<br />
<br />
The attributes for this monitor have been tested and [http://pymol.org/zalman/ described rather extensively by Warren DeLano] on the PyMOL site. Please read it for important details and suggested purchasing sources.<br />
<br />
The [[coot zalman]] page describes specifically how to get this to work with coot on Mac OS X, but the instructions should be generalizable to linux and Windoze.<br />
<br />
Note that the stereo effect is very sensitive to the vertical position of your eyes relative to the screen: if you don't see stereo, try tilting the screen.<br />
<br />
=== Stereo: left/right (and front/back) interchanged? === <br />
<br />
Establish an additional toolbutton "swap stereo":<br />
<br />
Main Toolbar -> right mouse click-> Manage buttons-> select Swap Stereo<br />
<br />
Or for the script minded:<br />
<br />
switch_stereo_sides()<br />
<br />
This will toggle the stereo images left and right.<br />
<br />
===External Links===<br />
====[http://www.ysbl.york.ac.uk/~emsley/coot/doc/user-manual.html On-line User Manual]====<br />
====[http://www.ysbl.york.ac.uk/~emsley/coot/ Coot's home page]====<br />
====[http://www.mail-archive.com/coot@jiscmail.ac.uk/ Current mailing list archives]====<br />
<br />
====[http://www.ysbl.york.ac.uk/~emsley/coot/mbox/ Mailing list archives: (no longer) current]====<br />
<br />
====[http://www.ysbl.york.ac.uk/~emsley/coot/mbox-2004-2005/ Mailing list archives: 2004-05]====<br />
<br />
==Scheme Scripts==<br />
<br />
Coot can be scripted in scheme (guile) or python - support for each is more or less equal these days. <br />
<br />
Several examples of coot extensions to the language can be seen by examining the 0-coot-state.scm file that coot leaves behind when it finishes.<br />
<br />
===COOPS===<br />
Coops generates a coot script from the output of molprobity, specifically probe, reduce, cluster and clashlist.<br />
<br />
For an explanation of the principals underlying reduce and clashlist see [http://kinemage.biochem.duke.edu/research/aac.php the Dots Page]. Get Molprobity software [http://kinemage.biochem.duke.edu/software/index.php here].<br />
<br />
Use Coot version 0.1 or higher.<br />
<br />
Invoke like this (from the directory in which you run coot):<br />
<br />
<pre>$ coops myfile.pdb</pre><br />
<br />
The use Calculate->Scripting to read in and run coops.scm<br />
<br />
Get COOPS [http://www.ysbl.york.ac.uk/~emsley/software/coops/coops-1.0.tar.gz here].<br />
<br />
===Example Scheme Script 1: Move to Molecule Centres===<br />
This example can be found in the coot scheme sources (the function name is molecule-centres-gui and is in the xxx/share/coot/scheme/coot-gui.scm file). It is a simple function that creates a button box - a button for each coordinates molecule in Coot. It is annotated. Reproduced as [[coot-scheme1]].<br />
<br />
===Example Scheme Script 2: Demo a Few of Coot's Features===<br />
<br />
[[ Composite Example Script | Reading in a pdb file, an MTZ file and manipulating the model ]]<br />
<br />
This is a composite script and demonstrate reading pdb file, an MTZ file, translations, zoom, spin zooms, contour level changing, map masking, real space refinement, water addition and loop fitting.<br />
<br />
The data files used in the example can be obtained [http://www.ysbl.york.ac.uk/~emsley/tutorial/rnasa-1.8-all_refmac1.mtz here] and [http://www.ysbl.york.ac.uk/~emsley/tutorial/tutorial-modern.pdb here]. Put them in the directory where you start coot. Save the script <br />
to your disk, then use Calculate -> Run Script... to activate it.<br />
<br />
===Example Scheme Script 3: Read CNS data===<br />
This [[ CNS data reading script ]] is a Cootenization of the CN2COOT script written by Joel Bard (it is based on his csh [http://www.ysbl.york.ac.uk/~emsley/coot/cn2coot script]) and can be used to compare and contrast scheme programming <br />
and shell script programming (the coot version is longer to some extent because it does extra error checking).<br />
<br />
As well as doing the conversion the resulting mtz/maps are loaded into Coot.<br />
<br />
It is part of Coot as of version 0.1.2.<br />
<br />
===Example Scheme Script 4: Load the Latest Data and PDB files Automatically ===<br />
<br />
To load the most recent files, do this:<br />
<br />
coot --[[script latest-files.scm]]<br />
<br />
which enables the scripting function: (load-latest-files)<br />
<br />
For extra gui goodness (you will need 0.1.2): <br />
<br />
coot --[[script latest-files.scm]] --[[script extensions.scm]]<br />
<br />
===Example Scheme Script 5: Saving a Partial model ===<br />
<br />
Here we create a small function to save part of a molecule and add a <br />
gui interface, it can be used in the usual way (i.e. with --script on the command line, Calculate->Run Script... or <br />
add the script to your ~/.coot file.<br />
<br />
[[save-partial.scm]]<br />
<br />
===Example Scheme Script 6: Creating an interface for the Powermate Dial ===<br />
<br />
The Powermate dial can be used with coot. One could just assign the rotations to +/-y keys and be done with it, but this script gives you a way of having positive and negative rotations in all three cartesian directions. The F1 key is mapped to positive rotation, the F2 key to negative rotation, and the F3 key permits you to toggle through x, y, and z, on successive key presses. I then map F1 and F2 into the ordinary rotations on the powermate (using send key equivalents) and then I map F3 into the single click on the dial, making it easy to toggle through x, y and z. The press-and-rotate options remain available; I map these into scroll up and down, and put them on the slowest response setting, which makes contouring density easier to control than it is from my mouse scroll wheel.<br />
<br />
[http://code.google.com/p/zsh-templates-osx/source/browse/trunk/Library/init/zsh/zshrc.d/local-functions/etc/dotfiles/cootrc_powermate_and_keybindings.scm powermate-coot.scm]<br />
<br />
===Example Scheme Script 7: Applying arbitrary value to "B" factor column ===<br />
<br />
Imagine you have a file of some property (Chemical Shifts, for example) of a residue that you wish to apply to the<br />
atoms of a particular model from a pdb file as pseudo B factors. Here's how to do that in Coot:<br />
<br />
We have a file "cs.tab" like this, the residue number then the chemical shift value (one for each residue in a particular chain):<br />
<br />
1 1.53159<br />
<br />
2 4.35884<br />
<br />
3 4.07123<br />
<br />
4 4.16932<br />
<br />
5 6.69103<br />
<br />
6 7.12071<br />
<br />
7 10.7419<br />
<br />
8 9.57176<br />
<br />
Use apply-cs.scm to apply these values as pseudo temperature factors. Typical usage, where "A" is the chain id, and cs.tab the file of values per residue.<br />
<br />
(apply-cs (read-pdb "test.pdb") "A" "cs.tab")<br />
<br />
--script [[apply-cs.scm]]<br />
<br />
<br />
===Example Script 8: Partial Occupancy Dialog===<br />
<br />
Imagine that you have a structure that has residues with partial occupancy. After refinement, it would be convenient to quickly navigate to all such residues. How can that be done?<br />
<br />
Start coot with command line arguments:<br />
<br />
--script [[partial-occupancy-navigation.scm]]<br />
<br />
This will provide an extra menu item called "Extras", clicking on "Residues with low occupancy..." therein will lead you through the process.<br />
Note that this will often work with SHELXL molecules, because they have atoms with negative (e.g -31) occupancies.<br />
<br />
Note also that you will need a recent version of Coot to use this, as it stands. This will not work on stock Coot version 0.4.x. You can enable this for use with 0.4.x if you update/replace your xxx/share/coot/scheme/coot-gui.scm file from [http://coot.googlecode.com/svn/trunk/scheme/coot-gui.scm here].<br />
<br />
===Example Script 9: A GUI for Chopping Back Sidechains from a Residue Range===<br />
<br />
This is a simple interface to the delete-sidechain-range function, it illustrates how arguments can be transfered<br />
from the GUI to the scripting function. It was written in response to a question from Byron DeLaBarre.<br />
<br />
Unfortunately (prior to 0.5) there was an error in the standard delete-sidechain-range function, which is why we over-ride it.<br />
<br />
--script [[chop-side-chains-gui.scm]]<br />
<br />
===Example 10: How do I bind a key to Toggle the display of NCS ghosts?===<br />
<br />
With this script: [[toggle-ncs-ghosts-script]]<br />
<br />
===Example 11: Paul Emsley's Key Bindings===<br />
<br />
Just so you get an idea of the customization by key bindings here are what Paul uses currently (add to your .coot file).<br />
<br />
[[pauls-key-bindings-for-coot]]<br />
<br />
===Optional Wrappers and (External) Shell Script Enhancements===<br />
<br />
I (wgscott) wrote a [http://xanana.ucsc.edu/Library/init/zsh/local-functions/xtal/coot coot] wrapper shell script that lets you convert xplor/cns maps on the fly (you need to install [http://xray.bmc.uu.se/usf mapman] first) and has a few other enhancements.<br />
<br />
I also made a Coot OS X applet that allows you to drag and drop a cns/xplor or ccp4 mapfile or any other coot-compatable file (mtz or pdb file, for example). Using the File > Get Info dialog, you can program this applet to open all .map and all .mtz files, if you want to, making these files double-clickable.<br />
<br />
[http://xanana.ucsc.edu/xtal/osx_xtal_applets.dmg.gz Download the Applet] (requires a separate working coot installation)<br />
<br />
==Python Scripts==<br />
<br />
===Example 1: Bernhard Lohkamp's Key Bindings===<br />
<br />
Just so you get an idea of the customization by key bindings here are what Bernhard/Paul uses currently (add to your .coot file or put the file in .coot-preferences directory).<br />
<br />
[[bernhards_key_bindings_for_coot.py]]<br />
<br />
===Example 2: More key bindings (inspired by the Coot BB)===<br />
<br />
For (re-)colouring maps blue:<br />
<br />
[[blueify_map_keys.py]]<br />
<br />
To (re-)colour coordinate molecules yellow:<br />
<br />
[[yellowify_molecule_keys.py]]<br />
<br />
===Example 3: NCS Rotamer differences===<br />
<br />
To show NCS where NCS-related side-chains have different rotamers:<br />
<br />
[[ncs_rotamer_differences.py]]<br />
<br />
===Example 4: Morphing GUI===<br />
<br />
GUI to easily access jiggle fit and morphing (currently pre-release Coot required, may be moved into trunk):<br />
<br />
[[morph_residues_gui.py]]<br />
<br />
===Example 5: Ensemble GUI===<br />
<br />
GUI to allow navigation through structural ensembles as obtained e.g. from ensemble refinement:<br />
<br />
[[ensemble_plugin.py]]<br />
<br />
==Python to Scheme and return==<br />
<br />
===Translating between Python and Scheme===<br />
Python scripting is different to (default) scheme scripting which is mainly described in Paul Emsley's documentation (although it's mentioned somewhere, fairly hidden). You have to change the commands in the following way:<br />
<br />
GUILE scripting: (guile-command argument1 argument2 ...)<br />
<br />
PYTHON scripting: python_command(argument1, argument2, ...)<br />
<br />
====Simple rules for Scheme to Python translations====<br />
Here some simple rules how to translate from Scheme to Python. To translate the other way around, i.e. Python to Scheme, just turn the rules around:<br />
<br />
# Replace all '-' with '_' (except in equation when you need arithmetic '-' minus signs)<br />
# Move the brackets around the argument(s)<br />
# Separate multiple arguments by commas rather than spaces<br />
# Replace 'define' with 'def' for functions and with '=' for assignments<br />
# Make sure to use indentation for the function content [Python is indentation sensitive] and a ':' after the function definition.<br />
<br />
Some additional/advanced(?) rules:<br />
# #f -> False<br />
# #t -> True<br />
# (set! variable value) -> variable=value<br />
<br />
====A simple example====<br />
<br />
In Scheme we may have the following script:<br />
<br />
(define mol2-pdbFile "somePDBfile.pdb" )<br />
(define mol2-model (read-pdb mol2-pdbFile))<br />
(define (read-mol-again)<br />
(clear-and-update-model-molecule-from-file mol2-model mol2-pdbFile))<br />
(read-mol-again)<br />
<br />
Which will translate into Python:<br />
<br />
mol2_pdbFile = "somePDBfile.pdb"<br />
mol2_model = read_pdb(mol2_pdbFile)<br />
def read_mol_again():<br />
clear_and_update_model_molecule_from_file(mol2_model, mol2_pdbFile)<br />
read_mol_again()<br />
<br />
===Running a Scheme/Python command from Python/Scheme===<br />
<br />
As of Coot 0.5 (and if you have both scripting languages available) you an use the following commands to run a script or command in the other language:<br />
<br />
(run-python-command "python_command(arg1, arg2, ...)") [from guile/scheme]<br />
<br />
run_scheme_command("(scheme-command arg1 arg2 ...)") [from python]<br />
<br />
=Enhanced Menu Appearance=<br />
<br />
<br />
==As of 0.4, coot works with gtk+2==<br />
<br />
This permits use of themes for a more OSX-like experience, among other things.<br />
<br />
Click on the thumbnail image below to see a full-size screenshot of Coot with a gtk+2 Aqua-like theme.<br />
<br />
[[Image:Thumb-coot-gtk2.png]]<br />
<br />
To get this effect, you need the Glossy_P gtk+2 theme:<br />
<br />
http://art.gnome.org/download/themes/gtk2/571/GTK2-Glossy_P.tar.gz<br />
<br />
Edit a file called ~/.gtkrc-2.0 and put into it the following line:<br />
<br />
include "/usr/share/themes/Glossy\ P/gtk-2.0/gtkrc"<br />
<br />
Alternatively, if you use gnome or xfce4, you can open the theme manager and just make it open the downloaded Glossy_P tarball, and it should add this as a theme.<br />
<br />
=Assorted questions and answers (from the mailinglist)=<br />
<br />
It should be noted that the answers ("A") are from Paul Emsley himself (and were maybe slightly edited).<br />
<br />
==Coot development==<br />
<br />
Q: How can I get involved with Coot development?<br />
<br />
A: Join the [[Coot Janitors]] project. This is a project to get new people involved in improving Coot, by acting as a clearing house for simple tasks which need doing, and providing documentation for doing them.<br />
<br />
<br />
== Get rid of the "fix nomenclature" check ==<br />
Q: Is it possible to deactivate the nomenclature errors check? Sometimes this check is not very useful and it becomes rather annoying when one has several molecules loaded only wants to look at the structures...<br />
<br />
A: Add to your ~/.coot or whatever:<br />
(set-nomenclature-errors-on-read "ignore")<br />
<br />
<br />
==NCS edits== <br />
Q: I am sure this exists somewhere through scripting in COOT, but can I apply NCS edits to only a subset of NCS copies? In other words, can I tell coot which are NCS related chains, and which aren't. I am working on this nightmarish case of asymmetrical homodimers, where the sequences are very similar, but the structures are not, so I need to tell coot which chains are actually related to each other.<br />
<br />
A: Nightmare. If you have a recent [1632 or later for the scheme version, 1646 for the python version] Coot, you can do this:<br />
<br />
(manual-ncs-ghosts imol resno-start resno-end chain-id-list)<br />
<br />
where <br />
* imol is 0 (say)<br />
* resno-start and resno-end is the residue range for the LSQ fitting to return the NCS matrix,<br />
* chain-id-list is the list of chain-ids, starting with the master/reference chain-id and followed by the peer chain-ids that are NCS related, e.g. (list "A" "B" "D")<br />
<br />
The python interface is similar.<br />
<br />
There is also a GUI to activate this feature under Extensions -> NCS.<br />
<br />
==SHELXL==<br />
Description of problematic situation: I am using [[SHELXL]] to refine my 1.2 Å data and I am refining the hydrogen atoms. Subsequent rebuilding in coot is difficult though since hydrogens often does not "follow" when you do side chain rebuilding. For the moment I have quit transfering hydrogens to coot and add the hydrogens every refinement cycle, though it would be good I think if I could see them in coot without bothering about wrong positions. So these are my specific questions:<br />
<br />
Specific Q1: Using "edit chi angles" does not work properly.<br />
<br />
A: This fails because for chi angles Coot uses the Refmac dictionary to know what is connected to what (if it can). The work-around is to rename the refmac dictionary so Coot can't find it - which will force Coot to find bond by distance criteria.<br />
<br />
Specific Q2: Using "real space refine" does not work properly.<br />
<br />
A: Yes this fails. Hydrogens are named differently to SHELX hydrogens. In principal this could be made to work if the dictionary was reworked to use SHELX hydrogen names. This would also fix the chi angles problem too of course.<br />
<br />
Specific Q3: I am unable to open the output pdb file from ShelXL in Coot.<br />
<br />
A: Well, it's hard to know what's the problem without details - the console should say something. But when handling the output of shelxl, I suggest you read the .res file rather than the pdb, then the subsequent .ins file contains lots of "header" information.<br />
<br />
Another answer to questions 1+2 is to rename the hydrogen atoms in the shelxl res-file to match the mmCIF dictionaries used by Coot. This only needs to be done once as shelxl does not modify these names. Except for a few manual editions, the renaming can be done semi-automatically using regular expressions (replacing A->1, B->2, etc).<br />
<br />
Concerning question 3, the Coot -> Extensions -> Module -> SHELXL menu entry works really well now. It reads in all relevant shelxl files and provides a menu highlighting the problematic areas in the model.<br />
<br />
==Image quality on NVidia cards==<br />
Q: improvement of image quality on machines with NVidia cards?<br />
<br />
A: After talking about antialiasing with Stuart McNicholas, I discovered a program called nvidia-settings. I found that if I do:<br />
Antialiasing Settings -> Override Application Setting, and set the slider to 16x<br />
then I start Coot, I see that it starts up in nice antialiasing mode (which is a lot better that the poor mode that Coot has built-in).<br />
I don't know if this works on other/newer systems, but it works for me on my oldish GeForce 6600. (Of course the FPS takes a hit.) <br />
<br />
<br />
==Setting default to show symmetry-related molecules==<br />
Q: How to set the default to display symmetry related molecules?<br />
<br />
A: Add (set-show-symmetry-master 1) to the appropriate file. <br />
<br />
==Startup files==<br />
Q: I still have a ".coot" file in my home folder for a few coot preferences that I couldn't find in the new ".coot-preferences/coot-preferences.scm". There is a warning that I should not add commands to this file. So is a "~/.coot" still the proper place to add default commands for coot? <br />
<br />
A: Coot does not create a ~/.coot file for you, but will read it if it exists. Likewise, ~/.coot.py in which you can write python commands.~/.coot-preferences is a directory in which all .scm files and .py files are executed. coot-preferences.scm and coot_preferences.py there are generated by using the Edit -> Preferences dialog (and thus it overwrites older versions - hence the warning).<br />
<br />
If you want to create a script that will be read by everyone then put those files into a directory defined by environment variable $COOT_SCHEME_EXTRAS_DIR (for *.scm) or $COOT_PYTHON_EXTRAS_DIR (*.py). All *.scm in $COOT_SCHEME_EXTRAS_DIR and *.py files in $COOT_PYTHON_EXTRAS_DIR will be executed at start up.<br />
<br />
So you have a variety of places. Personally I mostly use ~/.coot. <br />
<br />
==Torsion general==<br />
Q: How do I use "torsion general"?<br />
<br />
A: Thanks for pointing out the lack of documentation on this. I'll make a note to add some.<br />
<br />
You need to click on the torsion-general icon, then click 4 atoms that describe the torsion - the first atom will be the base (non moving) part of the atom tree, on clicking the 4th atom a dialog will pop up with a "Reverse" button [1].<br />
<br />
Move this dialog out of the way and then left mouse click and drag in the main<br />
window will rotate the moving/"top" part of the residue round the clicked atoms<br />
2 and 3. When you are happy, click "Accept".<br />
<br />
Window focus may be an issue - depending on your setting, the window manager may<br />
eat one of your clicks as you change focus between the dialog and the main<br />
graphics window (this I find annoying and there are instructions in the FAQ on<br />
how to turn that off for various systems).<br />
<br />
[1] which may not work in 0.6-pre (grumble/sigh/sorry). If it doesn't not work,<br />
the "Reverse" button should invert the moving and "base" part of the residue.<br />
<br />
==Peak heights in maps==<br />
Q: I have some peaks in my map which take water or sodium/magnesium or chlorine atom with out giving out any positive or negative density upon further refinement. Is there any easy way of calculating the peak height / number of electrons at a given position, say a mouse click point in coot? Is there any formula to calculate the number of electrons based on sigma level and peak height, as given in difference map peaks in coot?<br />
<br />
A: First, go to the Coot wiki and pick up the [[Coot#Scheme_Scripts|scheme key bindings]]. <br />
<br />
If you want density information at a given cursor point: point at the blob, press the 'g' key (which recentres on the biggest density under the cursor).<br />
<br />
using the Scheme scripting window:<br />
(apply density-at-point (imol-refinement-map) (rotation-centre))<br />
There is no user access to the peak integration code of coot as yet.<br />
<br />
==Disulfide bond across crystallographic axis==<br />
Q: I have a pair of disulfide bonds which link two monomers in separate asymmetric units. There is a single monomer in the asymmetric unit, and two monomers come together to form disulfides between Cys 26-Cys45, and Cys45-Cys26. When I real-space-refine these residues, they do not form a nice disulfide, and Coot does not seem to recognize them as a disulfide. <br />
<br />
A: For the record, you can't refine symmetry-related disulfides in Coot (as of Nov 3, 2009).<br />
<br />
==Macros in COOT==<br />
<br />
Q: How to use macros in COOT? Do they need to be written in Python or another language that I had not heard of before? Where can I find a low level description of how to write macros with some examples (I know nothing about Python, except that it is fashionable)?<br />
<br />
A: The other language is a form of Lisp, called [http://en.wikipedia.org/wiki/Scheme_(programming_language) Scheme]. You can learn about programming python in many ways of course (not least the [http://docs.python.org/tutorial/ python tutorial], which is what I read first). The coot python extensions are described in the documentation. There is a standard trivial formatting change that has to be made to get the syntax right for python, see "Python Scripting" [[http://www.ysbl.york.ac.uk/~lohkamp/coot/wincoot-faq.html|here]]. There is a growing collection of coot scripts in this Wiki article.<br />
<br />
== building loops ==<br />
<br />
Q: Is there any similar function in COOT as lego_auto_mainchain command in O program?<br />
<br />
A: there are 2 loop fitting tools in Coot<br />
<br />
# C alpha -> Mainchain [http://www2.mrc-lmb.cam.ac.uk/personal/pemsley/coot/web/docs/coot.html#C_002dalpha-_002d_003e-Mainchain],[http://www2.mrc-lmb.cam.ac.uk/personal/pemsley/coot/web/docs/coot.html#Building-Links-and-Loops]<br />
# DB Loop: (No good documentation) [http://www2.mrc-lmb.cam.ac.uk/personal/pemsley/coot/web/docs/coot.html#protein_002ddb_002dloops] Extensions -> Modelling -> DB Loop...<br />
<br />
==LSQ superpositions==<br />
<br />
Q: Do an LSQ superposition using specified residues in multiple chains (superposing one oligomer on another).<br />
<br />
A: Something like this then? <br />
<br />
clear_lsq_matches()<br />
# specs for reference then moving<br />
add_lsq_match(20, 90, "A", 20, 90, "A", 1)<br />
add_lsq_match(20, 90, "B", 20, 90, "B", 1)<br />
add_lsq_match(15, 75, "D", 15, 75, "D", 1)<br />
apply_lsq_matches(1, 2)<br />
<br />
which presumes that the reference molecule is in 1 and the moving molecule 2.<br />
<br />
Q: How to do a LSQ superposition of a homologous structure onto my working structure using ± N residues about the current position, where N is a variable (not essential, could be fixed) and the current position is the last residue that I clicked on.<br />
<br />
A: That is more involved - and more useful because it can be dynamic. Something like the following perhaps (in Scheme, just for amusement (not tested)). You will need to set imol-ref, perhaps by reading in the reference pdb, as demonstrated below. The function is bound to Shift-Y.<br />
<br />
(define dynamic-lsq-range-extent 2) ;; ± 2 residues either side of centre residue<br />
(define imol-ref (read-pdb "reference.pdb"))<br />
<br />
;; convert between the input reference chain id and the chain id of<br />
;; the moving molecule that corresponds to that chain<br />
;;<br />
(define (mov-match-chain ref-chain-id)<br />
ref-chain-id)<br />
<br />
(define (dynamic-lsq-match)<br />
<br />
;; get the current residue and use that to make residue ranges for<br />
;; an LSQ fit<br />
;;<br />
<br />
(using-active-atom<br />
(clear-lsq-matches)<br />
(add-lsq-match (- aa-res-no dynamic-lsq-range-extent)<br />
(+ aa-res-no dynamic-lsq-range-extent)<br />
aa-chain-id<br />
(- aa-res-no dynamic-lsq-range-extent)<br />
(+ aa-res-no dynamic-lsq-range-extent)<br />
(mov-match-chain aa-chain-id)<br />
1)<br />
(apply-lsq-matches aa-imol imol-ref)))<br />
<br />
<br />
(add-key-binding "Dynamic LSQ overlay" "Y" dynamic-lsq-match)<br />
<br />
== reading MTZ file with experimental PHI and FOM using --auto ==<br />
<br />
Q: There is the --auto <filename> commandline option for auto-reading mtz files (mtz file has the default labels FWT, PHWT). Can this be made to work with a SHELXE .phs output file after converting with convert2mtz ? - the resulting MTZ file has labels F PHI FOM.<br />
<br />
A: use: coot --python -c 'make_and_draw_map("sad.mtz", "F", "PHI", "FOM", "/HKL_base/HKL_base/FOM",1, 0)'<br />
<br />
== NCS Rotamer differences ==<br />
<br />
Show me where NCS-related side-chains have different rotamers<br />
<br />
<br />
(define (compare-ncs-rotamer imol chain-A chain-B)<br />
(let ((n-residues (chain-n-residues chain-A imol))<br />
(mismatched-rotamers '()))<br />
(for-each <br />
(lambda (serial-number)<br />
<br />
(let ((res-name-A (resname-from-serial-number imol chain-A serial-number))<br />
(res-no-A (seqnum-from-serial-number imol chain-A serial-number))<br />
(ins-code-A (insertion-code-from-serial-number imol chain-A serial-number))<br />
(res-name-B (resname-from-serial-number imol chain-A serial-number))<br />
(res-no-B (seqnum-from-serial-number imol chain-A serial-number))<br />
(ins-code-B (insertion-code-from-serial-number imol chain-A serial-number)))<br />
(if (not (= res-no-A res-no-B))<br />
(begin<br />
(format #t "sequence number for ~s do not match~%" res-no-A))<br />
(if (not (string=? res-name-A res-name-B))<br />
(begin<br />
(format #t "residue names for ~s do not match~%" res-no-A))<br />
(let ((rot-name-A (get-rotamer-name imol chain-A res-no-A ins-code-A))<br />
(rot-name-B (get-rotamer-name imol chain-B res-no-B ins-code-B)))<br />
(if (not (string=? rot-name-A rot-name-B))<br />
(begin<br />
(set! mismatched-rotamers <br />
(cons (list imol chain-A res-no-A ins-code-A <br />
res-name-A <br />
(if (string=? rot-name-A "") "-" rot-name-A)<br />
(if (string=? rot-name-B "") "-" rot-name-B))<br />
mismatched-rotamers))))<br />
)))))<br />
(range n-residues))<br />
(dialog-box-of-buttons "Mismatched Rotamers"<br />
(cons 300 300)<br />
(map (lambda(rotamer) <br />
(let ((label (string-append " " <br />
(list-ref rotamer 1)<br />
" "<br />
(number->string (list-ref rotamer 2))<br />
(list-ref rotamer 3) <br />
" "<br />
(list-ref rotamer 4) ;; res-name<br />
": "<br />
(list-ref rotamer 5)<br />
" vs. "<br />
(list-ref rotamer 6)))<br />
(thunk (lambda ()<br />
(set-go-to-atom-molecule imol)<br />
(set-go-to-atom-chain-residue-atom-name <br />
(list-ref rotamer 1) <br />
(list-ref rotamer 2) " CA "))))<br />
(list label thunk)))<br />
mismatched-rotamers)<br />
" Close ")))<br />
<br />
<br />
<br />
And one would use this something like:<br />
<br />
;; example usage:<br />
(let ((imol (read-pdb "test.pdb")))<br />
(compare-ncs-rotamer imol "A" "B"))<br />
<br />
== make RSR in coot 0.8.1 behave like in earlier versions ==<br />
<br />
Q: We've noticed a new behavior in real space refinement in coot 0.8.1 whereby dragged atoms are more tightly restrained to their initial positions than in earlier versions. This seems to be described in the release notes by:<br />
<br />
o BUG-FIX: The amount that the other atoms ove with moving the picked atom has been reduced (but is configurable) <br />
<br />
A: Add e.g. this to your ~/.coot.py file:<br />
<br />
set_refinement_drag_elasticity(0.8)<br />
<br />
Q: I'm wondering why this was changed. Does the optimum elasticity change with resolution, map quality, or another experimental limitation? Or does it more of a user preference?<br />
<br />
A: Because of cis-peptides. My worry was that in the previous regime, it was <br />
too easy to introduce cis-peptides when fitting to low resolution maps. <br />
I believe the current default setting is much less likely to do that.<br />
<br />
Q: I've tried various settings of refinement_drag_elasticity and I need to lower it to 0.5 or so before any semblance of earlier behavior appears.<br />
<br />
A: It used to be 0.167, I think.<br />
<br />
== Molprobity not active in COOT ==<br />
<br />
Q: I am using COOT 0.8.1 EL that comes with the CCP4 6.5.010 on my Mac OS X 10.10.2. I wanted to run molprobity but the Validate > Probe clashes button in my pull down menu is not active. Is this function available in this COOT version?<br />
<br />
A: Reduce and probe are separate programs available from the Richardson’s lab at Duke http://kinemage.biochem.duke.edu/. Download and install on your box. Then coot needs to be told in some instances where it can find these executables. I have the following lines in my ~/.coot file in Linux. <br />
<br />
<pre><br />
;; .coot<br />
;; This file is required. As of coot 0.8pre no other mechanism for<br />
;; enabling probe in coot works<br />
;;<br />
;; This is full pathname of molprobity's probe program<br />
(define *probe-command* "/apps/xray/bin/probe")<br />
;; This is full pathname of molprobity's reduce program<br />
(define *reduce-command* "/apps/xray/bin/reduce")<br />
</pre><br />
Untried: if you have Phenix installed: it comes with phenix.probe and phenix.reduce - you could insert the paths to these binaries into the above definitions.<br />
<br />
== some symmetry mates not shown ==<br />
<br />
Q: This structure has been solved and refined using phenix in the hexagonal setting of space group R 3. There is one copy per asymmetric unit in R 3. As you can see from the attached image, coot is rendering some but not all of the symmetry mates.<br />
<br />
A: Turn up the radius a bit and use (set-symmetry-shift-search-size 3) . I would have thought that 2 is big enough, but maybe not in this case.</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Coot&diff=2673Coot2019-09-10T07:01:27Z<p>Karsten: </p>
<hr />
<div>[[Image:Coot-with-ATP-vector.png|60%|thumb|right]]<br />
Coot is a graphics program for building, refining and analysing macromolecular models obtained with crystallographic procedures.<br />
<br />
There is a [http://www2.mrc-lmb.cam.ac.uk/Personal/pemsley/coot/ homepage] with extensive [http://www2.mrc-lmb.cam.ac.uk/Personal/pemsley/coot/web/docs/ documentation]. The program may be downloaded for Linux and Windows computers from the [http://www2.mrc-lmb.cam.ac.uk/Personal/pemsley/coot/binaries/pre-release/ primary server]. The license of Coot is GNU GPL. <br />
<br />
=Installing Coot=<br />
==Installing Coot on OS X==<br />
<br />
OS X install packages for nightly builds that work on 10.8.X and 10.9.X are available here: [http://scottlab.ucsc.edu/~wgscott/xtal/wiki/index.php/Stand-Alone_Coot Coot OS X package installers]<br />
<br />
Please refer to the [http://sage.ucsc.edu/xtal/wiki/index.php/Installing_Coot_on_OS_X Installing Coot on OS X] page<br />
<br />
==Installing Coot on Windows==<br />
Please refer to the [http://www.ysbl.york.ac.uk/~lohkamp/coot/wincoot-download.html WinCoot install and download] page.<br />
<br />
==Installing Coot on Linux==<br />
<br />
Installing coot on linux is rather more straightforward than on OS X, because most linux systems are based on gnome and/or kde, and tend to have many of the required components already installed. Most of the other dependencies are also readily available.<br />
<br />
=== Installation from a distributed binary tarball package ===<br />
This is the recommended way for those who do not want to delve into the mysteries of compiling and linking a great but complex piece of software. Read the (somewhat outdated, it seems) [http://www.ysbl.york.ac.uk/%7Eemsley/coot/coot-faq.html Coot FAQ] to find "Additional Notes" for your operating system.<br />
<br />
In short, just go to http://www.ysbl.york.ac.uk/~emsley/software/binaries/nightlies/pre-release/ (a mirror is at ftp://turn5.biologie.uni-konstanz.de/coot/software/binaries/nightlies/pre-release/ ) and pick a suitable binary, e.g.<br />
coot-0.5-pre-1-revision-1003-binary-Linux-i386-fedora-5.tar.gz for a Red Hat Enterprise Linux 5 or CentOS-5 system (Fedora 6 corresponds to RHEL5, thus Fedora 5 binaries are OK). If you prefer a "stable" binary, these are at http://www.ysbl.york.ac.uk/~emsley/software/binaries/stable/.<br />
<br />
Then un-tar it under /usr/local/src (or in your $HOME), and establish a symlink (ln -s) between /usr/local/bin/coot and the bin/coot of the freshly unpacked distribution.<br />
<br />
If you then run coot, and the loader complains that a certain library is missing, just ask<br />
yum whatprovides <thatlibrary><br />
and install the library, again using yum (assuming yum is available in your distribution, otherwise use apt or whatever is there for this purpose).<br />
<br />
==== Example: installing a 64bit nightly CentOS5 binary build on 64bit SL6.1 ====<br />
First of all, SL (Scientific Linux) is a derivative of RHEL, as is CentOS. So all three OSs behave exactly the same.<br />
The binaries with "x86_64" binaries are for 64bit systems; the "i386" binaries are for 32bit systems. Since my notebook is 64bits ("uname -a" reports "x86_64" more than once), I download coot-0.7-pre-1-revision-3999-binary-Linux-x86_64-centos-5-python-gtk2.tar.gz. As root, I "cd /usr/local/src" and un-tar. Next, have to find out which libraries are missing. This can be achieved by (''note the use of LD_LIBRARY_PATH in the second command - do not permanently modify LD_LIBRARY_PATH !''):<br />
[root@localhost]# cd coot-Linux-x86_64-centos-5-gtk2-python<br />
[root@localhost]# LD_LIBRARY_PATH=lib ldd bin/coot-real | grep found <br />
libssl.so.6 => not found<br />
libcrypto.so.6 => not found<br />
libssl.so.6 => not found<br />
libcrypto.so.6 => not found<br />
<br />
So only two libraries are missing! Either they can be installed using yum, or they are already available, but have a higher version.<br />
* First possibility: find out about installable RPM packages (preferred way):<br />
<br />
[root@localhost src]# yum provides libssl.so.6 libcrypto.so.6<br />
Loaded plugins: refresh-packagekit<br />
openssl098e-0.9.8e-17.el6.i686 : A compatibility version of a general<br />
: cryptography and TLS library<br />
Repo : sl<br />
Matched from:<br />
Other : libssl.so.6<br />
... (the package is repeated, and libcrypto.so.6 is also mentioned)<br />
: Now don't just install the openssl098e-0.9.8e-17.el6.i686 and its dependencies - it is a 32bit library (the name ends with ".i686")! Installing it does not solve the problem - we need a 64bit library. Unfortunately "yum provides" does not tell us about the 64bit library (is that a yum bug?). By specifying just the package name (openssl098e.x86_64 would also work, and would avoid any 32bit package)<br />
yum install openssl098e<br />
: we install both libssl.so.6 and libcrypto.so.6 in their 64bit versions - done! <br />
<br />
* Second possibility: find out if the system already has a higher version of the two libraries:<br />
[root@localhost locate libssl.so<br />
/usr/lib64/.libssl.so.1.0.0.hmac<br />
/usr/lib64/.libssl.so.10.hmac<br />
/usr/lib64/libssl.so<br />
/usr/lib64/libssl.so.1.0.0<br />
/usr/lib64/libssl.so.10<br />
<br />
: So the answer is: there is /usr/lib64/libssl.so which is at version 10, which is compatible with the version we need (6). For libcrypto.so the same is true. So just <br />
cd coot-Linux-x86_64-centos-5-gtk2-python/lib/<br />
ln -s /usr/lib64/libssl.so libssl.so.6<br />
ln -s /usr/lib64/libcrypto.so libcrypto.so.6<br />
: The way these symlinks are made they would even work if RHEL upgrades libssl or libcrypto to higher versions. Works for me.<br />
<br />
Final step (this does not need to be repeated for a new coot version): create /usr/local/bin/coot with<br />
#!/bin/csh -f<br />
setenv LANG C<br />
exec /usr/local/src/coot-Linux-x86_64-centos-5-gtk2-python/bin/coot $*<br />
and make it executable with <br />
chmod a+x /usr/local/bin/coot<br />
<br />
=== Installation from source code via autobuild scripts ===<br />
<br />
Installation of coot and all of its dependencies are handled automatically through the autobuild scripts. There are two versions:<br />
* [http://www.ysbl.york.ac.uk/~emsley/build-logs/build-it GTK1] - the old user interface. This script builds coot and all its dependencies.<br />
* [http://www.ysbl.york.ac.uk/~emsley/build-logs/build-it-gtk2-simple GTK2] - the new user interface. This script builds coot and most of the dependencies, excluding GTK2.<br />
<br />
To build Coot, all you should need to do is edit a few settings in the top of the build script, or alternatively specify those settings as environment variables. For example, the following sequence of instructions will build the latest pre-release of the GTK 2 version with python support:<br />
<br />
wget http://www.ysbl.york.ac.uk/~emsley/build-logs/build-it-gtk2-simple<br />
<br />
export AUTOBUILD_INSTALLED=${HOME}/autobuild/coot<br />
export AUTOBUILD_BUILD=${HOME}/autobuild/<br />
export LOGS=$AUTOBUILD_BUILD/logs<br />
export NIGHTLY_DEST_DIR=$AUTOBUILD_BUILD<br />
export STABLE_DEST_DIR=$AUTOBUILD_BUILD<br />
export build_coot_prerelease=1<br />
<br />
bash build-it-gtk2-simple python > build.log<br />
(This script works in bash. For tcsh, replace 'export' with 'setenv' and '=' with ' '.<br />
<br />
In some cases you may need to download additional development packages in order to build all the components.<br />
<br />
=== Installation from source code manually ===<br />
<br />
There are also instructions for [[Custom building Coot from source code]].<br />
<br />
=Running Coot=<br />
==General Topics==<br />
[[Image:Coot-screenshot-small.png]]<br />
<br />
===Controls===<br />
<br />
[[Image:Coot-controls-small.png]]<br />
<br />
<br />
===Stereographic Display ===<br />
<br />
Coot has several options for stereographic display, ranging from cross-eyed and wall-eyed split-screen stereo, to hardware-stereo modes that work with CRT systems and most recently the new Zalman 3-D LCD monitor.<br />
<br />
==== Side-by-Side ====<br />
<br />
Either cross-eyed or wall-eyed split-screen stereo mode can be invoked using the "Stereo" menu item under "Draw", as is shown in the image below:<br />
<br />
[[Image:stereo_menu_screenshot.png]]<br />
<br />
==== Hardware Stereo ====<br />
<br />
Similarly, hardware stereo can be invoked (assuming you have the CRT, correct graphics card, emitter, etc) using the same menu item, by selecting "Hardware Stereo".<br />
<br />
[[Image:a_zalman_zm_m220w__2d_35_pic.jpg|150px|thumb|right|3d lcd]]<br />
<br />
==== Zalman Stereo ====<br />
<br />
The first viable LCD monitor for stereographics display is made by Zalman and costs about $300: [http://www.zalman.co.kr/eng/product/product_read.asp?idx=219 Zalman ZM-M220W]<br />
<br />
The attributes for this monitor have been tested and [http://pymol.org/zalman/ described rather extensively by Warren DeLano] on the PyMOL site. Please read it for important details and suggested purchasing sources.<br />
<br />
The [[coot zalman]] page describes specifically how to get this to work with coot on Mac OS X, but the instructions should be generalizable to linux and Windoze.<br />
<br />
Note that the stereo effect is very sensitive to the vertical position of your eyes relative to the screen: if you don't see stereo, try tilting the screen.<br />
<br />
=== Stereo: left/right (and front/back) interchanged? === <br />
<br />
Establish an additional toolbutton "swap stereo":<br />
<br />
Main Toolbar -> right mouse click-> Manage buttons-> select Swap Stereo<br />
<br />
Or for the script minded:<br />
<br />
switch_stereo_sides()<br />
<br />
This will toggle the stereo images left and right.<br />
<br />
===External Links===<br />
====[http://www.ysbl.york.ac.uk/~emsley/coot/doc/user-manual.html On-line User Manual]====<br />
====[http://www.ysbl.york.ac.uk/~emsley/coot/ Coot's home page]====<br />
====[http://www.mail-archive.com/coot@jiscmail.ac.uk/ Current mailing list archives]====<br />
<br />
====[http://www.ysbl.york.ac.uk/~emsley/coot/mbox/ Mailing list archives: (no longer) current]====<br />
<br />
====[http://www.ysbl.york.ac.uk/~emsley/coot/mbox-2004-2005/ Mailing list archives: 2004-05]====<br />
<br />
==Scheme Scripts==<br />
<br />
Coot can be scripted in scheme (guile) or python - support for each is more or less equal these days. <br />
<br />
Several examples of coot extensions to the language can be seen by examining the 0-coot-state.scm file that coot leaves behind when it finishes.<br />
<br />
===COOPS===<br />
Coops generates a coot script from the output of molprobity, specifically probe, reduce, cluster and clashlist.<br />
<br />
For an explanation of the principals underlying reduce and clashlist see [http://kinemage.biochem.duke.edu/research/aac.php the Dots Page]. Get Molprobity software [http://kinemage.biochem.duke.edu/software/index.php here].<br />
<br />
Use Coot version 0.1 or higher.<br />
<br />
Invoke like this (from the directory in which you run coot):<br />
<br />
<pre>$ coops myfile.pdb</pre><br />
<br />
The use Calculate->Scripting to read in and run coops.scm<br />
<br />
Get COOPS [http://www.ysbl.york.ac.uk/~emsley/software/coops/coops-1.0.tar.gz here].<br />
<br />
===Example Scheme Script 1: Move to Molecule Centres===<br />
This example can be found in the coot scheme sources (the function name is molecule-centres-gui and is in the xxx/share/coot/scheme/coot-gui.scm file). It is a simple function that creates a button box - a button for each coordinates molecule in Coot. It is annotated. Reproduced as [[coot-scheme1]].<br />
<br />
===Example Scheme Script 2: Demo a Few of Coot's Features===<br />
<br />
[[ Composite Example Script | Reading in a pdb file, an MTZ file and manipulating the model ]]<br />
<br />
This is a composite script and demonstrate reading pdb file, an MTZ file, translations, zoom, spin zooms, contour level changing, map masking, real space refinement, water addition and loop fitting.<br />
<br />
The data files used in the example can be obtained [http://www.ysbl.york.ac.uk/~emsley/tutorial/rnasa-1.8-all_refmac1.mtz here] and [http://www.ysbl.york.ac.uk/~emsley/tutorial/tutorial-modern.pdb here]. Put them in the directory where you start coot. Save the script <br />
to your disk, then use Calculate -> Run Script... to activate it.<br />
<br />
===Example Scheme Script 3: Read CNS data===<br />
This [[ CNS data reading script ]] is a Cootenization of the CN2COOT script written by Joel Bard (it is based on his csh [http://www.ysbl.york.ac.uk/~emsley/coot/cn2coot script]) and can be used to compare and contrast scheme programming <br />
and shell script programming (the coot version is longer to some extent because it does extra error checking).<br />
<br />
As well as doing the conversion the resulting mtz/maps are loaded into Coot.<br />
<br />
It is part of Coot as of version 0.1.2.<br />
<br />
===Example Scheme Script 4: Load the Latest Data and PDB files Automatically ===<br />
<br />
To load the most recent files, do this:<br />
<br />
coot --[[script latest-files.scm]]<br />
<br />
which enables the scripting function: (load-latest-files)<br />
<br />
For extra gui goodness (you will need 0.1.2): <br />
<br />
coot --[[script latest-files.scm]] --[[script extensions.scm]]<br />
<br />
===Example Scheme Script 5: Saving a Partial model ===<br />
<br />
Here we create a small function to save part of a molecule and add a <br />
gui interface, it can be used in the usual way (i.e. with --script on the command line, Calculate->Run Script... or <br />
add the script to your ~/.coot file.<br />
<br />
[[save-partial.scm]]<br />
<br />
===Example Scheme Script 6: Creating an interface for the Powermate Dial ===<br />
<br />
The Powermate dial can be used with coot. One could just assign the rotations to +/-y keys and be done with it, but this script gives you a way of having positive and negative rotations in all three cartesian directions. The F1 key is mapped to positive rotation, the F2 key to negative rotation, and the F3 key permits you to toggle through x, y, and z, on successive key presses. I then map F1 and F2 into the ordinary rotations on the powermate (using send key equivalents) and then I map F3 into the single click on the dial, making it easy to toggle through x, y and z. The press-and-rotate options remain available; I map these into scroll up and down, and put them on the slowest response setting, which makes contouring density easier to control than it is from my mouse scroll wheel.<br />
<br />
[http://code.google.com/p/zsh-templates-osx/source/browse/trunk/Library/init/zsh/zshrc.d/local-functions/etc/dotfiles/cootrc_powermate_and_keybindings.scm powermate-coot.scm]<br />
<br />
===Example Scheme Script 7: Applying arbitrary value to "B" factor column ===<br />
<br />
Imagine you have a file of some property (Chemical Shifts, for example) of a residue that you wish to apply to the<br />
atoms of a particular model from a pdb file as pseudo B factors. Here's how to do that in Coot:<br />
<br />
We have a file "cs.tab" like this, the residue number then the chemical shift value (one for each residue in a particular chain):<br />
<br />
1 1.53159<br />
<br />
2 4.35884<br />
<br />
3 4.07123<br />
<br />
4 4.16932<br />
<br />
5 6.69103<br />
<br />
6 7.12071<br />
<br />
7 10.7419<br />
<br />
8 9.57176<br />
<br />
Use apply-cs.scm to apply these values as pseudo temperature factors. Typical usage, where "A" is the chain id, and cs.tab the file of values per residue.<br />
<br />
(apply-cs (read-pdb "test.pdb") "A" "cs.tab")<br />
<br />
--script [[apply-cs.scm]]<br />
<br />
<br />
===Example Script 8: Partial Occupancy Dialog===<br />
<br />
Imagine that you have a structure that has residues with partial occupancy. After refinement, it would be convenient to quickly navigate to all such residues. How can that be done?<br />
<br />
Start coot with command line arguments:<br />
<br />
--script [[partial-occupancy-navigation.scm]]<br />
<br />
This will provide an extra menu item called "Extras", clicking on "Residues with low occupancy..." therein will lead you through the process.<br />
Note that this will often work with SHELXL molecules, because they have atoms with negative (e.g -31) occupancies.<br />
<br />
Note also that you will need a recent version of Coot to use this, as it stands. This will not work on stock Coot version 0.4.x. You can enable this for use with 0.4.x if you update/replace your xxx/share/coot/scheme/coot-gui.scm file from [http://coot.googlecode.com/svn/trunk/scheme/coot-gui.scm here].<br />
<br />
===Example Script 9: A GUI for Chopping Back Sidechains from a Residue Range===<br />
<br />
This is a simple interface to the delete-sidechain-range function, it illustrates how arguments can be transfered<br />
from the GUI to the scripting function. It was written in response to a question from Byron DeLaBarre.<br />
<br />
Unfortunately (prior to 0.5) there was an error in the standard delete-sidechain-range function, which is why we over-ride it.<br />
<br />
--script [[chop-side-chains-gui.scm]]<br />
<br />
===Example 10: How do I bind a key to Toggle the display of NCS ghosts?===<br />
<br />
With this script: [[toggle-ncs-ghosts-script]]<br />
<br />
===Example 11: Paul Emsley's Key Bindings===<br />
<br />
Just so you get an idea of the customization by key bindings here are what Paul uses currently (add to your .coot file).<br />
<br />
[[pauls-key-bindings-for-coot]]<br />
<br />
===Optional Wrappers and (External) Shell Script Enhancements===<br />
<br />
I (wgscott) wrote a [http://xanana.ucsc.edu/Library/init/zsh/local-functions/xtal/coot coot] wrapper shell script that lets you convert xplor/cns maps on the fly (you need to install [http://xray.bmc.uu.se/usf mapman] first) and has a few other enhancements.<br />
<br />
I also made a Coot OS X applet that allows you to drag and drop a cns/xplor or ccp4 mapfile or any other coot-compatable file (mtz or pdb file, for example). Using the File > Get Info dialog, you can program this applet to open all .map and all .mtz files, if you want to, making these files double-clickable.<br />
<br />
[http://xanana.ucsc.edu/xtal/osx_xtal_applets.dmg.gz Download the Applet] (requires a separate working coot installation)<br />
<br />
==Python Scripts==<br />
<br />
===Example 1: Bernhard Lohkamp's Key Bindings===<br />
<br />
Just so you get an idea of the customization by key bindings here are what Bernhard/Paul uses currently (add to your .coot file or put the file in .coot-preferences directory).<br />
<br />
[[bernhards_key_bindings_for_coot.py]]<br />
<br />
===Example 2: More key bindings (inspired by the Coot BB)===<br />
<br />
For (re-)colouring maps blue:<br />
<br />
[[blueify_map_keys.py]]<br />
<br />
To (re-)colour coordinate molecules yellow:<br />
<br />
[[yellowify_molecule_keys.py]]<br />
<br />
===Example 3: NCS Rotamer differences===<br />
<br />
To show NCS where NCS-related side-chains have different rotamers:<br />
<br />
[[ncs_rotamer_differences.py]]<br />
<br />
===Example 4: Morphing GUI===<br />
<br />
GUI to easily access jiggle fit and morphing (currently pre-release Coot required, may be moved into trunk):<br />
<br />
[[morph_residues_gui.py]]<br />
<br />
===Example 5: Ensemble GUI===<br />
<br />
GUI to allow navigation through structural ensembles as obtained e.g. from ensemble refinement:<br />
<br />
[[ensemble_plugin.py]]<br />
<br />
==Python to Scheme and return==<br />
<br />
===Translating between Python and Scheme===<br />
Python scripting is different to (default) scheme scripting which is mainly described in Paul Emsley's documentation (although it's mentioned somewhere, fairly hidden). You have to change the commands in the following way:<br />
<br />
GUILE scripting: (guile-command argument1 argument2 ...)<br />
<br />
PYTHON scripting: python_command(argument1, argument2, ...)<br />
<br />
====Simple rules for Scheme to Python translations====<br />
Here some simple rules how to translate from Scheme to Python. To translate the other way around, i.e. Python to Scheme, just turn the rules around:<br />
<br />
# Replace all '-' with '_' (except in equation when you need arithmetic '-' minus signs)<br />
# Move the brackets around the argument(s)<br />
# Separate multiple arguments by commas rather than spaces<br />
# Replace 'define' with 'def' for functions and with '=' for assignments<br />
# Make sure to use indentation for the function content [Python is indentation sensitive] and a ':' after the function definition.<br />
<br />
Some additional/advanced(?) rules:<br />
# #f -> False<br />
# #t -> True<br />
# (set! variable value) -> variable=value<br />
<br />
====A simple example====<br />
<br />
In Scheme we may have the following script:<br />
<br />
(define mol2-pdbFile "somePDBfile.pdb" )<br />
(define mol2-model (read-pdb mol2-pdbFile))<br />
(define (read-mol-again)<br />
(clear-and-update-model-molecule-from-file mol2-model mol2-pdbFile))<br />
(read-mol-again)<br />
<br />
Which will translate into Python:<br />
<br />
mol2_pdbFile = "somePDBfile.pdb"<br />
mol2_model = read_pdb(mol2_pdbFile)<br />
def read_mol_again():<br />
clear_and_update_model_molecule_from_file(mol2_model, mol2_pdbFile)<br />
read_mol_again()<br />
<br />
===Running a Scheme/Python command from Python/Scheme===<br />
<br />
As of Coot 0.5 (and if you have both scripting languages available) you an use the following commands to run a script or command in the other language:<br />
<br />
(run-python-command "python_command(arg1, arg2, ...)") [from guile/scheme]<br />
<br />
run_scheme_command("(scheme-command arg1 arg2 ...)") [from python]<br />
<br />
=Enhanced Menu Appearance=<br />
<br />
<br />
==As of 0.4, coot works with gtk+2==<br />
<br />
This permits use of themes for a more OSX-like experience, among other things.<br />
<br />
Click on the thumbnail image below to see a full-size screenshot of Coot with a gtk+2 Aqua-like theme.<br />
<br />
[[Image:Thumb-coot-gtk2.png]]<br />
<br />
To get this effect, you need the Glossy_P gtk+2 theme:<br />
<br />
http://art.gnome.org/download/themes/gtk2/571/GTK2-Glossy_P.tar.gz<br />
<br />
Edit a file called ~/.gtkrc-2.0 and put into it the following line:<br />
<br />
include "/usr/share/themes/Glossy\ P/gtk-2.0/gtkrc"<br />
<br />
Alternatively, if you use gnome or xfce4, you can open the theme manager and just make it open the downloaded Glossy_P tarball, and it should add this as a theme.<br />
<br />
=Assorted questions and answers (from the mailinglist)=<br />
<br />
It should be noted that the answers ("A") are from Paul Emsley himself (and were maybe slightly edited).<br />
<br />
==Coot development==<br />
<br />
Q: How can I get involved with Coot development?<br />
<br />
A: Join the [[Coot Janitors]] project. This is a project to get new people involved in improving Coot, by acting as a clearing house for simple tasks which need doing, and providing documentation for doing them.<br />
<br />
<br />
== Get rid of the "fix nomenclature" check ==<br />
Q: Is it possible to deactivate the nomenclature errors check? Sometimes this check is not very useful and it becomes rather annoying when one has several molecules loaded only wants to look at the structures...<br />
<br />
A: Add to your ~/.coot or whatever:<br />
(set-nomenclature-errors-on-read "ignore")<br />
<br />
<br />
==NCS edits== <br />
Q: I am sure this exists somewhere through scripting in COOT, but can I apply NCS edits to only a subset of NCS copies? In other words, can I tell coot which are NCS related chains, and which aren't. I am working on this nightmarish case of asymmetrical homodimers, where the sequences are very similar, but the structures are not, so I need to tell coot which chains are actually related to each other.<br />
<br />
A: Nightmare. If you have a recent [1632 or later for the scheme version, 1646 for the python version] Coot, you can do this:<br />
<br />
(manual-ncs-ghosts imol resno-start resno-end chain-id-list)<br />
<br />
where <br />
* imol is 0 (say)<br />
* resno-start and resno-end is the residue range for the LSQ fitting to return the NCS matrix,<br />
* chain-id-list is the list of chain-ids, starting with the master/reference chain-id and followed by the peer chain-ids that are NCS related, e.g. (list "A" "B" "D")<br />
<br />
The python interface is similar.<br />
<br />
There is also a GUI to activate this feature under Extensions -> NCS.<br />
<br />
==SHELXL==<br />
Description of problematic situation: I am using [[SHELXL]] to refine my 1.2 Å data and I am refining the hydrogen atoms. Subsequent rebuilding in coot is difficult though since hydrogens often does not "follow" when you do side chain rebuilding. For the moment I have quit transfering hydrogens to coot and add the hydrogens every refinement cycle, though it would be good I think if I could see them in coot without bothering about wrong positions. So these are my specific questions:<br />
<br />
Specific Q1: Using "edit chi angles" does not work properly.<br />
<br />
A: This fails because for chi angles Coot uses the Refmac dictionary to know what is connected to what (if it can). The work-around is to rename the refmac dictionary so Coot can't find it - which will force Coot to find bond by distance criteria.<br />
<br />
Specific Q2: Using "real space refine" does not work properly.<br />
<br />
A: Yes this fails. Hydrogens are named differently to SHELX hydrogens. In principal this could be made to work if the dictionary was reworked to use SHELX hydrogen names. This would also fix the chi angles problem too of course.<br />
<br />
Specific Q3: I am unable to open the output pdb file from ShelXL in Coot.<br />
<br />
A: Well, it's hard to know what's the problem without details - the console should say something. But when handling the output of shelxl, I suggest you read the .res file rather than the pdb, then the subsequent .ins file contains lots of "header" information.<br />
<br />
Another answer to questions 1+2 is to rename the hydrogen atoms in the shelxl res-file to match the mmCIF dictionaries used by Coot. This only needs to be done once as shelxl does not modify these names. Except for a few manual editions, the renaming can be done semi-automatically using regular expressions (replacing A->1, B->2, etc).<br />
<br />
Concerning question 3, the Coot -> Extensions -> Module -> SHELXL menu entry works really well now. It reads in all relevant shelxl files and provides a menu highlighting the problematic areas in the model.<br />
<br />
==Image quality on NVidia cards==<br />
Q: improvement of image quality on machines with NVidia cards?<br />
<br />
A: After talking about antialiasing with Stuart McNicholas, I discovered a program called nvidia-settings. I found that if I do:<br />
Antialiasing Settings -> Override Application Setting, and set the slider to 16x<br />
then I start Coot, I see that it starts up in nice antialiasing mode (which is a lot better that the poor mode that Coot has built-in).<br />
I don't know if this works on other/newer systems, but it works for me on my oldish GeForce 6600. (Of course the FPS takes a hit.) <br />
<br />
<br />
==Setting default to show symmetry-related molecules==<br />
Q: How to set the default to display symmetry related molecules?<br />
<br />
A: Add (set-show-symmetry-master 1) to the appropriate file. <br />
<br />
==Startup files==<br />
Q: I still have a ".coot" file in my home folder for a few coot preferences that I couldn't find in the new ".coot-preferences/coot-preferences.scm". There is a warning that I should not add commands to this file. So is a "~/.coot" still the proper place to add default commands for coot? <br />
<br />
A: Coot does not create a ~/.coot file for you, but will read it if it exists. Likewise, ~/.coot.py in which you can write python commands.~/.coot-preferences is a directory in which all .scm files and .py files are executed. coot-preferences.scm and coot_preferences.py there are generated by using the Edit -> Preferences dialog (and thus it overwrites older versions - hence the warning).<br />
<br />
If you want to create a script that will be read by everyone then put those files into a directory defined by environment variable $COOT_SCHEME_EXTRAS_DIR (for *.scm) or $COOT_PYTHON_EXTRAS_DIR (*.py). All *.scm in $COOT_SCHEME_EXTRAS_DIR and *.py files in $COOT_PYTHON_EXTRAS_DIR will be executed at start up.<br />
<br />
So you have a variety of places. Personally I mostly use ~/.coot. <br />
<br />
==Torsion general==<br />
Q: How do I use "torsion general"?<br />
<br />
A: Thanks for pointing out the lack of documentation on this. I'll make a note to add some.<br />
<br />
You need to click on the torsion-general icon, then click 4 atoms that describe the torsion - the first atom will be the base (non moving) part of the atom tree, on clicking the 4th atom a dialog will pop up with a "Reverse" button [1].<br />
<br />
Move this dialog out of the way and then left mouse click and drag in the main<br />
window will rotate the moving/"top" part of the residue round the clicked atoms<br />
2 and 3. When you are happy, click "Accept".<br />
<br />
Window focus may be an issue - depending on your setting, the window manager may<br />
eat one of your clicks as you change focus between the dialog and the main<br />
graphics window (this I find annoying and there are instructions in the FAQ on<br />
how to turn that off for various systems).<br />
<br />
[1] which may not work in 0.6-pre (grumble/sigh/sorry). If it doesn't not work,<br />
the "Reverse" button should invert the moving and "base" part of the residue.<br />
<br />
==Peak heights in maps==<br />
Q: I have some peaks in my map which take water or sodium/magnesium or chlorine atom with out giving out any positive or negative density upon further refinement. Is there any easy way of calculating the peak height / number of electrons at a given position, say a mouse click point in coot? Is there any formula to calculate the number of electrons based on sigma level and peak height, as given in difference map peaks in coot?<br />
<br />
A: First, go to the Coot wiki and pick up the [[Coot#Scheme_Scripts|scheme key bindings]]. <br />
<br />
If you want density information at a given cursor point: point at the blob, press the 'g' key (which recentres on the biggest density under the cursor).<br />
<br />
using the Scheme scripting window:<br />
(apply density-at-point (imol-refinement-map) (rotation-centre))<br />
There is no user access to the peak integration code of coot as yet.<br />
<br />
==Disulfide bond across crystallographic axis==<br />
Q: I have a pair of disulfide bonds which link two monomers in separate asymmetric units. There is a single monomer in the asymmetric unit, and two monomers come together to form disulfides between Cys 26-Cys45, and Cys45-Cys26. When I real-space-refine these residues, they do not form a nice disulfide, and Coot does not seem to recognize them as a disulfide. <br />
<br />
A: For the record, you can't refine symmetry-related disulfides in Coot (as of Nov 3, 2009).<br />
<br />
==Macros in COOT==<br />
<br />
Q: How to use macros in COOT? Do they need to be written in Python or another language that I had not heard of before? Where can I find a low level description of how to write macros with some examples (I know nothing about Python, except that it is fashionable)?<br />
<br />
A: The other language is a form of Lisp, called [http://en.wikipedia.org/wiki/Scheme_(programming_language) Scheme]. You can learn about programming python in many ways of course (not least the [http://docs.python.org/tutorial/ python tutorial], which is what I read first). The coot python extensions are described in the documentation. There is a standard trivial formatting change that has to be made to get the syntax right for python, see "Python Scripting" [[http://www.ysbl.york.ac.uk/~lohkamp/coot/wincoot-faq.html|here]]. There is a growing collection of coot scripts in this Wiki article.<br />
<br />
== building loops ==<br />
<br />
Q: Is there any similar function in COOT as lego_auto_mainchain command in O program?<br />
<br />
A: there are 2 loop fitting tools in Coot<br />
<br />
# C alpha -> Mainchain [http://www2.mrc-lmb.cam.ac.uk/personal/pemsley/coot/web/docs/coot.html#C_002dalpha-_002d_003e-Mainchain],[http://www2.mrc-lmb.cam.ac.uk/personal/pemsley/coot/web/docs/coot.html#Building-Links-and-Loops]<br />
# DB Loop: (No good documentation) [http://www2.mrc-lmb.cam.ac.uk/personal/pemsley/coot/web/docs/coot.html#protein_002ddb_002dloops] Extensions -> Modelling -> DB Loop...<br />
<br />
==LSQ superpositions==<br />
<br />
Q: Do an LSQ superposition using specified residues in multiple chains (superposing one oligomer on another).<br />
<br />
A: Something like this then? <br />
<br />
clear_lsq_matches()<br />
# specs for reference then moving<br />
add_lsq_match(20, 90, "A", 20, 90, "A", 1)<br />
add_lsq_match(20, 90, "B", 20, 90, "B", 1)<br />
add_lsq_match(15, 75, "D", 15, 75, "D", 1)<br />
apply_lsq_matches(1, 2)<br />
<br />
which presumes that the reference molecule is in 1 and the moving molecule 2.<br />
<br />
Q: How to do a LSQ superposition of a homologous structure onto my working structure using ± N residues about the current position, where N is a variable (not essential, could be fixed) and the current position is the last residue that I clicked on.<br />
<br />
A: That is more involved - and more useful because it can be dynamic. Something like the following perhaps (in Scheme, just for amusement (not tested)). You will need to set imol-ref, perhaps by reading in the reference pdb, as demonstrated below. The function is bound to Shift-Y.<br />
<br />
(define dynamic-lsq-range-extent 2) ;; ± 2 residues either side of centre residue<br />
(define imol-ref (read-pdb "reference.pdb"))<br />
<br />
;; convert between the input reference chain id and the chain id of<br />
;; the moving molecule that corresponds to that chain<br />
;;<br />
(define (mov-match-chain ref-chain-id)<br />
ref-chain-id)<br />
<br />
(define (dynamic-lsq-match)<br />
<br />
;; get the current residue and use that to make residue ranges for<br />
;; an LSQ fit<br />
;;<br />
<br />
(using-active-atom<br />
(clear-lsq-matches)<br />
(add-lsq-match (- aa-res-no dynamic-lsq-range-extent)<br />
(+ aa-res-no dynamic-lsq-range-extent)<br />
aa-chain-id<br />
(- aa-res-no dynamic-lsq-range-extent)<br />
(+ aa-res-no dynamic-lsq-range-extent)<br />
(mov-match-chain aa-chain-id)<br />
1)<br />
(apply-lsq-matches aa-imol imol-ref)))<br />
<br />
<br />
(add-key-binding "Dynamic LSQ overlay" "Y" dynamic-lsq-match)<br />
<br />
== reading MTZ file with experimental PHI and FOM using --auto ==<br />
<br />
Q: There is the --auto <filename> commandline option for auto-reading mtz files (mtz file has the default labels FWT, PHWT). Can this be made to work with a SHELXE .phs output file after converting with convert2mtz ? - the resulting MTZ file has labels F PHI FOM.<br />
<br />
A: use: coot --python -c 'make_and_draw_map("sad.mtz", "F", "PHI", "FOM", "/HKL_base/HKL_base/FOM",1, 0)'<br />
<br />
== NCS Rotamer differences ==<br />
<br />
Show me where NCS-related side-chains have different rotamers<br />
<br />
<br />
(define (compare-ncs-rotamer imol chain-A chain-B)<br />
(let ((n-residues (chain-n-residues chain-A imol))<br />
(mismatched-rotamers '()))<br />
(for-each <br />
(lambda (serial-number)<br />
<br />
(let ((res-name-A (resname-from-serial-number imol chain-A serial-number))<br />
(res-no-A (seqnum-from-serial-number imol chain-A serial-number))<br />
(ins-code-A (insertion-code-from-serial-number imol chain-A serial-number))<br />
(res-name-B (resname-from-serial-number imol chain-A serial-number))<br />
(res-no-B (seqnum-from-serial-number imol chain-A serial-number))<br />
(ins-code-B (insertion-code-from-serial-number imol chain-A serial-number)))<br />
(if (not (= res-no-A res-no-B))<br />
(begin<br />
(format #t "sequence number for ~s do not match~%" res-no-A))<br />
(if (not (string=? res-name-A res-name-B))<br />
(begin<br />
(format #t "residue names for ~s do not match~%" res-no-A))<br />
(let ((rot-name-A (get-rotamer-name imol chain-A res-no-A ins-code-A))<br />
(rot-name-B (get-rotamer-name imol chain-B res-no-B ins-code-B)))<br />
(if (not (string=? rot-name-A rot-name-B))<br />
(begin<br />
(set! mismatched-rotamers <br />
(cons (list imol chain-A res-no-A ins-code-A <br />
res-name-A <br />
(if (string=? rot-name-A "") "-" rot-name-A)<br />
(if (string=? rot-name-B "") "-" rot-name-B))<br />
mismatched-rotamers))))<br />
)))))<br />
(range n-residues))<br />
(dialog-box-of-buttons "Mismatched Rotamers"<br />
(cons 300 300)<br />
(map (lambda(rotamer) <br />
(let ((label (string-append " " <br />
(list-ref rotamer 1)<br />
" "<br />
(number->string (list-ref rotamer 2))<br />
(list-ref rotamer 3) <br />
" "<br />
(list-ref rotamer 4) ;; res-name<br />
": "<br />
(list-ref rotamer 5)<br />
" vs. "<br />
(list-ref rotamer 6)))<br />
(thunk (lambda ()<br />
(set-go-to-atom-molecule imol)<br />
(set-go-to-atom-chain-residue-atom-name <br />
(list-ref rotamer 1) <br />
(list-ref rotamer 2) " CA "))))<br />
(list label thunk)))<br />
mismatched-rotamers)<br />
" Close ")))<br />
<br />
<br />
<br />
And one would use this something like:<br />
<br />
;; example usage:<br />
(let ((imol (read-pdb "test.pdb")))<br />
(compare-ncs-rotamer imol "A" "B"))<br />
<br />
== make RSR in coot 0.8.1 behave like in earlier versions ==<br />
<br />
Q: We've noticed a new behavior in real space refinement in coot 0.8.1 whereby dragged atoms are more tightly restrained to their initial positions than in earlier versions. This seems to be described in the release notes by:<br />
<br />
o BUG-FIX: The amount that the other atoms ove with moving the picked atom has been reduced (but is configurable) <br />
<br />
A: Add e.g. this to your ~/.coot.py file:<br />
<br />
set_refinement_drag_elasticity(0.8)<br />
<br />
Q: I'm wondering why this was changed. Does the optimum elasticity change with resolution, map quality, or another experimental limitation? Or does it more of a user preference?<br />
<br />
A: Because of cis-peptides. My worry was that in the previous regime, it was <br />
too easy to introduce cis-peptides when fitting to low resolution maps. <br />
I believe the current default setting is much less likely to do that.<br />
<br />
Q: I've tried various settings of refinement_drag_elasticity and I need to lower it to 0.5 or so before any semblance of earlier behavior appears.<br />
<br />
A: It used to be 0.167, I think.<br />
<br />
== Molprobity not active in COOT ==<br />
<br />
Q: I am using COOT 0.8.1 EL that comes with the CCP4 6.5.010 on my Mac OS X 10.10.2. I wanted to run molprobity but the Validate > Probe clashes button in my pull down menu is not active. Is this function available in this COOT version?<br />
<br />
A: Reduce and probe are separate programs available from the Richardson’s lab at Duke http://kinemage.biochem.duke.edu/. Download and install on your box. Then coot needs to be told in some instances where it can find these executables. I have the following lines in my ~/.coot file in Linux. <br />
<br />
<pre><br />
;; .coot<br />
;; This file is required. As of coot 0.8pre no other mechanism for<br />
;; enabling probe in coot works<br />
;;<br />
;; This is full pathname of molprobity's probe program<br />
(define *probe-command* "/apps/xray/bin/probe")<br />
;; This is full pathname of molprobity's reduce program<br />
(define *reduce-command* "/apps/xray/bin/reduce")<br />
</pre><br />
Untried: if you have Phenix installed: it comes with phenix.probe and phenix.reduce - you could insert the paths to these binaries into the above definitions.<br />
<br />
== some symmetry mates not shown ==<br />
<br />
Q: This structure has been solved and refined using phenix in the hexagonal setting of space group R 3. There is one copy per asymmetric unit in R 3. As you can see from the attached image, coot is rendering some but not all of the symmetry mates.<br />
<br />
A: Turn up the radius a bit and use (set-symmetry-shift-search-size 3) . I would have thought that 2 is big enough, but maybe not in this case.</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Test&diff=2646Test2019-01-18T14:11:15Z<p>Karsten: </p>
<hr />
<div>hallo<br />
<br />
We test tex:<br />
<br />
<math><br />
F_{hkl} = \sum_{i=1}^{N} e^{-2\pi\imath\left( h\frac{x}{a}+k\frac{y}{b}+l\frac{z}{c}\right)}<br />
</math><br />
<br />
[[Image:Acrb.jpg|thumb|test]]<br />
<br />
All crystals should look like this (or better)<br />
[[Image:xtals.jpg||pretty xtals]]<br />
<br />
Problem? - I tried uploading xtals.png but Wiki does not let me upload, quoting that 'wrong file type or extension' - yet it allowed me to load up the same image in jpg format. Png is a nice (some would even say the best) format so it'd be good to be able to use it! -AGE<br />
This is fixed now. --[[User:Kay|Kay]] 10:12, 5 February 2008 (CET)<br />
<br/><br />
Remark: <br />
MIME type of PNG files was not correctly identified. To fix this, we added the lines<br />
# PNG<br />
1 string PNG image/png<br />
to the file /etc/httpd/conf/magic<br />
<br />
Logo prototype #1<br />
[[Image:logo1.png]]<br />
<br />
Logo prototype #2<br />
[[Image:logo2.png]]<br />
<br />
do you like any of these?<br />
<br />
Testing the gnuplot extension (the plot is generated in realtime, not a bitmap)<br />
<gnuplot><br />
set hidden3d<br />
set isosamples 50<br />
splot sin(x)*cos(y)<br />
</gnuplot></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Stereo&diff=2636Stereo2018-11-30T10:38:18Z<p>Karsten: </p>
<hr />
<div>== Hardware ==<br />
<br />
=== Zalman Stereo ===<br />
<br />
has its own [[Zalman Stereo|article]].<br />
<br />
=== Nvidia 3D Vision 2 ===<br />
# '''3D Vision Pro''' is ''not'' what you want, this is for professional CAD/CAM applications, and is expensive.<br />
# The quality of Nvidia 3D Vision 2 is better than that of Zalman, because Zalman stereo means halved vertical resolution.<br />
# The Nvidia 3D vision emitter must be connected to both USB and Quadro 3-pin DIN Stereo socket. This setup works on Windows and nowadays Linux (consult this [http://cismm.cs.unc.edu/core-projects/visualization-and-analysis/setting-up-a-simple-stereo-system/ link], and [http://us.download.nvidia.com/XFree86/Linux-x86_64/346.35/README/xconfigoptions.html search this link for the word "USB"]. The cheapest (2018+) Nvidia Quadro with 3-pin DIN Stereo connector is the P4000 (http://www.nvidia.de/object/quadro-desktop-gpu-specs-de.html) which starts at ~ €780; cheaper and older ones can be found e.g. on EBay; they may require one of the [http://www.nvidia.com/object/unix.html long-term Legacy (Long Lived) Linux drivers]. For specific products, see also [http://www.nvidia.com/Download/index.aspx] .<br />
# For Linux, a Nvidia 3D vision emitter "workaround" requires the DIN 3-pin connector found on the high end Quadros and [http://www.nuvision3d.com/the60gx.html NuVision] or [http://www.reald-corporate.com/scientific CrystalEyes] stereo glasses and emitter. The 3-pin DIN cable is difficult to find if you do not have one, and it is recommended that you splice your own. This author purchased a 3-pin mini-DIN connector here: [http://www.vetco.net/catalog/product_info.php?products_id=6588], spliced it onto a 2.5mm stereo audio cable like from here: [http://www.vetco.net/catalog/product_info.php?products_id=6952], using information adapted from this page explaining how to adapt connections for the older VESA standard for stereo here: [http://www.stereo3d.com/vesa3.htm] (it works quite well). The Figures on [https://forums.geforce.com/default/topic/411219/3d-vision/3d-vision-vesa-cable/7/] or [https://forums.geforce.com/default/topic/513899/3d-vision/link-nvidia-3d-vision-kit-with-dlp-display-projector-connection-problem-with-vesa-stereo-cable-ir-tr/] show details on connecting the plugs. Note that the 5V power line does ''not'' need to be connected as the receiver is powered by USB. <br />
# The most affordable NVIDIA 3D Vision solution on Linux used to be a monitor with built-in IR emitter (for example BenQ XL2420TX or ASUS VG278HR), and a cheap Quadro, e.g. the K420 or FX380. The latter has a Dual-Link DVI (DVI-D) and a Displayport outlet, so can drive the stereo monitor, and an additional monitor. This solution avoids the USB/3-pin hassle altogether. See below for xorg.conf! GeForce cards (instead of Quadro) ''do not give openGL Quad Buffered Stereo'' on Linux (on Windows neither). Unfortunately, monitors with built-in IR emitter are seemingly no longer produced (2017).<br />
# [http://www.nvidia.com/object/3d-vision-displays.html The Nvidia page that names monitors with built-in emitter] also has not changed for years. http://geizhals.eu/?cat=monlcd19wide now has a "inkl. 3D-emitter" attribute. This currently only returns the Asus 278HR which can only be bought in Poland, or through EBay.<br />
# Cheap Quadros (e.g. K420) with DVI-I Dual-Link Connector (DVI-I DL / DVI-D) or DisplayPort work well. Make sure the card can do dual-link DVI if your monitor has only DVI-D input. Any card (including the "Windows only" ones!) [http://www.nvidia.com/object/3d-vision-pro-requirements.html#Quadro listed] should work if a) it can do dual-link DVI if the monitor has only DVI-D input, and b) if the monitor has built-in emitter. The DisplayPort 1.2 (Quadro) to DisplayPort 1.2 (monitor) connection works well (e.g. with the BenQ XL2420TX).<br />
# To connect a DVI-D monitor to ar Quadro with DisplayPort 1.2, you will need a Club-3D CAC-1051 active DisplayPort/Dual-Link DVI Adapter 330MHz (110 €: do not buy the cheaper 270 MHz model, it will not work because the bandwidth is too low for 1920x1080@120hz) or for cards with miniDisplay ports, the Club-3D CAC-1151 active miniDisplayPort/Dual-Link DVI Adapter 330MHz.<br />
# specs of latest ("Maxwell") and previous ("Kepler") generation of Quadro cards are at http://www.nvidia.com/object/quadro-desktop-gpus.html . A comparison of all PCIexpress Quadro cards is at http://en.wikipedia.org/wiki/Nvidia_Quadro#PCI_Express . Latest (2014+) with prices are at http://www.heise.de/preisvergleich/?cat=gra16_512&asd=on&asuch=quadro&xf=3312_2014&sort=p . Currently (2015), the K620 is the entry system with the best price/performance ratio; in the middle range the K2200 still seems affordable.<br />
<br />
== Software ==<br />
=== Linux ===<br />
[[Zalman Stereo]] is supported by [[Coot]]; no drivers or other software must be installed.<br />
<br />
NVidia 3D Vision and 3D Vision 2: Requires the appropriate hardware as detailed above. With the NVidia proprietary graphics drivers installed, 3D Vision will work with a few minor tweaks. The Xorg configuration file (usually located at /etc/X11/xorg.conf) must be adapted.<br />
# It can be edited -- see [http://us.download.nvidia.com/XFree86/Linux-x86_64/346.35/README/xconfigoptions.html] for more information), or <br />
# <code>nvidia-settings</code> could be run, or<br />
# <code>nvidia-xconfig --stereo=10</code> (The value of 10 is specific to the 3D Vision system)<br />
<br />
The resulting xorg.conf has an option for stereo mode under the section labeled "Device" in this minimal example:<br />
<pre><br />
Section "Device"<br />
Identifier "Device0"<br />
Driver "nvidia"<br />
VendorName "NVIDIA Corporation"<br />
Option "Stereo" "10"<br />
EndSection<br />
</pre><br />
The following is a longer example, for a dual-monitor setup - this was generated with <code>nvidia-settings</code> and has the stereo option in the "Screen" section::<br />
<br />
<pre><br />
Section "Monitor"<br />
Identifier "Monitor0"<br />
VendorName "Unknown"<br />
ModelName "BenQ XL2420TX"<br />
HorizSync 30.0 - 140.0<br />
VertRefresh 56.0 - 120.0<br />
EndSection<br />
<br />
Section "Device"<br />
Identifier "Videocard0"<br />
Driver "nvidia"<br />
EndSection<br />
<br />
Section "Device"<br />
Identifier "Device0"<br />
Driver "nvidia"<br />
VendorName "NVIDIA Corporation"<br />
BoardName "Quadro K420"<br />
EndSection<br />
<br />
Section "Screen"<br />
Identifier "Default Screen"<br />
Device "Videocard0"<br />
EndSection<br />
<br />
Section "Screen"<br />
Identifier "Screen0"<br />
Device "Device0"<br />
Monitor "Monitor0"<br />
DefaultDepth 24<br />
Option "Stereo" "10"<br />
Option "nvidiaXineramaInfoOrder" "DFP-0"<br />
Option "metamodes" "DVI-I-1: 1920x1080_120 +0+0, DP-1: nvidia-auto-select<br />
+1920+0"<br />
Option "SLI" "Off"<br />
Option "MultiGPU" "Off"<br />
Option "BaseMosaic" "off"<br />
SubSection "Display"<br />
Depth 24<br />
EndSubSection<br />
EndSection<br />
</pre><br />
<br />
This should be enough to get stereo working. However, note that there is a major issue on linux with window compositing causing stereo images to display improperly. This is a problem with the window display manager, and is not inherent to Xorg or NVidia. If you try to turn on stereo in Coot in a display manager that makes use of the Xorg compositing extension (e.g., Gnome3, or Unity in Ubuntu) then what you will see when you activate hardware stereo is a slight rotation of the view, but stereo remains disabled. In order to get around this problem, you must use a display manager that does not make use of compositing as part of its eye candy. This author has found the MATE desktop to work quite well for this purpose (and may in fact be one of the few that still does not use software compositing by default). Note that you ''do not'' need to disable the Compositing extension in the Xorg configuration file to make this work -- this will allow you to switch back to Gnome3, Unity, etc when you don't need stereo if you prefer! Not disabling the window compositing extension globally allows for a more flexible setup depending on your preferred workflow.<br />
<br />
===Mac OS X===<br />
The following command needs to be run for Macs to be able to support stereo in X11 programs, such as Coot [http://xanana.ucsc.edu/~wgscott/xtal/wiki/index.php/Installing_Coot_on_OS_X] :<br />
defaults write com.apple.x11 enable_stereo -bool true<br />
<br />
===Windows===<br />
There's some information at http://bernhardcl.github.io/coot/wincoot-faq.html#mozTocId910462 . This says:<br />
<br />
'''WinCoot with NVIDIA and Win 10?'''<br />
<br />
If you run into problems with stereo on Windows 10 (using an NVIDA card), you can try the following (thanks to Jose Brito):<br />
# Open NVIDIA Control Panel --> Manage 3D settings.<br />
# Go to Program Settings tab.<br />
# Add the file you launch the program from by clicking on Add button.<br />
# In the setting window, change the 'Power Management Mode' to 'Prefer Maximum Performance'. Also below settings as well:<br />
# Vertical sync - off<br />
# Threaded optimization - off<br />
# Triple buffering - off<br />
# Stereo enable on<br />
# Run the program and check<br />
<br />
==Ono==<br />
You also need to set the environment variable STEREO for the stereo to work properly in ono:<br />
setenv STEREO on (tcsh)<br />
STEREO = on; export STEREO (bash)<br />
[http://dbb.urmc.rochester.edu/labs/wedekind/Wedekind-Lab/X-ray%20Lab/Methods/Nvidia-ONO.html]<br />
<br />
== Stereo on conventional CRT monitors ==<br />
<br />
Some of the NVidia Quadro cards support stereo. The cards that have an output called "stereo" under "Display Connectors" listed at [http://www.nvidia.com/object/IO_11761.html Nvidia's Quadro overview page] have a 3-pin DIN outlet that fits with [http://www.nuvision3d.com/the60gx.html NuVision] or [http://www.reald-corporate.com/scientific CrystalEyes] stereo glasses.<br />
<br />
The cheapest of these used to be the FX1400 (difficult to find these days, around 450 €), but now appears to be the FX3450 (around 750 €). These cards are by far fast enough for protein crystallography or modelling.<br />
<br />
For stereo, the xorg.conf might need the following lines<br />
Section "Extensions"<br />
Option "Composite" "Disable"<br />
EndSection<br />
if the X log file (e.g. at /var/log/Xorg.0.log) says that stereo is not supported by composite.<br />
<br />
Another option that will be required in xorg.conf by programs running stereo is<br />
<br />
Section "Device"<br />
Driver "nvidia"<br />
Option "Stereo" "3"<br />
End Section<br />
<br />
For an example of how else to configure xorg.conf, see old versions of this article.<br />
<br />
==See also==<br />
Stereo on TFT: see [[Zalman Stereo]]<br />
<br />
http://pymolwiki.org/index.php/Stereo_3D_Display_Options</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=R-factors&diff=2527R-factors2018-01-26T14:05:35Z<p>Karsten: /* Definitions */</p>
<hr />
<div>== Definitions ==<br />
=== Data quality indicators ===<br />
In the following, all sums over hkl extend only over unique reflections with more than one observation!<br />
* R<sub>sym</sub> and R<sub>merge</sub> - the formula for both is:<br />
<br />
<br />
: <math><br />
R = \frac{\sum_{hkl} \sum_{j} \vert I_{hkl,j}-\langle I_{hkl}\rangle\vert}{\sum_{hkl} \sum_{j}I_{hkl,j}}<br />
</math><br />
<br />
<br />
where <math>\langle I_{hkl}\rangle</math> is the average of symmetry- (or Friedel-) related observations of a unique reflection. The formula is due to Arndt, U.W., Crowther, R.A. & Mallet, J.F.W. A computer-linked cathode ray tube microdensitometer for X-ray crystallography. J. Phys. E:Sci. Instr. 1, 510−516 (1968). Any unique reflection with n=2 or more observations enters the sums.<br />
<br />
It can be shown that this formula results in higher R-factors when the redundancy is higher (Diederichs and Karplus <ref name="DiKa97">K. Diederichs and P.A. Karplus (1997). Improved R-factors for diffraction data analysis in macromolecular crystallography. Nature Struct. Biol. 4, 269-275 [http://strucbio.biologie.uni-konstanz.de/strucbio/files/nsb-1997.pdf]</ref>). In other words, low-redundancy datasets appear better than high-redundancy ones, which obviously violates the intention of having an indicator of data quality!<br />
* Redundancy-independant version of the above: <br />
<br />
<br />
: <math><br />
R_{meas} = \frac{\sum_{hkl} \sqrt \frac{n}{n-1} \sum_{j=1}^{n} \vert I_{hkl,j}-\langle I_{hkl}\rangle\vert}{\sum_{hkl} \sum_{j}I_{hkl,j}}<br />
</math><br />
<br />
<br />
which unfortunately results in higher (but more realistic) numerical values than R<sub>sym</sub> / R<sub>merge</sub> <br />
(Diederichs and Karplus <ref name="DiKa97"/> , <br />
Weiss and Hilgenfeld <ref name="WeHi97">M.S. Weiss and R. Hilgenfeld (1997) On the use of the merging R-factor as a quality indicator for X-ray data. J. Appl. Crystallogr. 30, 203-205[http://dx.doi.org/10.1107/S0021889897003907]</ref>).<br />
<br />
==== measuring precision of averaged intensities/amplitudes ====<br />
<br />
for intensities use <br />
(Weiss <ref name="We01">M.S. Weiss. Global indicators of X-ray data quality. J. Appl. Cryst. (2001). 34, 130-135 [http://dx.doi.org/10.1107/S0021889800018227]</ref>)<br />
<br />
<br />
: <math><br />
R_{p.i.m.} = \frac{\sum_{hkl} \sqrt \frac{1}{n-1} \sum_{j=1}^{n} \vert I_{hkl,j}-\langle I_{hkl}\rangle\vert}{\sum_{hkl} \sum_{j}I_{hkl,j}}<br />
</math><br />
<br />
R<sub>mrgd-I</sub> (defined in Diederichs and Karplus <ref name="DiKa97"/>) only differs by a factor (FIXME: what is the factor? 0.5 or 1.4142 or ?) since it likewise takes the improvement in precision from multiplicity into account. R<sub>split</sub> , which is what the X-FEL community uses, is the same as R<sub>mrgd-I</sub> but that community seems not to be aware of this. <br />
<br />
Similarly, one should use R<sub>mrgd-F</sub> as a quality indicator for amplitudes <ref name="DiKa97"/>, which may be calculated as: <br />
<br />
<br />
: <math><br />
R_{mrgd-F} = \frac{\sum_{hkl} \sqrt \frac{1}{n-1} \sum_{j=1}^{n} \vert F_{hkl,j}-\langle F_{hkl}\rangle\vert}{\sum_{hkl} \sum_{j}F_{hkl,j}}<br />
</math><br />
<br />
<br />
with <math>\langle F_{hkl}\rangle</math> defined analogously as <math>\langle I_{hkl}\rangle</math>.<br />
<br />
In the sums above, the summation omits those reflections with just one observation.<br />
<br />
==== measuring radiation damage ====<br />
<br />
We can plot (Diederichs <ref name="Di06">K. Diederichs (2006). Some aspects of quantitative analysis and correction of radiation damage. Acta Cryst D62, 96-101 [http://strucbio.biologie.uni-konstanz.de/strucbio/files/Diederichs_ActaD62_96.pdf]</ref>)<br />
<br />
<br />
: <math><br />
R_{d} = \frac{\sum_{hkl} \sum_{|i-j|=d} \vert I_{hkl,i} - I_{hkl,j}\vert}{\sum_{hkl} \sum_{|i-j|=d} (I_{hkl,i} + I_{hkl,j})/2}<br />
</math><br />
<br />
<br />
which gives us the average R-factor of two reflections measured d frames apart. As long as the plot is parallel to the x axis there is no radiation damage. As soon as the plot starts to rise, we see that there's a systematical error contribution due to radiation damage.<br />
<br />
Strong wiggles at very high d are irrelevant as only few reflections contribute.<br />
<br />
To my knowledge, the only program that implements this currently (December 2008) is [[xds:XDSSTAT|XDSSTAT]].<br />
<br />
=== Comparing two sets of structure factor amplitudes or intensities ===<br />
The following is symmetric, and suitable for comparing two data sets, or two model amplitudes:<br />
<br />
<br />
: <math><br />
R_{scale}=\frac{\sum_{hkl}\vert F_{hkl,i}-F_{hkl,j}\vert}{0.5\sum_{hkl} F_{hkl,i}+F_{hkl,j}}<br />
</math><br />
<br />
<br />
for amplitudes, and analogously for intensities.<br />
<br />
=== Model quality indicators ===<br />
* R and [[iucr:Free_R_factor|R<sub>free</sub>]] : the formula for both is <br />
<br />
<br />
: <math><br />
R=\frac{\sum_{hkl}\vert F_{hkl}^{obs}-F_{hkl}^{calc}\vert}{\sum_{hkl} F_{hkl}^{obs}}<br />
</math><br />
<br />
<br />
where <math>F_{hkl}^{obs}</math> and <math>F_{hkl}^{calc}</math> have to be scaled w.r.t. each other. R and R<sub>free</sub> differ in the set of reflections they are calculated from: R is calculated for the [[working set]], whereas R<sub>free</sub> is calculated for the [[test set]].<br />
<br />
==== Relation between R and R<sub>free</sub> as a function of resolution ====<br />
<br />
* The PDBe provide a service to plot many different statistical properties in the PDB against other properties. The link is http://www.ebi.ac.uk/pdbe-as/pdbestatistics/PDBeStatistics.jsp . You can see that there is an option of RDiff which is the difference between R and R-Free for all structures that contain both data. Take a look at this first. There is a second parameter which you can set to resolution and this will allow you to draw a plot that you want. This will draw a 3D isometric plot which you can scale, and pick data points to view particular entries.<br />
* formula from Kleywegt and Jones (2002): R<sub>free</sub> = 1.065*R + 0.036<br />
* plot with empirical data: http://xray.bmc.uu.se/gerard/supmat/rfree2000/rfminusr_vs_resolution.gif<br />
* many more plots: http://xray.bmc.uu.se/gerard/supmat/rfree2000<br />
* harry plotter (java): http://xray.bmc.uu.se/gerard/supmat/rfree2000/plotter.html<br />
* When the resolution is plotted on a logarithmic scale, the most frequent values (modes) are practically linear functions allowing their easy interpolation / extrapolation as (Urzhumtsev et al, 2009)<br />
mode(R) = 0.091*ln(resolution) + 0.134<br />
mode(Rfree-R) = 0.024*ln(resolution) + 0.020<br />
<br />
References:<br />
* Tickle IJ, Laskowski RA and Moss DS. Rfree and the Rfree Ratio. I. Derivation of Expected Values of Cross-Validation Residuals Used in Macromolecular Least-Squares Refinement. Acta Cryst. (1998). D54, 547-557 [http://dx.doi.org/10.1107/S0907444997013875] <br />
<br />
* Tickle IJ, Laskowski RA and Moss DS. Rfree and the Rfree ratio. II. Calculation of the expected values and variances of cross-validation statistics in macromolecular least-squares refinement. Acta Cryst. (2000). D56, 442-450 [http://dx.doi.org/10.1107/S0907444999016868]<br />
<br />
* GJ Kleywegt and TA Jones (2002). Homo Crystallographicus - Quo vadis? Structure 10, 465-472. (reprint from http://xray.bmc.uu.se/cgi-bin/gerard/reprint_mailer.pl?pref=65)<br />
<br />
* Urzhumtsev, Afonine & Adams (2009) Acta Cryst., D65, 1283-1291.<br />
<br />
== what kinds of problems exist with these indicators? ==<br />
* (R<sub>sym</sub> / R<sub>merge</sub> ) should not be used to judge [[data quality]], R<sub>meas</sub> should be used instead. The reason is that the former depend on multiplicity, whereas the latter doesn't.<br />
<br />
* R/R<sub>free</sub> and NCS: reflections in work and test set are not independent if chosen randomly. It is better to choose the test set reflections in thin resolution shells. Since the twin related reflections have the same sin(theta)/lambda values they will not be split over the working and reference sets. [http://xray.bmc.uu.se/usf/dataman_man.html DATAMAN] from the [http://xray.bmc.uu.se/usf/ Uppsala Software Factory] and [[XPREP]] (a program which may be obtained from Bruker) offer this option. The "RFREE SHELL" command in sftools is another way to select thin shells. A disadvantage is the the maps may not be quite as good as when the free R reflections are selected randomly. (FIXME: which [[Phenix]] program does this?). A paper investigating this thoroughly is Fabiola, F., A. Korostelev, et al. (2006). "Bias in cross-validated free R factors: mitigation of the effects of non-crystallographic symmetry." Acta Cryst. D 62: 227-38. <br />
<br />
* Sets of reflections used for calculating R<sub>free</sub> should be maintained throughout a project. This is nicely discussed at http://www.bmsc.washington.edu/people/merritt/xplor/rfree_example.html . Note that none of the programs mentioned for selecting thin shells will allow you to extend the set of shells to higher resolution if you want to preserve your existing R-free set.<br />
<br />
* R-values and twinning: [http://www.ysbl.york.ac.uk/refmac/papers/Rfactor.pdf Garib N. Murshudov (2011) "Some properties of crystallographic reliability index - Rfactor: effect of twinning" Appl. Comput. Math., V.10, N.2, 2011, pp.250-261]. From the paper, the R-value table for random models is:<br />
twinning twinning not<br />
modelled modelled<br />
twin 0.41 0.49<br />
normal 0.52 0.58<br />
Another paper which investigates the properties of R-values in the presence of twinning is [http://journals.iucr.org/d/issues/2013/07/00/ba5190/index.html P. R. Evans and G. N. Murshudov (2013) "How good are my data and what is the resolution?" Acta Cryst. (2013). D69, 1204-1214]. As the title indicates, this paper discusses at what resolution the data should be cut. One important finding is that a perfect model gives an R value of 42.0% (for a perfect twin, 29.1%) against pure noise. This tells us that a model that gives significantly lower R<sub>free</sub> in the (current) high resolution shell may benefit from including higher resolution data.<br />
<br />
* R-values and [[pseudo-translation]]: if you have pseudotranslation you should be aware that if you solve the structure by molecular replacement, starting R factors could be 70-80%.<br />
<br />
* data R-values are not meaningful at high resolution. This is discussed by [http://strucbio.biologie.uni-konstanz.de//strucbio/files/karplus2012_science.pdf Karplus and Diederichs (2012) "Linking crystallographic data and model quality". ''Science'' '''336''', 1030]<br />
<br />
==Notes==<br />
<references/></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=R-factors&diff=2526R-factors2018-01-26T14:03:53Z<p>Karsten: /* Data quality indicators */</p>
<hr />
<div>== Definitions ==<br />
=== Data quality indicators ===<br />
In the following, all sums over hkl extend only over unique reflections with more than one observation!<br />
* R<sub>sym</sub> and R<sub>merge</sub> - the formula for both is:<br />
<br />
<br />
: <math><br />
R = \frac{\sum_{hkl} \sum_{j} \vert I_{hkl,j}-\langle I_{hkl}\rangle\vert}{\sum_{hkl} \sum_{j}I_{hkl,j}}<br />
</math><br />
<br />
<br />
where <math>\langle I_{hkl}\rangle</math> is the average of symmetry- (or Friedel-) related observations of a unique reflection. The formula is due to Arndt, U.W., Crowther, R.A. & Mallet, J.F.W. A computer-linked cathode ray tube microdensitometer for X-ray crystallography. J. Phys. E:Sci. Instr. 1, 510−516 (1968). Any unique reflection with n=2 or more observations enters the sums.<br />
<br />
It can be shown that this formula results in higher R-factors when the redundancy is higher (Diederichs and Karplus <ref name="DiKa97">K. Diederichs and P.A. Karplus (1997). Improved R-factors for diffraction data analysis in macromolecular crystallography. Nature Struct. Biol. 4, 269-275 [http://strucbio.biologie.uni-konstanz.de/strucbio/files/nsb-1997.pdf]</ref>). In other words, low-redundancy datasets appear better than high-redundancy ones, which obviously violates the intention of having an indicator of data quality!<br />
* Redundancy-independant version of the above: <br />
<br />
<br />
: <math><br />
R_{meas} = \frac{\sum_{hkl} \sqrt \frac{n}{n-1} \sum_{j=1}^{n} \vert I_{hkl,j}-\langle I_{hkl}\rangle\vert}{\sum_{hkl} \sum_{j}I_{hkl,j}}<br />
</math><br />
<br />
<br />
which unfortunately results in higher (but more realistic) numerical values than R<sub>sym</sub> / R<sub>merge</sub> <br />
(Diederichs and Karplus <ref name="DiKa97"/> , <br />
Weiss and Hilgenfeld <ref name="WeHi97">M.S. Weiss and R. Hilgenfeld (1997) On the use of the merging R-factor as a quality indicator for X-ray data. J. Appl. Crystallogr. 30, 203-205[http://dx.doi.org/10.1107/S0021889897003907]</ref>).<br />
<br />
==== measuring precision of averaged intensities/amplitudes ====<br />
<br />
for intensities use <br />
(Weiss <ref name="We01">M.S. Weiss. Global indicators of X-ray data quality. J. Appl. Cryst. (2001). 34, 130-135 [http://dx.doi.org/10.1107/S0021889800018227]</ref>)<br />
<br />
<br />
: <math><br />
R_{p.i.m.} = \frac{\sum_{hkl} \sqrt \frac{1}{n-1} \sum_{j=1}^{n} \vert I_{hkl,j}-\langle I_{hkl}\rangle\vert}{\sum_{hkl} \sum_{j}I_{hkl,j}}<br />
</math><br />
<br />
R<sub>mrgd-I</sub> (defined in Diederichs and Karplus <ref name="DiKa97"/>) only differs by a factor (FIXME: what is the factor? 0.5 or 1.4142 or ?) since it likewise takes the improvement in precision from multiplicity into account. R<sub>split</sub> , which is what the X-FEL community uses, is the same as R<sub>mrgd-I</sub> but that community seems not to be aware of this. <br />
<br />
Similarly, one should use R<sub>mrgd-F</sub> as a quality indicator for amplitudes <ref name="DiKa97"/>, which may be calculated as: <br />
<br />
<br />
: <math><br />
R_{mrgd-F} = \frac{\sum_{hkl} \sqrt \frac{1}{n-1} \sum_{j=1}^{n} \vert F_{hkl,j}-\langle F_{hkl}\rangle\vert}{\sum_{hkl} \sum_{j}F_{hkl,j}}<br />
</math><br />
<br />
<br />
with <math>\langle F_{hkl}\rangle</math> defined analogously as <math>\langle I_{hkl}\rangle</math>.<br />
<br />
In the sums above, the summation omits those reflections with just one observation.<br />
<br />
==== measuring radiation damage ====<br />
<br />
We can plot (Diederichs <ref name="Di06">K. Diederichs (2006). Some aspects of quantitative analysis and correction of radiation damage. Acta Cryst D62, 96-101 [http://strucbio.biologie.uni-konstanz.de/strucbio/files/Diederichs_ActaD62_96.pdf]</ref>)<br />
<br />
<br />
: <math><br />
R_{d} = \frac{\sum_{hkl} \sum_{|i-j|=d} \vert I_{hkl,i} - I_{hkl,j}\vert}{\sum_{hkl} \sum_{|i-j|=d} (I_{hkl,i} + I_{hkl,j})/2}<br />
</math><br />
<br />
<br />
which gives us the average R-factor of two reflections measured d frames apart. As long as the plot is parallel to the x axis there is no radiation damage. As soon as the plot starts to rise, we see that there's a systematical error contribution due to radiation damage.<br />
<br />
Strong wiggles at very high d are irrelevant as only few reflections contribute.<br />
<br />
To my knowledge, the only program that implements this currently (December 2008) is [[xds:XDSSTAT|XDSSTAT]].<br />
<br />
=== Comparing two sets of structure factor amplitudes or intensities ===<br />
The following is symmetric, and suitable for comparing two data sets, or two model amplitudes:<br />
<math><br />
R_{scale}=\frac{\sum_{hkl}\vert F_{hkl,i}-F_{hkl,j}\vert}{0.5\sum_{hkl} F_{hkl,i}+F_{hkl,j}}<br />
</math><br />
for amplitudes, and analogously for intensities.<br />
<br />
=== Model quality indicators ===<br />
* R and [[iucr:Free_R_factor|R<sub>free</sub>]] : the formula for both is <br />
<math><br />
R=\frac{\sum_{hkl}\vert F_{hkl}^{obs}-F_{hkl}^{calc}\vert}{\sum_{hkl} F_{hkl}^{obs}}<br />
</math><br />
where <math>F_{hkl}^{obs}</math> and <math>F_{hkl}^{calc}</math> have to be scaled w.r.t. each other. R and R<sub>free</sub> differ in the set of reflections they are calculated from: R is calculated for the [[working set]], whereas R<sub>free</sub> is calculated for the [[test set]].<br />
<br />
==== Relation between R and R<sub>free</sub> as a function of resolution ====<br />
<br />
* The PDBe provide a service to plot many different statistical properties in the PDB against other properties. The link is http://www.ebi.ac.uk/pdbe-as/pdbestatistics/PDBeStatistics.jsp . You can see that there is an option of RDiff which is the difference between R and R-Free for all structures that contain both data. Take a look at this first. There is a second parameter which you can set to resolution and this will allow you to draw a plot that you want. This will draw a 3D isometric plot which you can scale, and pick data points to view particular entries.<br />
* formula from Kleywegt and Jones (2002): R<sub>free</sub> = 1.065*R + 0.036<br />
* plot with empirical data: http://xray.bmc.uu.se/gerard/supmat/rfree2000/rfminusr_vs_resolution.gif<br />
* many more plots: http://xray.bmc.uu.se/gerard/supmat/rfree2000<br />
* harry plotter (java): http://xray.bmc.uu.se/gerard/supmat/rfree2000/plotter.html<br />
* When the resolution is plotted on a logarithmic scale, the most frequent values (modes) are practically linear functions allowing their easy interpolation / extrapolation as (Urzhumtsev et al, 2009)<br />
mode(R) = 0.091*ln(resolution) + 0.134<br />
mode(Rfree-R) = 0.024*ln(resolution) + 0.020<br />
<br />
References:<br />
* Tickle IJ, Laskowski RA and Moss DS. Rfree and the Rfree Ratio. I. Derivation of Expected Values of Cross-Validation Residuals Used in Macromolecular Least-Squares Refinement. Acta Cryst. (1998). D54, 547-557 [http://dx.doi.org/10.1107/S0907444997013875] <br />
<br />
* Tickle IJ, Laskowski RA and Moss DS. Rfree and the Rfree ratio. II. Calculation of the expected values and variances of cross-validation statistics in macromolecular least-squares refinement. Acta Cryst. (2000). D56, 442-450 [http://dx.doi.org/10.1107/S0907444999016868]<br />
<br />
* GJ Kleywegt and TA Jones (2002). Homo Crystallographicus - Quo vadis? Structure 10, 465-472. (reprint from http://xray.bmc.uu.se/cgi-bin/gerard/reprint_mailer.pl?pref=65)<br />
<br />
* Urzhumtsev, Afonine & Adams (2009) Acta Cryst., D65, 1283-1291.<br />
<br />
== what kinds of problems exist with these indicators? ==<br />
* (R<sub>sym</sub> / R<sub>merge</sub> ) should not be used to judge [[data quality]], R<sub>meas</sub> should be used instead. The reason is that the former depend on multiplicity, whereas the latter doesn't.<br />
<br />
* R/R<sub>free</sub> and NCS: reflections in work and test set are not independent if chosen randomly. It is better to choose the test set reflections in thin resolution shells. Since the twin related reflections have the same sin(theta)/lambda values they will not be split over the working and reference sets. [http://xray.bmc.uu.se/usf/dataman_man.html DATAMAN] from the [http://xray.bmc.uu.se/usf/ Uppsala Software Factory] and [[XPREP]] (a program which may be obtained from Bruker) offer this option. The "RFREE SHELL" command in sftools is another way to select thin shells. A disadvantage is the the maps may not be quite as good as when the free R reflections are selected randomly. (FIXME: which [[Phenix]] program does this?). A paper investigating this thoroughly is Fabiola, F., A. Korostelev, et al. (2006). "Bias in cross-validated free R factors: mitigation of the effects of non-crystallographic symmetry." Acta Cryst. D 62: 227-38. <br />
<br />
* Sets of reflections used for calculating R<sub>free</sub> should be maintained throughout a project. This is nicely discussed at http://www.bmsc.washington.edu/people/merritt/xplor/rfree_example.html . Note that none of the programs mentioned for selecting thin shells will allow you to extend the set of shells to higher resolution if you want to preserve your existing R-free set.<br />
<br />
* R-values and twinning: [http://www.ysbl.york.ac.uk/refmac/papers/Rfactor.pdf Garib N. Murshudov (2011) "Some properties of crystallographic reliability index - Rfactor: effect of twinning" Appl. Comput. Math., V.10, N.2, 2011, pp.250-261]. From the paper, the R-value table for random models is:<br />
twinning twinning not<br />
modelled modelled<br />
twin 0.41 0.49<br />
normal 0.52 0.58<br />
Another paper which investigates the properties of R-values in the presence of twinning is [http://journals.iucr.org/d/issues/2013/07/00/ba5190/index.html P. R. Evans and G. N. Murshudov (2013) "How good are my data and what is the resolution?" Acta Cryst. (2013). D69, 1204-1214]. As the title indicates, this paper discusses at what resolution the data should be cut. One important finding is that a perfect model gives an R value of 42.0% (for a perfect twin, 29.1%) against pure noise. This tells us that a model that gives significantly lower R<sub>free</sub> in the (current) high resolution shell may benefit from including higher resolution data.<br />
<br />
* R-values and [[pseudo-translation]]: if you have pseudotranslation you should be aware that if you solve the structure by molecular replacement, starting R factors could be 70-80%.<br />
<br />
* data R-values are not meaningful at high resolution. This is discussed by [http://strucbio.biologie.uni-konstanz.de//strucbio/files/karplus2012_science.pdf Karplus and Diederichs (2012) "Linking crystallographic data and model quality". ''Science'' '''336''', 1030]<br />
<br />
==Notes==<br />
<references/></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Bootable_Linux_USB_stick&diff=2525Bootable Linux USB stick2018-01-26T10:38:26Z<p>Karsten: /* generating images and copies of the stick */</p>
<hr />
<div>We regularly and successfully use USB sticks for courses where participants bring their own notebooks. The big benefit is that students<br />
learn Linux, and realize that they can easily use the hardware they already own. Current notebook hardware is by far fast enough for data<br />
processing, structure solution and coot visualization. What we found:<br />
# the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB or bigger; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do ''not'' have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk.<br />
# we use Fedora (FC23 and higher) because its hardware support is very good. We always use the 64bit distro.<br />
# The stick can be booted on MacBooks as well (press the <code>alt</code> key at the boot sound); their hardware works well with Fedora. For Windows clients (press F11 or F12 or sometimes F9 or F10 for the boot menu; if that does not work press F2 or DEL for the BIOS menu and change the boot order), one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes <code>powercfg -H off</code> (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from the stick because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should.<br />
# if the WiFi does not work out-of-the-box on MacBook Pro, connect temporarily to the Internet by other means (ethernet cable, WiFi via USB key, tether to your phone via Bluetooth), become root, and install the latest kernel and tools with <code>dnf install -y akmods kernel kernel-devel broadcom-wl --best --allowerasing</code>. After installing, reboot into the new kernel. <br />
# we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks.<br />
# any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync.<br />
<br />
To make students familiar with the sticks and how to boot them, one needs 30+ minutes and a few tutors. <br />
<br />
'''Some more details'''<br />
<br />
# we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes a biosboot partition (1MB; required for booting bios system from a GPT disk), an efi partition (e.g. 250MB; required for UEFI/EFI boot on PC's/MAC's) and the Linux partition, with 13.9GB, so that the sum of the four partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB stick; the /data partition then does not fit and cannot be used on that stick, but the operating system will then work on the small stick just as well. This requires that the /data partition is not fsck'ed automatically from /etc/fstab (0 in the 6th field).<br />
# it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use<br />
# it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up.<br />
# (if the installation not already does it for you) '''you should label the partitions and use the labels for mounting the partitions in /etc/fstab, alternatively use UUID's; don't use /dev/sda1 or the like because depending on the hardware it is booted on, after booting the stick, and depending on the actual hardware, the stick may be /dev/sdb or /dev/sdc or ... !'''<br />
<br />
<br />
<br />
== How to create a bootabled GPT-partitioned USB-stick with Fedora ==<br />
<br />
This will create a USB stick capable to<br />
* EFI boot on Macs<br />
* UEFI boot on PCs<br />
* BIOS boot on newer hardware (legacy boot / CSM enabled in BIOS)<br />
* BIOS boot on medium aged hardware<br />
* not boot on really old BIOS hardware which doesn't support booting from GPT (see below)<br />
<br />
=== GPT partition the stick using <code>gdisk</code> ===<br />
The example shows a 32 GB Sandisk Extreme:<br />
Part 1: +1G VFAT (0700) (Windows versions before Windows 10 1709 needs it to be the first partition on usbsticks) <br />
Part 2: +1M BIOSBOOT (ef02)<br />
Part 3: +250M EFI (ef00)<br />
Part 4: +13500M / (0083)<br />
Part 5: +15000M /mnt/data (0083)<br />
For performance, make sure that the big partitions are aligned to 8192-sector boundaries; gdisk default is 2048-sector boundaries - the default can be adjusted. If you use the above sizes and the + sign for specifying them, this happens to work out automatically.<br />
<br />
=== install Fedora on an UEFI machine (UEFI enabled ; LegacyBoot / CSM disabled) ===<br />
Note: in the following, whenever you see the X in sdX, you must fill in the appropriate drive letter (e.g. a or b or c or d or ...). <br />
sdX3: efi /boot/efi<br />
sdX4: ext4 / <br />
sdX5: ext4 /mnt/data<br />
<br />
Not much to say about the installation of programs (CCP4 ?, Phenix ?, XDS ?, ...?). You should also install the <br />
* hfsplus-tools RPM to access Mac harddisks,<br />
* fuse-exfat RPM to access exFAT SD cards,<br />
* binutils RPM to get the "strings" and other useful GNU commands,<br />
* atop, a better top; lbzip2, a parallel bzip2; xxdiff, a diff GUI; nedit, editor<br />
* tcsh RPM to allow installation of CCP4. <br />
dnf -y install tcsh xterm tk qt-x11 xxdiff atop nedit lbzip2 hfsplus-tools binutils<br />
<br />
=== adjust the USB-stick for UEFI/EFI boot (before reboot from chroot environment) ===<br />
<br />
chroot to the installed system:<br />
chroot /mnt/sysimage<br />
<br />
install Fedora updates:<br />
dnf -y update # Fedora's dnf is successor to yum<br />
<br />
<br />
configure grub2 for (U)EFI systems:<br />
<br />
*disable auto recognition of other installed Operating systems (specific to current computer), and<br />
*update grub2-efi.cfg<br />
echo 'GRUB_DISABLE_OS_PROBER=”true”' >> /etc/default/grub<br />
grub2-mkconfig -o /etc/grub2-efi.cfg<br />
(last step automatically puts UUID's to recognise the boot device in grub2-efi.cfg ; default after fresh Fedora installation is the device name which might be different on another computer)<br />
<br />
<br />
Performance tuning (not strictly required):<br />
<br />
We use ext4 filesystems and <code>data=writeback,nobarrier </code> in /etc/fstab. To be able to set these options also on the / filesystem, we use tune2fs to set both as default mount options on a Linux machine where the stick (dev/sdX) is just plugged in (i.e. not booted from):<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX4<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX5<br />
<br />
create & label vfat filesystem:<br />
mkfs.vfat /dev/sdX1<br />
dosfslabel /dev/sdX1 VFAT<br />
<br />
shutdown or reboot:<br />
exit<br />
systemctl poweroff (or reboot for testing)<br />
<br />
=== Install grub2 for BIOS boot (from chroot environment) ===<br />
<br />
boot Fedora Live DVD on a BIOS machine and chroot to the stick:<br />
mount /dev/sdX4 /mnt<br />
mount -o bind /dev /mnt/dev<br />
mount -t proc /proc /mnt/proc<br />
mount -t sysfs /sys /mnt/sys<br />
chroot /mnt<br />
<br />
install grub2 for BIOS boot and configure it:<br />
rm /boot/grub2/grubenv<br />
grub2-install /dev/sdX<br />
grub2-mkconfig -o /etc/grub2.cfg<br />
(on UEFI systems /boot/grub2/grubenv is a symlink to /boot/efi/EFI/fedora/grubenv (from package grub2-efi.rpm). The symlink has to be removed before grub2-install can finish successfully, it automatically creates a new file /boot/grub2/grubenv (as in package grub2.rpm)<br />
<br />
check /etc/grub2.cfg and if needed, change at all positions:<br />
“linuxefi” to “linux16”, and<br />
“intrdefi” to “intrd16”<br />
<br />
=== generating images and copies of the stick ===<br />
<br />
Save a compressed disk image - we use the parallel gzip program called pigz:<br />
pigz -c < /dev/sdX > usbstick.img.gz<br />
(time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB)<br />
write compressed image back to a stick:<br />
unpigz -c usbstick.img.gz > /dev/sdX<br />
(speed: ~ 98MB/s)<br />
<br />
For creation/maintainance a bunch of sticks, a multiple port USB-HUB is a very usefull tool. E.g. the Raid Sonic Icy Box IB-AC6113 accepts 12 Sandisk Extreme sticks at once (port 7 has to stay empty due to spatial restrictions) and all 12 stick are recognized by the OS (CentOS7).<br />
<br />
== Hardware considerations ==<br />
<br />
Cheap USB sticks can hold a lot of data, but when it comes to writing small files (which happens when used as the operating system disk), they are either slow from the beginning, or their write performance quickly degrades as soon as free space becomes low for the first time. We have been using [https://crystalmark.info/software/CrystalDiskMark/index-e.html CrystalDiskMark] (Windows) for characterizing USB media: as a rule of thumb, the "Random write 4k blocks" disciplines should result in at least 1 MB/s, and for sequential writes the numbers should be >50 MB/s. However, CrystalDiskMark does not capture the effect of degrading write performance, which can only be mitigated by TRIMming the media - a feature that few USB sticks support (see below).<br />
<br />
The following is about performance and durability of the USB media. If you don't know what TRIM is and how it relates to flash media, read [https://en.wikipedia.org/wiki/Trim_(computing) this].<br />
<br />
=== types of USB media ===<br />
<br />
The suitability of the media makes a notable difference, in particular when the operating system or CCP4 is upgraded, or when e.g. a new Phenix version is installed.<br />
<br />
* very good: USB sticks that support TRIM. The ones known to us are the Sandisk Extreme USB sticks bought before 2017 (e.g. SDCZ80-032G 32GB for ~22€ or SDCZ80-064G 64GB for ~40€; unfortunately, these are no longer available, at least not at reasonable prices). These sticks are very fast (CrystalDiskMark scores >10MB/s for random 4K writes). The successor are the differently-named current (2018) Sandisk Extreme Pro USB sticks, but only 128 and 256 GB models exist, and they are expensive. There exist also fast and big, but costly Mushkin USB sticks which support TRIM. USB sticks which support TRIM also usually support [https://en.wikipedia.org/wiki/S.M.A.R.T. S.M.A.R.T.], which is useful for detecting problems.<br />
* good: we have verified that a USB3 adapter with (micro)SD card (separate items) is suitable. USB3 adapters are cheap (around 5€ at Amazon, like https://www.amazon.de/EX1-Kartenleser-Micro-Karte-Adapter/dp/B06XX3T219; cheaper at Ebay). microSD cards usually come with a SD adapter, and can be TRIMmed (but only in a SD slot; see below!). microSD cards ideally should be in the [https://en.wikipedia.org/wiki/Secure_Digital#Speed_class_rating Class 1 (A1) application performance class] since this is what is defined as requirement for use as extended operating system disk in Android smartphones. 64GB A1 microSD cards cost around 25€. In practice, slightly cheaper non-A1 microSD cards also seem to work well, but may be slower and may require more often TRIMming. The price/performance ratio of the microSD/USB3 combination is very good.<br />
* acceptable: USB sticks that do not support TRIM but are fast and high quality, e.g. Sandisk Extreme Go or Sandisk Ultra bought 2017 and later. However, their write performance may degrade with time (or rather, with the number of writes to a partition), and not much can be done about that.<br />
* inacceptable: tiny sticks like Sandisk Ultra Fit become hot, degrade fast and are unreliable. Cheap sticks are usually just too slow.<br />
<br />
The drawback of the "good" solution above is: from time to time (e.g. once every week), a knowledgeable person may want to<br />
# remove the microSD card from the USB3 adapter<br />
# insert it into the SD adapter, and that into a SD slot - it is usually auto-mounted by the operating system<br />
# run the <code>fstrim -v /mount/point</code> command (this takes a few seconds)<br />
# unmount the card, and revert steps 2. and 1.<br />
<br />
This naturally begs the question whether it wouldn't be much smarter to not use an USB adapter for microSD cards, but rather just use the SD adapter in a SD slot right away. That would have advantages - the SD card vanishes in most SD slots which is better than an USB stick sticking out, <code>fstrim</code> works, and possibly the SD controller is faster than the USB3 controller (depending on the notebook model). However, we tried to boot a microSD card in a SD lot and found that it doesn't boot, whereas it does boot in an USB adapter. Thus, setup of a microSD card for booting in SD slot needs to be investigated.<br />
<br />
=== TRIM ===<br />
==== USB sticks ==== <br />
TRIMming informs the firmware about blocks of the filesystem that do not contain file data. This is available for USB sticks for which <code>hdparm -I /dev/sdX</code> returns "Data Set Management TRIM supported". <br />
<br />
The <code>wiper.sh</code> script (part of the <code>hdparm</code> source distribution) TRIMs ext3/ext4/xfs filesystems, but does not reproducibly seem to work for our preferred SanDisk Extreme 32GB stick if the filesystem is mounted. This is probably due to the fact that this stick has a limitation of max 65536 blocks in one TRIM command (https://sourceforge.net/p/hdparm/bugs/63/). Since <code>wiper.sh</code> also works for unmounted filesystems, and the mapping of unused space is then obtained with a different tool, one should try with an unmounted filesystem as well - this worked for us in all cases tried so far (tested with [http://lightrush.ndoytchev.com/random-1/checkiftrimonext4isenabledandworking this script]). <br />
<br />
All other methods, like the <code>fstrim</code> command and the <code>discard</code> mount option '''do not work for USB sticks''', because the usb-storage kernel module does not pass the ATA TRIM command through the USB bridge and controller to the device. The same goes for [http://unix.stackexchange.com/questions/97143/utility-to-trim-unallocated-space-on-drive <code>blkdiscard</code>].<br />
<br />
==== (micro)SD cards ====<br />
These support TRIM when in a SD slot, but not when inserted in a USB adapter. When in a SD slot, the <code>lsblk -D</code> command reveals that the card can be TRIMmed, and <code>fstrim -v /dev/mmcXXXXX</code> works.<br />
<br />
=== fill empty space of a partition with “zeroes” ===<br />
This could be done after updating the software or data on the USB media, and before saving and compressing an image of it. <br />
USB sticks that support “Deterministic read data after TRIM” according to <code>hdparm -I /dev/sdX</code>, and microSD cards (which have zeroes in free space after <code>fstrim</code> in a SD slot) should be TRIMmed because this does not wear out the storage cells.<br />
<br />
Other media: as root<br />
dd if=/dev/zero of=/mountpoint_of_partition/delete.me bs=10M # this will stop when filesystem is full<br />
rm /mountpoint_of_partition/delete.me<br />
However, this stresses the flash cells; they sustain only a limited number of write cycles.<br />
<br />
=== initializing the media ===<br />
Low-level formatting restores the write performance, and removes any sensitive information (e.g. passwords) - important for workshops.<br />
==== Low-level formatting of "very good" USB sticks ==== <br />
<code>hdparm</code>'s (ENHANCED) SECURITY ERASE initializes the whole stick in a few seconds, and restores it to an almost factory-fresh condition, including zeroing the device. This works on all sticks where <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" but if used wrongly (e.g. on the wrong device, or with the wrong options, or ...) it may brick your device, or delete valuable data! So use at your own risk:<br />
* check if <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" and "not frozen" (to unfreeze a frozen diks, suspend your system with <code>pm-suspend</code> and wake it up again)<br />
* set disk password, and erase it (which unsets the password):<br />
hdparm --user-master u --security-set-pass Eins /dev/sdX<br />
hdparm --user-master u --security-erase-enhanced Eins /dev/sdX<br />
If you tried with --security-erase-enhanced, but the disk does not support it, then an error will occur and the disk will still be locked after the failed command, and you will not be able to write anything to it! If this happens, you can unlock it with <br />
hdparm --user-master u --security-unlock Eins /dev/sdX</code><br />
<br />
==== Low-level formatting of (micro)SD cards ====<br />
In a SD slot, <code>blkdiscard /dev/mmcblk0pX</code> initializes the card.<br />
<br />
==== Low-level formatting of other USB sticks ====<br />
One may try to write zeroes on the whole stick <br />
dd if=/dev/zero of=/dev/sdX bs=8192k<br />
hoping that the firmware will understand this as a request for low-level formatting. This seems to work with some sticks, but takes a long time and may stress the flash cells.<br />
<br />
== installation of crystallographic software ==<br />
* XDS and related programs (XDS-viewer, xdsstat, xdsgui, generate_XDS.INP): [http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Installation#Linux see XDSwiki]<br />
* CCP4: download and install from [http://www.ccp4.ac.uk/download/#os=linux CCP4 Homepage]<br />
* PHENIX: download and install from [https://www.phenix-online.org/download/ Phenix-online]<br />
* SHELX: download and install from [http://shelx.uni-ac.gwdg.de/SHELX/download.php SHELX Homepage] or together with [http://www.ccp4.ac.uk/download/#os=linux CCP4]<br />
* [http://webapps.embl-hamburg.de/bundle/download.php?hkl2mp=true&sitcom=false HKL2MAP]</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Bootable_Linux_USB_stick&diff=2524Bootable Linux USB stick2018-01-25T14:58:14Z<p>Karsten: /* GPT partition the stick using gdisk */</p>
<hr />
<div>We regularly and successfully use USB sticks for courses where participants bring their own notebooks. The big benefit is that students<br />
learn Linux, and realize that they can easily use the hardware they already own. Current notebook hardware is by far fast enough for data<br />
processing, structure solution and coot visualization. What we found:<br />
# the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB or bigger; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do ''not'' have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk.<br />
# we use Fedora (FC23 and higher) because its hardware support is very good. We always use the 64bit distro.<br />
# The stick can be booted on MacBooks as well (press the <code>alt</code> key at the boot sound); their hardware works well with Fedora. For Windows clients (press F11 or F12 or sometimes F9 or F10 for the boot menu; if that does not work press F2 or DEL for the BIOS menu and change the boot order), one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes <code>powercfg -H off</code> (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from the stick because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should.<br />
# if the WiFi does not work out-of-the-box on MacBook Pro, connect temporarily to the Internet by other means (ethernet cable, WiFi via USB key, tether to your phone via Bluetooth), become root, and install the latest kernel and tools with <code>dnf install -y akmods kernel kernel-devel broadcom-wl --best --allowerasing</code>. After installing, reboot into the new kernel. <br />
# we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks.<br />
# any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync.<br />
<br />
To make students familiar with the sticks and how to boot them, one needs 30+ minutes and a few tutors. <br />
<br />
'''Some more details'''<br />
<br />
# we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes a biosboot partition (1MB; required for booting bios system from a GPT disk), an efi partition (e.g. 250MB; required for UEFI/EFI boot on PC's/MAC's) and the Linux partition, with 13.9GB, so that the sum of the four partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB stick; the /data partition then does not fit and cannot be used on that stick, but the operating system will then work on the small stick just as well. This requires that the /data partition is not fsck'ed automatically from /etc/fstab (0 in the 6th field).<br />
# it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use<br />
# it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up.<br />
# (if the installation not already does it for you) '''you should label the partitions and use the labels for mounting the partitions in /etc/fstab, alternatively use UUID's; don't use /dev/sda1 or the like because depending on the hardware it is booted on, after booting the stick, and depending on the actual hardware, the stick may be /dev/sdb or /dev/sdc or ... !'''<br />
<br />
<br />
<br />
== How to create a bootabled GPT-partitioned USB-stick with Fedora ==<br />
<br />
This will create a USB stick capable to<br />
* EFI boot on Macs<br />
* UEFI boot on PCs<br />
* BIOS boot on newer hardware (legacy boot / CSM enabled in BIOS)<br />
* BIOS boot on medium aged hardware<br />
* not boot on really old BIOS hardware which doesn't support booting from GPT (see below)<br />
<br />
=== GPT partition the stick using <code>gdisk</code> ===<br />
The example shows a 32 GB Sandisk Extreme:<br />
Part 1: +1G VFAT (0700) (Windows versions before Windows 10 1709 needs it to be the first partition on usbsticks) <br />
Part 2: +1M BIOSBOOT (ef02)<br />
Part 3: +250M EFI (ef00)<br />
Part 4: +13500M / (0083)<br />
Part 5: +15000M /mnt/data (0083)<br />
For performance, make sure that the big partitions are aligned to 8192-sector boundaries; gdisk default is 2048-sector boundaries - the default can be adjusted. If you use the above sizes and the + sign for specifying them, this happens to work out automatically.<br />
<br />
=== install Fedora on an UEFI machine (UEFI enabled ; LegacyBoot / CSM disabled) ===<br />
Note: in the following, whenever you see the X in sdX, you must fill in the appropriate drive letter (e.g. a or b or c or d or ...). <br />
sdX3: efi /boot/efi<br />
sdX4: ext4 / <br />
sdX5: ext4 /mnt/data<br />
<br />
Not much to say about the installation of programs (CCP4 ?, Phenix ?, XDS ?, ...?). You should also install the <br />
* hfsplus-tools RPM to access Mac harddisks,<br />
* fuse-exfat RPM to access exFAT SD cards,<br />
* binutils RPM to get the "strings" and other useful GNU commands,<br />
* atop, a better top; lbzip2, a parallel bzip2; xxdiff, a diff GUI; nedit, editor<br />
* tcsh RPM to allow installation of CCP4. <br />
dnf -y install tcsh xterm tk qt-x11 xxdiff atop nedit lbzip2 hfsplus-tools binutils<br />
<br />
=== adjust the USB-stick for UEFI/EFI boot (before reboot from chroot environment) ===<br />
<br />
chroot to the installed system:<br />
chroot /mnt/sysimage<br />
<br />
install Fedora updates:<br />
dnf -y update # Fedora's dnf is successor to yum<br />
<br />
<br />
configure grub2 for (U)EFI systems:<br />
<br />
*disable auto recognition of other installed Operating systems (specific to current computer), and<br />
*update grub2-efi.cfg<br />
echo 'GRUB_DISABLE_OS_PROBER=”true”' >> /etc/default/grub<br />
grub2-mkconfig -o /etc/grub2-efi.cfg<br />
(last step automatically puts UUID's to recognise the boot device in grub2-efi.cfg ; default after fresh Fedora installation is the device name which might be different on another computer)<br />
<br />
<br />
Performance tuning (not strictly required):<br />
<br />
We use ext4 filesystems and <code>data=writeback,nobarrier </code> in /etc/fstab. To be able to set these options also on the / filesystem, we use tune2fs to set both as default mount options on a Linux machine where the stick (dev/sdX) is just plugged in (i.e. not booted from):<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX4<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX5<br />
<br />
create & label vfat filesystem:<br />
mkfs.vfat /dev/sdX1<br />
dosfslabel /dev/sdX1 VFAT<br />
<br />
shutdown or reboot:<br />
exit<br />
systemctl poweroff (or reboot for testing)<br />
<br />
=== Install grub2 for BIOS boot (from chroot environment) ===<br />
<br />
boot Fedora Live DVD on a BIOS machine and chroot to the stick:<br />
mount /dev/sdX4 /mnt<br />
mount -o bind /dev /mnt/dev<br />
mount -t proc /proc /mnt/proc<br />
mount -t sysfs /sys /mnt/sys<br />
chroot /mnt<br />
<br />
install grub2 for BIOS boot and configure it:<br />
rm /boot/grub2/grubenv<br />
grub2-install /dev/sdX<br />
grub2-mkconfig -o /etc/grub2.cfg<br />
(on UEFI systems /boot/grub2/grubenv is a symlink to /boot/efi/EFI/fedora/grubenv (from package grub2-efi.rpm). The symlink has to be removed before grub2-install can finish successfully, it automatically creates a new file /boot/grub2/grubenv (as in package grub2.rpm)<br />
<br />
check /etc/grub2.cfg and if needed, change at all positions:<br />
“linuxefi” to “linux16”, and<br />
“intrdefi” to “intrd16”<br />
<br />
=== generating images and copies of the stick ===<br />
<br />
Save a compressed disk image - we use the parallel gzip program called pigz:<br />
pigz -c < /dev/sdX > usbstick.img.gz<br />
(time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB)<br />
write compressed image back to a stick:<br />
unpigz -c < usbstick.img.gz > /dev/sdX<br />
(speed: ~ 98MB/s)<br />
<br />
For creation/maintainance a bunch of sticks, a multiple port USB-HUB is a very usefull tool. E.g. the Raid Sonic Icy Box IB-AC6113 accepts 12 Sandisk Extreme sticks at once (port 7 has to stay empty due to spatial restrictions) and all 12 stick are recognized by the OS (CentOS7).<br />
<br />
== Hardware considerations ==<br />
<br />
Cheap USB sticks can hold a lot of data, but when it comes to writing small files (which happens when used as the operating system disk), they are either slow from the beginning, or their write performance quickly degrades as soon as free space becomes low for the first time. We have been using [https://crystalmark.info/software/CrystalDiskMark/index-e.html CrystalDiskMark] (Windows) for characterizing USB media: as a rule of thumb, the "Random write 4k blocks" disciplines should result in at least 1 MB/s, and for sequential writes the numbers should be >50 MB/s. However, CrystalDiskMark does not capture the effect of degrading write performance, which can only be mitigated by TRIMming the media - a feature that few USB sticks support (see below).<br />
<br />
The following is about performance and durability of the USB media. If you don't know what TRIM is and how it relates to flash media, read [https://en.wikipedia.org/wiki/Trim_(computing) this].<br />
<br />
=== types of USB media ===<br />
<br />
The suitability of the media makes a notable difference, in particular when the operating system or CCP4 is upgraded, or when e.g. a new Phenix version is installed.<br />
<br />
* very good: USB sticks that support TRIM. The ones known to us are the Sandisk Extreme USB sticks bought before 2017 (e.g. SDCZ80-032G 32GB for ~22€ or SDCZ80-064G 64GB for ~40€; unfortunately, these are no longer available, at least not at reasonable prices). These sticks are very fast (CrystalDiskMark scores >10MB/s for random 4K writes). The successor are the differently-named current (2018) Sandisk Extreme Pro USB sticks, but only 128 and 256 GB models exist, and they are expensive. There exist also fast and big, but costly Mushkin USB sticks which support TRIM. USB sticks which support TRIM also usually support [https://en.wikipedia.org/wiki/S.M.A.R.T. S.M.A.R.T.], which is useful for detecting problems.<br />
* good: we have verified that a USB3 adapter with (micro)SD card (separate items) is suitable. USB3 adapters are cheap (around 5€ at Amazon, like https://www.amazon.de/EX1-Kartenleser-Micro-Karte-Adapter/dp/B06XX3T219; cheaper at Ebay). microSD cards usually come with a SD adapter, and can be TRIMmed (but only in a SD slot; see below!). microSD cards ideally should be in the [https://en.wikipedia.org/wiki/Secure_Digital#Speed_class_rating Class 1 (A1) application performance class] since this is what is defined as requirement for use as extended operating system disk in Android smartphones. 64GB A1 microSD cards cost around 25€. In practice, slightly cheaper non-A1 microSD cards also seem to work well, but may be slower and may require more often TRIMming. The price/performance ratio of the microSD/USB3 combination is very good.<br />
* acceptable: USB sticks that do not support TRIM but are fast and high quality, e.g. Sandisk Extreme Go or Sandisk Ultra bought 2017 and later. However, their write performance may degrade with time (or rather, with the number of writes to a partition), and not much can be done about that.<br />
* inacceptable: tiny sticks like Sandisk Ultra Fit become hot, degrade fast and are unreliable. Cheap sticks are usually just too slow.<br />
<br />
The drawback of the "good" solution above is: from time to time (e.g. once every week), a knowledgeable person may want to<br />
# remove the microSD card from the USB3 adapter<br />
# insert it into the SD adapter, and that into a SD slot - it is usually auto-mounted by the operating system<br />
# run the <code>fstrim -v /mount/point</code> command (this takes a few seconds)<br />
# unmount the card, and revert steps 2. and 1.<br />
<br />
This naturally begs the question whether it wouldn't be much smarter to not use an USB adapter for microSD cards, but rather just use the SD adapter in a SD slot right away. That would have advantages - the SD card vanishes in most SD slots which is better than an USB stick sticking out, <code>fstrim</code> works, and possibly the SD controller is faster than the USB3 controller (depending on the notebook model). However, we tried to boot a microSD card in a SD lot and found that it doesn't boot, whereas it does boot in an USB adapter. Thus, setup of a microSD card for booting in SD slot needs to be investigated.<br />
<br />
=== TRIM ===<br />
==== USB sticks ==== <br />
TRIMming informs the firmware about blocks of the filesystem that do not contain file data. This is available for USB sticks for which <code>hdparm -I /dev/sdX</code> returns "Data Set Management TRIM supported". <br />
<br />
The <code>wiper.sh</code> script (part of the <code>hdparm</code> source distribution) TRIMs ext3/ext4/xfs filesystems, but does not reproducibly seem to work for our preferred SanDisk Extreme 32GB stick if the filesystem is mounted. This is probably due to the fact that this stick has a limitation of max 65536 blocks in one TRIM command (https://sourceforge.net/p/hdparm/bugs/63/). Since <code>wiper.sh</code> also works for unmounted filesystems, and the mapping of unused space is then obtained with a different tool, one should try with an unmounted filesystem as well - this worked for us in all cases tried so far (tested with [http://lightrush.ndoytchev.com/random-1/checkiftrimonext4isenabledandworking this script]). <br />
<br />
All other methods, like the <code>fstrim</code> command and the <code>discard</code> mount option '''do not work for USB sticks''', because the usb-storage kernel module does not pass the ATA TRIM command through the USB bridge and controller to the device. The same goes for [http://unix.stackexchange.com/questions/97143/utility-to-trim-unallocated-space-on-drive <code>blkdiscard</code>].<br />
<br />
==== (micro)SD cards ====<br />
These support TRIM when in a SD slot, but not when inserted in a USB adapter. When in a SD slot, the <code>lsblk -D</code> command reveals that the card can be TRIMmed, and <code>fstrim -v /dev/mmcXXXXX</code> works.<br />
<br />
=== fill empty space of a partition with “zeroes” ===<br />
This could be done after updating the software or data on the USB media, and before saving and compressing an image of it. <br />
USB sticks that support “Deterministic read data after TRIM” according to <code>hdparm -I /dev/sdX</code>, and microSD cards (which have zeroes in free space after <code>fstrim</code> in a SD slot) should be TRIMmed because this does not wear out the storage cells.<br />
<br />
Other media: as root<br />
dd if=/dev/zero of=/mountpoint_of_partition/delete.me bs=10M # this will stop when filesystem is full<br />
rm /mountpoint_of_partition/delete.me<br />
However, this stresses the flash cells; they sustain only a limited number of write cycles.<br />
<br />
=== initializing the media ===<br />
Low-level formatting restores the write performance, and removes any sensitive information (e.g. passwords) - important for workshops.<br />
==== Low-level formatting of "very good" USB sticks ==== <br />
<code>hdparm</code>'s (ENHANCED) SECURITY ERASE initializes the whole stick in a few seconds, and restores it to an almost factory-fresh condition, including zeroing the device. This works on all sticks where <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" but if used wrongly (e.g. on the wrong device, or with the wrong options, or ...) it may brick your device, or delete valuable data! So use at your own risk:<br />
* check if <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" and "not frozen" (to unfreeze a frozen diks, suspend your system with <code>pm-suspend</code> and wake it up again)<br />
* set disk password, and erase it (which unsets the password):<br />
hdparm --user-master u --security-set-pass Eins /dev/sdX<br />
hdparm --user-master u --security-erase-enhanced Eins /dev/sdX<br />
If you tried with --security-erase-enhanced, but the disk does not support it, then an error will occur and the disk will still be locked after the failed command, and you will not be able to write anything to it! If this happens, you can unlock it with <br />
hdparm --user-master u --security-unlock Eins /dev/sdX</code><br />
<br />
==== Low-level formatting of (micro)SD cards ====<br />
In a SD slot, <code>blkdiscard /dev/mmcblk0pX</code> initializes the card.<br />
<br />
==== Low-level formatting of other USB sticks ====<br />
One may try to write zeroes on the whole stick <br />
dd if=/dev/zero of=/dev/sdX bs=8192k<br />
hoping that the firmware will understand this as a request for low-level formatting. This seems to work with some sticks, but takes a long time and may stress the flash cells.<br />
<br />
== installation of crystallographic software ==<br />
* XDS and related programs (XDS-viewer, xdsstat, xdsgui, generate_XDS.INP): [http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Installation#Linux see XDSwiki]<br />
* CCP4: download and install from [http://www.ccp4.ac.uk/download/#os=linux CCP4 Homepage]<br />
* PHENIX: download and install from [https://www.phenix-online.org/download/ Phenix-online]<br />
* SHELX: download and install from [http://shelx.uni-ac.gwdg.de/SHELX/download.php SHELX Homepage] or together with [http://www.ccp4.ac.uk/download/#os=linux CCP4]<br />
* [http://webapps.embl-hamburg.de/bundle/download.php?hkl2mp=true&sitcom=false HKL2MAP]</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Bootable_Linux_USB_stick&diff=2523Bootable Linux USB stick2018-01-25T14:44:56Z<p>Karsten: /* Low-level formatting of (micro)SD cards */</p>
<hr />
<div>We regularly and successfully use USB sticks for courses where participants bring their own notebooks. The big benefit is that students<br />
learn Linux, and realize that they can easily use the hardware they already own. Current notebook hardware is by far fast enough for data<br />
processing, structure solution and coot visualization. What we found:<br />
# the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB or bigger; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do ''not'' have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk.<br />
# we use Fedora (FC23 and higher) because its hardware support is very good. We always use the 64bit distro.<br />
# The stick can be booted on MacBooks as well (press the <code>alt</code> key at the boot sound); their hardware works well with Fedora. For Windows clients (press F11 or F12 or sometimes F9 or F10 for the boot menu; if that does not work press F2 or DEL for the BIOS menu and change the boot order), one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes <code>powercfg -H off</code> (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from the stick because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should.<br />
# if the WiFi does not work out-of-the-box on MacBook Pro, connect temporarily to the Internet by other means (ethernet cable, WiFi via USB key, tether to your phone via Bluetooth), become root, and install the latest kernel and tools with <code>dnf install -y akmods kernel kernel-devel broadcom-wl --best --allowerasing</code>. After installing, reboot into the new kernel. <br />
# we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks.<br />
# any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync.<br />
<br />
To make students familiar with the sticks and how to boot them, one needs 30+ minutes and a few tutors. <br />
<br />
'''Some more details'''<br />
<br />
# we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes a biosboot partition (1MB; required for booting bios system from a GPT disk), an efi partition (e.g. 250MB; required for UEFI/EFI boot on PC's/MAC's) and the Linux partition, with 13.9GB, so that the sum of the four partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB stick; the /data partition then does not fit and cannot be used on that stick, but the operating system will then work on the small stick just as well. This requires that the /data partition is not fsck'ed automatically from /etc/fstab (0 in the 6th field).<br />
# it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use<br />
# it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up.<br />
# (if the installation not already does it for you) '''you should label the partitions and use the labels for mounting the partitions in /etc/fstab, alternatively use UUID's; don't use /dev/sda1 or the like because depending on the hardware it is booted on, after booting the stick, and depending on the actual hardware, the stick may be /dev/sdb or /dev/sdc or ... !'''<br />
<br />
<br />
<br />
== How to create a bootabled GPT-partitioned USB-stick with Fedora ==<br />
<br />
This will create a USB stick capable to<br />
* EFI boot on Macs<br />
* UEFI boot on PCs<br />
* BIOS boot on newer hardware (legacy boot / CSM enabled in BIOS)<br />
* BIOS boot on medium aged hardware<br />
* not boot on really old BIOS hardware which doesn't support booting from GPT (see below)<br />
<br />
=== GPT partition the stick using <code>gdisk</code> ===<br />
The example shows a 32 GB Sandisk Extreme:<br />
Part 1: +1G VFAT (0700) (Windows needs it to be the first partition) <br />
Part 2: +1M BIOSBOOT (ef02)<br />
Part 3: +250M EFI (ef00)<br />
Part 4: +13500M / (0083)<br />
Part 5: +15000M /mnt/data (0083)<br />
For performance, make sure that the big partitions are aligned to 8192-sector boundaries; gdisk default is 2048-sector boundaries - the default can be adjusted. If you use the above sizes and the + sign for specifying them, this happens to work out automatically.<br />
<br />
=== install Fedora on an UEFI machine (UEFI enabled ; LegacyBoot / CSM disabled) ===<br />
Note: in the following, whenever you see the X in sdX, you must fill in the appropriate drive letter (e.g. a or b or c or d or ...). <br />
sdX3: efi /boot/efi<br />
sdX4: ext4 / <br />
sdX5: ext4 /mnt/data<br />
<br />
Not much to say about the installation of programs (CCP4 ?, Phenix ?, XDS ?, ...?). You should also install the <br />
* hfsplus-tools RPM to access Mac harddisks,<br />
* fuse-exfat RPM to access exFAT SD cards,<br />
* binutils RPM to get the "strings" and other useful GNU commands,<br />
* atop, a better top; lbzip2, a parallel bzip2; xxdiff, a diff GUI; nedit, editor<br />
* tcsh RPM to allow installation of CCP4. <br />
dnf -y install tcsh xterm tk qt-x11 xxdiff atop nedit lbzip2 hfsplus-tools binutils<br />
<br />
=== adjust the USB-stick for UEFI/EFI boot (before reboot from chroot environment) ===<br />
<br />
chroot to the installed system:<br />
chroot /mnt/sysimage<br />
<br />
install Fedora updates:<br />
dnf -y update # Fedora's dnf is successor to yum<br />
<br />
<br />
configure grub2 for (U)EFI systems:<br />
<br />
*disable auto recognition of other installed Operating systems (specific to current computer), and<br />
*update grub2-efi.cfg<br />
echo 'GRUB_DISABLE_OS_PROBER=”true”' >> /etc/default/grub<br />
grub2-mkconfig -o /etc/grub2-efi.cfg<br />
(last step automatically puts UUID's to recognise the boot device in grub2-efi.cfg ; default after fresh Fedora installation is the device name which might be different on another computer)<br />
<br />
<br />
Performance tuning (not strictly required):<br />
<br />
We use ext4 filesystems and <code>data=writeback,nobarrier </code> in /etc/fstab. To be able to set these options also on the / filesystem, we use tune2fs to set both as default mount options on a Linux machine where the stick (dev/sdX) is just plugged in (i.e. not booted from):<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX4<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX5<br />
<br />
create & label vfat filesystem:<br />
mkfs.vfat /dev/sdX1<br />
dosfslabel /dev/sdX1 VFAT<br />
<br />
shutdown or reboot:<br />
exit<br />
systemctl poweroff (or reboot for testing)<br />
<br />
=== Install grub2 for BIOS boot (from chroot environment) ===<br />
<br />
boot Fedora Live DVD on a BIOS machine and chroot to the stick:<br />
mount /dev/sdX4 /mnt<br />
mount -o bind /dev /mnt/dev<br />
mount -t proc /proc /mnt/proc<br />
mount -t sysfs /sys /mnt/sys<br />
chroot /mnt<br />
<br />
install grub2 for BIOS boot and configure it:<br />
rm /boot/grub2/grubenv<br />
grub2-install /dev/sdX<br />
grub2-mkconfig -o /etc/grub2.cfg<br />
(on UEFI systems /boot/grub2/grubenv is a symlink to /boot/efi/EFI/fedora/grubenv (from package grub2-efi.rpm). The symlink has to be removed before grub2-install can finish successfully, it automatically creates a new file /boot/grub2/grubenv (as in package grub2.rpm)<br />
<br />
check /etc/grub2.cfg and if needed, change at all positions:<br />
“linuxefi” to “linux16”, and<br />
“intrdefi” to “intrd16”<br />
<br />
=== generating images and copies of the stick ===<br />
<br />
Save a compressed disk image - we use the parallel gzip program called pigz:<br />
pigz -c < /dev/sdX > usbstick.img.gz<br />
(time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB)<br />
write compressed image back to a stick:<br />
unpigz -c < usbstick.img.gz > /dev/sdX<br />
(speed: ~ 98MB/s)<br />
<br />
For creation/maintainance a bunch of sticks, a multiple port USB-HUB is a very usefull tool. E.g. the Raid Sonic Icy Box IB-AC6113 accepts 12 Sandisk Extreme sticks at once (port 7 has to stay empty due to spatial restrictions) and all 12 stick are recognized by the OS (CentOS7).<br />
<br />
== Hardware considerations ==<br />
<br />
Cheap USB sticks can hold a lot of data, but when it comes to writing small files (which happens when used as the operating system disk), they are either slow from the beginning, or their write performance quickly degrades as soon as free space becomes low for the first time. We have been using [https://crystalmark.info/software/CrystalDiskMark/index-e.html CrystalDiskMark] (Windows) for characterizing USB media: as a rule of thumb, the "Random write 4k blocks" disciplines should result in at least 1 MB/s, and for sequential writes the numbers should be >50 MB/s. However, CrystalDiskMark does not capture the effect of degrading write performance, which can only be mitigated by TRIMming the media - a feature that few USB sticks support (see below).<br />
<br />
The following is about performance and durability of the USB media. If you don't know what TRIM is and how it relates to flash media, read [https://en.wikipedia.org/wiki/Trim_(computing) this].<br />
<br />
=== types of USB media ===<br />
<br />
The suitability of the media makes a notable difference, in particular when the operating system or CCP4 is upgraded, or when e.g. a new Phenix version is installed.<br />
<br />
* very good: USB sticks that support TRIM. The ones known to us are the Sandisk Extreme USB sticks bought before 2017 (e.g. SDCZ80-032G 32GB for ~22€ or SDCZ80-064G 64GB for ~40€; unfortunately, these are no longer available, at least not at reasonable prices). These sticks are very fast (CrystalDiskMark scores >10MB/s for random 4K writes). The successor are the differently-named current (2018) Sandisk Extreme Pro USB sticks, but only 128 and 256 GB models exist, and they are expensive. There exist also fast and big, but costly Mushkin USB sticks which support TRIM. USB sticks which support TRIM also usually support [https://en.wikipedia.org/wiki/S.M.A.R.T. S.M.A.R.T.], which is useful for detecting problems.<br />
* good: we have verified that a USB3 adapter with (micro)SD card (separate items) is suitable. USB3 adapters are cheap (around 5€ at Amazon, like https://www.amazon.de/EX1-Kartenleser-Micro-Karte-Adapter/dp/B06XX3T219; cheaper at Ebay). microSD cards usually come with a SD adapter, and can be TRIMmed (but only in a SD slot; see below!). microSD cards ideally should be in the [https://en.wikipedia.org/wiki/Secure_Digital#Speed_class_rating Class 1 (A1) application performance class] since this is what is defined as requirement for use as extended operating system disk in Android smartphones. 64GB A1 microSD cards cost around 25€. In practice, slightly cheaper non-A1 microSD cards also seem to work well, but may be slower and may require more often TRIMming. The price/performance ratio of the microSD/USB3 combination is very good.<br />
* acceptable: USB sticks that do not support TRIM but are fast and high quality, e.g. Sandisk Extreme Go or Sandisk Ultra bought 2017 and later. However, their write performance may degrade with time (or rather, with the number of writes to a partition), and not much can be done about that.<br />
* inacceptable: tiny sticks like Sandisk Ultra Fit become hot, degrade fast and are unreliable. Cheap sticks are usually just too slow.<br />
<br />
The drawback of the "good" solution above is: from time to time (e.g. once every week), a knowledgeable person may want to<br />
# remove the microSD card from the USB3 adapter<br />
# insert it into the SD adapter, and that into a SD slot - it is usually auto-mounted by the operating system<br />
# run the <code>fstrim -v /mount/point</code> command (this takes a few seconds)<br />
# unmount the card, and revert steps 2. and 1.<br />
<br />
This naturally begs the question whether it wouldn't be much smarter to not use an USB adapter for microSD cards, but rather just use the SD adapter in a SD slot right away. That would have advantages - the SD card vanishes in most SD slots which is better than an USB stick sticking out, <code>fstrim</code> works, and possibly the SD controller is faster than the USB3 controller (depending on the notebook model). However, we tried to boot a microSD card in a SD lot and found that it doesn't boot, whereas it does boot in an USB adapter. Thus, setup of a microSD card for booting in SD slot needs to be investigated.<br />
<br />
=== TRIM ===<br />
==== USB sticks ==== <br />
TRIMming informs the firmware about blocks of the filesystem that do not contain file data. This is available for USB sticks for which <code>hdparm -I /dev/sdX</code> returns "Data Set Management TRIM supported". <br />
<br />
The <code>wiper.sh</code> script (part of the <code>hdparm</code> source distribution) TRIMs ext3/ext4/xfs filesystems, but does not reproducibly seem to work for our preferred SanDisk Extreme 32GB stick if the filesystem is mounted. This is probably due to the fact that this stick has a limitation of max 65536 blocks in one TRIM command (https://sourceforge.net/p/hdparm/bugs/63/). Since <code>wiper.sh</code> also works for unmounted filesystems, and the mapping of unused space is then obtained with a different tool, one should try with an unmounted filesystem as well - this worked for us in all cases tried so far (tested with [http://lightrush.ndoytchev.com/random-1/checkiftrimonext4isenabledandworking this script]). <br />
<br />
All other methods, like the <code>fstrim</code> command and the <code>discard</code> mount option '''do not work for USB sticks''', because the usb-storage kernel module does not pass the ATA TRIM command through the USB bridge and controller to the device. The same goes for [http://unix.stackexchange.com/questions/97143/utility-to-trim-unallocated-space-on-drive <code>blkdiscard</code>].<br />
<br />
==== (micro)SD cards ====<br />
These support TRIM when in a SD slot, but not when inserted in a USB adapter. When in a SD slot, the <code>lsblk -D</code> command reveals that the card can be TRIMmed, and <code>fstrim -v /dev/mmcXXXXX</code> works.<br />
<br />
=== fill empty space of a partition with “zeroes” ===<br />
This could be done after updating the software or data on the USB media, and before saving and compressing an image of it. <br />
USB sticks that support “Deterministic read data after TRIM” according to <code>hdparm -I /dev/sdX</code>, and microSD cards (which have zeroes in free space after <code>fstrim</code> in a SD slot) should be TRIMmed because this does not wear out the storage cells.<br />
<br />
Other media: as root<br />
dd if=/dev/zero of=/mountpoint_of_partition/delete.me bs=10M # this will stop when filesystem is full<br />
rm /mountpoint_of_partition/delete.me<br />
However, this stresses the flash cells; they sustain only a limited number of write cycles.<br />
<br />
=== initializing the media ===<br />
Low-level formatting restores the write performance, and removes any sensitive information (e.g. passwords) - important for workshops.<br />
==== Low-level formatting of "very good" USB sticks ==== <br />
<code>hdparm</code>'s (ENHANCED) SECURITY ERASE initializes the whole stick in a few seconds, and restores it to an almost factory-fresh condition, including zeroing the device. This works on all sticks where <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" but if used wrongly (e.g. on the wrong device, or with the wrong options, or ...) it may brick your device, or delete valuable data! So use at your own risk:<br />
* check if <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" and "not frozen" (to unfreeze a frozen diks, suspend your system with <code>pm-suspend</code> and wake it up again)<br />
* set disk password, and erase it (which unsets the password):<br />
hdparm --user-master u --security-set-pass Eins /dev/sdX<br />
hdparm --user-master u --security-erase-enhanced Eins /dev/sdX<br />
If you tried with --security-erase-enhanced, but the disk does not support it, then an error will occur and the disk will still be locked after the failed command, and you will not be able to write anything to it! If this happens, you can unlock it with <br />
hdparm --user-master u --security-unlock Eins /dev/sdX</code><br />
<br />
==== Low-level formatting of (micro)SD cards ====<br />
In a SD slot, <code>blkdiscard /dev/mmcblk0pX</code> initializes the card.<br />
<br />
==== Low-level formatting of other USB sticks ====<br />
One may try to write zeroes on the whole stick <br />
dd if=/dev/zero of=/dev/sdX bs=8192k<br />
hoping that the firmware will understand this as a request for low-level formatting. This seems to work with some sticks, but takes a long time and may stress the flash cells.<br />
<br />
== installation of crystallographic software ==<br />
* XDS and related programs (XDS-viewer, xdsstat, xdsgui, generate_XDS.INP): [http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Installation#Linux see XDSwiki]<br />
* CCP4: download and install from [http://www.ccp4.ac.uk/download/#os=linux CCP4 Homepage]<br />
* PHENIX: download and install from [https://www.phenix-online.org/download/ Phenix-online]<br />
* SHELX: download and install from [http://shelx.uni-ac.gwdg.de/SHELX/download.php SHELX Homepage] or together with [http://www.ccp4.ac.uk/download/#os=linux CCP4]<br />
* [http://webapps.embl-hamburg.de/bundle/download.php?hkl2mp=true&sitcom=false HKL2MAP]</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Bootable_Linux_USB_stick&diff=2522Bootable Linux USB stick2018-01-25T14:38:18Z<p>Karsten: /* generating images and copies of the stick */</p>
<hr />
<div>We regularly and successfully use USB sticks for courses where participants bring their own notebooks. The big benefit is that students<br />
learn Linux, and realize that they can easily use the hardware they already own. Current notebook hardware is by far fast enough for data<br />
processing, structure solution and coot visualization. What we found:<br />
# the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB or bigger; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do ''not'' have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk.<br />
# we use Fedora (FC23 and higher) because its hardware support is very good. We always use the 64bit distro.<br />
# The stick can be booted on MacBooks as well (press the <code>alt</code> key at the boot sound); their hardware works well with Fedora. For Windows clients (press F11 or F12 or sometimes F9 or F10 for the boot menu; if that does not work press F2 or DEL for the BIOS menu and change the boot order), one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes <code>powercfg -H off</code> (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from the stick because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should.<br />
# if the WiFi does not work out-of-the-box on MacBook Pro, connect temporarily to the Internet by other means (ethernet cable, WiFi via USB key, tether to your phone via Bluetooth), become root, and install the latest kernel and tools with <code>dnf install -y akmods kernel kernel-devel broadcom-wl --best --allowerasing</code>. After installing, reboot into the new kernel. <br />
# we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks.<br />
# any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync.<br />
<br />
To make students familiar with the sticks and how to boot them, one needs 30+ minutes and a few tutors. <br />
<br />
'''Some more details'''<br />
<br />
# we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes a biosboot partition (1MB; required for booting bios system from a GPT disk), an efi partition (e.g. 250MB; required for UEFI/EFI boot on PC's/MAC's) and the Linux partition, with 13.9GB, so that the sum of the four partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB stick; the /data partition then does not fit and cannot be used on that stick, but the operating system will then work on the small stick just as well. This requires that the /data partition is not fsck'ed automatically from /etc/fstab (0 in the 6th field).<br />
# it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use<br />
# it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up.<br />
# (if the installation not already does it for you) '''you should label the partitions and use the labels for mounting the partitions in /etc/fstab, alternatively use UUID's; don't use /dev/sda1 or the like because depending on the hardware it is booted on, after booting the stick, and depending on the actual hardware, the stick may be /dev/sdb or /dev/sdc or ... !'''<br />
<br />
<br />
<br />
== How to create a bootabled GPT-partitioned USB-stick with Fedora ==<br />
<br />
This will create a USB stick capable to<br />
* EFI boot on Macs<br />
* UEFI boot on PCs<br />
* BIOS boot on newer hardware (legacy boot / CSM enabled in BIOS)<br />
* BIOS boot on medium aged hardware<br />
* not boot on really old BIOS hardware which doesn't support booting from GPT (see below)<br />
<br />
=== GPT partition the stick using <code>gdisk</code> ===<br />
The example shows a 32 GB Sandisk Extreme:<br />
Part 1: +1G VFAT (0700) (Windows needs it to be the first partition) <br />
Part 2: +1M BIOSBOOT (ef02)<br />
Part 3: +250M EFI (ef00)<br />
Part 4: +13500M / (0083)<br />
Part 5: +15000M /mnt/data (0083)<br />
For performance, make sure that the big partitions are aligned to 8192-sector boundaries; gdisk default is 2048-sector boundaries - the default can be adjusted. If you use the above sizes and the + sign for specifying them, this happens to work out automatically.<br />
<br />
=== install Fedora on an UEFI machine (UEFI enabled ; LegacyBoot / CSM disabled) ===<br />
Note: in the following, whenever you see the X in sdX, you must fill in the appropriate drive letter (e.g. a or b or c or d or ...). <br />
sdX3: efi /boot/efi<br />
sdX4: ext4 / <br />
sdX5: ext4 /mnt/data<br />
<br />
Not much to say about the installation of programs (CCP4 ?, Phenix ?, XDS ?, ...?). You should also install the <br />
* hfsplus-tools RPM to access Mac harddisks,<br />
* fuse-exfat RPM to access exFAT SD cards,<br />
* binutils RPM to get the "strings" and other useful GNU commands,<br />
* atop, a better top; lbzip2, a parallel bzip2; xxdiff, a diff GUI; nedit, editor<br />
* tcsh RPM to allow installation of CCP4. <br />
dnf -y install tcsh xterm tk qt-x11 xxdiff atop nedit lbzip2 hfsplus-tools binutils<br />
<br />
=== adjust the USB-stick for UEFI/EFI boot (before reboot from chroot environment) ===<br />
<br />
chroot to the installed system:<br />
chroot /mnt/sysimage<br />
<br />
install Fedora updates:<br />
dnf -y update # Fedora's dnf is successor to yum<br />
<br />
<br />
configure grub2 for (U)EFI systems:<br />
<br />
*disable auto recognition of other installed Operating systems (specific to current computer), and<br />
*update grub2-efi.cfg<br />
echo 'GRUB_DISABLE_OS_PROBER=”true”' >> /etc/default/grub<br />
grub2-mkconfig -o /etc/grub2-efi.cfg<br />
(last step automatically puts UUID's to recognise the boot device in grub2-efi.cfg ; default after fresh Fedora installation is the device name which might be different on another computer)<br />
<br />
<br />
Performance tuning (not strictly required):<br />
<br />
We use ext4 filesystems and <code>data=writeback,nobarrier </code> in /etc/fstab. To be able to set these options also on the / filesystem, we use tune2fs to set both as default mount options on a Linux machine where the stick (dev/sdX) is just plugged in (i.e. not booted from):<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX4<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX5<br />
<br />
create & label vfat filesystem:<br />
mkfs.vfat /dev/sdX1<br />
dosfslabel /dev/sdX1 VFAT<br />
<br />
shutdown or reboot:<br />
exit<br />
systemctl poweroff (or reboot for testing)<br />
<br />
=== Install grub2 for BIOS boot (from chroot environment) ===<br />
<br />
boot Fedora Live DVD on a BIOS machine and chroot to the stick:<br />
mount /dev/sdX4 /mnt<br />
mount -o bind /dev /mnt/dev<br />
mount -t proc /proc /mnt/proc<br />
mount -t sysfs /sys /mnt/sys<br />
chroot /mnt<br />
<br />
install grub2 for BIOS boot and configure it:<br />
rm /boot/grub2/grubenv<br />
grub2-install /dev/sdX<br />
grub2-mkconfig -o /etc/grub2.cfg<br />
(on UEFI systems /boot/grub2/grubenv is a symlink to /boot/efi/EFI/fedora/grubenv (from package grub2-efi.rpm). The symlink has to be removed before grub2-install can finish successfully, it automatically creates a new file /boot/grub2/grubenv (as in package grub2.rpm)<br />
<br />
check /etc/grub2.cfg and if needed, change at all positions:<br />
“linuxefi” to “linux16”, and<br />
“intrdefi” to “intrd16”<br />
<br />
=== generating images and copies of the stick ===<br />
<br />
Save a compressed disk image - we use the parallel gzip program called pigz:<br />
pigz -c < /dev/sdX > usbstick.img.gz<br />
(time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB)<br />
write compressed image back to a stick:<br />
unpigz -c < usbstick.img.gz > /dev/sdX<br />
(speed: ~ 98MB/s)<br />
<br />
For creation/maintainance a bunch of sticks, a multiple port USB-HUB is a very usefull tool. E.g. the Raid Sonic Icy Box IB-AC6113 accepts 12 Sandisk Extreme sticks at once (port 7 has to stay empty due to spatial restrictions) and all 12 stick are recognized by the OS (CentOS7).<br />
<br />
== Hardware considerations ==<br />
<br />
Cheap USB sticks can hold a lot of data, but when it comes to writing small files (which happens when used as the operating system disk), they are either slow from the beginning, or their write performance quickly degrades as soon as free space becomes low for the first time. We have been using [https://crystalmark.info/software/CrystalDiskMark/index-e.html CrystalDiskMark] (Windows) for characterizing USB media: as a rule of thumb, the "Random write 4k blocks" disciplines should result in at least 1 MB/s, and for sequential writes the numbers should be >50 MB/s. However, CrystalDiskMark does not capture the effect of degrading write performance, which can only be mitigated by TRIMming the media - a feature that few USB sticks support (see below).<br />
<br />
The following is about performance and durability of the USB media. If you don't know what TRIM is and how it relates to flash media, read [https://en.wikipedia.org/wiki/Trim_(computing) this].<br />
<br />
=== types of USB media ===<br />
<br />
The suitability of the media makes a notable difference, in particular when the operating system or CCP4 is upgraded, or when e.g. a new Phenix version is installed.<br />
<br />
* very good: USB sticks that support TRIM. The ones known to us are the Sandisk Extreme USB sticks bought before 2017 (e.g. SDCZ80-032G 32GB for ~22€ or SDCZ80-064G 64GB for ~40€; unfortunately, these are no longer available, at least not at reasonable prices). These sticks are very fast (CrystalDiskMark scores >10MB/s for random 4K writes). The successor are the differently-named current (2018) Sandisk Extreme Pro USB sticks, but only 128 and 256 GB models exist, and they are expensive. There exist also fast and big, but costly Mushkin USB sticks which support TRIM. USB sticks which support TRIM also usually support [https://en.wikipedia.org/wiki/S.M.A.R.T. S.M.A.R.T.], which is useful for detecting problems.<br />
* good: we have verified that a USB3 adapter with (micro)SD card (separate items) is suitable. USB3 adapters are cheap (around 5€ at Amazon, like https://www.amazon.de/EX1-Kartenleser-Micro-Karte-Adapter/dp/B06XX3T219; cheaper at Ebay). microSD cards usually come with a SD adapter, and can be TRIMmed (but only in a SD slot; see below!). microSD cards ideally should be in the [https://en.wikipedia.org/wiki/Secure_Digital#Speed_class_rating Class 1 (A1) application performance class] since this is what is defined as requirement for use as extended operating system disk in Android smartphones. 64GB A1 microSD cards cost around 25€. In practice, slightly cheaper non-A1 microSD cards also seem to work well, but may be slower and may require more often TRIMming. The price/performance ratio of the microSD/USB3 combination is very good.<br />
* acceptable: USB sticks that do not support TRIM but are fast and high quality, e.g. Sandisk Extreme Go or Sandisk Ultra bought 2017 and later. However, their write performance may degrade with time (or rather, with the number of writes to a partition), and not much can be done about that.<br />
* inacceptable: tiny sticks like Sandisk Ultra Fit become hot, degrade fast and are unreliable. Cheap sticks are usually just too slow.<br />
<br />
The drawback of the "good" solution above is: from time to time (e.g. once every week), a knowledgeable person may want to<br />
# remove the microSD card from the USB3 adapter<br />
# insert it into the SD adapter, and that into a SD slot - it is usually auto-mounted by the operating system<br />
# run the <code>fstrim -v /mount/point</code> command (this takes a few seconds)<br />
# unmount the card, and revert steps 2. and 1.<br />
<br />
This naturally begs the question whether it wouldn't be much smarter to not use an USB adapter for microSD cards, but rather just use the SD adapter in a SD slot right away. That would have advantages - the SD card vanishes in most SD slots which is better than an USB stick sticking out, <code>fstrim</code> works, and possibly the SD controller is faster than the USB3 controller (depending on the notebook model). However, we tried to boot a microSD card in a SD lot and found that it doesn't boot, whereas it does boot in an USB adapter. Thus, setup of a microSD card for booting in SD slot needs to be investigated.<br />
<br />
=== TRIM ===<br />
==== USB sticks ==== <br />
TRIMming informs the firmware about blocks of the filesystem that do not contain file data. This is available for USB sticks for which <code>hdparm -I /dev/sdX</code> returns "Data Set Management TRIM supported". <br />
<br />
The <code>wiper.sh</code> script (part of the <code>hdparm</code> source distribution) TRIMs ext3/ext4/xfs filesystems, but does not reproducibly seem to work for our preferred SanDisk Extreme 32GB stick if the filesystem is mounted. This is probably due to the fact that this stick has a limitation of max 65536 blocks in one TRIM command (https://sourceforge.net/p/hdparm/bugs/63/). Since <code>wiper.sh</code> also works for unmounted filesystems, and the mapping of unused space is then obtained with a different tool, one should try with an unmounted filesystem as well - this worked for us in all cases tried so far (tested with [http://lightrush.ndoytchev.com/random-1/checkiftrimonext4isenabledandworking this script]). <br />
<br />
All other methods, like the <code>fstrim</code> command and the <code>discard</code> mount option '''do not work for USB sticks''', because the usb-storage kernel module does not pass the ATA TRIM command through the USB bridge and controller to the device. The same goes for [http://unix.stackexchange.com/questions/97143/utility-to-trim-unallocated-space-on-drive <code>blkdiscard</code>].<br />
<br />
==== (micro)SD cards ====<br />
These support TRIM when in a SD slot, but not when inserted in a USB adapter. When in a SD slot, the <code>lsblk -D</code> command reveals that the card can be TRIMmed, and <code>fstrim -v /dev/mmcXXXXX</code> works.<br />
<br />
=== fill empty space of a partition with “zeroes” ===<br />
This could be done after updating the software or data on the USB media, and before saving and compressing an image of it. <br />
USB sticks that support “Deterministic read data after TRIM” according to <code>hdparm -I /dev/sdX</code>, and microSD cards (which have zeroes in free space after <code>fstrim</code> in a SD slot) should be TRIMmed because this does not wear out the storage cells.<br />
<br />
Other media: as root<br />
dd if=/dev/zero of=/mountpoint_of_partition/delete.me bs=10M # this will stop when filesystem is full<br />
rm /mountpoint_of_partition/delete.me<br />
However, this stresses the flash cells; they sustain only a limited number of write cycles.<br />
<br />
=== initializing the media ===<br />
Low-level formatting restores the write performance, and removes any sensitive information (e.g. passwords) - important for workshops.<br />
==== Low-level formatting of "very good" USB sticks ==== <br />
<code>hdparm</code>'s (ENHANCED) SECURITY ERASE initializes the whole stick in a few seconds, and restores it to an almost factory-fresh condition, including zeroing the device. This works on all sticks where <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" but if used wrongly (e.g. on the wrong device, or with the wrong options, or ...) it may brick your device, or delete valuable data! So use at your own risk:<br />
* check if <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" and "not frozen" (to unfreeze a frozen diks, suspend your system with <code>pm-suspend</code> and wake it up again)<br />
* set disk password, and erase it (which unsets the password):<br />
hdparm --user-master u --security-set-pass Eins /dev/sdX<br />
hdparm --user-master u --security-erase-enhanced Eins /dev/sdX<br />
If you tried with --security-erase-enhanced, but the disk does not support it, then an error will occur and the disk will still be locked after the failed command, and you will not be able to write anything to it! If this happens, you can unlock it with <br />
hdparm --user-master u --security-unlock Eins /dev/sdX</code><br />
<br />
==== Low-level formatting of (micro)SD cards ====<br />
In a SD slot, <code>blkdiscard /dev/mmcXXXXXXX</code> initializes the card.<br />
<br />
==== Low-level formatting of other USB sticks ====<br />
One may try to write zeroes on the whole stick <br />
dd if=/dev/zero of=/dev/sdX bs=8192k<br />
hoping that the firmware will understand this as a request for low-level formatting. This seems to work with some sticks, but takes a long time and may stress the flash cells.<br />
<br />
== installation of crystallographic software ==<br />
* XDS and related programs (XDS-viewer, xdsstat, xdsgui, generate_XDS.INP): [http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Installation#Linux see XDSwiki]<br />
* CCP4: download and install from [http://www.ccp4.ac.uk/download/#os=linux CCP4 Homepage]<br />
* PHENIX: download and install from [https://www.phenix-online.org/download/ Phenix-online]<br />
* SHELX: download and install from [http://shelx.uni-ac.gwdg.de/SHELX/download.php SHELX Homepage] or together with [http://www.ccp4.ac.uk/download/#os=linux CCP4]<br />
* [http://webapps.embl-hamburg.de/bundle/download.php?hkl2mp=true&sitcom=false HKL2MAP]</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Test&diff=2485Test2017-11-24T15:03:14Z<p>Karsten: </p>
<hr />
<div>We test tex:<br />
<br />
<math><br />
F_{hkl} = \sum_{i=1}^{N} e^{-2\pi\imath\left( h\frac{x}{a}+k\frac{y}{b}+l\frac{z}{c}\right)}<br />
</math><br />
<br />
[[Image:Acrb.jpg|thumb|test]]<br />
<br />
All crystals should look like this (or better)<br />
[[Image:xtals.jpg||pretty xtals]]<br />
<br />
Problem? - I tried uploading xtals.png but Wiki does not let me upload, quoting that 'wrong file type or extension' - yet it allowed me to load up the same image in jpg format. Png is a nice (some would even say the best) format so it'd be good to be able to use it! -AGE<br />
This is fixed now. --[[User:Kay|Kay]] 10:12, 5 February 2008 (CET)<br />
<br/><br />
Remark: <br />
MIME type of PNG files was not correctly identified. To fix this, we added the lines<br />
# PNG<br />
1 string PNG image/png<br />
to the file /etc/httpd/conf/magic<br />
<br />
Logo prototype #1<br />
[[Image:logo1.png]]<br />
<br />
Logo prototype #2<br />
[[Image:logo2.png]]<br />
<br />
do you like any of these?<br />
<br />
Testing the gnuplot extension (the plot is generated in realtime, not a bitmap)<br />
<gnuplot><br />
set hidden3d<br />
set isosamples 50<br />
splot sin(x)*cos(y)<br />
</gnuplot></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Test&diff=2484Test2017-11-24T15:02:31Z<p>Karsten: </p>
<hr />
<div>We test tex:<br />
<br />
<math> 1 + b = mist </math><br />
<math><br />
F_{hkl} = \sum_{i=1}^{N} e^{-2\pi\imath\left( h\frac{x}{a}+k\frac{y}{b}+l\frac{z}{c}\right)}<br />
</math><br />
<br />
[[Image:Acrb.jpg|thumb|test]]<br />
<br />
All crystals should look like this (or better)<br />
[[Image:xtals.jpg||pretty xtals]]<br />
<br />
Problem? - I tried uploading xtals.png but Wiki does not let me upload, quoting that 'wrong file type or extension' - yet it allowed me to load up the same image in jpg format. Png is a nice (some would even say the best) format so it'd be good to be able to use it! -AGE<br />
This is fixed now. --[[User:Kay|Kay]] 10:12, 5 February 2008 (CET)<br />
<br/><br />
Remark: <br />
MIME type of PNG files was not correctly identified. To fix this, we added the lines<br />
# PNG<br />
1 string PNG image/png<br />
to the file /etc/httpd/conf/magic<br />
<br />
Logo prototype #1<br />
[[Image:logo1.png]]<br />
<br />
Logo prototype #2<br />
[[Image:logo2.png]]<br />
<br />
do you like any of these?<br />
<br />
Testing the gnuplot extension (the plot is generated in realtime, not a bitmap)<br />
<gnuplot><br />
set hidden3d<br />
set isosamples 50<br />
splot sin(x)*cos(y)<br />
</gnuplot></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Bootable_Linux_USB_stick&diff=2467Bootable Linux USB stick2017-08-10T12:38:34Z<p>Karsten: /* installation of crystallographic software */</p>
<hr />
<div>We regularly and successfully use USB sticks for courses where participants bring their own notebooks. The big benefit is that students<br />
learn Linux, and realize that they can easily use the hardware they already own. Current notebook hardware is by far fast enough for data<br />
processing, structure solution and coot visualization. What we found:<br />
# the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB or bigger; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do ''not'' have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk.<br />
# we use Fedora 23 (and higher) because its hardware support is very good. We always use the 64bit distro.<br />
# The stick can be booted on MacBooks as well (press the <code>alt</code> key at the boot sound); their hardware works well with Fedora. For Windows clients (press F11 or F12 or sometimes F9 or F10 for the boot menu; if that does not work press F2 or DEL for the BIOS menu and change the boot order), one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes <code>powercfg -H off</code> (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from the stick because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should.<br />
# if the WiFi does not work out-of-the-box on MacBook Pro, connect temporarily to the Internet by other means (ethernet cable, WiFi via USB key, tether to your phone via Bluetooth), become root, and install the latest kernel and tools with <code>dnf install -y akmods kernel kernel-devel broadcom-wl --best --allowerasing</code>. After installing, reboot into the new kernel. <br />
# we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks.<br />
# any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync.<br />
<br />
To make students familiar with the sticks and how to boot them, one needs 30+ minutes and a few tutors. <br />
<br />
'''Some more details'''<br />
<br />
# we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes a biosboot partition (1MB; required for booting bios system from a GPT disk), an efi partition (e.g. 250MB; required for UEFI/EFI boot on PC's/MAC's) and the Linux partition, with 13.9GB, so that the sum of the four partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB stick; the /data partition then does not fit and cannot be used on that stick, but the operating system will then work on the small stick just as well. This requires that the /data partition is not fsck'ed automatically from /etc/fstab (0 in the 6th field).<br />
# it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use<br />
# it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up.<br />
# (if the installation not already does it for you) '''you should label the partitions and use the labels for mounting the partitions in /etc/fstab, alternatively use UUID's; don't use /dev/sda1 or the like because depending on the hardware it is booted on, after booting the stick, and depending on the actual hardware, the stick may be /dev/sdb or /dev/sdc or ... !'''<br />
<br />
<br />
<br />
== How to create a bootabled GPT-partitioned USB-stick with Fedora 23 ==<br />
<br />
This will create a USB stick capable to<br />
* EFI boot on Macs<br />
* UEFI boot on PCs<br />
* BIOS boot on newer hardware (legacy boot / CSM enabled in BIOS)<br />
* BIOS boot on medium aged hardware<br />
* not boot on really old BIOS hardware which doesn't support booting from GPT (see below)<br />
<br />
=== GPT partition the stick using <code>gdisk</code> ===<br />
The example shows a 32 GB Sandisk Extreme:<br />
Part 1: +1G VFAT (0700) (Windows needs it to be the first partition) <br />
Part 2: +1M BIOSBOOT (ef02)<br />
Part 3: +250M EFI (ef00)<br />
Part 4: +13500M / (0083)<br />
Part 5: +15000M /mnt/data (0083)<br />
For performance, make sure that the big partitions are aligned to 8192-sector boundaries; gdisk default is 2048-sector boundaries - the default can be adjusted. If you use the above sizes and the + sign for specifying them, this happens to work out automatically.<br />
<br />
=== install Fedora23 on an UEFI machine (UEFI enabled ; LegacyBoot / CSM disabled) ===<br />
Note: in the following, whenever you see the X in sdX, you must fill in the appropriate drive letter (e.g. a or b or c or d or ...). <br />
sdX3: efi /boot/efi<br />
sdX4: ext4 / <br />
sdX5: ext4 /mnt/data<br />
<br />
Not much to say about the installation of programs (CCP4 ?, Phenix ?, XDS ?, ...?). You should also install the hfsplus-tools to enable Mac users to access their data on harddisk. Make sure to install the binutils RPM to get the "strings" and other useful GNU commands.<br />
<br />
=== adjust the USB-stick for UEFI/EFI boot (before reboot from chroot environment) ===<br />
<br />
chroot to the installed system:<br />
chroot /mnt/sysimage<br />
<br />
install Fedora updates:<br />
dnf -y update # dnf on FC23 is successor to yum<br />
<br />
<br />
configure grub2 for (U)EFI systems:<br />
<br />
*disable auto recognition of other installed Operating systems (specific to current computer), and<br />
*update grub2-efi.cfg<br />
echo 'GRUB_DISABLE_OS_PROBER=”true”' >> /etc/default/grub<br />
grub2-mkconfig -o /etc/grub2-efi.cfg<br />
(last step automatically puts UUID's to recognise the boot device in grub2-efi.cfg ; default after fresh Fedora installation is the device name which might be different on another computer)<br />
<br />
<br />
Performance tuning (not strictly required):<br />
<br />
We use ext4 filesystems and <code>data=writeback,nobarrier </code> in /etc/fstab. To be able to set these options also on the / filesystem, we use tune2fs to set both as default mount options on a Linux machine where the stick (dev/sdX) is just plugged in (i.e. not booted from):<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX4<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX5<br />
<br />
create & label vfat filesystem:<br />
mkfs.vfat /dev/sdX1<br />
dosfslabel /dev/sdX1 VFAT<br />
<br />
shutdown or reboot:<br />
exit<br />
systemctl poweroff (or reboot for testing)<br />
<br />
=== Install grub2 for BIOS boot (from chroot environment) ===<br />
<br />
boot Fedora Live DVD on a BIOS machine and chroot to the stick:<br />
mount /dev/sdX4 /mnt<br />
mount -o bind /dev /mnt/dev<br />
mount -t proc /proc /mnt/proc<br />
mount -t sysfs /sys /mnt/sys<br />
chroot /mnt<br />
<br />
install grub2 for BIOS boot and configure it:<br />
rm /boot/grub2/grubenv<br />
grub2-install /dev/sdX<br />
grub2-mkconfig -o /etc/grub2.cfg<br />
(on UEFI systems /boot/grub2/grubenv is a symlink to /boot/efi/EFI/fedora/grubenv (from package grub2-efi.rpm). The symlink has to be removed before grub2-install can finish successfully, it automatically creates a new file /boot/grub2/grubenv (as in package grub2.rpm)<br />
<br />
<br />
check /etc/grub2.cfg and if needed, change at all positions:<br />
“linuxefi” to “linux16”, and<br />
“intrdefi” to “intrd16”<br />
<br />
=== Problems on old BIOS hardware ===<br />
Some old computers still might not boot since they don't support booting from GPT.<br />
This can be fixed to boot on such old hardware, however the stick afterwards cannot boot on EFI/UEFI systems anymore:<br />
<br />
*set the boot flag (active) on the single 0xEE partition in the protective MBR by using fdisk version <= 2.22 (versions 1.23 and newer have GPT support and thus don't make changes to the protective MBR but to the GPT) (I used fdisk (util-linux-ng 2.17.2) on a ScientificLinux 6.7 machine)<br />
fdisk /dev/sdX<br />
a<br />
part 1<br />
w<br />
<br />
*recompute CHS values using option “h” in gdisk's expert menu:<br />
gdisk /dev/sdX<br />
x<br />
h<br />
m<br />
q<br />
<br />
this can be reversed:<br />
- toggle bootflag off in old fdisk<br />
- recompute CHS values in gdisk<br />
<br />
(using a sgdisk backup of the GPT should also be fine)<br />
<br />
helpful links:<br />
<br />
http://www.rodsbooks.com/gdisk/bios.html#bios --> option h of gdisks experts menu did the trick!<br />
<br />
https://fedoraproject.org/wiki/GRUB_2?rd=Grub2#Updating_GRUB_2_configuration_on_BIOS_systems<br />
<br />
=== generating images and copies of the stick ===<br />
<br />
Save a compressed disk image - we use the parallel gzip program called pigz:<br />
pigz -c < /dev/sdX > usbstick.img.gz<br />
(time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB)<br />
write compressed image back to a stick:<br />
unpigz -c usbstick.img.gz > /dev/sdX<br />
(speed: ~ 98MB/s)<br />
<br />
For creation/maintainance a bunch of sticks, a multiple port USB-HUB is a very usefull tool. E.g. the Raid Sonic Icy Box IB-AC6113 accepts 12 Sandisk Extreme sticks at once (port 7 has to stay empty due to spatial restrictions) and all 12 stick are recognized by the OS (CentOS7).<br />
<br />
=== for techies: taking care of the stick ===<br />
<br />
The following is about performance and durability of the USB stick, and reading it is not required for the functionality described above. If you don't know what TRIM is and how it relates to flash media, this section is probably not for you.<br />
<br />
To fill empty part of all partitions on the stick with “zeros”: for each partition, do (as root)<br />
dd if=/dev/zero of=/mountpoint_of_partition/delete.me bs=10M # this will stop when filesystem is full<br />
rm /mountpoint_of_partition/delete.me<br />
<br />
This could be done after deleting large amounts of data from the USB stick, and before saving and compressing an image of it. USB sticks that support “Deterministic read data after TRIM” should instead be TRIMmed (see below) because this does not wear out the storage cells.<br />
<br />
==== Initializing the stick ====<br />
<code>hdparm</code>'s (ENHANCED) SECURITY ERASE initializes the whole stick in a few seconds, and restores it to an almost factory-fresh condition, including zeroing the device. This works on all sticks where <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" but if used wrongly (e.g. on the wrong device, or with the wrong options, or ...) it may brick your device, or delete valuable data! So use at your own risk:<br />
* check if <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" and "not frozen" (to unfreeze a frozen diks, suspend your system with <code>pm-suspend</code> and wake it up again)<br />
* set disk password, and erase it (which unsets the password):<br />
hdparm --user-master u --security-set-pass Eins /dev/sdX<br />
hdparm --user-master u --security-erase-enhanced Eins /dev/sdX<br />
If you tried with --security-erase-enhanced, but the disk does not support it, then an error will occur and the disk will still be locked after the failed command, and you will not be able to write anything to it! If this happens, you can unlock it with <br />
hdparm --user-master u --security-unlock Eins /dev/sdX</code><br />
<br />
==== TRIMming the stick ====<br />
TRIMming informs the USB stick firmware about blocks of the filesystem that do not contain file data. This is available for USB sticks for which <code>hdparm -I /dev/sdX</code> returns "Data Set Management TRIM supported"; if that command also returns "Deterministic read data after TRIM" then zeros are returned upon reads of TRIMmed blocks.<br />
<br />
The <code>wiper.sh</code> script (part of the <code>hdparm</code> source distribution) TRIMs ext3/ext4/xfs filesystems, but does not reproducibly seem to work for the SanDisk Extreme 32GB stick if the filesystem is mounted. This is probably due to the fact that this stick has a limitation of max 65536 blocks in one TRIM command (https://sourceforge.net/p/hdparm/bugs/63/). Since <code>wiper.sh</code> also works for unmounted filesystems, and the mapping of unused space is then obtained with a different tool, one should try with an unmounted filesystem as well - this worked for us in all cases tried so far (tested with [http://lightrush.ndoytchev.com/random-1/checkiftrimonext4isenabledandworking this script]).<br />
<br />
All other methods, like the <code>fstrim</code> command and the <code>discard</code> mount option '''do not work for USB sticks''', presumably because the usb-storage kernel module does not pass the ATA trim command through the USB bridge and controller to the device. The same goes for [http://unix.stackexchange.com/questions/97143/utility-to-trim-unallocated-space-on-drive <code>blkdiscard</code>].<br />
<br />
== installation of crystallographic software ==<br />
* XDS and related programs (XDS-viewer, xdsstat, xdsgui, generate_XDS.INP): [http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Installation#Linux see XDSwiki]<br />
* CCP4: download and install from [http://www.ccp4.ac.uk/download/#os=linux CCP4 Hompage]<br />
* PHENIX: download and install from [https://www.phenix-online.org/download/ Phenix-online]<br />
* SHELX: download and install from [http://shelx.uni-ac.gwdg.de/SHELX/download.php SHELX Homepage] or together with [http://www.ccp4.ac.uk/download/#os=linux CCP4]<br />
* [http://webapps.embl-hamburg.de/bundle/download.php?hkl2mp=true&sitcom=false HKL2MAP]<br />
install required (+ additional) software from the Fedora repository:<br />
dnf -y install tcsh xterm tk qt-x11 xxdiff atop nedit lbzip2</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Bootable_Linux_USB_stick&diff=2466Bootable Linux USB stick2017-08-10T11:53:29Z<p>Karsten: /* installation of crystallographic software */</p>
<hr />
<div>We regularly and successfully use USB sticks for courses where participants bring their own notebooks. The big benefit is that students<br />
learn Linux, and realize that they can easily use the hardware they already own. Current notebook hardware is by far fast enough for data<br />
processing, structure solution and coot visualization. What we found:<br />
# the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB or bigger; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do ''not'' have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk.<br />
# we use Fedora 23 (and higher) because its hardware support is very good. We always use the 64bit distro.<br />
# The stick can be booted on MacBooks as well (press the <code>alt</code> key at the boot sound); their hardware works well with Fedora. For Windows clients (press F11 or F12 or sometimes F9 or F10 for the boot menu; if that does not work press F2 or DEL for the BIOS menu and change the boot order), one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes <code>powercfg -H off</code> (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from the stick because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should.<br />
# if the WiFi does not work out-of-the-box on MacBook Pro, connect temporarily to the Internet by other means (ethernet cable, WiFi via USB key, tether to your phone via Bluetooth), become root, and install the latest kernel and tools with <code>dnf install -y akmods kernel kernel-devel broadcom-wl --best --allowerasing</code>. After installing, reboot into the new kernel. <br />
# we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks.<br />
# any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync.<br />
<br />
To make students familiar with the sticks and how to boot them, one needs 30+ minutes and a few tutors. <br />
<br />
'''Some more details'''<br />
<br />
# we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes a biosboot partition (1MB; required for booting bios system from a GPT disk), an efi partition (e.g. 250MB; required for UEFI/EFI boot on PC's/MAC's) and the Linux partition, with 13.9GB, so that the sum of the four partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB stick; the /data partition then does not fit and cannot be used on that stick, but the operating system will then work on the small stick just as well. This requires that the /data partition is not fsck'ed automatically from /etc/fstab (0 in the 6th field).<br />
# it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use<br />
# it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up.<br />
# (if the installation not already does it for you) '''you should label the partitions and use the labels for mounting the partitions in /etc/fstab, alternatively use UUID's; don't use /dev/sda1 or the like because depending on the hardware it is booted on, after booting the stick, and depending on the actual hardware, the stick may be /dev/sdb or /dev/sdc or ... !'''<br />
<br />
<br />
<br />
== How to create a bootabled GPT-partitioned USB-stick with Fedora 23 ==<br />
<br />
This will create a USB stick capable to<br />
* EFI boot on Macs<br />
* UEFI boot on PCs<br />
* BIOS boot on newer hardware (legacy boot / CSM enabled in BIOS)<br />
* BIOS boot on medium aged hardware<br />
* not boot on really old BIOS hardware which doesn't support booting from GPT (see below)<br />
<br />
=== GPT partition the stick using <code>gdisk</code> ===<br />
The example shows a 32 GB Sandisk Extreme:<br />
Part 1: +1G VFAT (0700) (Windows needs it to be the first partition) <br />
Part 2: +1M BIOSBOOT (ef02)<br />
Part 3: +250M EFI (ef00)<br />
Part 4: +13500M / (0083)<br />
Part 5: +15000M /mnt/data (0083)<br />
For performance, make sure that the big partitions are aligned to 8192-sector boundaries; gdisk default is 2048-sector boundaries - the default can be adjusted. If you use the above sizes and the + sign for specifying them, this happens to work out automatically.<br />
<br />
=== install Fedora23 on an UEFI machine (UEFI enabled ; LegacyBoot / CSM disabled) ===<br />
Note: in the following, whenever you see the X in sdX, you must fill in the appropriate drive letter (e.g. a or b or c or d or ...). <br />
sdX3: efi /boot/efi<br />
sdX4: ext4 / <br />
sdX5: ext4 /mnt/data<br />
<br />
Not much to say about the installation of programs (CCP4 ?, Phenix ?, XDS ?, ...?). You should also install the hfsplus-tools to enable Mac users to access their data on harddisk. Make sure to install the binutils RPM to get the "strings" and other useful GNU commands.<br />
<br />
=== adjust the USB-stick for UEFI/EFI boot (before reboot from chroot environment) ===<br />
<br />
chroot to the installed system:<br />
chroot /mnt/sysimage<br />
<br />
install Fedora updates:<br />
dnf -y update # dnf on FC23 is successor to yum<br />
<br />
<br />
configure grub2 for (U)EFI systems:<br />
<br />
*disable auto recognition of other installed Operating systems (specific to current computer), and<br />
*update grub2-efi.cfg<br />
echo 'GRUB_DISABLE_OS_PROBER=”true”' >> /etc/default/grub<br />
grub2-mkconfig -o /etc/grub2-efi.cfg<br />
(last step automatically puts UUID's to recognise the boot device in grub2-efi.cfg ; default after fresh Fedora installation is the device name which might be different on another computer)<br />
<br />
<br />
Performance tuning (not strictly required):<br />
<br />
We use ext4 filesystems and <code>data=writeback,nobarrier </code> in /etc/fstab. To be able to set these options also on the / filesystem, we use tune2fs to set both as default mount options on a Linux machine where the stick (dev/sdX) is just plugged in (i.e. not booted from):<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX4<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX5<br />
<br />
create & label vfat filesystem:<br />
mkfs.vfat /dev/sdX1<br />
dosfslabel /dev/sdX1 VFAT<br />
<br />
shutdown or reboot:<br />
exit<br />
systemctl poweroff (or reboot for testing)<br />
<br />
=== Install grub2 for BIOS boot (from chroot environment) ===<br />
<br />
boot Fedora Live DVD on a BIOS machine and chroot to the stick:<br />
mount /dev/sdX4 /mnt<br />
mount -o bind /dev /mnt/dev<br />
mount -t proc /proc /mnt/proc<br />
mount -t sysfs /sys /mnt/sys<br />
chroot /mnt<br />
<br />
install grub2 for BIOS boot and configure it:<br />
rm /boot/grub2/grubenv<br />
grub2-install /dev/sdX<br />
grub2-mkconfig -o /etc/grub2.cfg<br />
(on UEFI systems /boot/grub2/grubenv is a symlink to /boot/efi/EFI/fedora/grubenv (from package grub2-efi.rpm). The symlink has to be removed before grub2-install can finish successfully, it automatically creates a new file /boot/grub2/grubenv (as in package grub2.rpm)<br />
<br />
<br />
check /etc/grub2.cfg and if needed, change at all positions:<br />
“linuxefi” to “linux16”, and<br />
“intrdefi” to “intrd16”<br />
<br />
=== Problems on old BIOS hardware ===<br />
Some old computers still might not boot since they don't support booting from GPT.<br />
This can be fixed to boot on such old hardware, however the stick afterwards cannot boot on EFI/UEFI systems anymore:<br />
<br />
*set the boot flag (active) on the single 0xEE partition in the protective MBR by using fdisk version <= 2.22 (versions 1.23 and newer have GPT support and thus don't make changes to the protective MBR but to the GPT) (I used fdisk (util-linux-ng 2.17.2) on a ScientificLinux 6.7 machine)<br />
fdisk /dev/sdX<br />
a<br />
part 1<br />
w<br />
<br />
*recompute CHS values using option “h” in gdisk's expert menu:<br />
gdisk /dev/sdX<br />
x<br />
h<br />
m<br />
q<br />
<br />
this can be reversed:<br />
- toggle bootflag off in old fdisk<br />
- recompute CHS values in gdisk<br />
<br />
(using a sgdisk backup of the GPT should also be fine)<br />
<br />
helpful links:<br />
<br />
http://www.rodsbooks.com/gdisk/bios.html#bios --> option h of gdisks experts menu did the trick!<br />
<br />
https://fedoraproject.org/wiki/GRUB_2?rd=Grub2#Updating_GRUB_2_configuration_on_BIOS_systems<br />
<br />
=== generating images and copies of the stick ===<br />
<br />
Save a compressed disk image - we use the parallel gzip program called pigz:<br />
pigz -c < /dev/sdX > usbstick.img.gz<br />
(time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB)<br />
write compressed image back to a stick:<br />
unpigz -c usbstick.img.gz > /dev/sdX<br />
(speed: ~ 98MB/s)<br />
<br />
For creation/maintainance a bunch of sticks, a multiple port USB-HUB is a very usefull tool. E.g. the Raid Sonic Icy Box IB-AC6113 accepts 12 Sandisk Extreme sticks at once (port 7 has to stay empty due to spatial restrictions) and all 12 stick are recognized by the OS (CentOS7).<br />
<br />
=== for techies: taking care of the stick ===<br />
<br />
The following is about performance and durability of the USB stick, and reading it is not required for the functionality described above. If you don't know what TRIM is and how it relates to flash media, this section is probably not for you.<br />
<br />
To fill empty part of all partitions on the stick with “zeros”: for each partition, do (as root)<br />
dd if=/dev/zero of=/mountpoint_of_partition/delete.me bs=10M # this will stop when filesystem is full<br />
rm /mountpoint_of_partition/delete.me<br />
<br />
This could be done after deleting large amounts of data from the USB stick, and before saving and compressing an image of it. USB sticks that support “Deterministic read data after TRIM” should instead be TRIMmed (see below) because this does not wear out the storage cells.<br />
<br />
==== Initializing the stick ====<br />
<code>hdparm</code>'s (ENHANCED) SECURITY ERASE initializes the whole stick in a few seconds, and restores it to an almost factory-fresh condition, including zeroing the device. This works on all sticks where <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" but if used wrongly (e.g. on the wrong device, or with the wrong options, or ...) it may brick your device, or delete valuable data! So use at your own risk:<br />
* check if <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" and "not frozen" (to unfreeze a frozen diks, suspend your system with <code>pm-suspend</code> and wake it up again)<br />
* set disk password, and erase it (which unsets the password):<br />
hdparm --user-master u --security-set-pass Eins /dev/sdX<br />
hdparm --user-master u --security-erase-enhanced Eins /dev/sdX<br />
If you tried with --security-erase-enhanced, but the disk does not support it, then an error will occur and the disk will still be locked after the failed command, and you will not be able to write anything to it! If this happens, you can unlock it with <br />
hdparm --user-master u --security-unlock Eins /dev/sdX</code><br />
<br />
==== TRIMming the stick ====<br />
TRIMming informs the USB stick firmware about blocks of the filesystem that do not contain file data. This is available for USB sticks for which <code>hdparm -I /dev/sdX</code> returns "Data Set Management TRIM supported"; if that command also returns "Deterministic read data after TRIM" then zeros are returned upon reads of TRIMmed blocks.<br />
<br />
The <code>wiper.sh</code> script (part of the <code>hdparm</code> source distribution) TRIMs ext3/ext4/xfs filesystems, but does not reproducibly seem to work for the SanDisk Extreme 32GB stick if the filesystem is mounted. This is probably due to the fact that this stick has a limitation of max 65536 blocks in one TRIM command (https://sourceforge.net/p/hdparm/bugs/63/). Since <code>wiper.sh</code> also works for unmounted filesystems, and the mapping of unused space is then obtained with a different tool, one should try with an unmounted filesystem as well - this worked for us in all cases tried so far (tested with [http://lightrush.ndoytchev.com/random-1/checkiftrimonext4isenabledandworking this script]).<br />
<br />
All other methods, like the <code>fstrim</code> command and the <code>discard</code> mount option '''do not work for USB sticks''', presumably because the usb-storage kernel module does not pass the ATA trim command through the USB bridge and controller to the device. The same goes for [http://unix.stackexchange.com/questions/97143/utility-to-trim-unallocated-space-on-drive <code>blkdiscard</code>].<br />
<br />
== installation of crystallographic software ==<br />
* XDS and related programs (XDS-viewer, xdsstat, xdsgui, generate_XDS.INP): [http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Installation#Linux see XDSwiki]<br />
* CCP4: download and install from [http://www.ccp4.ac.uk/download/#os=linux CCP4 Hompage]<br />
* PHENIX: download and install from [https://www.phenix-online.org/download/ Phenix-online]<br />
* SHELX: download and install from [http://shelx.uni-ac.gwdg.de/SHELX/download.php SHELX Homepage] or together with [http://www.ccp4.ac.uk/download/#os=linux CCP4]<br />
* [http://webapps.embl-hamburg.de/bundle/download.php?hkl2mp=true&sitcom=false HKL2MAP]<br />
install required (+ additional) software from the Fedora repository<br />
dnf -y install tcsh xterm tk qt-x11 xxdiff atop nedit lbzip2</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Bootable_Linux_USB_stick&diff=2465Bootable Linux USB stick2017-08-10T11:14:03Z<p>Karsten: </p>
<hr />
<div>We regularly and successfully use USB sticks for courses where participants bring their own notebooks. The big benefit is that students<br />
learn Linux, and realize that they can easily use the hardware they already own. Current notebook hardware is by far fast enough for data<br />
processing, structure solution and coot visualization. What we found:<br />
# the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB or bigger; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do ''not'' have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk.<br />
# we use Fedora 23 (and higher) because its hardware support is very good. We always use the 64bit distro.<br />
# The stick can be booted on MacBooks as well (press the <code>alt</code> key at the boot sound); their hardware works well with Fedora. For Windows clients (press F11 or F12 or sometimes F9 or F10 for the boot menu; if that does not work press F2 or DEL for the BIOS menu and change the boot order), one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes <code>powercfg -H off</code> (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from the stick because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should.<br />
# if the WiFi does not work out-of-the-box on MacBook Pro, connect temporarily to the Internet by other means (ethernet cable, WiFi via USB key, tether to your phone via Bluetooth), become root, and install the latest kernel and tools with <code>dnf install -y akmods kernel kernel-devel broadcom-wl --best --allowerasing</code>. After installing, reboot into the new kernel. <br />
# we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks.<br />
# any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync.<br />
<br />
To make students familiar with the sticks and how to boot them, one needs 30+ minutes and a few tutors. <br />
<br />
'''Some more details'''<br />
<br />
# we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes a biosboot partition (1MB; required for booting bios system from a GPT disk), an efi partition (e.g. 250MB; required for UEFI/EFI boot on PC's/MAC's) and the Linux partition, with 13.9GB, so that the sum of the four partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB stick; the /data partition then does not fit and cannot be used on that stick, but the operating system will then work on the small stick just as well. This requires that the /data partition is not fsck'ed automatically from /etc/fstab (0 in the 6th field).<br />
# it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use<br />
# it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up.<br />
# (if the installation not already does it for you) '''you should label the partitions and use the labels for mounting the partitions in /etc/fstab, alternatively use UUID's; don't use /dev/sda1 or the like because depending on the hardware it is booted on, after booting the stick, and depending on the actual hardware, the stick may be /dev/sdb or /dev/sdc or ... !'''<br />
<br />
<br />
<br />
== How to create a bootabled GPT-partitioned USB-stick with Fedora 23 ==<br />
<br />
This will create a USB stick capable to<br />
* EFI boot on Macs<br />
* UEFI boot on PCs<br />
* BIOS boot on newer hardware (legacy boot / CSM enabled in BIOS)<br />
* BIOS boot on medium aged hardware<br />
* not boot on really old BIOS hardware which doesn't support booting from GPT (see below)<br />
<br />
=== GPT partition the stick using <code>gdisk</code> ===<br />
The example shows a 32 GB Sandisk Extreme:<br />
Part 1: +1G VFAT (0700) (Windows needs it to be the first partition) <br />
Part 2: +1M BIOSBOOT (ef02)<br />
Part 3: +250M EFI (ef00)<br />
Part 4: +13500M / (0083)<br />
Part 5: +15000M /mnt/data (0083)<br />
For performance, make sure that the big partitions are aligned to 8192-sector boundaries; gdisk default is 2048-sector boundaries - the default can be adjusted. If you use the above sizes and the + sign for specifying them, this happens to work out automatically.<br />
<br />
=== install Fedora23 on an UEFI machine (UEFI enabled ; LegacyBoot / CSM disabled) ===<br />
Note: in the following, whenever you see the X in sdX, you must fill in the appropriate drive letter (e.g. a or b or c or d or ...). <br />
sdX3: efi /boot/efi<br />
sdX4: ext4 / <br />
sdX5: ext4 /mnt/data<br />
<br />
Not much to say about the installation of programs (CCP4 ?, Phenix ?, XDS ?, ...?). You should also install the hfsplus-tools to enable Mac users to access their data on harddisk. Make sure to install the binutils RPM to get the "strings" and other useful GNU commands.<br />
<br />
=== adjust the USB-stick for UEFI/EFI boot (before reboot from chroot environment) ===<br />
<br />
chroot to the installed system:<br />
chroot /mnt/sysimage<br />
<br />
install Fedora updates:<br />
dnf -y update # dnf on FC23 is successor to yum<br />
<br />
<br />
configure grub2 for (U)EFI systems:<br />
<br />
*disable auto recognition of other installed Operating systems (specific to current computer), and<br />
*update grub2-efi.cfg<br />
echo 'GRUB_DISABLE_OS_PROBER=”true”' >> /etc/default/grub<br />
grub2-mkconfig -o /etc/grub2-efi.cfg<br />
(last step automatically puts UUID's to recognise the boot device in grub2-efi.cfg ; default after fresh Fedora installation is the device name which might be different on another computer)<br />
<br />
<br />
Performance tuning (not strictly required):<br />
<br />
We use ext4 filesystems and <code>data=writeback,nobarrier </code> in /etc/fstab. To be able to set these options also on the / filesystem, we use tune2fs to set both as default mount options on a Linux machine where the stick (dev/sdX) is just plugged in (i.e. not booted from):<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX4<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX5<br />
<br />
create & label vfat filesystem:<br />
mkfs.vfat /dev/sdX1<br />
dosfslabel /dev/sdX1 VFAT<br />
<br />
shutdown or reboot:<br />
exit<br />
systemctl poweroff (or reboot for testing)<br />
<br />
=== Install grub2 for BIOS boot (from chroot environment) ===<br />
<br />
boot Fedora Live DVD on a BIOS machine and chroot to the stick:<br />
mount /dev/sdX4 /mnt<br />
mount -o bind /dev /mnt/dev<br />
mount -t proc /proc /mnt/proc<br />
mount -t sysfs /sys /mnt/sys<br />
chroot /mnt<br />
<br />
install grub2 for BIOS boot and configure it:<br />
rm /boot/grub2/grubenv<br />
grub2-install /dev/sdX<br />
grub2-mkconfig -o /etc/grub2.cfg<br />
(on UEFI systems /boot/grub2/grubenv is a symlink to /boot/efi/EFI/fedora/grubenv (from package grub2-efi.rpm). The symlink has to be removed before grub2-install can finish successfully, it automatically creates a new file /boot/grub2/grubenv (as in package grub2.rpm)<br />
<br />
<br />
check /etc/grub2.cfg and if needed, change at all positions:<br />
“linuxefi” to “linux16”, and<br />
“intrdefi” to “intrd16”<br />
<br />
=== Problems on old BIOS hardware ===<br />
Some old computers still might not boot since they don't support booting from GPT.<br />
This can be fixed to boot on such old hardware, however the stick afterwards cannot boot on EFI/UEFI systems anymore:<br />
<br />
*set the boot flag (active) on the single 0xEE partition in the protective MBR by using fdisk version <= 2.22 (versions 1.23 and newer have GPT support and thus don't make changes to the protective MBR but to the GPT) (I used fdisk (util-linux-ng 2.17.2) on a ScientificLinux 6.7 machine)<br />
fdisk /dev/sdX<br />
a<br />
part 1<br />
w<br />
<br />
*recompute CHS values using option “h” in gdisk's expert menu:<br />
gdisk /dev/sdX<br />
x<br />
h<br />
m<br />
q<br />
<br />
this can be reversed:<br />
- toggle bootflag off in old fdisk<br />
- recompute CHS values in gdisk<br />
<br />
(using a sgdisk backup of the GPT should also be fine)<br />
<br />
helpful links:<br />
<br />
http://www.rodsbooks.com/gdisk/bios.html#bios --> option h of gdisks experts menu did the trick!<br />
<br />
https://fedoraproject.org/wiki/GRUB_2?rd=Grub2#Updating_GRUB_2_configuration_on_BIOS_systems<br />
<br />
=== generating images and copies of the stick ===<br />
<br />
Save a compressed disk image - we use the parallel gzip program called pigz:<br />
pigz -c < /dev/sdX > usbstick.img.gz<br />
(time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB)<br />
write compressed image back to a stick:<br />
unpigz -c usbstick.img.gz > /dev/sdX<br />
(speed: ~ 98MB/s)<br />
<br />
For creation/maintainance a bunch of sticks, a multiple port USB-HUB is a very usefull tool. E.g. the Raid Sonic Icy Box IB-AC6113 accepts 12 Sandisk Extreme sticks at once (port 7 has to stay empty due to spatial restrictions) and all 12 stick are recognized by the OS (CentOS7).<br />
<br />
=== for techies: taking care of the stick ===<br />
<br />
The following is about performance and durability of the USB stick, and reading it is not required for the functionality described above. If you don't know what TRIM is and how it relates to flash media, this section is probably not for you.<br />
<br />
To fill empty part of all partitions on the stick with “zeros”: for each partition, do (as root)<br />
dd if=/dev/zero of=/mountpoint_of_partition/delete.me bs=10M # this will stop when filesystem is full<br />
rm /mountpoint_of_partition/delete.me<br />
<br />
This could be done after deleting large amounts of data from the USB stick, and before saving and compressing an image of it. USB sticks that support “Deterministic read data after TRIM” should instead be TRIMmed (see below) because this does not wear out the storage cells.<br />
<br />
==== Initializing the stick ====<br />
<code>hdparm</code>'s (ENHANCED) SECURITY ERASE initializes the whole stick in a few seconds, and restores it to an almost factory-fresh condition, including zeroing the device. This works on all sticks where <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" but if used wrongly (e.g. on the wrong device, or with the wrong options, or ...) it may brick your device, or delete valuable data! So use at your own risk:<br />
* check if <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" and "not frozen" (to unfreeze a frozen diks, suspend your system with <code>pm-suspend</code> and wake it up again)<br />
* set disk password, and erase it (which unsets the password):<br />
hdparm --user-master u --security-set-pass Eins /dev/sdX<br />
hdparm --user-master u --security-erase-enhanced Eins /dev/sdX<br />
If you tried with --security-erase-enhanced, but the disk does not support it, then an error will occur and the disk will still be locked after the failed command, and you will not be able to write anything to it! If this happens, you can unlock it with <br />
hdparm --user-master u --security-unlock Eins /dev/sdX</code><br />
<br />
==== TRIMming the stick ====<br />
TRIMming informs the USB stick firmware about blocks of the filesystem that do not contain file data. This is available for USB sticks for which <code>hdparm -I /dev/sdX</code> returns "Data Set Management TRIM supported"; if that command also returns "Deterministic read data after TRIM" then zeros are returned upon reads of TRIMmed blocks.<br />
<br />
The <code>wiper.sh</code> script (part of the <code>hdparm</code> source distribution) TRIMs ext3/ext4/xfs filesystems, but does not reproducibly seem to work for the SanDisk Extreme 32GB stick if the filesystem is mounted. This is probably due to the fact that this stick has a limitation of max 65536 blocks in one TRIM command (https://sourceforge.net/p/hdparm/bugs/63/). Since <code>wiper.sh</code> also works for unmounted filesystems, and the mapping of unused space is then obtained with a different tool, one should try with an unmounted filesystem as well - this worked for us in all cases tried so far (tested with [http://lightrush.ndoytchev.com/random-1/checkiftrimonext4isenabledandworking this script]).<br />
<br />
All other methods, like the <code>fstrim</code> command and the <code>discard</code> mount option '''do not work for USB sticks''', presumably because the usb-storage kernel module does not pass the ATA trim command through the USB bridge and controller to the device. The same goes for [http://unix.stackexchange.com/questions/97143/utility-to-trim-unallocated-space-on-drive <code>blkdiscard</code>].<br />
<br />
== installation of crystallographic software ==<br />
* XDS and related programs (XDS-viewer, xdsstat, xdsgui, generate_XDS.INP): [http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Installation#Linux see XDSwiki]<br />
* CCP4: download and install from [http://www.ccp4.ac.uk/download/#os=linux CCP4 Hompage]<br />
* PHENIX: download and install from [https://www.phenix-online.org/download/ Phenix-online]<br />
* SHELX: download and install from [http://shelx.uni-ac.gwdg.de/SHELX/download.php SHELX Homepage] or together with [http://www.ccp4.ac.uk/download/#os=linux CCP4]<br />
install required (+ additional) software from the Fedora repository<br />
dnf -y install tcsh xterm tk qt-x11 xxdiff atop nedit lbzip2</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Bootable_Linux_USB_stick&diff=2424Bootable Linux USB stick2016-11-25T14:30:15Z<p>Karsten: /* generating images and copies of the stick */</p>
<hr />
<div>We regularly and successfully use USB sticks for courses where participants bring their own notebooks. The big benefit is that students<br />
learn Linux, and realize that they can easily use the hardware they already own. Current notebook hardware is by far fast enough for data<br />
processing, structure solution and coot visualization. What we found:<br />
# the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB or bigger; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do ''not'' have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk.<br />
# we use Fedora 23 (and higher) because its hardware support is very good. We always use the 64bit distro.<br />
# The stick can be booted on MacBooks as well (press the <code>alt</code> key at the boot sound); their hardware works well with Fedora. For Windows clients (press F11 or F12 or sometimes F9 or F10 for the boot menu; if that does not work press F2 or DEL for the BIOS menu and change the boot order), one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes <code>powercfg -H off</code> (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from the stick because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should.<br />
# if the WiFi does not work out-of-the-box on MacBook Pro, connect temporarily to the Internet by other means (ethernet cable, WiFi via USB key, tether to your phone via Bluetooth), become root, and install the latest kernel and tools with <code>dnf install -y akmods kernel kernel-devel broadcom-wl --best --allowerasing</code>. After installing, reboot into the new kernel. <br />
# we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks.<br />
# any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync.<br />
<br />
To make students familiar with the sticks and how to boot them, one needs 30+ minutes and a few tutors. <br />
<br />
'''Some more details'''<br />
<br />
# we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes a biosboot partition (1MB; required for booting bios system from a GPT disk), an efi partition (e.g. 250MB; required for UEFI/EFI boot on PC's/MAC's) and the Linux partition, with 13.9GB, so that the sum of the four partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB stick; the /data partition then does not fit and cannot be used on that stick, but the operating system will then work on the small stick just as well. This requires that the /data partition is not fsck'ed automatically from /etc/fstab (0 in the 6th field).<br />
# it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use<br />
# it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up.<br />
# (if the installation not already does it for you) '''you should label the partitions and use the labels for mounting the partitions in /etc/fstab, alternatively use UUID's; don't use /dev/sda1 or the like because depending on the hardware it is booted on, after booting the stick, and depending on the actual hardware, the stick may be /dev/sdb or /dev/sdc or ... !'''<br />
<br />
<br />
<br />
== How to create a bootabled GPT-partitioned USB-stick with Fedora 23 ==<br />
<br />
This will create a USB stick capable to<br />
* EFI boot on Macs<br />
* UEFI boot on PCs<br />
* BIOS boot on newer hardware (legacy boot / CSM enabled in BIOS)<br />
* BIOS boot on medium aged hardware<br />
* not boot on really old BIOS hardware which doesn't support booting from GPT (see below)<br />
<br />
=== GPT partition the stick using <code>gdisk</code> ===<br />
The example shows a 32 GB Sandisk Extreme:<br />
Part 1: +1G VFAT (0700) (Windows needs it to be the first partition) <br />
Part 2: +1M BIOSBOOT (ef02)<br />
Part 3: +250M EFI (ef00)<br />
Part 4: +13500M / (0083)<br />
Part 5: +15000M /mnt/data (0083)<br />
For performance, make sure that the big partitions are aligned to 8192-sector boundaries; gdisk default is 2048-sector boundaries - the default can be adjusted. If you use the above sizes and the + sign for specifying them, this happens to work out automatically.<br />
<br />
=== install Fedora23 on an UEFI machine (UEFI enabled ; LegacyBoot / CSM disabled) ===<br />
Note: in the following, whenever you see the X in sdX, you must fill in the appropriate drive letter (e.g. a or b or c or d or ...). <br />
sdX3: efi /boot/efi<br />
sdX4: ext4 / <br />
sdX5: ext4 /mnt/data<br />
<br />
Not much to say about the installation of programs (CCP4 ?, Phenix ?, XDS ?, ...?). You should also install the hfsplus-tools to enable Mac users to access their data on harddisk. Make sure to install the binutils RPM to get the "strings" and other useful GNU commands.<br />
<br />
=== adjust the USB-stick for UEFI/EFI boot (before reboot from chroot environment) ===<br />
<br />
chroot to the installed system:<br />
chroot /mnt/sysimage<br />
<br />
install Fedora updates:<br />
dnf -y update # dnf on FC23 is successor to yum<br />
<br />
<br />
configure grub2 for (U)EFI systems:<br />
<br />
*disable auto recognition of other installed Operating systems (specific to current computer), and<br />
*update grub2-efi.cfg<br />
echo 'GRUB_DISABLE_OS_PROBER=”true”' >> /etc/default/grub<br />
grub2-mkconfig -o /etc/grub2-efi.cfg<br />
(last step automatically puts UUID's to recognise the boot device in grub2-efi.cfg ; default after fresh Fedora installation is the device name which might be different on another computer)<br />
<br />
<br />
Performance tuning (not strictly required):<br />
<br />
We use ext4 filesystems and <code>data=writeback,nobarrier </code> in /etc/fstab. To be able to set these options also on the / filesystem, we use tune2fs to set both as default mount options on a Linux machine where the stick (dev/sdX) is just plugged in (i.e. not booted from):<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX4<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX5<br />
<br />
create & label vfat filesystem:<br />
mkfs.vfat /dev/sdX1<br />
dosfslabel /dev/sdX1 VFAT<br />
<br />
shutdown or reboot:<br />
exit<br />
systemctl poweroff (or reboot for testing)<br />
<br />
=== Install grub2 for BIOS boot (from chroot environment) ===<br />
<br />
boot Fedora Live DVD on a BIOS machine and chroot to the stick:<br />
mount /dev/sdX4 /mnt<br />
mount -o bind /dev /mnt/dev<br />
mount -t proc /proc /mnt/proc<br />
mount -t sysfs /sys /mnt/sys<br />
chroot /mnt<br />
<br />
install grub2 for BIOS boot and configure it:<br />
rm /boot/grub2/grubenv<br />
grub2-install /dev/sdX<br />
grub2-mkconfig -o /etc/grub2.cfg<br />
(on UEFI systems /boot/grub2/grubenv is a symlink to /boot/efi/EFI/fedora/grubenv (from package grub2-efi.rpm). The symlink has to be removed before grub2-install can finish successfully, it automatically creates a new file /boot/grub2/grubenv (as in package grub2.rpm)<br />
<br />
<br />
check /etc/grub2.cfg and if needed, change at all positions:<br />
“linuxefi” to “linux16”, and<br />
“intrdefi” to “intrd16”<br />
<br />
=== Problems on old BIOS hardware ===<br />
Some old computers still might not boot since they don't support booting from GPT.<br />
This can be fixed to boot on such old hardware, however the stick afterwards cannot boot on EFI/UEFI systems anymore:<br />
<br />
*set the boot flag (active) on the single 0xEE partition in the protective MBR by using fdisk version <= 2.22 (versions 1.23 and newer have GPT support and thus don't make changes to the protective MBR but to the GPT) (I used fdisk (util-linux-ng 2.17.2) on a ScientificLinux 6.7 machine)<br />
fdisk /dev/sdX<br />
a<br />
part 1<br />
w<br />
<br />
*recompute CHS values using option “h” in gdisk's expert menu:<br />
gdisk /dev/sdX<br />
x<br />
h<br />
m<br />
q<br />
<br />
this can be reversed:<br />
- toggle bootflag off in old fdisk<br />
- recompute CHS values in gdisk<br />
<br />
(using a sgdisk backup of the GPT should also be fine)<br />
<br />
helpful links:<br />
<br />
http://www.rodsbooks.com/gdisk/bios.html#bios --> option h of gdisks experts menu did the trick!<br />
<br />
https://fedoraproject.org/wiki/GRUB_2?rd=Grub2#Updating_GRUB_2_configuration_on_BIOS_systems<br />
<br />
=== generating images and copies of the stick ===<br />
<br />
Save a compressed disk image - we use the parallel gzip program called pigz:<br />
pigz -c < /dev/sdX > usbstick.img.gz<br />
(time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB)<br />
write compressed image back to a stick:<br />
unpigz -c usbstick.img.gz > /dev/sdX<br />
(speed: ~ 98MB/s)<br />
<br />
For creation/maintainance a bunch of sticks, a multiple port USB-HUB is a very usefull tool. E.g. the Raid Sonic Icy Box IB-AC6113 accepts 12 Sandisk Extreme sticks at once (port 7 has to stay empty due to spatial restrictions) and all 12 stick are recognized by the OS (CentOS7).<br />
<br />
=== for techies: taking care of the stick ===<br />
<br />
The following is about performance and durability of the USB stick, and reading it is not required for the functionality described above. If you don't know what TRIM is and how it relates to flash media, this section is probably not for you.<br />
<br />
To fill empty part of all partitions on the stick with “zeros”: for each partition, do (as root)<br />
dd if=/dev/zero of=/mountpoint_of_partition/delete.me bs=10M # this will stop when filesystem is full<br />
rm /mountpoint_of_partition/delete.me<br />
<br />
This could be done after deleting large amounts of data from the USB stick, and before saving and compressing an image of it. USB sticks that support “Deterministic read data after TRIM” should instead be TRIMmed (see below) because this does not wear out the storage cells.<br />
<br />
==== Initializing the stick ====<br />
<code>hdparm</code>'s (ENHANCED) SECURITY ERASE initializes the whole stick in a few seconds, and restores it to an almost factory-fresh condition, including zeroing the device. This works on all sticks where <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" but if used wrongly (e.g. on the wrong device, or with the wrong options, or ...) it may brick your device, or delete valuable data! So use at your own risk:<br />
* check if <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" and "not frozen" (to unfreeze a frozen diks, suspend your system with <code>pm-suspend</code> and wake it up again)<br />
* set disk password, and erase it (which unsets the password):<br />
hdparm --user-master u --security-set-pass Eins /dev/sdX<br />
hdparm --user-master u --security-erase-enhanced Eins /dev/sdX<br />
If you tried with --security-erase-enhanced, but the disk does not support it, then an error will occur and the disk will still be locked after the failed command, and you will not be able to write anything to it! If this happens, you can unlock it with <br />
hdparm --user-master u --security-unlock Eins /dev/sdX</code><br />
<br />
==== TRIMming the stick ====<br />
TRIMming informs the USB stick firmware about blocks of the filesystem that do not contain file data. This is available for USB sticks for which <code>hdparm -I /dev/sdX</code> returns "Data Set Management TRIM supported"; if that command also returns "Deterministic read data after TRIM" then zeros are returned upon reads of TRIMmed blocks.<br />
<br />
The <code>wiper.sh</code> script (part of the <code>hdparm</code> source distribution) TRIMs ext3/ext4/xfs filesystems, but does not reproducibly seem to work for the SanDisk Extreme 32GB stick if the filesystem is mounted. This is probably due to the fact that this stick has a limitation of max 65536 blocks in one TRIM command (https://sourceforge.net/p/hdparm/bugs/63/). Since <code>wiper.sh</code> also works for unmounted filesystems, and the mapping of unused space is then obtained with a different tool, one should try with an unmounted filesystem as well - this worked for us in all cases tried so far (tested with [http://lightrush.ndoytchev.com/random-1/checkiftrimonext4isenabledandworking this script]).<br />
<br />
All other methods, like the <code>fstrim</code> command and the <code>discard</code> mount option '''do not work for USB sticks''', presumably because the usb-storage kernel module does not pass the ATA trim command through the USB bridge and controller to the device. The same goes for [http://unix.stackexchange.com/questions/97143/utility-to-trim-unallocated-space-on-drive <code>blkdiscard</code>].</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Bootable_Linux_USB_stick&diff=2388Bootable Linux USB stick2016-03-30T14:48:52Z<p>Karsten: /* generating images and copies of the stick */</p>
<hr />
<div>We regularly and successfully use USB sticks for courses where participants bring their own notebooks. The big benefit is that students<br />
learn Linux, and realize that they can easily use the hardware they already own. Current notebook hardware is by far fast enough for data<br />
processing, structure solution and coot visualization. What we found:<br />
# the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do ''not'' have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk.<br />
# we use Fedora 23 because its hardware support is very good. We always use the 64bit distro.<br />
# The stick can be booted on MacBooks as well (press the <code>alt</code> key at the boot sound); their hardware works well with Fedora 23. For Windows clients (press F11 or F12 or sometimes F9 or F10 for the boot menu; if that does not work press F2 or DEL for the BIOS menu and change the boot order), one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes <code>powercfg -H off</code> (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from the stick because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should.<br />
# we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks.<br />
# any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync.<br />
<br />
To make students familiar with the sticks and how to boot them, one needs 30+ minutes and a few tutors. <br />
<br />
'''Some more details'''<br />
<br />
# we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes a biosboot partition (1MB; required for booting bios system from a GPT disk), an efi partition (e.g. 250MB; required for UEFI/EFI boot on PC's/MAC's) and the Linux partition, with 13.9GB, so that the sum of the four partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB stick; the /data partition then does not fit and cannot be used on that stick, but the operating system will then work on the small stick just as well. This requires that the /data partition is not fsck'ed automatically from /etc/fstab (0 in the 6th field).<br />
# it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use<br />
# it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up.<br />
# (if the installation not already does it for you) '''you should label the partitions and use the labels for mounting the partitions in /etc/fstab, alternatively use UUID's; don't use /dev/sda1 or the like because depending on the hardware it is booted on, after booting the stick, and depending on the actual hardware, the stick may be /dev/sdb or /dev/sdc or ... !'''<br />
<br />
<br />
<br />
== How to create a bootabled GPT-partitioned USB-stick with Fedora 23 ==<br />
<br />
This will create a USB stick capable to<br />
* EFI boot on Macs<br />
* UEFI boot on PCs<br />
* BIOS boot on newer hardware (legacy boot / CSM enabled in BIOS)<br />
* BIOS boot on medium aged hardware<br />
* not boot on really old BIOS hardware which doesn't support booting from GPT (see below)<br />
<br />
=== GPT partition the stick using <code>gdisk</code> ===<br />
The example shows a 32 GB Sandisk Extreme:<br />
Part 1: +1G VFAT (0700) (Windows needs it to be the first partition) <br />
Part 2: +1M BIOSBOOT (ef02)<br />
Part 3: +250M EFI (ef00)<br />
Part 4: +13500M / (0083)<br />
Part 5: +15000M /mnt/data (0083)<br />
For performance, make sure that the big partitions are aligned to 8192-sector boundaries; gdisk default is 2048-sector boundaries - the default can be adjusted. If you use the above sizes and the + sign for specifying them, this happens to work out automatically.<br />
<br />
=== install Fedora23 on an UEFI machine (UEFI enabled ; LegacyBoot / CSM disabled) ===<br />
Note: in the following, whenever you see the X in sdX, you must fill in the appropriate drive letter (e.g. a or b or c or d or ...). <br />
sdX3: efi /boot/efi<br />
sdX4: ext4 / <br />
sdX5: ext4 /mnt/data<br />
<br />
=== adjust the USB-stick for UEFI/EFI boot (before reboot from chroot environment) ===<br />
<br />
chroot to the installed system:<br />
chroot /mnt/sysimage<br />
<br />
install Fedora updates:<br />
dnf -y update # dnf on FC23 is successor to yum<br />
<br />
<br />
configure grub2 for (U)EFI systems:<br />
<br />
*disable auto recognition of other installed Operating systems (specific to current computer), and<br />
*update grub2-efi.cfg<br />
echo 'GRUB_DISABLE_OS_PROBER=”true”' >> /etc/default/grub<br />
grub2-mkconfig -o /etc/grub2-efi.cfg<br />
(last step automatically puts UUID's to recognise the boot device in grub2-efi.cfg ; default after fresh Fedora installation is the device name which might be different on another computer)<br />
<br />
<br />
Performance tuning (not strictly required):<br />
<br />
We use ext4 filesystems and <code>data=writeback,nobarrier </code> in /etc/fstab. To be able to set these options also on the / filesystem, we use tune2fs to set both as default mount options on a Linux machine where the stick (dev/sdX) is just plugged in (i.e. not booted from):<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX4<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX5<br />
<br />
create & label vfat filesystem:<br />
mkfs.vfat /dev/sdX1<br />
dosfslabel /dev/sdX1 VFAT<br />
<br />
shutdown or reboot:<br />
exit<br />
systemctl poweroff (or reboot for testing)<br />
<br />
=== Install grub2 for BIOS boot (from chroot environment) ===<br />
<br />
boot Fedora Live DVD on a BIOS machine and chroot to the stick:<br />
mount /dev/sdX4 /mnt<br />
mount -o bind /dev /mnt/dev<br />
mount -t proc /proc /mnt/proc<br />
mount -t sysfs /sys /mnt/sys<br />
chroot /mnt<br />
<br />
install grub2 for BIOS boot and configure it:<br />
rm /boot/grub2/grubenv<br />
grub2-install /dev/sdX<br />
grub2-mkconfig -o /etc/grub2.cfg<br />
(on UEFI systems /boot/grub2/grubenv is a symlink to /boot/efi/EFI/fedora/grubenv (from package grub2-efi.rpm). The symlink has to be removed before grub2-install can finish successfully, it automatically creates a new file /boot/grub2/grubenv (as in package grub2.rpm)<br />
<br />
<br />
check /etc/grub2.cfg and if needed, change at all positions:<br />
“linuxefi” to “linux16”, and<br />
“intrdefi” to “intrd16”<br />
<br />
=== Problems on old BIOS hardware ===<br />
Some old computers still might not boot since they don't support booting from GPT.<br />
This can be fixed to boot on such old hardware, however the stick afterwards cannot boot on EFI/UEFI systems anymore:<br />
<br />
*set the boot flag (active) on the single 0xEE partition in the protective MBR by using fdisk version <= 2.22 (versions 1.23 and newer have GPT support and thus don't make changes to the protective MBR but to the GPT) (I used fdisk (util-linux-ng 2.17.2) on a ScientificLinux 6.7 machine)<br />
fdisk /dev/sdX<br />
a<br />
part 1<br />
w<br />
<br />
*recompute CHS values using option “h” in gdisk's expert menu:<br />
gdisk /dev/sdX<br />
x<br />
h<br />
m<br />
q<br />
<br />
this can be reversed:<br />
- toggle bootflag off in old fdisk<br />
- recompute CHS values in gdisk<br />
<br />
(using a sgdisk backup of the GPT should also be fine)<br />
<br />
helpful links:<br />
<br />
http://www.rodsbooks.com/gdisk/bios.html#bios --> option h of gdisks experts menu did the trick!<br />
<br />
https://fedoraproject.org/wiki/GRUB_2?rd=Grub2#Updating_GRUB_2_configuration_on_BIOS_systems<br />
<br />
=== generating images and copies of the stick ===<br />
<br />
Save a compressed disk image - we use the parallel gzip program called pigz:<br />
pigz -c < /dev/sdX > usbstick.img.gz<br />
(time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB)<br />
write compressed image back to a stick:<br />
unpigz -c usbstick.img.gz > /dev/sdX<br />
(speed: ~ 98MB/s)<br />
<br />
=== for techies: taking care of the stick ===<br />
<br />
The following is about performance and durability of the USB stick, and reading it is not required for the functionality described above. If you don't know what TRIM is and how it relates to flash media, this section is probably not for you.<br />
<br />
To fill empty part of all partitions on the stick with “zeros”: for each partition, do (as root)<br />
dd if=/dev/zero of=/mountpoint_of_partition/delete.me bs=10M # this will stop when filesystem is full<br />
rm /mountpoint_of_partition/delete.me<br />
<br />
This is sensible to do after deleting large amounts of data from the USB stick, and before saving and compressing an image of it. It also usually results in faster writes afterwards i.e. it may be a way to restore the write speed of the stick (of course this depends on the specific hardware). <br />
<br />
<code>hdparm</code>'s (ENHANCED) SECURITY ERASE initializes the whole stick in a few seconds, and restores it to an almost factory-fresh condition, including zeroing the device. This works on all sticks where hdparm -I /dev/sdX reports "supported: enhanced erase" but if used wrongly (e.g. on the wrong device, or with the wrong options, or ...) it may brick your device, or delete valuable data! So use at your own risk:<br />
* check if <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" and "not frozen" (to unfreeze a frozen diks, suspend your system with <code>pm-suspend</code> and wake it up again)<br />
* set disk password, and erase it (which unsets the password):<br />
hdparm --user-master u --security-set-pass Eins /dev/sdX<br />
hdparm --user-master u --security-erase-enhanced Eins /dev/sdX<br />
If you tried with --security-erase-enhanced, but the disk does not support it, then an error will occur and the disk will still be locked after the failed command, and you will not be able to write anything to it! If this happens, you can unlock it with <code> hdparm --user-master u --security-unlock Eins /dev/sdX</code><br />
<br />
The <code>wiper.sh</code> script (which is part of the <code>hdparm</code> source distribution) gives no error message, but does not reproducibly fill the empty space of the filesystem with zeroes (it sometimes does).<br />
<br />
All other methods, like the <code>fstrim</code> command and the <code>discard</code> mount option '''do not work for USB sticks''', presumably because the usb-storage kernel module does not pass the ATA trim command through the USB bridge and controller to the device. The same goes for [http://unix.stackexchange.com/questions/97143/utility-to-trim-unallocated-space-on-drive <code>blkdiscard</code>].</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Bootable_Linux_USB_stick&diff=2387Bootable Linux USB stick2016-03-30T14:41:51Z<p>Karsten: /* generating images and copies of the stick */</p>
<hr />
<div>We regularly and successfully use USB sticks for courses where participants bring their own notebooks. The big benefit is that students<br />
learn Linux, and realize that they can easily use the hardware they already own. Current notebook hardware is by far fast enough for data<br />
processing, structure solution and coot visualization. What we found:<br />
# the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do ''not'' have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk.<br />
# we use Fedora 23 because its hardware support is very good. We always use the 64bit distro.<br />
# The stick can be booted on MacBooks as well (press the <code>alt</code> key at the boot sound); their hardware works well with Fedora 23. For Windows clients (press F11 or F12 or sometimes F9 or F10 for the boot menu; if that does not work press F2 or DEL for the BIOS menu and change the boot order), one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes <code>powercfg -H off</code> (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from the stick because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should.<br />
# we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks.<br />
# any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync.<br />
<br />
To make students familiar with the sticks and how to boot them, one needs 30+ minutes and a few tutors. <br />
<br />
'''Some more details'''<br />
<br />
# we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes a biosboot partition (1MB; required for booting bios system from a GPT disk), an efi partition (e.g. 250MB; required for UEFI/EFI boot on PC's/MAC's) and the Linux partition, with 13.9GB, so that the sum of the four partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB stick; the /data partition then does not fit and cannot be used on that stick, but the operating system will then work on the small stick just as well. This requires that the /data partition is not fsck'ed automatically from /etc/fstab (0 in the 6th field).<br />
# it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use<br />
# it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up.<br />
# (if the installation not already does it for you) '''you should label the partitions and use the labels for mounting the partitions in /etc/fstab, alternatively use UUID's; don't use /dev/sda1 or the like because depending on the hardware it is booted on, after booting the stick, and depending on the actual hardware, the stick may be /dev/sdb or /dev/sdc or ... !'''<br />
<br />
<br />
<br />
== How to create a bootabled GPT-partitioned USB-stick with Fedora 23 ==<br />
<br />
This will create a USB stick capable to<br />
* EFI boot on Macs<br />
* UEFI boot on PCs<br />
* BIOS boot on newer hardware (legacy boot / CSM enabled in BIOS)<br />
* BIOS boot on medium aged hardware<br />
* not boot on really old BIOS hardware which doesn't support booting from GPT (see below)<br />
<br />
=== GPT partition the stick using <code>gdisk</code> ===<br />
The example shows a 32 GB Sandisk Extreme:<br />
Part 1: +1G VFAT (0700) (Windows needs it to be the first partition) <br />
Part 2: +1M BIOSBOOT (ef02)<br />
Part 3: +250M EFI (ef00)<br />
Part 4: +13500M / (0083)<br />
Part 5: +15000M /mnt/data (0083)<br />
For performance, make sure that the big partitions are aligned to 8192-sector boundaries; gdisk default is 2048-sector boundaries - the default can be adjusted. If you use the above sizes and the + sign for specifying them, this happens to work out automatically.<br />
<br />
=== install Fedora23 on an UEFI machine (UEFI enabled ; LegacyBoot / CSM disabled) ===<br />
Note: in the following, whenever you see the X in sdX, you must fill in the appropriate drive letter (e.g. a or b or c or d or ...). <br />
sdX3: efi /boot/efi<br />
sdX4: ext4 / <br />
sdX5: ext4 /mnt/data<br />
<br />
=== adjust the USB-stick for UEFI/EFI boot (before reboot from chroot environment) ===<br />
<br />
chroot to the installed system:<br />
chroot /mnt/sysimage<br />
<br />
install Fedora updates:<br />
dnf -y update # dnf on FC23 is successor to yum<br />
<br />
<br />
configure grub2 for (U)EFI systems:<br />
<br />
*disable auto recognition of other installed Operating systems (specific to current computer), and<br />
*update grub2-efi.cfg<br />
echo 'GRUB_DISABLE_OS_PROBER=”true”' >> /etc/default/grub<br />
grub2-mkconfig -o /etc/grub2-efi.cfg<br />
(last step automatically puts UUID's to recognise the boot device in grub2-efi.cfg ; default after fresh Fedora installation is the device name which might be different on another computer)<br />
<br />
<br />
Performance tuning (not strictly required):<br />
<br />
We use ext4 filesystems and <code>data=writeback,nobarrier </code> in /etc/fstab. To be able to set these options also on the / filesystem, we use tune2fs to set both as default mount options on a Linux machine where the stick (dev/sdX) is just plugged in (i.e. not booted from):<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX4<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX5<br />
<br />
create & label vfat filesystem:<br />
mkfs.vfat /dev/sdX1<br />
dosfslabel /dev/sdX1 VFAT<br />
<br />
shutdown or reboot:<br />
exit<br />
systemctl poweroff (or reboot for testing)<br />
<br />
=== Install grub2 for BIOS boot (from chroot environment) ===<br />
<br />
boot Fedora Live DVD on a BIOS machine and chroot to the stick:<br />
mount /dev/sdX4 /mnt<br />
mount -o bind /dev /mnt/dev<br />
mount -t proc /proc /mnt/proc<br />
mount -t sysfs /sys /mnt/sys<br />
chroot /mnt<br />
<br />
install grub2 for BIOS boot and configure it:<br />
rm /boot/grub2/grubenv<br />
grub2-install /dev/sdX<br />
grub2-mkconfig -o /etc/grub2.cfg<br />
(on UEFI systems /boot/grub2/grubenv is a symlink to /boot/efi/EFI/fedora/grubenv (from package grub2-efi.rpm). The symlink has to be removed before grub2-install can finish successfully, it automatically creates a new file /boot/grub2/grubenv (as in package grub2.rpm)<br />
<br />
<br />
check /etc/grub2.cfg and if needed, change at all positions:<br />
“linuxefi” to “linux16”, and<br />
“intrdefi” to “intrd16”<br />
<br />
=== Problems on old BIOS hardware ===<br />
Some old computers still might not boot since they don't support booting from GPT.<br />
This can be fixed to boot on such old hardware, however the stick afterwards cannot boot on EFI/UEFI systems anymore:<br />
<br />
*set the boot flag (active) on the single 0xEE partition in the protective MBR by using fdisk version <= 2.22 (versions 1.23 and newer have GPT support and thus don't make changes to the protective MBR but to the GPT) (I used fdisk (util-linux-ng 2.17.2) on a ScientificLinux 6.7 machine)<br />
fdisk /dev/sdX<br />
a<br />
part 1<br />
w<br />
<br />
*recompute CHS values using option “h” in gdisk's expert menu:<br />
gdisk /dev/sdX<br />
x<br />
h<br />
m<br />
q<br />
<br />
this can be reversed:<br />
- toggle bootflag off in old fdisk<br />
- recompute CHS values in gdisk<br />
<br />
(using a sgdisk backup of the GPT should also be fine)<br />
<br />
helpful links:<br />
<br />
http://www.rodsbooks.com/gdisk/bios.html#bios --> option h of gdisks experts menu did the trick!<br />
<br />
https://fedoraproject.org/wiki/GRUB_2?rd=Grub2#Updating_GRUB_2_configuration_on_BIOS_systems<br />
<br />
=== generating images and copies of the stick ===<br />
<br />
Save a compressed disk image - we use the parallel gzip program called pigz:<br />
dd if=/dev/sdX bs=4096 | pigz -c usbstick.img.gz<br />
(time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB)<br />
write compressed image back to a stick:<br />
unpigz -c usbstick.img.gz > /dev/sdX<br />
(speed: ~ 98MB/s)<br />
<br />
=== for techies: taking care of the stick ===<br />
<br />
The following is about performance and durability of the USB stick, and reading it is not required for the functionality described above. If you don't know what TRIM is and how it relates to flash media, this section is probably not for you.<br />
<br />
To fill empty part of all partitions on the stick with “zeros”: for each partition, do (as root)<br />
dd if=/dev/zero of=/mountpoint_of_partition/delete.me bs=10M # this will stop when filesystem is full<br />
rm /mountpoint_of_partition/delete.me<br />
<br />
This is sensible to do after deleting large amounts of data from the USB stick, and before saving and compressing an image of it. It also usually results in faster writes afterwards i.e. it may be a way to restore the write speed of the stick (of course this depends on the specific hardware). <br />
<br />
<code>hdparm</code>'s (ENHANCED) SECURITY ERASE initializes the whole stick in a few seconds, and restores it to an almost factory-fresh condition, including zeroing the device. This works on all sticks where hdparm -I /dev/sdX reports "supported: enhanced erase" but if used wrongly (e.g. on the wrong device, or with the wrong options, or ...) it may brick your device, or delete valuable data! So use at your own risk:<br />
* check if <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" and "not frozen" (to unfreeze a frozen diks, suspend your system with <code>pm-suspend</code> and wake it up again)<br />
* set disk password, and erase it (which unsets the password):<br />
hdparm --user-master u --security-set-pass Eins /dev/sdX<br />
hdparm --user-master u --security-erase-enhanced Eins /dev/sdX<br />
If you tried with --security-erase-enhanced, but the disk does not support it, then an error will occur and the disk will still be locked after the failed command, and you will not be able to write anything to it! If this happens, you can unlock it with <code> hdparm --user-master u --security-unlock Eins /dev/sdX</code><br />
<br />
The <code>wiper.sh</code> script (which is part of the <code>hdparm</code> source distribution) gives no error message, but does not reproducibly fill the empty space of the filesystem with zeroes (it sometimes does).<br />
<br />
All other methods, like the <code>fstrim</code> command and the <code>discard</code> mount option '''do not work for USB sticks''', presumably because the usb-storage kernel module does not pass the ATA trim command through the USB bridge and controller to the device. The same goes for [http://unix.stackexchange.com/questions/97143/utility-to-trim-unallocated-space-on-drive <code>blkdiscard</code>].</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Bootable_Linux_USB_stick&diff=2386Bootable Linux USB stick2016-03-30T14:41:29Z<p>Karsten: /* generating images and copies of the stick */</p>
<hr />
<div>We regularly and successfully use USB sticks for courses where participants bring their own notebooks. The big benefit is that students<br />
learn Linux, and realize that they can easily use the hardware they already own. Current notebook hardware is by far fast enough for data<br />
processing, structure solution and coot visualization. What we found:<br />
# the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do ''not'' have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk.<br />
# we use Fedora 23 because its hardware support is very good. We always use the 64bit distro.<br />
# The stick can be booted on MacBooks as well (press the <code>alt</code> key at the boot sound); their hardware works well with Fedora 23. For Windows clients (press F11 or F12 or sometimes F9 or F10 for the boot menu; if that does not work press F2 or DEL for the BIOS menu and change the boot order), one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes <code>powercfg -H off</code> (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from the stick because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should.<br />
# we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks.<br />
# any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync.<br />
<br />
To make students familiar with the sticks and how to boot them, one needs 30+ minutes and a few tutors. <br />
<br />
'''Some more details'''<br />
<br />
# we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes a biosboot partition (1MB; required for booting bios system from a GPT disk), an efi partition (e.g. 250MB; required for UEFI/EFI boot on PC's/MAC's) and the Linux partition, with 13.9GB, so that the sum of the four partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB stick; the /data partition then does not fit and cannot be used on that stick, but the operating system will then work on the small stick just as well. This requires that the /data partition is not fsck'ed automatically from /etc/fstab (0 in the 6th field).<br />
# it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use<br />
# it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up.<br />
# (if the installation not already does it for you) '''you should label the partitions and use the labels for mounting the partitions in /etc/fstab, alternatively use UUID's; don't use /dev/sda1 or the like because depending on the hardware it is booted on, after booting the stick, and depending on the actual hardware, the stick may be /dev/sdb or /dev/sdc or ... !'''<br />
<br />
<br />
<br />
== How to create a bootabled GPT-partitioned USB-stick with Fedora 23 ==<br />
<br />
This will create a USB stick capable to<br />
* EFI boot on Macs<br />
* UEFI boot on PCs<br />
* BIOS boot on newer hardware (legacy boot / CSM enabled in BIOS)<br />
* BIOS boot on medium aged hardware<br />
* not boot on really old BIOS hardware which doesn't support booting from GPT (see below)<br />
<br />
=== GPT partition the stick using <code>gdisk</code> ===<br />
The example shows a 32 GB Sandisk Extreme:<br />
Part 1: +1G VFAT (0700) (Windows needs it to be the first partition) <br />
Part 2: +1M BIOSBOOT (ef02)<br />
Part 3: +250M EFI (ef00)<br />
Part 4: +13500M / (0083)<br />
Part 5: +15000M /mnt/data (0083)<br />
For performance, make sure that the big partitions are aligned to 8192-sector boundaries; gdisk default is 2048-sector boundaries - the default can be adjusted. If you use the above sizes and the + sign for specifying them, this happens to work out automatically.<br />
<br />
=== install Fedora23 on an UEFI machine (UEFI enabled ; LegacyBoot / CSM disabled) ===<br />
Note: in the following, whenever you see the X in sdX, you must fill in the appropriate drive letter (e.g. a or b or c or d or ...). <br />
sdX3: efi /boot/efi<br />
sdX4: ext4 / <br />
sdX5: ext4 /mnt/data<br />
<br />
=== adjust the USB-stick for UEFI/EFI boot (before reboot from chroot environment) ===<br />
<br />
chroot to the installed system:<br />
chroot /mnt/sysimage<br />
<br />
install Fedora updates:<br />
dnf -y update # dnf on FC23 is successor to yum<br />
<br />
<br />
configure grub2 for (U)EFI systems:<br />
<br />
*disable auto recognition of other installed Operating systems (specific to current computer), and<br />
*update grub2-efi.cfg<br />
echo 'GRUB_DISABLE_OS_PROBER=”true”' >> /etc/default/grub<br />
grub2-mkconfig -o /etc/grub2-efi.cfg<br />
(last step automatically puts UUID's to recognise the boot device in grub2-efi.cfg ; default after fresh Fedora installation is the device name which might be different on another computer)<br />
<br />
<br />
Performance tuning (not strictly required):<br />
<br />
We use ext4 filesystems and <code>data=writeback,nobarrier </code> in /etc/fstab. To be able to set these options also on the / filesystem, we use tune2fs to set both as default mount options on a Linux machine where the stick (dev/sdX) is just plugged in (i.e. not booted from):<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX4<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sdX5<br />
<br />
create & label vfat filesystem:<br />
mkfs.vfat /dev/sdX1<br />
dosfslabel /dev/sdX1 VFAT<br />
<br />
shutdown or reboot:<br />
exit<br />
systemctl poweroff (or reboot for testing)<br />
<br />
=== Install grub2 for BIOS boot (from chroot environment) ===<br />
<br />
boot Fedora Live DVD on a BIOS machine and chroot to the stick:<br />
mount /dev/sdX4 /mnt<br />
mount -o bind /dev /mnt/dev<br />
mount -t proc /proc /mnt/proc<br />
mount -t sysfs /sys /mnt/sys<br />
chroot /mnt<br />
<br />
install grub2 for BIOS boot and configure it:<br />
rm /boot/grub2/grubenv<br />
grub2-install /dev/sdX<br />
grub2-mkconfig -o /etc/grub2.cfg<br />
(on UEFI systems /boot/grub2/grubenv is a symlink to /boot/efi/EFI/fedora/grubenv (from package grub2-efi.rpm). The symlink has to be removed before grub2-install can finish successfully, it automatically creates a new file /boot/grub2/grubenv (as in package grub2.rpm)<br />
<br />
<br />
check /etc/grub2.cfg and if needed, change at all positions:<br />
“linuxefi” to “linux16”, and<br />
“intrdefi” to “intrd16”<br />
<br />
=== Problems on old BIOS hardware ===<br />
Some old computers still might not boot since they don't support booting from GPT.<br />
This can be fixed to boot on such old hardware, however the stick afterwards cannot boot on EFI/UEFI systems anymore:<br />
<br />
*set the boot flag (active) on the single 0xEE partition in the protective MBR by using fdisk version <= 2.22 (versions 1.23 and newer have GPT support and thus don't make changes to the protective MBR but to the GPT) (I used fdisk (util-linux-ng 2.17.2) on a ScientificLinux 6.7 machine)<br />
fdisk /dev/sdX<br />
a<br />
part 1<br />
w<br />
<br />
*recompute CHS values using option “h” in gdisk's expert menu:<br />
gdisk /dev/sdX<br />
x<br />
h<br />
m<br />
q<br />
<br />
this can be reversed:<br />
- toggle bootflag off in old fdisk<br />
- recompute CHS values in gdisk<br />
<br />
(using a sgdisk backup of the GPT should also be fine)<br />
<br />
helpful links:<br />
<br />
http://www.rodsbooks.com/gdisk/bios.html#bios --> option h of gdisks experts menu did the trick!<br />
<br />
https://fedoraproject.org/wiki/GRUB_2?rd=Grub2#Updating_GRUB_2_configuration_on_BIOS_systems<br />
<br />
=== generating images and copies of the stick ===<br />
<br />
Save a compressed disk image - we use the parallel gzip program called pigz:<br />
dd if=/dev/sdX bs=4096 | pigz -c usbstick.img.gz<br />
(time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB)<br />
write compressed image back to a stick:<br />
unpigz -c usbstick.img.gz > /dev/sdX bs=4096<br />
(speed: ~ 98MB/s)<br />
<br />
=== for techies: taking care of the stick ===<br />
<br />
The following is about performance and durability of the USB stick, and reading it is not required for the functionality described above. If you don't know what TRIM is and how it relates to flash media, this section is probably not for you.<br />
<br />
To fill empty part of all partitions on the stick with “zeros”: for each partition, do (as root)<br />
dd if=/dev/zero of=/mountpoint_of_partition/delete.me bs=10M # this will stop when filesystem is full<br />
rm /mountpoint_of_partition/delete.me<br />
<br />
This is sensible to do after deleting large amounts of data from the USB stick, and before saving and compressing an image of it. It also usually results in faster writes afterwards i.e. it may be a way to restore the write speed of the stick (of course this depends on the specific hardware). <br />
<br />
<code>hdparm</code>'s (ENHANCED) SECURITY ERASE initializes the whole stick in a few seconds, and restores it to an almost factory-fresh condition, including zeroing the device. This works on all sticks where hdparm -I /dev/sdX reports "supported: enhanced erase" but if used wrongly (e.g. on the wrong device, or with the wrong options, or ...) it may brick your device, or delete valuable data! So use at your own risk:<br />
* check if <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" and "not frozen" (to unfreeze a frozen diks, suspend your system with <code>pm-suspend</code> and wake it up again)<br />
* set disk password, and erase it (which unsets the password):<br />
hdparm --user-master u --security-set-pass Eins /dev/sdX<br />
hdparm --user-master u --security-erase-enhanced Eins /dev/sdX<br />
If you tried with --security-erase-enhanced, but the disk does not support it, then an error will occur and the disk will still be locked after the failed command, and you will not be able to write anything to it! If this happens, you can unlock it with <code> hdparm --user-master u --security-unlock Eins /dev/sdX</code><br />
<br />
The <code>wiper.sh</code> script (which is part of the <code>hdparm</code> source distribution) gives no error message, but does not reproducibly fill the empty space of the filesystem with zeroes (it sometimes does).<br />
<br />
All other methods, like the <code>fstrim</code> command and the <code>discard</code> mount option '''do not work for USB sticks''', presumably because the usb-storage kernel module does not pass the ATA trim command through the USB bridge and controller to the device. The same goes for [http://unix.stackexchange.com/questions/97143/utility-to-trim-unallocated-space-on-drive <code>blkdiscard</code>].</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Bootable_Linux_USB_stick&diff=2381Bootable Linux USB stick2016-03-29T12:27:13Z<p>Karsten: </p>
<hr />
<div>We regularly and successfully use USB sticks for courses where participants bring their own notebooks. The big benefit is that students<br />
learn Linux, and realize that they can easily use the hardware they already own. Current notebook hardware is by far fast enough for data<br />
processing, structure solution and coot visualization. What we found:<br />
# the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do ''not'' have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk.<br />
# we use Fedora 23 because its hardware support is very good. We always use the 64bit distro.<br />
# The stick can be booted on MacBooks as well (press the <code>alt</code> key at the boot sound); their hardware works well with Fedora 23. For Windows clients (press F11 or F12 or sometimes F9 or F10 for the boot menu; if that does not work press F2 or DEL for the BIOS menu and change the boot order), one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes <code>powercfg -H off</code> (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from the stick because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should.<br />
# we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks.<br />
# any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync.<br />
<br />
To make students familiar with the sticks and how to boot them, one needs 30+ minutes and a few tutors. <br />
<br />
'''Some more details'''<br />
<br />
# we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes a biosboot partition (1MB; required for booting bios system from a GPT disk), an efi partition (e.g. 250MB; required for UEFI/EFI boot on PC's/MAC's) and the Linux partition, with 13.9GB, so that the sum of the four partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB stick; the /data partition then does not fit and cannot be used on that stick, but the operating system will then work on the small stick just as well. This requires that the /data partition is not fsck'ed automatically from /etc/fstab (0 in the 6th field).<br />
# it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use<br />
# it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up.<br />
# (if the installation not already does it for you) '''you should label the partitions and use the labels for mounting the partitions in /etc/fstab, alternatively use UUID's; don't use /dev/sda1 or the like because depending on the hardware it is booted on, after booting the stick, and depending on the actual hardware, the stick may be /dev/sdb or /dev/sdc or ... !'''<br />
<br />
<br />
<br />
== How to create a bootabled GPT-partitioned USB-stick with Fedora 23 ==<br />
<br />
This will create a USB stick capable to<br />
* EFI boot on Macs<br />
* UEFI boot on PCs<br />
* BIOS boot on newer hardware (legacy boot / CSM enabled in BIOS)<br />
* BIOS boot on medium aged hardware<br />
* not boot on really old BIOS hardware which doesn't support booting from GPT (see below)<br />
<br />
=== GPT partition the stick using <code>gdisk</code> ===<br />
The example shows a 32 GB Sandisk Extreme:<br />
Part 1: +1G VFAT (0700) (Windows needs it to be the first partition) <br />
Part 2: +1M BIOSBOOT (ef02)<br />
Part 3: +250M EFI (ef00)<br />
Part 4: +13500M / (0083)<br />
Part 5: +15000M /mnt/data (0083)<br />
For performance, make sure that the big partitions are aligned to 8192-sector boundaries; gdisk default is 2048-sector boundaries - the default can be adjusted. If you use the above sizes and the + sign for specifying them, this happens to work out automatically.<br />
<br />
=== install Fedora23 on an UEFI machine (UEFI enabled ; LegacyBoot / CSM disabled) ===<br />
Whenever you see a ?, you must fill in the appropriate drive letter (e.g. c or d). <br />
sd?3: efi /boot/efi<br />
sd?4: ext4 / <br />
sd?5: ext4 /mnt/data<br />
<br />
=== adjust the USB-stick for UEFI/EFI boot (before reboot from chroot environment) ===<br />
<br />
chroot to the installed system:<br />
chroot /mnt/sysimage<br />
<br />
install Fedora updates:<br />
dnf -y update # dnf on FC23 is successor to yum<br />
<br />
<br />
configure grub2 for (U)EFI systems:<br />
<br />
*disable auto recognition of other installed Operating systems (specific to current computer), and<br />
*update grub2-efi.cfg<br />
echo 'GRUB_DISABLE_OS_PROBER=”true”' >> /etc/default/grub<br />
grub2-mkconfig -o /etc/grub2-efi.cfg<br />
(last step automatically puts UUID's to recognise the boot device in grub2-efi.cfg ; default after fresh Fedora installation is the device name which might be different on another computer)<br />
<br />
<br />
Performance tuning (not strictly required):<br />
<br />
We use ext4 filesystems and <code>data=writeback,nobarrier </code> in /etc/fstab. To be able to set these options also on the / filesystem, we use tune2fs to set both as default mount options on a Linux machine where the stick (dev/sd?) is just plugged in (i.e. not booted from):<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sd?4<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sd?5<br />
<br />
create & label vfat filesystem:<br />
mkfs.vfat /dev/sd?1<br />
dosfslabel /dev/sd?1 VFAT<br />
<br />
shutdown or reboot:<br />
exit<br />
systemctl poweroff (or reboot for testing)<br />
<br />
=== Install grub2 for BIOS boot (from chroot environment) ===<br />
<br />
boot Fedora Live DVD on a BIOS machine and chroot to the stick:<br />
mount /dev/sd?4 /mnt<br />
mount -o bind /dev /mnt/dev<br />
mount -t proc /proc /mnt/proc<br />
mount -t sysfs /sys /mnt/sys<br />
chroot /mnt<br />
<br />
install grub2 for BIOS boot and configure it:<br />
rm /boot/grub2/grubenv<br />
grub2-install /dev/sd?<br />
grub2-mkconfig -o /etc/grub2.cfg<br />
(on UEFI systems /boot/grub2/grubenv is a symlink to /boot/efi/EFI/fedora/grubenv (from package grub2-efi.rpm). The symlink has to be removed before grub2-install can finish successfully, it automatically creates a new file /boot/grub2/grubenv (as in package grub2.rpm)<br />
<br />
<br />
check /etc/grub2.cfg and if needed, change at all positions:<br />
“linuxefi” to “linux16”, and<br />
“intrdefi” to “intrd16”<br />
<br />
=== Problems on old BIOS hardware ===<br />
Some old computers still might not boot since they don't support booting from GPT.<br />
This can be fixed to boot on such old hardware, however the stick afterwards cannot boot on EFI/UEFI systems anymore:<br />
<br />
*set the boot flag (active) on the single 0xEE partition in the protective MBR by using fdisk version <= 2.22 (versions 1.23 and newer have GPT support and thus don't make changes to the protective MBR but to the GPT) (I used fdisk (util-linux-ng 2.17.2) on a ScientificLinux 6.7 machine)<br />
fdisk /dev/sd? <br />
a<br />
part 1<br />
w<br />
<br />
*recompute CHS values using option “h” in gdisk's expert menu:<br />
gdisk /dev/sd?<br />
x<br />
h<br />
m<br />
q<br />
<br />
this can be reversed:<br />
- toggle bootflag off in old fdisk<br />
- recompute CHS values in gdisk<br />
<br />
(using a sgdisk backup of the GPT should also be fine)<br />
<br />
helpful links:<br />
<br />
http://www.rodsbooks.com/gdisk/bios.html#bios --> option h of gdisks experts menu did the trick!<br />
<br />
https://fedoraproject.org/wiki/GRUB_2?rd=Grub2#Updating_GRUB_2_configuration_on_BIOS_systems<br />
<br />
=== generating images and copies of the stick ===<br />
<br />
Save a compressed disk image - we use the parallel gzip program called pigz:<br />
dd if=/dev/sd? bs=4096 | pigz -c usbstick.img.gz<br />
(time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB)<br />
write compressed image back to a stick:<br />
unpigz -c usbstick.img.gz | dd of=/dev/sd? bs=4096<br />
(speed should be >100MB/s)<br />
<br />
=== for techies: taking care of the stick ===<br />
<br />
The following is about performance and durability of the USB stick, and reading it is not required for the functionality described above. If you don't know what TRIM is and how it relates to flash media, this section is probably not for you.<br />
<br />
To fill empty part of all partitions on the stick with “zeros”: for each partition, do (as root)<br />
dd if=/dev/zero of=/mountpoint_of_partition/delete.me bs=10M # this will stop when filesystem is full<br />
rm /mountpoint_of_partition/delete.me<br />
<br />
This is sensible to do after deleting large amounts of data from the USB stick, and before saving and compressing an image of it. It also seems to result in faster writes afterwards i.e. it may be a way to restore the write speed of the stick. <br />
<br />
A better method that does not wear the stick (flash media support a limited number of writes!) is to use TRIM. The SANdisk Extreme supports TRIM, as <code>hdparm -I</code> shows. However, neither the <code>fstrim</code> command nor the <code>discard</code> mount option work for USB sticks (at least not the ones that I tested), presumably because the usb-storage kernel module does not pass the ATA trim command through the USB bridge and controller to the device. <br />
<br />
The workaround I found is to use the <code>wiper.sh</code> script which is part of the <code>hdparm</code> package - this uses <code>hdparm</code> directly. For me, this seems to work, as verified with the [https://sites.google.com/site/lightrush/random-1/checkiftrimonext4isenabledandworking test_trim.sh] script. Use it on your own risk!<br />
<br />
For initializing the whole stick or a specific partition, in principle [http://unix.stackexchange.com/questions/97143/utility-to-trim-unallocated-space-on-drive <code>blkdiscard</code>] may work - but it gave me an error message (seems also to be using usb-storage). <br />
<br />
However, <code>hdparm</code>'s (ENHANCED) SECURITY ERASE initializes the whole stick - I checked the result.</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Bootable_Linux_USB_stick&diff=2380Bootable Linux USB stick2016-03-29T12:21:52Z<p>Karsten: </p>
<hr />
<div>We regularly and successfully use USB sticks for courses where participants bring their own notebooks. The big benefit is that students<br />
learn Linux, and realize that they can easily use the hardware they already own. Current notebook hardware is by far fast enough for data<br />
processing, structure solution and coot visualization. What we found:<br />
# the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do ''not'' have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk.<br />
# we use Fedora 23 because its hardware support is very good. We always use the 64bit distro.<br />
# The stick can be booted on MacBooks as well (press the <code>alt</code> key at the boot sound); their hardware works well with Fedora 23. For Windows clients (press F11 or F12 or sometimes F9 or F10 for the boot menu; if that does not work press F2 or DEL for the BIOS menu and change the boot order), one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes <code>powercfg -H off</code> (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from the stick because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should.<br />
# we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks.<br />
# any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync.<br />
<br />
To make students familiar with the sticks and how to boot them, one needs 30+ minutes and a few tutors. <br />
<br />
'''Some more details'''<br />
<br />
# we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes a biosboot partition (1MB; required for booting bios system from a GPT disk), an efi partition (e.g. 250MB; required for UEFI/EFI boot on PC's/MAC's) and the Linux partition, with 13.9GB, so that the sum of the four partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB stick; the /data partition then does not fit and cannot be used on that stick, but the operating system will then work on the small stick just as well. This requires that the /data partition is not fsck'ed automatically from /etc/fstab (0 in the 6th field).<br />
# it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use<br />
# it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up.<br />
# (if the installation not already does it for you) '''you should label the partitions and use the labels for mounting the partitions in /etc/fstab; don't use /dev/sda1 or the like because depending on the hardware it is booted on, after booting the stick, and depending on the actual hardware, the stick may be /dev/sdb or /dev/sdc or ... !'''<br />
<br />
<br />
<br />
== How to create a bootabled GPT-partitioned USB-stick with Fedora 23 ==<br />
<br />
This will create a USB stick capable to<br />
* EFI boot on Macs<br />
* UEFI boot on PCs<br />
* BIOS boot on newer hardware (legacy boot / CSM enabled in BIOS)<br />
* BIOS boot on medium aged hardware<br />
* not boot on really old BIOS hardware which doesn't support booting from GPT (see below)<br />
<br />
=== GPT partition the stick using <code>gdisk</code> ===<br />
The example shows a 32 GB Sandisk Extreme:<br />
Part 1: +1G VFAT (0700) (Windows needs it to be the first partition) <br />
Part 2: +1M BIOSBOOT (ef02)<br />
Part 3: +250M EFI (ef00)<br />
Part 4: +13500M / (0083)<br />
Part 5: +15000M /mnt/data (0083)<br />
For performance, make sure that the big partitions are aligned to 8192-sector boundaries; gdisk default is 2048-sector boundaries - the default can be adjusted. If you use the above sizes and the + sign for specifying them, this happens to work out automatically.<br />
<br />
=== Install Fedora23 on an UEFI machine (UEFI enabled ; LegacyBoot / CSM disabled) ===<br />
Whenever you see a ?, you must fill in the appropriate drive letter (e.g. c or d). <br />
sd?3: efi /boot/efi<br />
sd?4: ext4 / <br />
sd?5: ext4 /mnt/data<br />
<br />
=== adjust the USB-stick for UEFI/EFI boot (before reboot from chroot environment) ===<br />
<br />
chroot to the installed system:<br />
chroot /mnt/sysimage<br />
<br />
install Fedora updates:<br />
dnf -y update # dnf on FC23 is successor to yum<br />
<br />
<br />
configure grub2 for (U)EFI systems:<br />
<br />
*disable auto recognition of other installed Operating systems (specific to current computer), and<br />
*update grub2-efi.cfg<br />
echo 'GRUB_DISABLE_OS_PROBER=”true”' >> /etc/default/grub<br />
grub2-mkconfig -o /etc/grub2-efi.cfg<br />
(last step automatically puts UUID's to recognise the boot device in grub2-efi.cfg ; default after fresh Fedora installation is the device name which might be different on another computer)<br />
<br />
<br />
Performance tuning (not strictly required):<br />
<br />
We use ext4 filesystems and <code>data=writeback,nobarrier </code> in /etc/fstab. To be able to set these options also on the / filesystem, we use tune2fs to set both as default mount options on a Linux machine where the stick (dev/sd?) is just plugged in (i.e. not booted from):<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sd?4<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sd?5<br />
<br />
create & label vfat filesystem:<br />
mkfs.vfat /dev/sd?1<br />
dosfslabel /dev/sd?1 VFAT<br />
<br />
shutdown or reboot<br />
exit<br />
systemctl poweroff (or reboot for testing)<br />
<br />
=== Install grub2 for BIOS boot (from chroot environment) ===<br />
<br />
boot Fedora Live DVD on a BIOS machine and chroot to the stick:<br />
mount /dev/sd?4 /mnt<br />
mount -o bind /dev /mnt/dev<br />
mount -t proc /proc /mnt/proc<br />
mount -t sysfs /sys /mnt/sys<br />
chroot /mnt<br />
<br />
install grub2 for BIOS boot and configure it:<br />
rm /boot/grub2/grubenv<br />
grub2-install /dev/sd?<br />
grub2-mkconfig -o /etc/grub2.cfg<br />
(on UEFI systems /boot/grub2/grubenv is a symlink to /boot/efi/EFI/fedora/grubenv (from package grub2-efi.rpm). The symlink has to be removed before grub2-install can finish successfully, it automatically creates a new file /boot/grub2/grubenv (as in package grub2.rpm)<br />
<br />
<br />
check /etc/grub2.cfg and if needed, change at all positions:<br />
“linuxefi” to “linux16”, and<br />
“intrdefi” to “intrd16”<br />
<br />
=== Problems on old BIOS hardware ===<br />
Some old computers still might not boot since they don't support booting from GPT.<br />
This can be fixed to boot on such old hardware, however the stick afterwards cannot boot on EFI/UEFI systems anymore:<br />
<br />
*set the boot flag (active) on the single 0xEE partition in the protective MBR by using fdisk version <= 2.22 (versions 1.23 and newer have GPT support and thus don't make changes to the protective MBR but to the GPT) (I used fdisk (util-linux-ng 2.17.2) on a ScientificLinux 6.7 machine)<br />
fdisk /dev/sd? <br />
a<br />
part 1<br />
w<br />
<br />
*recompute CHS values using option “h” in gdisk's expert menu:<br />
gdisk /dev/sd?<br />
x<br />
h<br />
m<br />
q<br />
<br />
this can be reversed:<br />
- toggle bootflag off in old fdisk<br />
- recompute CHS values in gdisk<br />
<br />
(using a sgdisk backup of the GPT should also be fine)<br />
<br />
helpful links:<br />
<br />
http://www.rodsbooks.com/gdisk/bios.html#bios --> option h of gdisks experts menu did the trick!<br />
<br />
https://fedoraproject.org/wiki/GRUB_2?rd=Grub2#Updating_GRUB_2_configuration_on_BIOS_systems<br />
<br />
=== generating images and copies of the stick ===<br />
<br />
Save a compressed disk image - we use the parallel gzip program called pigz:<br />
dd if=/dev/sd? bs=4096 | pigz -c usbstick.img.gz<br />
(time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB)<br />
write compressed image back to a stick:<br />
unpigz -c usbstick.img.gz | dd of=/dev/sd? bs=4096<br />
(speed should be >100MB/s)<br />
<br />
=== for techies: taking care of the stick ===<br />
<br />
The following is about performance and durability of the USB stick, and reading it is not required for the functionality described above. If you don't know what TRIM is and how it relates to flash media, this section is probably not for you.<br />
<br />
To fill empty part of all partitions on the stick with “zeros”: for each partition, do (as root)<br />
dd if=/dev/zero of=/mountpoint_of_partition/delete.me bs=10M # this will stop when filesystem is full<br />
rm /mountpoint_of_partition/delete.me<br />
<br />
This is sensible to do after deleting large amounts of data from the USB stick, and before saving and compressing an image of it. It also seems to result in faster writes afterwards i.e. it may be a way to restore the write speed of the stick. <br />
<br />
A better method that does not wear the stick (flash media support a limited number of writes!) is to use TRIM. The SANdisk Extreme supports TRIM, as <code>hdparm -I</code> shows. However, neither the <code>fstrim</code> command nor the <code>discard</code> mount option work for USB sticks (at least not the ones that I tested), presumably because the usb-storage kernel module does not pass the ATA trim command through the USB bridge and controller to the device. <br />
<br />
The workaround I found is to use the <code>wiper.sh</code> script which is part of the <code>hdparm</code> package - this uses <code>hdparm</code> directly. For me, this seems to work, as verified with the [https://sites.google.com/site/lightrush/random-1/checkiftrimonext4isenabledandworking test_trim.sh] script. Use it on your own risk!<br />
<br />
For initializing the whole stick or a specific partition, in principle [http://unix.stackexchange.com/questions/97143/utility-to-trim-unallocated-space-on-drive <code>blkdiscard</code>] may work - but it gave me an error message (seems also to be using usb-storage). <br />
<br />
However, <code>hdparm</code>'s (ENHANCED) SECURITY ERASE initializes the whole stick - I checked the result.</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Bootable_Linux_USB_stick&diff=2378Bootable Linux USB stick2016-03-29T11:31:17Z<p>Karsten: /* adjust the USB-stick for UEFI/EFI boot (before reboot from chroot environment) */</p>
<hr />
<div>We regularly and successfully use USB sticks for courses where participants bring their own notebooks. The big benefit is that students<br />
learn Linux, and realize that they can easily use the hardware they already own. Current notebook hardware is by far fast enough for data<br />
processing, structure solution and coot visualization. What we found:<br />
# the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do ''not'' have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk.<br />
# we use Fedora 23 because its hardware support is very good. We always use the 64bit distro.<br />
# The stick can be booted on MacBooks as well (press the <code>alt</code> key at the boot sound); their hardware works well with Fedora 23. For Windows clients (press F11 or F12 or sometimes F9 or F10 for the boot menu; if that does not work press F2 or DEL for the BIOS menu and change the boot order), one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes <code>powercfg -H off</code> (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from the stick because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should.<br />
# we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks.<br />
# any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync.<br />
<br />
To make students familiar with the sticks and how to boot them, one needs 30+ minutes and a few tutors. <br />
<br />
'''Some more details'''<br />
<br />
# we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes the Linux partition, with 13.9GB, so that the sum of the two partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB stick; the /data partition then does not fit and cannot be used on that stick, but the operating system will then work on the small stick just as well. This requires that the /data partition is not fsck'ed automatically from /etc/fstab (0 in the 6th field).<br />
# it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use<br />
# it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up.<br />
# (if the installation not already does it for you) '''you should label the partitions and use the labels for mounting the partitions in /etc/fstab; don't use /dev/sda1 or the like because depending on the hardware it is booted on, after booting the stick, and depending on the actual hardware, the stick may be /dev/sdb or /dev/sdc or ... !'''<br />
<br />
<br />
<br />
== How to create a bootabled GPT-partitioned USB-stick with Fedora 23 ==<br />
<br />
This will create a USB stick capable to<br />
* EFI boot on Macs<br />
* UEFI boot on PCs<br />
* BIOS boot on newer hardware (legacy boot / CSM enabled in BIOS)<br />
* BIOS boot on medium aged hardware<br />
* not boot on really old BIOS hardware which doesn't support booting from GPT (see below)<br />
<br />
=== GPT partition the stick using <code>gdisk</code> ===<br />
The example shows a 32 GB Sandisk Extreme:<br />
Part 1: +1G VFAT (0700) (Windows needs it to be the first partition) <br />
Part 2: +1M Biosboot (ef02)<br />
Part 3: +250M EFI (ef00)<br />
Part 4: +13500M / (0083)<br />
Part 5: +15000M /mnt/data (0083)<br />
For performance, make sure that the big partitions are aligned to 8192-sector boundaries; gdisk default is 2048-sector boundaries - the default can be adjusted. If you use the above sizes and the + sign for specifying them, this happens to work out automatically.<br />
<br />
=== Install Fedora23 on an UEFI machine (UEFI enabled; LegacyBoot / CSM disabled) ===<br />
Whenever you see a ?, you must fill in the appropriate drive letter (e.g. c or d). <br />
sd?3: efi /boot/efi<br />
sd?4: ext4 / <br />
sd?5: ext4 /mnt/data<br />
<br />
=== adjust the USB-stick for UEFI/EFI boot (before reboot from chroot environment) ===<br />
<br />
chroot:<br />
chroot /mnt/sysimage<br />
<br />
install Fedora updates:<br />
dnf -y update # dnf on FC23 is successor to yum<br />
<br />
configure grub2 for (U)EFI systems:<br />
<br />
*disable auto recognition of other installed Operating systems (specific to current computer), and<br />
*update grub2-efi.cfg<br />
echo 'GRUB_DISABLE_OS_PROBER=”true”' >> /etc/default/grub<br />
grub2-mkconfig -o /etc/grub2-efi.cfg<br />
(last step automatically puts UUID's to recognise the boot device in grub2-efi.cfg ; default after fresh Fedora installation is the device name which might be different on another computer)<br />
<br />
Performance tuning (not strictly required):<br />
<br />
We use ext4 filesystems and <code>data=writeback,nobarrier </code> in /etc/fstab. To be able to set these options also on the / filesystem, we use tune2fs to set both as default mount options on a Linux machine where the stick (dev/sd?) is just plugged in (i.e. not booted from):<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sd?4<br />
tune2fs -o journal_data_writeback,nobarrier /dev/sd?5<br />
<br />
create & label vfat filesystem:<br />
mkfs.vfat /dev/sd?1<br />
dosfslabel /dev/sd?1 VFAT<br />
<br />
exit<br />
systemctl poweroff (or reboot for testing)<br />
<br />
=== Install grub2 for BIOS boot (from chroot environment) ===<br />
<br />
boot Fedora Live DVD on a BIOS machine and chroot to the stick:<br />
mount /dev/sd?4 /mnt<br />
mount -o bind /dev /mnt/dev<br />
mount -t proc /proc /mnt/proc<br />
mount -t sysfs /sys /mnt/sys<br />
chroot /mnt<br />
<br />
Install grub2 for BIOS boot and configure it:<br />
rm /boot/grub2/grubenv<br />
(symlink to /boot/efi/EFI/fedora/grubenv in grub2-efi.rpm; binary file in grub2.rpm)<br />
grub2-install /dev/sd?<br />
(also creates a new /boot/grub2/grubenv<br />
grub2-mkconfig -o /etc/grub2.cfg<br />
<br />
check /etc/grub2.cfg and if needed, change at all positions<br />
“linuxefi” to “linux16”, and<br />
“intrdefi” to “intrd16”<br />
<br />
=== Problems on old BIOS hardware ===<br />
some old computers still might not boot (they don't support booting from GPT).<br />
This can be fixed but the stick afterwards cannot boot on EFI/UEFI systems anymore.<br />
a) set the boot flag (active) on the single 0xEE partition in the protective MBR by using fdisk version <= 2.22 (versions 1.23 and newer have GPT support and thus don't make changes to the protective MBR but to the GPT) (I used fdisk (util-linux-ng 2.17.2) on a ScientificLinux 6.7 machine)<br />
fdisk /dev/sd? <br />
a<br />
part 1<br />
w<br />
b) recompute CHS values using option “h” in gdisk's expert menu:<br />
gdisk /dev/sd?<br />
x<br />
h<br />
m<br />
q<br />
<br />
this can be reversed:<br />
- toggle bootflag off in old fdisk<br />
- recompute CHS values in gdisk<br />
<br />
(using a sgdisk backup of the GPT should also be fine)<br />
<br />
helpful links:<br />
<br />
http://www.rodsbooks.com/gdisk/bios.html#bios --> option h of gdisks experts menu did the trick!<br />
<br />
https://fedoraproject.org/wiki/GRUB_2?rd=Grub2#Updating_GRUB_2_configuration_on_BIOS_systems<br />
<br />
=== generating images and copies of the stick ===<br />
<br />
Save a compressed disk image - we use the parallel gzip program called pigz:<br />
dd if=/dev/sd? bs=4096 | pigz -c usbstick.img.gz<br />
(time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB)<br />
write compressed image back to a stick:<br />
unpigz -c usbstick.img.gz | dd of=/dev/sd? bs=4096<br />
(speed should be >100MB/s)<br />
<br />
=== for techies: taking care of the stick ===<br />
<br />
The following is about performance and durability of the USB stick, and reading it is not required for the functionality described above. If you don't know what TRIM is and how it relates to flash media, this section is probably not for you.<br />
<br />
To fill empty part of all partitions on the stick with “zeros”: for each partition, do (as root)<br />
dd if=/dev/zero of=/mountpoint_of_partition/delete.me bs=10M # this will stop when filesystem is full<br />
rm /mountpoint_of_partition/delete.me<br />
<br />
This is sensible to do after deleting large amounts of data from the USB stick, and before saving and compressing an image of it. It also seems to result in faster writes afterwards i.e. it may be a way to restore the write speed of the stick. <br />
<br />
A better method that does not wear the stick (flash media support a limited number of writes!) is to use TRIM. The SANdisk Extreme supports TRIM, as <code>hdparm -I</code> shows. However, neither the <code>fstrim</code> command nor the <code>discard</code> mount option work for USB sticks (at least not the ones that I tested), presumably because the usb-storage kernel module does not pass the ATA trim command through the USB bridge and controller to the device. <br />
<br />
The workaround I found is to use the <code>wiper.sh</code> script which is part of the <code>hdparm</code> package - this uses <code>hdparm</code> directly. For me, this seems to work, as verified with the [https://sites.google.com/site/lightrush/random-1/checkiftrimonext4isenabledandworking test_trim.sh] script. Use it on your own risk!<br />
<br />
For initializing the whole stick or a specific partition, [http://unix.stackexchange.com/questions/97143/utility-to-trim-unallocated-space-on-drive <code>blkdiscard</code>] may work - I have not yet checked it. I do believe that <code>hdparm</code>'s (ENHANCED) SECURITY ERASE should initialize the whole stick - again, I have not yet tried it.</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Bootable_Linux_USB_stick&diff=2351Bootable Linux USB stick2016-03-08T15:35:46Z<p>Karsten: Created page with "We regularly and successfully use USB sticks for courses where participants bring their own notebooks. The big benefit is that students learn Linux, and realize that they can..."</p>
<hr />
<div>We regularly and successfully use USB sticks for courses where participants bring their own notebooks. The big benefit is that students<br />
learn Linux, and realize that they can easily use the hardware they<br />
already own. Current notebook hardware is by far fast enough for data<br />
processing, structure solution and coot visualization. What we found:<br />
# the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB). The computers do_not_ have to be recent, nor do they have to have USB3 ports<br />
# we use Fedora 23 because its hardware support is very good<br />
# we produce >50% BIOS-bootable sticks and the rest are EFI-bootable sticks (we have not yet found out how to combine this into one). The latter can be booted on Macbooks as well; their hardware works well with Fedora 23. For Windows clients, one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes powercfg -H off (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from any of the sticks because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should.<br />
# performance tuning (not required): We use an ext4 filesystem and <pre> data=writeback,nobarrier </pre> in /etc/fstab. To set the writeback option on the / filesystem one needs to use tune2fs on a Linux machine where the stick is just mounted (i.e. not booted from), with the option data_journal_writeback; one should also set nobarrier that way. With this tuning the stick is as fast as a local harddisk.<br />
# we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple<br />
# we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks.<br />
# any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync.<br />
<br />
To make students familiar with the sticks and how to boot them, one<br />
needs 30+ minutes and a few tutors. <br />
<br />
If somebody figures out how to install Fedora23 sticks that boot on<br />
both BIOS- and EFI hardware, I'd like to hear about this.<br />
<br />
<br />
== Some more details ==<br />
<br />
# As for normal OS installation it matters if for the installation of the stick your computer is booted in BIOS mode (on newer machines "CSM enabled") --> you obtain a stick which boots BIOS systems, or if for the installation your computer is booted in EFI mode --> you obtain a stick which boots in EFI mode on newer computers and Macs<br />
# the SANdisk Extreme is the best stick we found; physically small sticks look nicer but are slow (even though they are called USB3)<br />
# we always use the 64bit distro<br />
# the FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes the Linux partition, with 13.9GB, so that the sum of the two partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB stick; the /data partition then does not fit and cannot be used on that stick, but the operating system will then work on the small stick just as well. This requires that the /data partition is not mounted automatically from /etc/fstab (i.e. it should be listed, but with the noauto option), but rather with a command like "mount /data &" from /etc/rc.local - because if the 16GB stick is booted, it will otherwise find a corrupt filesystem and the boot will fail, leaving you with a rudimentary shell prompt which only experts can recover from. Also this means that the image of the stick need only comprise the first 16GB (unless the /data partition already has something on it) - that makes it faster to copy it, and it is quite fast to re-create an empty /data partition after booting the stick.<br />
# it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use<br />
# it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up.<br />
# (if the installation not already does it for you) '''you should label the partitions and use the labels for mounting the partitions in /etc/fstab; don't use /dev/sda1 or the like because depending on the hardware it is booted on, after booting the stick, and depending on the actual hardware, the stick may be /dev/sdb or /dev/sdc !'''</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Centric_and_acentric_reflections&diff=2350Centric and acentric reflections2016-03-08T15:34:40Z<p>Karsten: </p>
<hr />
<div>Centric reflections play the same role in reciprocal space as special positions in real space: they may occur at the borders of the asymmetric unit. It makes sense to discuss the reciprocal case and the real space together.<br />
<br />
A definition and a theorem about centric reflections are stated here (see reference [1]) before the role of centrics is examined.<br />
<br />
Definition: '''A reflection (h,k,l) is said to be centric if in the space group there is at least one symmetry operation g(x)=R_g*x+t_g whose rotational part R_g sends the reflection to minus itself''', i.e.:<br />
<br />
(h,k,l) is centric if there is a symop g(x)=R_g*x+t_g in G such that R_g*(h,k,l)=(-h,-k,-l)<br />
<br />
For example all reflection in the zone (h,k,0) are centrics in all space groups with twofold axes down c.<br />
<br />
Theorem: '''The phase of a centric reflection is restricted to phi(h,k,l)=pi*(h*tx_g+k*ty_g+l*tz_g) plus or minus any integer multiple of pi'''<br />
<br />
where the vector t_g=(tx_g,ty_g,tz_g) is the translational part of the symop g that causes the reflection to be centric.<br />
<br />
== Real space ==<br />
<br />
Let's first look at real space. A special position results if there exists one or more symmetry operators, other than the trivial operator {x,y,z}, which map this position upon itself. As an example: take spacegroup P2 with its symmetry operators {x,y,z} and {-x,y,-z}. Now consider any point with x=0 and z=0, and some value of y. Obviously this point, when transformed with -x,y,-z , yields 0,y,0 - just the same point! Thus this is a special position. Generally, positions on n-fold symmetry axes are special.<br />
<br />
Mathematically, to find special positions we have to solve the Eigenproblem A v = v where A is the symmetry operator (expressed as rotation matrix and translation vector), and v, the Eigenvector, represents the special position(s). For a given space group, we need to check all symmetry operators. <br />
<br />
An atom at a special position usually has (at most) an occupancy of 0.5. However, it may happen that more than one symmetry operator maps the special position upon itself; in that case the occupancy is 1/(number of positions generated by all symmetry operators that map the point onto itself). Thus, a point on a n-fold rotation axis has (maximum) occupancy of 1/n. Disorder or partial occupation will result in lower occupancy. <br />
<br />
In space group P2<sub>1</sub>, there are no special positions - the Eigenproblem has no solution.<br />
<br />
== Reciprocal space ==<br />
<br />
In reciprocal space the situation is quite similar. A reflection is centric if there is a reciprocal space symmetry operator which maps it onto itself (or rather its Friedel mate). Reciprocal space symmetry operators can be obtained from the real space symmetry operators by following two rules:<br />
<br />
a) take the rotation matrix and transpose it<br />
<br />
b) omit the translation vector<br />
<br />
Whereas rule b) makes things slightly easier in reciprocal space, we must be aware that in reciprocal space we have additional symmetry, namely Friedel symmetry. This means for each reciprocal symmetry operator we also have to consider the Friedel-related operator (all elements of the matrix multiplied by -1).<br />
<br />
To find centric reflections, we just solve the Eigenvalue problem A v = -v, now considering each reciprocal space symmetry operator in turn.<br />
Centric reflections in space group P2 and P2<sub>1</sub> are thus those with 0,k,0. There exist space groups without centric reflections, like R3.<br />
<br />
Properties: centric reflections have only two phase possibilities, e.g. 0° and 180° (but in any case 180° apart), and centric reflections do not have an anomalous signal (can these properties be easily derived here?). <br />
<br />
Furthermore, the "intensity statistics" [2] of acentric reflections <br />
(<math> P(|E|) = 2 |E| e^{-|E|^2} </math> ) <br />
<br />
[[file:I_acentrics.png]]<br />
<br />
are different from those of centric reflections<br />
(<math> P(|E|) = \sqrt{\frac{2}{\pi}} e^{-|E|^2/2} </math> ) <br />
<br />
[[file:I_centrics.png]]<br />
.<br />
<br />
Centric reflections have a special role in experimental [[phasing]].<br />
<br />
== References ==<br />
[1] G. Bricogne, "Fourier Transforms in Crystallography", page 68, Chapter 1.3, International Tables for Crystallography Volume B, Kluwer Publishers (2001)<br />
<br />
[2] U. Shmueli and A. J. C. Wilson, "Statistical properties of the weighted reciprocal lattice", page 190-209, Chapter 2.1, International Tables for Crystallography Volume B, Kluwer Publishers (2006)</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=File:I_centrics.png&diff=2349File:I centrics.png2016-03-08T15:34:11Z<p>Karsten: </p>
<hr />
<div></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=File:I_acentrics.png&diff=2348File:I acentrics.png2016-03-08T15:33:45Z<p>Karsten: </p>
<hr />
<div></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Current_events&diff=2347Current events2016-03-08T15:31:56Z<p>Karsten: </p>
<hr />
<div><br />
* [[Course_or_Conference_1| put text and date here]]<br />
<br />
<br />
<br />
* [[Course_or_Conference_10| another conference/time]]<br />
<br />
=== Note those who add new items to this page===<br />
To there are articles with generic names Course_or_Conference_1, Course_or_Conference_2 and so on. If you add a course or conference, please edit an empty or expired article with generic name, and adjust the name associated with the link on this page accordingly. Please, regardless of the _n name put the most recent even first</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Test&diff=2278Test2015-01-23T13:52:05Z<p>Karsten: </p>
<hr />
<div>We test tex:<br />
<br />
<math><br />
F_{hkl} = \sum_{i=1}^{N} e^{-2\pi\imath\left( h\frac{x}{a}+k\frac{y}{b}+l\frac{z}{c}\right)}<br />
</math><br />
<br />
[[Image:Acrb.jpg|thumb|test]]<br />
<br />
All crystals should look like this (or better)<br />
[[Image:xtals.jpg||pretty xtals]]<br />
<br />
Problem? - I tried uploading xtals.png but Wiki does not let me upload, quoting that 'wrong file type or extension' - yet it allowed me to load up the same image in jpg format. Png is a nice (some would even say the best) format so it'd be good to be able to use it! -AGE<br />
This is fixed now. --[[User:Kay|Kay]] 10:12, 5 February 2008 (CET)<br />
<br/><br />
Remark: <br />
MIME type of PNG files was not correctly identified. To fix this, we added the lines<br />
# PNG<br />
1 string PNG image/png<br />
to the file /etc/httpd/conf/magic<br />
<br />
Logo prototype #1<br />
[[Image:logo1.png]]<br />
<br />
Logo prototype #2<br />
[[Image:logo2.png]]<br />
<br />
do you like any of these?<br />
<br />
Testing the gnuplot extension (the plot is generated in realtime, not a bitmap)<br />
<gnuplot><br />
set hidden3d<br />
set isosamples 50<br />
splot sin(x)*cos(y)<br />
</gnuplot></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Test&diff=2277Test2015-01-23T13:51:55Z<p>Karsten: </p>
<hr />
<div>xxxWe test tex:<br />
<br />
<math><br />
F_{hkl} = \sum_{i=1}^{N} e^{-2\pi\imath\left( h\frac{x}{a}+k\frac{y}{b}+l\frac{z}{c}\right)}<br />
</math><br />
<br />
[[Image:Acrb.jpg|thumb|test]]<br />
<br />
All crystals should look like this (or better)<br />
[[Image:xtals.jpg||pretty xtals]]<br />
<br />
Problem? - I tried uploading xtals.png but Wiki does not let me upload, quoting that 'wrong file type or extension' - yet it allowed me to load up the same image in jpg format. Png is a nice (some would even say the best) format so it'd be good to be able to use it! -AGE<br />
This is fixed now. --[[User:Kay|Kay]] 10:12, 5 February 2008 (CET)<br />
<br/><br />
Remark: <br />
MIME type of PNG files was not correctly identified. To fix this, we added the lines<br />
# PNG<br />
1 string PNG image/png<br />
to the file /etc/httpd/conf/magic<br />
<br />
Logo prototype #1<br />
[[Image:logo1.png]]<br />
<br />
Logo prototype #2<br />
[[Image:logo2.png]]<br />
<br />
do you like any of these?<br />
<br />
Testing the gnuplot extension (the plot is generated in realtime, not a bitmap)<br />
<gnuplot><br />
set hidden3d<br />
set isosamples 50<br />
splot sin(x)*cos(y)<br />
</gnuplot></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Test&diff=2242Test2014-10-31T14:28:44Z<p>Karsten: </p>
<hr />
<div>We test tex:<br />
<br />
<math><br />
F_{hkl} = \sum_{i=1}^{N} e^{-2\pi\imath\left( h\frac{x}{a}+k\frac{y}{b}+l\frac{z}{c}\right)}<br />
</math><br />
<br />
[[Image:Acrb.jpg|thumb|test]]<br />
<br />
All crystals should look like this (or better)<br />
[[Image:xtals.jpg||pretty xtals]]<br />
<br />
Problem? - I tried uploading xtals.png but Wiki does not let me upload, quoting that 'wrong file type or extension' - yet it allowed me to load up the same image in jpg format. Png is a nice (some would even say the best) format so it'd be good to be able to use it! -AGE<br />
This is fixed now. --[[User:Kay|Kay]] 10:12, 5 February 2008 (CET)<br />
<br/><br />
Remark: <br />
MIME type of PNG files was not correctly identified. To fix this, we added the lines<br />
# PNG<br />
1 string PNG image/png<br />
to the file /etc/httpd/conf/magic<br />
<br />
Logo prototype #1<br />
[[Image:logo1.png]]<br />
<br />
Logo prototype #2<br />
[[Image:logo2.png]]<br />
<br />
do you like any of these?<br />
<br />
Testing the gnuplot extension (the plot is generated in realtime, not a bitmap)<br />
<gnuplot><br />
set hidden3d<br />
set isosamples 50<br />
splot sin(x)*cos(y)<br />
</gnuplot></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Test&diff=2241Test2014-10-31T14:28:34Z<p>Karsten: </p>
<hr />
<div>We test tex:<br />
haha<br />
<math><br />
F_{hkl} = \sum_{i=1}^{N} e^{-2\pi\imath\left( h\frac{x}{a}+k\frac{y}{b}+l\frac{z}{c}\right)}<br />
</math><br />
<br />
[[Image:Acrb.jpg|thumb|test]]<br />
<br />
All crystals should look like this (or better)<br />
[[Image:xtals.jpg||pretty xtals]]<br />
<br />
Problem? - I tried uploading xtals.png but Wiki does not let me upload, quoting that 'wrong file type or extension' - yet it allowed me to load up the same image in jpg format. Png is a nice (some would even say the best) format so it'd be good to be able to use it! -AGE<br />
This is fixed now. --[[User:Kay|Kay]] 10:12, 5 February 2008 (CET)<br />
<br/><br />
Remark: <br />
MIME type of PNG files was not correctly identified. To fix this, we added the lines<br />
# PNG<br />
1 string PNG image/png<br />
to the file /etc/httpd/conf/magic<br />
<br />
Logo prototype #1<br />
[[Image:logo1.png]]<br />
<br />
Logo prototype #2<br />
[[Image:logo2.png]]<br />
<br />
do you like any of these?<br />
<br />
Testing the gnuplot extension (the plot is generated in realtime, not a bitmap)<br />
<gnuplot><br />
set hidden3d<br />
set isosamples 50<br />
splot sin(x)*cos(y)<br />
</gnuplot></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Test&diff=2240Test2014-10-31T14:28:01Z<p>Karsten: </p>
<hr />
<div>We test tex:<br />
<br />
<math><br />
F_{hkl} = \sum_{i=1}^{N} e^{-2\pi\imath\left( h\frac{x}{a}+k\frac{y}{b}+l\frac{z}{c}\right)}<br />
</math><br />
<br />
[[Image:Acrb.jpg|thumb|test]]<br />
<br />
All crystals should look like this (or better)<br />
[[Image:xtals.jpg||pretty xtals]]<br />
<br />
Problem? - I tried uploading xtals.png but Wiki does not let me upload, quoting that 'wrong file type or extension' - yet it allowed me to load up the same image in jpg format. Png is a nice (some would even say the best) format so it'd be good to be able to use it! -AGE<br />
This is fixed now. --[[User:Kay|Kay]] 10:12, 5 February 2008 (CET)<br />
<br/><br />
Remark: <br />
MIME type of PNG files was not correctly identified. To fix this, we added the lines<br />
# PNG<br />
1 string PNG image/png<br />
to the file /etc/httpd/conf/magic<br />
<br />
Logo prototype #1<br />
[[Image:logo1.png]]<br />
<br />
Logo prototype #2<br />
[[Image:logo2.png]]<br />
<br />
do you like any of these?<br />
<br />
Testing the gnuplot extension (the plot is generated in realtime, not a bitmap)<br />
<gnuplot><br />
set hidden3d<br />
set isosamples 50<br />
splot sin(x)*cos(y)<br />
</gnuplot></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Test&diff=2239Test2014-10-31T14:26:13Z<p>Karsten: </p>
<hr />
<div>We test tex:<br />
huhu<br />
<math><br />
F_{hkl} = \sum_{i=1}^{N} e^{-2\pi\imath\left( h\frac{x}{a}+k\frac{y}{b}+l\frac{z}{c}\right)}<br />
</math><br />
<br />
[[Image:Acrb.jpg|thumb|test]]<br />
<br />
All crystals should look like this (or better)<br />
[[Image:xtals.jpg||pretty xtals]]<br />
<br />
Problem? - I tried uploading xtals.png but Wiki does not let me upload, quoting that 'wrong file type or extension' - yet it allowed me to load up the same image in jpg format. Png is a nice (some would even say the best) format so it'd be good to be able to use it! -AGE<br />
This is fixed now. --[[User:Kay|Kay]] 10:12, 5 February 2008 (CET)<br />
<br/><br />
Remark: <br />
MIME type of PNG files was not correctly identified. To fix this, we added the lines<br />
# PNG<br />
1 string PNG image/png<br />
to the file /etc/httpd/conf/magic<br />
<br />
Logo prototype #1<br />
[[Image:logo1.png]]<br />
<br />
Logo prototype #2<br />
[[Image:logo2.png]]<br />
<br />
do you like any of these?<br />
<br />
Testing the gnuplot extension (the plot is generated in realtime, not a bitmap)<br />
<gnuplot><br />
set hidden3d<br />
set isosamples 50<br />
splot sin(x)*cos(y)<br />
</gnuplot></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Test&diff=2233Test2014-10-14T11:20:54Z<p>Karsten: </p>
<hr />
<div>We test tex:<br />
<br />
<math><br />
F_{hkl} = \sum_{i=1}^{N} e^{-2\pi\imath\left( h\frac{x}{a}+k\frac{y}{b}+l\frac{z}{c}\right)}<br />
</math><br />
<br />
[[Image:Acrb.jpg|thumb|test]]<br />
<br />
All crystals should look like this (or better)<br />
[[Image:xtals.jpg||pretty xtals]]<br />
<br />
Problem? - I tried uploading xtals.png but Wiki does not let me upload, quoting that 'wrong file type or extension' - yet it allowed me to load up the same image in jpg format. Png is a nice (some would even say the best) format so it'd be good to be able to use it! -AGE<br />
This is fixed now. --[[User:Kay|Kay]] 10:12, 5 February 2008 (CET)<br />
<br/><br />
Remark: <br />
MIME type of PNG files was not correctly identified. To fix this, we added the lines<br />
# PNG<br />
1 string PNG image/png<br />
to the file /etc/httpd/conf/magic<br />
<br />
Logo prototype #1<br />
[[Image:logo1.png]]<br />
<br />
Logo prototype #2<br />
[[Image:logo2.png]]<br />
<br />
do you like any of these?<br />
<br />
Testing the gnuplot extension (the plot is generated in realtime, not a bitmap)<br />
<gnuplot><br />
set hidden3d<br />
set isosamples 50<br />
splot sin(x)*cos(y)<br />
</gnuplot></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Test&diff=2232Test2014-10-14T11:20:29Z<p>Karsten: </p>
<hr />
<div>We test tex:<br />
<br />
<math><br />
F_{hkl} = \sum_{i=1}^{N} e^{-2*5/3\pi\imath\left( h\frac{x}{a}+k\frac{y}{b}+l\frac{z}{c}\right)}<br />
</math><br />
<br />
[[Image:Acrb.jpg|thumb|test]]<br />
<br />
All crystals should look like this (or better)<br />
[[Image:xtals.jpg||pretty xtals]]<br />
<br />
Problem? - I tried uploading xtals.png but Wiki does not let me upload, quoting that 'wrong file type or extension' - yet it allowed me to load up the same image in jpg format. Png is a nice (some would even say the best) format so it'd be good to be able to use it! -AGE<br />
This is fixed now. --[[User:Kay|Kay]] 10:12, 5 February 2008 (CET)<br />
<br/><br />
Remark: <br />
MIME type of PNG files was not correctly identified. To fix this, we added the lines<br />
# PNG<br />
1 string PNG image/png<br />
to the file /etc/httpd/conf/magic<br />
<br />
Logo prototype #1<br />
[[Image:logo1.png]]<br />
<br />
Logo prototype #2<br />
[[Image:logo2.png]]<br />
<br />
do you like any of these?<br />
<br />
Testing the gnuplot extension (the plot is generated in realtime, not a bitmap)<br />
<gnuplot><br />
set hidden3d<br />
set isosamples 50<br />
splot sin(x)*cos(y)<br />
</gnuplot></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Test&diff=2131Test2012-09-13T14:04:14Z<p>Karsten: </p>
<hr />
<div>We test tex:<br />
<br />
<math><br />
F_{hkl} = \sum_{i=1}^{N} e^{-2\pi\imath\left( h\frac{x}{a}+k\frac{y}{b}+l\frac{z}{c}\right)}<br />
</math><br />
<br />
[[Image:Acrb.jpg|thumb|test]]<br />
<br />
All crystals should look like this (or better)<br />
[[Image:xtals.jpg||pretty xtals]]<br />
<br />
Problem? - I tried uploading xtals.png but Wiki does not let me upload, quoting that 'wrong file type or extension' - yet it allowed me to load up the same image in jpg format. Png is a nice (some would even say the best) format so it'd be good to be able to use it! -AGE<br />
This is fixed now. --[[User:Kay|Kay]] 10:12, 5 February 2008 (CET)<br />
<br/><br />
Remark: <br />
MIME type of PNG files was not correctly identified. To fix this, we added the lines<br />
# PNG<br />
1 string PNG image/png<br />
to the file /etc/httpd/conf/magic<br />
<br />
Logo prototype #1<br />
[[Image:logo1.png]]<br />
<br />
Logo prototype #2<br />
[[Image:logo2.png]]<br />
<br />
do you like any of these?<br />
<br />
Testing the gnuplot extension (the plot is generated in realtime, not a bitmap)<br />
<gnuplot><br />
set hidden3d<br />
set isosamples 50<br />
splot sin(x)*cos(y)<br />
</gnuplot></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Test&diff=2130Test2012-09-13T14:01:26Z<p>Karsten: </p>
<hr />
<div>We test tex:<br />
<math>a+b=c</math><br />
<br />
<math><br />
F_{hkl} = \sum_{i=1}^{N} e^{-2\pi\imath\left( h\frac{x}{a}+k\frac{y}{b}+l\frac{z}{c}\right)}<br />
</math><br />
<br />
[[Image:Acrb.jpg|thumb|test]]<br />
<br />
All crystals should look like this (or better)<br />
[[Image:xtals.jpg||pretty xtals]]<br />
<br />
Problem? - I tried uploading xtals.png but Wiki does not let me upload, quoting that 'wrong file type or extension' - yet it allowed me to load up the same image in jpg format. Png is a nice (some would even say the best) format so it'd be good to be able to use it! -AGE<br />
This is fixed now. --[[User:Kay|Kay]] 10:12, 5 February 2008 (CET)<br />
<br/><br />
Remark: <br />
MIME type of PNG files was not correctly identified. To fix this, we added the lines<br />
# PNG<br />
1 string PNG image/png<br />
to the file /etc/httpd/conf/magic<br />
<br />
Logo prototype #1<br />
[[Image:logo1.png]]<br />
<br />
Logo prototype #2<br />
[[Image:logo2.png]]<br />
<br />
do you like any of these?<br />
<br />
Testing the gnuplot extension (the plot is generated in realtime, not a bitmap)<br />
<gnuplot><br />
set hidden3d<br />
set isosamples 50<br />
splot sin(x)*cos(y)<br />
</gnuplot></div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Stereo&diff=1917Stereo2011-02-02T10:21:02Z<p>Karsten: /* Stereo on conventional CRT monitors */</p>
<hr />
<div>== Stereo on TFT monitors ==<br />
Question: is someone using Nvidia 3D vision + a compatible 1920x1080 23.5" Desktop Display e.g. ACER GD245HQ 120 Hz LCD display OR Alienware OptX AW2310 120 Hz LCD display? Is it running nicely with Linux + Nvidia's Linux driver? How is the stereo quality compared to Zalman's 3D-LCDs or the old CrystalEyes shutter glasses + CRT monitor?<br />
<br />
Answers:<br />
# I am using the combination you are asking for with alienware monitor. The driver I am using is 195.30 beta. It works very well. Quality is much much better than Zalman and somewhat better than CRTs.<br />
# I have Nvidia 3D vision running on the Samsung SyncMaster 2233. One note of caution, for the Quadra FX3800 if you are using dual monitors I haven't found a way to get TwinView to work with the Nvidia 195.30 beta linux driver. You can configure for 2 X-screens to drive the two screens as a work around. This is most likely a 3800 problem since the 3800 has 1 DVI and 2 HDMI outputs instead of 2 DVI outputs. If you are looking for a new graphics card as well and like dual monitors I would stay away from the Quadro FX1800 and FX3800. The Nvidia 3D vision quality is better than Zalman but the Zalman glasses are much lighter if you do a lot of stereo.<br />
<br />
Follow up question: Is your Nvidia 3D vision emitter connected via the 3-pin DIN or USB? The Nvidia 3D vision emitter I have only connects via USB and I could not get this working under linux.<br />
<br />
== Stereo on conventional CRT monitors ==<br />
<br />
Some of the NVidia Quadro cards support stereo. The cards that have an output called "stereo" under "Display Connectors" listed at [http://www.nvidia.com/object/IO_11761.html Nvidia's Quadro overview page] have a 3-pin DIN outlet that fits with [http://www.nuvision3d.com/the60gx.html NuVision] or [http://www.reald-corporate.com/scientific CrystalEyes] stereo glasses.<br />
<br />
The cheapest of these used to be the FX1400 (difficult to find these days, around 450 €), but now appears to be the FX3450 (around 750 €). These cards are by far fast enough for protein crystallography or modelling.<br />
<br />
For stereo, the xorg.conf might need the following lines<br />
Section "Extensions"<br />
Option "Composite" "Disable"<br />
EndSection<br />
if the X log file (e.g. at /var/log/Xorg.0.log) says that stereo is not supported by composite.<br />
<br />
Another option that will be required in xorg.conf by programs running stereo is<br />
<br />
Section "Device"<br />
Driver "nvidia"<br />
Option "Stereo" "3"<br />
<br />
----<br />
<br />
Sometimes it is handy to configure two Desktops: one on a CRT monitor that can do stereo-graphics, and one on an LCD monitor for the more regular work, eg your refinement jobs with CCP4. For Nvidia cards you need to modify the xorg.conf file to have a section more or less like that:<br />
<br />
<code><br />
Section "Monitor"<br />
Identifier "Monitor0"<br />
VendorName "Iiyama"<br />
ModelName "Vision Master Pro 512"<br />
DisplaySize 450 330<br />
HorizSync 31.5 - 120.0<br />
VertRefresh 50.0 - 150.0<br />
Option "dpms"<br />
EndSection<br />
Section "Monitor"<br />
Identifier "Monitor1"<br />
VendorName "Philips"<br />
ModelName "150B"<br />
EndSection<br />
Section "Device"<br />
Identifier "Videocard0"<br />
Driver "nvidia"<br />
VendorName "Videocard vendor"<br />
BoardName "NVIDIA Quadro FX (generic)"<br />
BusId "PCI:1:0:0"<br />
Screen 0<br />
EndSection<br />
Section "Device"<br />
Identifier "Videocard1"<br />
Driver "nvidia"<br />
VendorName "Videocard vendor"<br />
BoardName "NVIDIA Quadro FX (generic)"<br />
Option "Stereo" "3"<br />
BusId "PCI:1:0:0"<br />
Screen 1<br />
EndSection<br />
Section "Screen"<br />
Identifier "Screen0"<br />
Device "Videocard0"<br />
Monitor "Monitor0"<br />
DefaultDepth 24 <br />
SubSection "Display"<br />
Viewport 0 0<br />
Depth 24 <br />
Modes "1024x768" "800x600"<br />
EndSubSection<br />
EndSection<br />
Section "Screen"<br />
Identifier "Screen1"<br />
Device "Videocard1"<br />
Monitor "Monitor1"<br />
DefaultDepth 24 <br />
SubSection "Display"<br />
Viewport 0 0<br />
Depth 24 <br />
Modes "1600x1280" "1280x1024" "1024x768" "800x600"<br />
<br />
</code><br />
<br />
Note that one can think this is rather silly to define two card, two monitors, two cards, and two screens.<br />
However, that the only way I know that one monitor can be stereo-enabled and the other one not.<br />
<br />
==Ono==<br />
You also need to set the environment variable STEREO for the stereo to work properly in ono:<br />
setenv STEREO on (tcsh)<br />
STEREO = on; export STEREO (bash)<br />
[http://dbb.urmc.rochester.edu/labs/wedekind/Wedekind-Lab/X-ray%20Lab/Methods/Nvidia-ONO.html]<br />
<br />
==Mac OS X==<br />
The following command needs to be run for Macs to be able to support stereo in X11 programs, such as Coot [http://xanana.ucsc.edu/~wgscott/xtal/wiki/index.php/Installing_Coot_on_OS_X] :<br />
defaults write com.apple.x11 enable_stereo -bool true<br />
<br />
<br />
==See also==<br />
Stereo on TFT: see [[Coot zalman]]<br />
<br />
http://pymol.sourceforge.net/stereo3d.html<br />
<br />
http://pymolwiki.org/index.php/Stereo_3D_Display_Options</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Stereo&diff=1916Stereo2011-02-02T10:20:46Z<p>Karsten: /* Stereo on conventional CRT monitors */</p>
<hr />
<div>== Stereo on TFT monitors ==<br />
Question: is someone using Nvidia 3D vision + a compatible 1920x1080 23.5" Desktop Display e.g. ACER GD245HQ 120 Hz LCD display OR Alienware OptX AW2310 120 Hz LCD display? Is it running nicely with Linux + Nvidia's Linux driver? How is the stereo quality compared to Zalman's 3D-LCDs or the old CrystalEyes shutter glasses + CRT monitor?<br />
<br />
Answers:<br />
# I am using the combination you are asking for with alienware monitor. The driver I am using is 195.30 beta. It works very well. Quality is much much better than Zalman and somewhat better than CRTs.<br />
# I have Nvidia 3D vision running on the Samsung SyncMaster 2233. One note of caution, for the Quadra FX3800 if you are using dual monitors I haven't found a way to get TwinView to work with the Nvidia 195.30 beta linux driver. You can configure for 2 X-screens to drive the two screens as a work around. This is most likely a 3800 problem since the 3800 has 1 DVI and 2 HDMI outputs instead of 2 DVI outputs. If you are looking for a new graphics card as well and like dual monitors I would stay away from the Quadro FX1800 and FX3800. The Nvidia 3D vision quality is better than Zalman but the Zalman glasses are much lighter if you do a lot of stereo.<br />
<br />
Follow up question: Is your Nvidia 3D vision emitter connected via the 3-pin DIN or USB? The Nvidia 3D vision emitter I have only connects via USB and I could not get this working under linux.<br />
<br />
== Stereo on conventional CRT monitors ==<br />
<br />
Some of the NVidia Quadro cards support stereo. The cards that have an output called "stereo" under "Display Connectors" listed at [http://www.nvidia.com/object/IO_11761.html Nvidia's Quadro overview page] have a 3-pin DIN outlet that fits with [http://www.nuvision3d.com/the60gx.html NuVision] or [http://www.reald-corporate.com/scientific CrystalEyes] stereo glasses.<br />
<br />
The cheapest of these used to be the FX1400 (difficult to find these days, around 450 €), but now appears to be the FX3450 (around 750 €). These cards are by far fast enough for protein crystallography or modelling.<br />
<br />
For stereo, the xorg.conf might need the following lines<br />
Section "Extensions"<br />
Option "Composite" "Disable"<br />
EndSection<br />
if the X log file (e.g. at /var/log/Xorg.0.log) says that stereo is not supported by composite.<br />
<br />
Another option that will be required in xorg.conf by programs running stereo is<br />
<br />
Section "Device"<br />
Driver "nvidia"<br />
Option "Stereo" "3"<br />
<br />
----<br />
<br />
Sometimes it is handy to configure two Desktops: one on a CRT monitor that can do stereo-graphics, and one on an LCD monitor for the more regular work, eg your refinement jobs with CCP4. For Nvidia cards you need to modify the xorg.conf file to have a section more or less like that:<br />
<br />
<code><br />
Section "Monitor"<br />
Identifier "Monitor0"<br />
VendorName "Iiyama"<br />
ModelName "Vision Master Pro 512"<br />
DisplaySize 450 330<br />
HorizSync 31.5 - 120.0<br />
VertRefresh 50.0 - 150.0<br />
Option "dpms"<br />
EndSection<br />
Section "Monitor"<br />
Identifier "Monitor1"<br />
VendorName "Philips"<br />
ModelName "150B"<br />
EndSection<br />
Section "Device"<br />
Identifier "Videocard0"<br />
Driver "nvidia"<br />
VendorName "Videocard vendor"<br />
BoardName "NVIDIA Quadro FX (generic)"<br />
BusId "PCI:1:0:0"<br />
Screen 0<br />
EndSection<br />
Section "Device"<br />
Identifier "Videocard1"<br />
Driver "nvidia"<br />
VendorName "Videocard vendor"<br />
BoardName "NVIDIA Quadro FX (generic)"<br />
Option "Stereo" "3"<br />
BusId "PCI:1:0:0"<br />
Screen 1<br />
EndSection<br />
Section "Screen"<br />
Identifier "Screen0"<br />
Device "Videocard0"<br />
Monitor "Monitor0"<br />
DefaultDepth 24 <br />
SubSection "Display"<br />
Viewport 0 0<br />
Depth 24 <br />
Modes "1024x768" "800x600"<br />
EndSubSection<br />
EndSection<br />
Section "Screen"<br />
Identifier "Screen1"<br />
Device "Videocard1"<br />
Monitor "Monitor1"<br />
DefaultDepth 24 <br />
SubSection "Display"<br />
Viewport 0 0<br />
Depth 24 <br />
Modes "1600x1280" "1280x1024" "1024x768" "800x600"<br />
<br />
</code><br />
<br />
Note that one can think this is rather silly to define two card, two monitors, two cards, and two screens.<br />
However, that the only way I know that one monitor can be stereo-enabled and the other one not.<br />
<br />
test link [http://www.pymolwiki.org/index.php/Stereo_3D_Display_Options]<br />
<br />
==Ono==<br />
You also need to set the environment variable STEREO for the stereo to work properly in ono:<br />
setenv STEREO on (tcsh)<br />
STEREO = on; export STEREO (bash)<br />
[http://dbb.urmc.rochester.edu/labs/wedekind/Wedekind-Lab/X-ray%20Lab/Methods/Nvidia-ONO.html]<br />
<br />
==Mac OS X==<br />
The following command needs to be run for Macs to be able to support stereo in X11 programs, such as Coot [http://xanana.ucsc.edu/~wgscott/xtal/wiki/index.php/Installing_Coot_on_OS_X] :<br />
defaults write com.apple.x11 enable_stereo -bool true<br />
<br />
<br />
==See also==<br />
Stereo on TFT: see [[Coot zalman]]<br />
<br />
http://pymol.sourceforge.net/stereo3d.html<br />
<br />
http://pymolwiki.org/index.php/Stereo_3D_Display_Options</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Stereo&diff=1915Stereo2011-02-02T10:20:12Z<p>Karsten: /* Stereo on conventional CRT monitors */</p>
<hr />
<div>== Stereo on TFT monitors ==<br />
Question: is someone using Nvidia 3D vision + a compatible 1920x1080 23.5" Desktop Display e.g. ACER GD245HQ 120 Hz LCD display OR Alienware OptX AW2310 120 Hz LCD display? Is it running nicely with Linux + Nvidia's Linux driver? How is the stereo quality compared to Zalman's 3D-LCDs or the old CrystalEyes shutter glasses + CRT monitor?<br />
<br />
Answers:<br />
# I am using the combination you are asking for with alienware monitor. The driver I am using is 195.30 beta. It works very well. Quality is much much better than Zalman and somewhat better than CRTs.<br />
# I have Nvidia 3D vision running on the Samsung SyncMaster 2233. One note of caution, for the Quadra FX3800 if you are using dual monitors I haven't found a way to get TwinView to work with the Nvidia 195.30 beta linux driver. You can configure for 2 X-screens to drive the two screens as a work around. This is most likely a 3800 problem since the 3800 has 1 DVI and 2 HDMI outputs instead of 2 DVI outputs. If you are looking for a new graphics card as well and like dual monitors I would stay away from the Quadro FX1800 and FX3800. The Nvidia 3D vision quality is better than Zalman but the Zalman glasses are much lighter if you do a lot of stereo.<br />
<br />
Follow up question: Is your Nvidia 3D vision emitter connected via the 3-pin DIN or USB? The Nvidia 3D vision emitter I have only connects via USB and I could not get this working under linux.<br />
<br />
== Stereo on conventional CRT monitors ==<br />
<br />
Some of the NVidia Quadro cards support stereo. The cards that have an output called "stereo" under "Display Connectors" listed at [http://www.nvidia.com/object/IO_11761.html Nvidia's Quadro overview page] have a 3-pin DIN outlet that fits with [http://www.nuvision3d.com/the60gx.html NuVision] or [http://www.reald-corporate.com/scientific CrystalEyes] stereo glasses.<br />
<br />
The cheapest of these used to be the FX1400 (difficult to find these days, around 450 €), but now appears to be the FX3450 (around 750 €). These cards are by far fast enough for protein crystallography or modelling.<br />
<br />
For stereo, the xorg.conf might need the following lines<br />
Section "Extensions"<br />
Option "Composite" "Disable"<br />
EndSection<br />
if the X log file (e.g. at /var/log/Xorg.0.log) says that stereo is not supported by composite.<br />
<br />
Another option that will be required in xorg.conf by programs running stereo is<br />
<br />
Section "Device"<br />
Driver "nvidia"<br />
Option "Stereo" "3"<br />
<br />
----<br />
<br />
Sometimes it is handy to configure two Desktops: one on a CRT monitor that can do stereo-graphics, and one on an LCD monitor for the more regular work, eg your refinement jobs with CCP4. For Nvidia cards you need to modify the xorg.conf file to have a section more or less like that:<br />
<br />
<code><br />
Section "Monitor"<br />
Identifier "Monitor0"<br />
VendorName "Iiyama"<br />
ModelName "Vision Master Pro 512"<br />
DisplaySize 450 330<br />
HorizSync 31.5 - 120.0<br />
VertRefresh 50.0 - 150.0<br />
Option "dpms"<br />
EndSection<br />
Section "Monitor"<br />
Identifier "Monitor1"<br />
VendorName "Philips"<br />
ModelName "150B"<br />
EndSection<br />
Section "Device"<br />
Identifier "Videocard0"<br />
Driver "nvidia"<br />
VendorName "Videocard vendor"<br />
BoardName "NVIDIA Quadro FX (generic)"<br />
BusId "PCI:1:0:0"<br />
Screen 0<br />
EndSection<br />
Section "Device"<br />
Identifier "Videocard1"<br />
Driver "nvidia"<br />
VendorName "Videocard vendor"<br />
BoardName "NVIDIA Quadro FX (generic)"<br />
Option "Stereo" "3"<br />
BusId "PCI:1:0:0"<br />
Screen 1<br />
EndSection<br />
Section "Screen"<br />
Identifier "Screen0"<br />
Device "Videocard0"<br />
Monitor "Monitor0"<br />
DefaultDepth 24 <br />
SubSection "Display"<br />
Viewport 0 0<br />
Depth 24 <br />
Modes "1024x768" "800x600"<br />
EndSubSection<br />
EndSection<br />
Section "Screen"<br />
Identifier "Screen1"<br />
Device "Videocard1"<br />
Monitor "Monitor1"<br />
DefaultDepth 24 <br />
SubSection "Display"<br />
Viewport 0 0<br />
Depth 24 <br />
Modes "1600x1280" "1280x1024" "1024x768" "800x600"<br />
<br />
</code><br />
<br />
Note that one can think this is rather silly to define two card, two monitors, two cards, and two screens.<br />
However, that the only way I know that one monitor can be stereo-enabled and the other one not.<br />
<br />
test link http://www.pymolwiki.org/index.php/Stereo_3D_Display_Options<br />
<br />
==Ono==<br />
You also need to set the environment variable STEREO for the stereo to work properly in ono:<br />
setenv STEREO on (tcsh)<br />
STEREO = on; export STEREO (bash)<br />
[http://dbb.urmc.rochester.edu/labs/wedekind/Wedekind-Lab/X-ray%20Lab/Methods/Nvidia-ONO.html]<br />
<br />
==Mac OS X==<br />
The following command needs to be run for Macs to be able to support stereo in X11 programs, such as Coot [http://xanana.ucsc.edu/~wgscott/xtal/wiki/index.php/Installing_Coot_on_OS_X] :<br />
defaults write com.apple.x11 enable_stereo -bool true<br />
<br />
<br />
==See also==<br />
Stereo on TFT: see [[Coot zalman]]<br />
<br />
http://pymol.sourceforge.net/stereo3d.html<br />
<br />
http://pymolwiki.org/index.php/Stereo_3D_Display_Options</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Main_Page&diff=1887Main Page2011-01-28T09:00:39Z<p>Karsten: /* How to use this wiki */</p>
<hr />
<div>This [http://www.ccp4.ac.uk/ CCP4] user community wiki ("CCP4 wiki" in short) is meant to be a collection of crystallographic knowledge as discussed on the [http://www.ccp4.ac.uk/ccp4bb.php CCP4 mailing list] (CCP4BB), and elsewhere. It may contain information about anything relevant to protein crystallographers, whether methods-related ("what is the best program for purpose X?"), problem-oriented ("my crystals melt if I look at them"), or concerning hardware ("what is your opinion on robot X / computer Y?"). <br />
<br />
The general [[Topics]] include articles about [[Expression_and_Purification| protein expression and purification]],<br />
[[Crystals| crystal growth, handling and data collection]],[[Crystallography| crystallographic theory and practice]],<br />
and crystal structure related [[Bioinformatics|bioinformatics]] and [[Xtal_computing|computer-related issues]]. There are also announcements for [[Positions|open positions]], [[Courses and Conferences|courses and conferences]] and a [[FAQ|FAQ]].<br />
<br />
Many interesting questions are currently being answered on CCP4BB, but the answers are not easily accessible for others after some time. The intention is that the wiki reflect the breadth of topics on CCP4BB, which will make it a useful resource e.g. for "frequently asked questions", offloading some of the question/answer traffic on CCP4BB to a more permanent mode of storage. A wiki has the additional advantage that mathematical formulas and images may be presented nicely.<br />
<br />
== News ==<br />
April 04, 2008: transparent inter-wiki linking is now very easy due to an extension called "[http://www.mediawiki.org/wiki/Extension:Special_page_to_work_with_the_interwiki_table SpecialInterwiki]". These links appear like internal links (see the next sentences!), and can point to subsections of an article, or to figures (untested!). This also means that if a wiki changes its DNS name, only a single inter-wiki link has to be updated. <br />
Currently, the [[ccp4dev:Main_Page|CCP4 developers' wiki]], the [[xds:Main_Page|XDSwiki]] and (as of May 02, 2008) the [[iucr:Main_Page|IUCr Online Dictionary of Crystallography]] are in the [[Special:Interwiki|Inter-Wiki table]]. <br />
<br />
Old news may be found [[Old_news|here]].<br />
<br />
== What is CCP4? ==<br />
[[CCP4]] is a crystallographic program system, and can be downloaded from [http://www.ccp4.ac.uk/download.php]. <br />
It is free for academic use; the license is at [http://www.ccp4.ac.uk/ccp4license.php].<br />
Documentation is in the [[ccp4dev:Main_Page|CCP4 developers' wiki]] and at [http://www.ccp4.ac.uk/docs.php], and a mailing list (CCP4BB) is at [http://www.ccp4.ac.uk/ccp4bb.php] that can also be viewed [http://www.mail-archive.com/ccp4bb@jiscmail.ac.uk/ here].<br />
<br />
== How to use this wiki ==<br />
* you may read the existing articles with a web browser (does not require an account).<br />
* you are invited to contribute to CCP4 wiki, by creating new articles, and by adding information to, or correcting existing articles. To do this, please [[create an account]]. <br />
<br />
A number of experienced crystallographers, who are also contributors to CCP4BB, have agreed to take a role as editors. This wiki is under the same [[Copyright]] (termed "GNU Free Documentation License") as [http://en.wikipedia.org Wikipedia], so any information in it may be transferred to other wikis subject to the [[Copyright]].<br />
<br />
== Getting started ==<br />
Consult the [http://meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software, and see [[Creating an article]].<br />
<br />
== Wiki contents ==<br />
<br />
* [[Topics]] - an attempt to list possible items hierarchically<br />
* [[Special:Allpages|All pages]] - list of all pages of this wiki<br />
* [[Courses and Conferences]]<br />
* [[Positions]]<br />
* articles can also be found using the search box in the left column</div>Karstenhttps://wiki.uni-konstanz.de/ccp4/index.php?title=Main_Page&diff=1886Main Page2011-01-28T08:59:40Z<p>Karsten: /* How to use this wiki */</p>
<hr />
<div>This [http://www.ccp4.ac.uk/ CCP4] user community wiki ("CCP4 wiki" in short) is meant to be a collection of crystallographic knowledge as discussed on the [http://www.ccp4.ac.uk/ccp4bb.php CCP4 mailing list] (CCP4BB), and elsewhere. It may contain information about anything relevant to protein crystallographers, whether methods-related ("what is the best program for purpose X?"), problem-oriented ("my crystals melt if I look at them"), or concerning hardware ("what is your opinion on robot X / computer Y?"). <br />
<br />
The general [[Topics]] include articles about [[Expression_and_Purification| protein expression and purification]],<br />
[[Crystals| crystal growth, handling and data collection]],[[Crystallography| crystallographic theory and practice]],<br />
and crystal structure related [[Bioinformatics|bioinformatics]] and [[Xtal_computing|computer-related issues]]. There are also announcements for [[Positions|open positions]], [[Courses and Conferences|courses and conferences]] and a [[FAQ|FAQ]].<br />
<br />
Many interesting questions are currently being answered on CCP4BB, but the answers are not easily accessible for others after some time. The intention is that the wiki reflect the breadth of topics on CCP4BB, which will make it a useful resource e.g. for "frequently asked questions", offloading some of the question/answer traffic on CCP4BB to a more permanent mode of storage. A wiki has the additional advantage that mathematical formulas and images may be presented nicely.<br />
<br />
== News ==<br />
April 04, 2008: transparent inter-wiki linking is now very easy due to an extension called "[http://www.mediawiki.org/wiki/Extension:Special_page_to_work_with_the_interwiki_table SpecialInterwiki]". These links appear like internal links (see the next sentences!), and can point to subsections of an article, or to figures (untested!). This also means that if a wiki changes its DNS name, only a single inter-wiki link has to be updated. <br />
Currently, the [[ccp4dev:Main_Page|CCP4 developers' wiki]], the [[xds:Main_Page|XDSwiki]] and (as of May 02, 2008) the [[iucr:Main_Page|IUCr Online Dictionary of Crystallography]] are in the [[Special:Interwiki|Inter-Wiki table]]. <br />
<br />
Old news may be found [[Old_news|here]].<br />
<br />
== What is CCP4? ==<br />
[[CCP4]] is a crystallographic program system, and can be downloaded from [http://www.ccp4.ac.uk/download.php]. <br />
It is free for academic use; the license is at [http://www.ccp4.ac.uk/ccp4license.php].<br />
Documentation is in the [[ccp4dev:Main_Page|CCP4 developers' wiki]] and at [http://www.ccp4.ac.uk/docs.php], and a mailing list (CCP4BB) is at [http://www.ccp4.ac.uk/ccp4bb.php] that can also be viewed [http://www.mail-archive.com/ccp4bb@jiscmail.ac.uk/ here].<br />
<br />
== How to use this wiki ==<br />
* you may read the existing articles with a web browser (does not require an account).<br />
* you are invited to contribute to CCP4 wiki, by creating new articles, and by adding information to, or correcting existing articles. To do this, please [[create an account]]. <br />
<br />
A number of experienced crystallographers, who are also contributors to CCP4BB, have agreed to take a role as editors. This wiki is under the same [[Copyright]] (termed "GNU Free Documentation License") as [http://en.wikipedia.org Wikipedia], so any information in it may be transferred to other wikis subject to the [[Copyright]].<br />
<br />
test<br />
<br />
== Getting started ==<br />
Consult the [http://meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software, and see [[Creating an article]].<br />
<br />
== Wiki contents ==<br />
<br />
* [[Topics]] - an attempt to list possible items hierarchically<br />
* [[Special:Allpages|All pages]] - list of all pages of this wiki<br />
* [[Courses and Conferences]]<br />
* [[Positions]]<br />
* articles can also be found using the search box in the left column</div>Karsten