Cygwin, A fish out of water!!!
Cygwin is a great tool for unix people who find themselves in a Microsoft world. I’ve been using Cygwin for years but never for more than some very basic things. Lately I’ve been spending some more time with it and I have to say I’m really impressed. I won’t go into any unix basics or how to install Cygwin as there are many good references on the internet.[1][2] What I will do is list some tips and techniques that I have found useful and not entirely obvious such as setting up a tool for managing packages similar to Debians venerable apt-get, managing ssh keys, replacing the standard windows dos shell as the terminal and using apt-cyg to install packages not in the standard Cygwin repositories.
How I setup Cygwin
You can always read the standard instructions for cygwin but after using it for a while I prefer something a little different. First of all I don’t like to use the UI to do the install. Also, the standard install creates a link to a batch file which runs inside of the standard dos terminal. If you like this shell then this post is not for you!!!
- Download the Cygwin setup program.
- Run the following command or something similar at the windows cmd prompt. You can read more about the options in the Cygwin FAQ or by typing setup.exe –help. ***NOTE: add util-linux to the list of packages to prevent things like getopt not being found errors in bash***
- Install mintty to replace the dos window that cygwin normally runs in.
Now you can add a shortcut to c:\cygwin\bin\mintty.exe and finally you’ll be working in a unix like environment with a reasonable shell.
You may want to finish your setup by updating your /etc/passwd and /etc/group files. The following commands will add both the local and domain users and groups to your environment. Note that I’ve changed the path separator used for domain accounts from ‘\’ to ‘_’. This is due issues with to how Cygwin will handle domain accounts.
me@cygwin ~ $ mkpasswd.exe -l -D -S '_' > /etc/passwd me@cygwin ~ $ mkgroup.exe -l -D -S '_' > /etc/group
The only problem is that your windows user directory is usually something like c:\Users on Windows 7 but Cygwin creates its own /home directory within the c:\cygwin windows path. I like to merge my windows and unix files when using cygwin so I don’t have to remember that I actually have user files in c:\cygwin. I’ve uninstalled Cygwin in the past only to remember just after clearing out my recycling bin. But others may not like to comingle things. It is up to you. If you want to change where /home points in cygwin then add the following to your /etc/fstab.
$ cat /etc/fstab # For a description of the file format, see the Users Guide # http://cygwin.com/cygwin-ug-net/using.html#mount-table # This is default anyway: # none /cygdrive cygdrive binary,posix=0,user 0 0 #C: on /cygdrive/c type ntfs (binary,posix=0,user,noumount,auto) c:/Users /home ntfs binary,posix=0,user 0 0
A note about Cygwin 1.7.9-1
The latest version of Cygwin has changed in a number of ways. One of them is how it handles MS-DOS style paths. For me this didn’t cause much of a problem but now Cygwin issues a warning when you run many commands which encounter a MS-DOS style path. To disable this create a CYGWIN environment variable in windows and add nodosfilewarning as a value for that variable. You can find more explicit details here.
apt-cyg
Ok so now you’ve got a very basic install but I would definitely download the apt-cyg[6] program to regain some sanity. This is a program that allows you to add, search, install, upgrade or uninstall packages just like you would with apt-get or yum which is convenient but apt-cyg is also a bit like portage on gentoo which means that you can install source packages as well. I find this very useful.
To make apt-cyg work a little better you may want to add these aliases to your shell. I use bash so I added the following;
alias apt-cyg=’apt-cyg -m http://cygwin.mirrors.pair.com/’
alias apt-cygports=’apt-cyg -m ftp://sourceware.org/pub/cygwinports/’
alias apt-cygports=’apt-cyg -m ftp://sourceware.org/pub/cygwinports/’
These are great because now I can just type apt-cyg to install the binary packages that are maintained by the Cygwin community proper while also installing the ports/source based packages as well. You can do quite a bit with apt-cyg. Take a look at the project site[6] for more details.
Getting SSH to work just right
SSH is a wonderful thing. Using ssh-agent, you can enter your password just once and log into many different workstations securely. With Cygwin this takes a little work. For some reason the way I normally start ssh-agent didn’t work in Cygwin. So here is how I did it.
Using our fancy new apt-cyg command we’ll install ssh and update our shell to start ssh-agent when we first log in. All subsequent shells will attach to this running agent and any keys it has been authorized to manage. When you log out the agent will shut down and thus no one can just log into your workstation and use that as a means to gain access to your other hosts.
$ apt-cyg find openssh Working directory is /setup Mirror is http://cygwin.mirrors.pair.com/ --2011-05-10 17:52:13-- http://cygwin.mirrors.pair.com//setup.bz2 Resolving cygwin.mirrors.pair.com (cygwin.mirrors.pair.com)... 216.92.2.149 Connecting to cygwin.mirrors.pair.com (cygwin.mirrors.pair.com)|216.92.2.149|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 268404 (262K) [text/plain] Saving to: `setup.bz2' 100%[======================================================================================================================>] 268,404 --.-K/s in 0.02s 2011-05-10 17:52:13 (12.7 MB/s) - `setup.bz2' saved [268404/268404] Updated setup.ini Searching for installed packages matching openssh: Searching for installable packages matching openssh: openssh
$ apt-cyg install openssh Working directory is /setup Mirror is http://cygwin.mirrors.pair.com/ --2011-05-10 17:52:31-- http://cygwin.mirrors.pair.com//setup.bz2 Resolving cygwin.mirrors.pair.com (cygwin.mirrors.pair.com)... 216.92.2.149 Connecting to cygwin.mirrors.pair.com (cygwin.mirrors.pair.com)|216.92.2.149|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 268404 (262K) [text/plain] Saving to: `setup.bz2' 100%[======================================================================================================================>] 268,404 --.-K/s in 0.02s 2011-05-10 17:52:31 (12.1 MB/s) - `setup.bz2' saved [268404/268404] Updated setup.ini Installing openssh Found package openssh
Package openssh installed
Now that we have ssh let’s update bash to start ssh-agent when we start our first shell. There are some good sources on different ways to set this up[7] but if you’re using bash you can add the following to your $HOME/.bash_profile
SSH_ENV="$HOME/.ssh/environment"
function start_agent {
echo "Initialising new SSH agent..."
/usr/bin/ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
echo succeeded
chmod 600 "${SSH_ENV}"
. "${SSH_ENV}" > /dev/null
/usr/bin/ssh-add;
}
# Source SSH settings, if applicable
if [ -f "${SSH_ENV}" ]; then
. "${SSH_ENV}" > /dev/null
#ps ${SSH_AGENT_PID} doesn't work under cywgin
ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
start_agent;
}
else
start_agent;
fi
Lastly, we have to create a ssh-key and add it to our ssh-agent.
$ ssh-keygen.exe Generating public/private rsa key pair. Enter file in which to save the key (/home/me/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/me/.ssh/id_rsa. Your public key has been saved in /home/me/.ssh/id_rsa.pub. The key fingerprint is: d1:29:8f:5b:f9:f3:88:12:c5:b5:aa:5a:18:bf:e4:7d me@cygwin The key's randomart image is: +--[ RSA 2048]----+ | | | . .. | | o.o. . | | =o.. | | . S.+. | | +.o.. | | . =o o | | +oo .E+ | | ..o.o.. . | +-----------------+
Restart mintty and you’ll be asked to enter the passphrase you used when creating your key.
Python
Ok so why did I go through the trouble of setting up apt-cyg? If it were just for the ease of installing packages it would have proved itself worthy but it isn’t just the convenience alone. apt-cyg uses a ports like approach to packages which in this case means that they are compiled on your machine. One benefit to this approach is that you can often find packages in the apt-cyg repository that aren’t available in the standard binary distribution.
Ok so lets look at python. I’ve installed the standard distribution of python that comes with Cygwin. Now I want to install some things that I use on a daily basis, namely setuptools, pip, virtualenv and others. But they aren’t available. I could just download the packages and install them manually. This isn’t a big deal but keep in mind that there are often differences in how you would build something to work properly in Cygwin than what you would do on many linux/bsd distributions. Why do all that work myself when there is a community of people who work on this very thing.
$ apt-cyg find setuptools Working directory is /setup Mirror is http://cygwin.mirrors.pair.com/ --2011-05-16 09:31:51-- http://cygwin.mirrors.pair.com//setup.bz2 Resolving cygwin.mirrors.pair.com (cygwin.mirrors.pair.com)... 216.92.2.149 Connecting to cygwin.mirrors.pair.com (cygwin.mirrors.pair.com)|216.92.2.149|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 268764 (262K) [text/plain] Saving to: `setup.bz2' 100%[======================================================================================================================>] 268,764 --.-K/s in 0.02s 2011-05-16 09:31:51 (11.5 MB/s) - `setup.bz2' saved [268764/268764] Updated setup.ini Searching for installed packages matching setuptools: Searching for installable packages matching setuptools:
You can see that there is no binary for setuptools in the Cygwin binary repository but low and behold the gods of the ports community have saved us the work of doing this ourselves.
$ apt-cygports find setuptools
Working directory is /setup
Mirror is ftp://sourceware.org/pub/cygwinports/
--2011-05-16 09:31:41-- ftp://sourceware.org/pub/cygwinports//setup.bz2
=> `.listing'
Resolving sourceware.org (sourceware.org)... 209.132.180.131
Connecting to sourceware.org (sourceware.org)|209.132.180.131|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /pub/cygwinports/ ... done.
==> PASV ... done. ==> LIST ... done.
[ <=> ] 728 --.-K/s in 0.06s
2011-05-16 09:31:42 (12.9 KB/s) - `.listing' saved [728]
Removed `.listing'.
--2011-05-16 09:31:42-- ftp://sourceware.org/pub/cygwinports//setup.bz2
=> `setup.bz2'
==> CWD not required.
==> PASV ... done. ==> RETR setup.bz2 ... done.
Length: 456183 (445K)
100%[======================================================================================================================>] 456,183 434K/s in 1.0s
2011-05-16 09:31:43 (434 KB/s) - `setup.bz2' saved [456183]
Updated setup.ini
Searching for installed packages matching setuptools:
Searching for installable packages matching setuptools:
python-setuptools
python3-setuptools
After running apt-cygports install python-setuptools I’m all set. A quick
$ python -c 'import setuptools'
shows me that setuptools is installed so now I can just install my python packages the way I like to.
$ easy_install pip virtualenv Searching for pip Reading http://pypi.python.org/simple/pip/ Reading http://pip.openplans.org Reading http://www.pip-installer.org Best match: pip 1.0.1 Downloading http://pypi.python.org/packages/source/p/pip/pip-1.0.1.tar.gz#md5=28dcc70225e5bf925532abc5b087a94b Processing pip-1.0.1.tar.gz Running pip-1.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-AmKs1m/pip-1.0.1/egg-dist-tmp-eM10iI warning: no files found matching '*.html' under directory 'docs' warning: no previously-included files matching '*.txt' found under directory 'docs/_build' no previously-included directories found matching 'docs/_build/_sources' Adding pip 1.0.1 to easy-install.pth file Installing pip script to /usr/bin Installing pip-2.6 script to /usr/bin Installed /usr/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg Processing dependencies for pip Finished processing dependencies for pip Searching for virtualenv Reading http://pypi.python.org/simple/virtualenv/ Reading http://virtualenv.openplans.org Reading http://www.virtualenv.org Best match: virtualenv 1.6.1 Downloading http://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.6.1.tar.gz#md5=1a475df2219457b6b4febb9fe595d915 Processing virtualenv-1.6.1.tar.gz Running virtualenv-1.6.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-azd5Po/virtualenv-1.6.1/egg-dist-tmp-L5w7AD warning: no previously-included files matching '*.*' found under directory 'docs/_templates' Adding virtualenv 1.6.1 to easy-install.pth file Installing virtualenv script to /usr/bin Installed /usr/lib/python2.6/site-packages/virtualenv-1.6.1-py2.6.egg Processing dependencies for virtualenv Finished processing dependencies for virtualenv
Now a word of warning. I don’t like to manually install things on my system in a way that I can’t keep them managable and distict from what the system is managing. The packages that easy_install just installed were put into /usr/ namespace. I could have had them installed into /usr/local but in my laziness I have let this one slide. To prevent any further cruft I have installed virtualenv. Virtualenv and the python virtual toolset could have a post all their own. In any case if you want to use virtualenv with cygwin you can see how I use it below.
$ pip install virtualenvwrapper
Downloading/unpacking virtualenvwrapper
Downloading virtualenvwrapper-2.7.1.tar.gz (690Kb): 690Kb downloaded
Running setup.py egg_info for package virtualenvwrapper
Requirement already satisfied (use --upgrade to upgrade): virtualenv in /usr/lib/python2.6/site-packages/virtualenv-1.6.1-py2.6.egg (from virtualenvwrapper)
Installing collected packages: virtualenvwrapper
Running setup.py install for virtualenvwrapper
2 [main] python 644 C:\cygwin\bin\python.exe: *** fatal error - unable to remap \\?\C:\cygwin\lib\python2.6\lib-dynload\time.dll to same address as parent: 0x18B40000 != 0x1A230000
Stack trace:
Frame Function Args
00289F08 6102796B (00289F08, 00000000, 00000000, 00000000)
0028A1F8 6102796B (6117EC60, 00008000, 00000000, 61180977)
0028B228 61004F1B (611A7FAC, 6124355C, 18B40000, 1A230000)
End of stack trace
2 [main] python 7940 fork: child 644 - died waiting for dll loading, errno 11
Error [Errno 11] Resource temporarily unavailable while executing command /usr/bin/python -c "import setuptools;__file__='/home/jrizzo/build/virtualenvwrapper/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/pip-vDBPql-record/install-record.txt
Exception:
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg/pip/basecommand.py", line 126, in main
self.run(options, args)
File "/usr/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg/pip/commands/install.py", line 228, in run
requirement_set.install(install_options, global_options)
File "/usr/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg/pip/req.py", line 1100, in install
requirement.install(install_options, global_options)
File "/usr/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg/pip/req.py", line 572, in install
cwd=self.source_dir, filter_stdout=self._filter_install, show_stdout=False)
File "/usr/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg/pip/__init__.py", line 220, in call_subprocess
cwd=cwd, env=env)
File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1049, in _execute_child
self.pid = os.fork()
OSError: [Errno 11] Resource temporarily unavailable
a forking mess
Before continuing with my installation of python I want to point out something you may run into from time to time. This happened to me when setting up python so I point it out now, but this could happen whenever you update the Cygwin binaries.
This error is due to difficulty implementing POSIX standard fork on Windows. In order to work around some issues there is some voodoo magic going on that requires a very special memory layout in assemblies such as the cygwin1.dll. To fix this problem you will probably need to run a rebaseall and possibly peflagsall. Just remember that you can’t run those commands while you have any cygwin processes running. Even if you are careful about not having any cygwin processes running you should probably to reboot after you run those commands. If you don’t reboot don’t yell at me 
. Others have written about this in more detail so take a look before blindly running rebaseall.
Continuing with your normally scheduled broadcast… back to setting up python
So now after running rebaseall you can see that pip installed virtualenvwrapper nicely.
$ pip install virtualenvwrapper
Downloading/unpacking virtualenvwrapper
Running setup.py egg_info for package virtualenvwrapper
Requirement already satisfied (use --upgrade to upgrade): virtualenv in /usr/lib/python2.6/site-packages/virtualenv-1.6.1-py2.6.egg (from virtualenvwrapper)
Installing collected packages: virtualenvwrapper
Running setup.py install for virtualenvwrapper
changing mode of build/scripts-2.6/virtualenvwrapper.sh from 644 to 755
Skipping installation of /usr/lib/python2.6/site-packages/virtualenvwrapper/__init__.py (namespace package)
Installing /usr/lib/python2.6/site-packages/virtualenvwrapper-2.7.1-py2.6-nspkg.pth
changing mode of /usr/bin/virtualenvwrapper.sh to 755
Successfully installed virtualenvwrapper
Cleaning up...
Now I can setup my python virtual environment and install the things I use on an every day basis. The following steps were taken directly from the virtualenvwrapper.sh script
- Create a directory to hold the virtual environments. (mkdir $HOME/.virtualenvs).
- Add a line like “export WORKON_HOME=$HOME/.virtualenvs” to your .bashrc.
- Add a line like “source /path/to/this/file/virtualenvwrapper.sh” to your .bashrc.
- Run: source ~/.bashrc
- Run: workon
- A list of environments, empty, is printed.
- Run: mkvirtualenv temp
- Run: workon
- This time, the “temp” environment is included.
- Run: workon temp
- The virtual environment is activated.
Generally speaking the standard steps for using virtualenvwrapper are good however a couple of notes. I usually use the following command when creating virtual environments so that I don’t have any conflict or dependency on the system installed packages.
me@cygwin ~/.virtualenvs $ mkvirtualenv --distribute --no-site-packages --python=python2.6.exe py26 Running virtualenv with interpreter /usr/bin/python2.6.exe New python executable in py26/bin/python2.6 Also creating executable in py26/bin/python.exe Installing distribute....................................................................................................................................................................................done. Installing pip...............done. virtualenvwrapper.user_scripts creating /home/me/.virtualenvs/py26/bin/predeactivate virtualenvwrapper.user_scripts creating /home/me/.virtualenvs/py26/bin/postdeactivate virtualenvwrapper.user_scripts creating /home/me/.virtualenvs/py26/bin/preactivate virtualenvwrapper.user_scripts creating /home/me/.virtualenvs/py26/bin/postactivate virtualenvwrapper.user_scripts creating /home/me/.virtualenvs/py26/bin/get_env_details (py26)
So now I can install the other programs I want without worrying about any cygwin specific packaging.
me@cygwin ~/.virtualenvs $ which pip /home/me/.virtualenvs/py26/bin/pip (py26) me@cygwin ~/.virtualenvs $ pip install ipython Downloading/unpacking ipython Downloading ipython-0.10.2.zip (6.4Mb): 6.4Mb downloaded Running setup.py egg_info for package ipython running egg_info Installing collected packages: ipython Running setup.py install for ipython running install Installing iptest script to /home/me/.virtualenvs/py26/bin Installing ipythonx script to /home/me/.virtualenvs/py26/bin Installing ipcluster script to /home/me/.virtualenvs/py26/bin Installing ipython script to /home/me/.virtualenvs/py26/bin Installing pycolor script to /home/me/.virtualenvs/py26/bin Installing ipcontroller script to /home/me/.virtualenvs/py26/bin Installing ipengine script to /home/me/.virtualenvs/py26/bin Successfully installed ipython Cleaning up... (py26)
A lot more can be done here. In fact I usually use virtualenvwrapper on top of virtualenv and several other things but I think this is enough for now. To read more take a look at the following links.
- Virtualenv: http://pypi.python.org/pypi/virtualenv
- Virtualenvwrapper: http://www.doughellmann.com/docs/virtualenvwrapper/
Summary
So now you can see some of the things that I do with Cygwin to make working in a Windows world more comfortable but what about you? Do you have any tips or techniques that give you that cozy Unix feeling? Let me know what you think and maybe I’ll follow up with other things I’ve found.
References
- http://www.cygwin.com/install.html
- http://lifehacker.com/179514/geek-to-live–introduction-to-cygwin-part-i
- http://www.e-texteditor.com/forum/viewtopic.php?p=1823
- http://sourceware.org/cygwinports/
- http://www.gentoo-wiki.info/HOWTO_Gentoo_on_Cygwin
- http://code.google.com/p/apt-cyg/
- http://mah.everybody.org/docs/ssh
