Cygwin, A fish out of water!!!

Cygwin is a great tool for unix people who find themselves in a Microsoft world.  I’ve been using Cygwin for years but never for more than some very basic things.  Lately I’ve been spending some more time with it and I have to say I’m really impressed.  I won’t go into any unix basics or how to install Cygwin as there are many good references on the internet.[1][2] What I will do is list some tips and techniques that I have found useful and not entirely obvious such as setting up a tool for managing packages similar to Debians venerable apt-get, managing ssh keys, replacing the standard windows dos shell as the terminal and using apt-cyg to install packages not in the standard Cygwin repositories.

How I setup Cygwin

You can always read the standard instructions for cygwin but after using it for a while I prefer something a little different.  First of all I don’t like to use the UI to do the install.  Also, the standard install creates a link to a batch file which runs inside of the standard dos terminal.  If you like this shell then this post is not for you!!!
  • Download the Cygwin setup program.
  • Run the following command or something similar at the windows cmd prompt.  You can read more about the options in the Cygwin FAQ or by typing setup.exe –help.  ***NOTE: add util-linux to the list of packages to prevent things like getopt not being found errors in bash***
  • Install mintty to replace the dos window that cygwin normally runs in.
Now you can add a shortcut to c:\cygwin\bin\mintty.exe and finally you’ll be working in a unix like environment with a reasonable shell.
You may want to finish your setup by updating your /etc/passwd and /etc/group files.  The following commands will add both the local and domain users and groups to your environment.  Note that I’ve changed the path separator used for domain accounts from ‘\’ to ‘_’.  This is due issues with to how Cygwin will handle domain accounts.
me@cygwin ~
$ mkpasswd.exe -l -D -S '_' > /etc/passwd

me@cygwin ~
$ mkgroup.exe -l -D -S '_' > /etc/group
The only problem is that your windows user directory is usually something like c:\Users on Windows 7 but Cygwin creates its own /home directory within the c:\cygwin windows path.  I like to merge my windows and unix files when using cygwin so I don’t have to remember that I actually have user files in c:\cygwin.  I’ve uninstalled Cygwin in the past only to remember just after clearing out my recycling bin.  But others may not like to comingle things.  It is up to you.  If you want to change where /home points in cygwin then add the following to your /etc/fstab.
$ cat /etc/fstab
# For a description of the file format, see the Users Guide
# http://cygwin.com/cygwin-ug-net/using.html#mount-table

# This is default anyway:
# none /cygdrive cygdrive binary,posix=0,user 0 0
#C: on /cygdrive/c type ntfs (binary,posix=0,user,noumount,auto)
c:/Users /home ntfs binary,posix=0,user 0 0

A note about Cygwin 1.7.9-1

The latest version of Cygwin has changed in a number of ways.  One of them is how it handles MS-DOS style paths.  For me this didn’t cause much of a problem but now Cygwin issues a warning when you run many commands which encounter a MS-DOS style path.  To disable this create a CYGWIN environment variable in windows and add nodosfilewarning as a value for that variable.  You can find more explicit details here.

apt-cyg

Ok so now you’ve got a very basic install but I would definitely download the apt-cyg[6] program to regain some sanity.  This is a program that allows you to add, search, install, upgrade or uninstall packages just like you would with apt-get or yum which is convenient but apt-cyg is also a bit like portage on gentoo which means that you can install source packages as well.  I find this very useful.
To make apt-cyg work a little better you may want to add these aliases to your shell.  I use bash so I added the following;
alias apt-cyg=’apt-cyg -m http://cygwin.mirrors.pair.com/’
alias apt-cygports=’apt-cyg -m ftp://sourceware.org/pub/cygwinports/’
These are great because now I can just type apt-cyg to install the binary packages that are maintained by the Cygwin community proper while also installing the ports/source based packages as well.  You can do quite a bit with apt-cyg.  Take a look at the project site[6] for more details.

Getting SSH to work just right

SSH is a wonderful thing.  Using ssh-agent, you can enter your password just once and log into many different workstations securely.  With Cygwin this takes a little work.  For some reason the way I normally start ssh-agent didn’t work in Cygwin.  So here is how I did it.
Using our fancy new apt-cyg command we’ll install ssh and update our shell to start ssh-agent when we first log in.  All subsequent shells will attach to this running agent and any keys it has been authorized to manage.  When you log out the agent will shut down and thus no one can just log into your workstation and use that as a means to gain access to your other hosts.
$ apt-cyg find openssh
Working directory is /setup
Mirror is http://cygwin.mirrors.pair.com/
--2011-05-10 17:52:13--  http://cygwin.mirrors.pair.com//setup.bz2
Resolving cygwin.mirrors.pair.com (cygwin.mirrors.pair.com)... 216.92.2.149
Connecting to cygwin.mirrors.pair.com (cygwin.mirrors.pair.com)|216.92.2.149|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 268404 (262K) [text/plain]
Saving to: `setup.bz2'

100%[======================================================================================================================>] 268,404     --.-K/s   in 0.02s

2011-05-10 17:52:13 (12.7 MB/s) - `setup.bz2' saved [268404/268404]

Updated setup.ini

Searching for installed packages matching openssh:

Searching for installable packages matching openssh:
openssh
$ apt-cyg install openssh
Working directory is /setup
Mirror is http://cygwin.mirrors.pair.com/
--2011-05-10 17:52:31--  http://cygwin.mirrors.pair.com//setup.bz2
Resolving cygwin.mirrors.pair.com (cygwin.mirrors.pair.com)... 216.92.2.149
Connecting to cygwin.mirrors.pair.com (cygwin.mirrors.pair.com)|216.92.2.149|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 268404 (262K) [text/plain]
Saving to: `setup.bz2'

100%[======================================================================================================================>] 268,404     --.-K/s   in 0.02s

2011-05-10 17:52:31 (12.1 MB/s) - `setup.bz2' saved [268404/268404]

Updated setup.ini

Installing openssh
Found package openssh
Package openssh installed
Now that we have ssh let’s update bash to start ssh-agent when we start our first shell.  There are some good sources on different ways to set this up[7] but if you’re using bash you can add the following to your $HOME/.bash_profile
SSH_ENV="$HOME/.ssh/environment"

function start_agent {
 echo "Initialising new SSH agent..."
 /usr/bin/ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
 echo succeeded
 chmod 600 "${SSH_ENV}"
 . "${SSH_ENV}" > /dev/null
 /usr/bin/ssh-add;
}

# Source SSH settings, if applicable

if [ -f "${SSH_ENV}" ]; then
 . "${SSH_ENV}" > /dev/null
 #ps ${SSH_AGENT_PID} doesn't work under cywgin
 ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
 start_agent;
 }
else
 start_agent;
fi
Lastly, we have to create a ssh-key and add it to our ssh-agent.
$ ssh-keygen.exe
Generating public/private rsa key pair.
Enter file in which to save the key (/home/me/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/me/.ssh/id_rsa.
Your public key has been saved in /home/me/.ssh/id_rsa.pub.
The key fingerprint is:
d1:29:8f:5b:f9:f3:88:12:c5:b5:aa:5a:18:bf:e4:7d me@cygwin
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|         . ..    |
|        o.o. .   |
|         =o..    |
|      . S.+.     |
|       +.o..     |
|      . =o  o    |
|       +oo .E+   |
|      ..o.o.. .  |
+-----------------+
Restart mintty and you’ll be asked to enter the passphrase you used when creating your key.

Python

Ok so why did I go through the trouble of setting up apt-cyg?  If it were just for the ease of installing packages it would have proved itself worthy but it isn’t just the convenience alone.  apt-cyg uses a ports like approach to packages which in this case means that they are compiled on your machine.  One benefit to this approach is that you can often find packages in the apt-cyg repository that aren’t available in the standard binary distribution.
Ok so lets look at python.  I’ve installed the standard distribution of python that comes with Cygwin.  Now I want to install some things that I use on a daily basis, namely setuptools, pip, virtualenv and others.  But they aren’t available.  I could just download the packages and install them manually.  This isn’t a big deal but keep in mind that there are often differences in how you would build something to work properly in Cygwin than what you would do on many linux/bsd distributions.  Why do all that work myself when there is a community of people who work on this very thing.
$ apt-cyg find setuptools
Working directory is /setup
Mirror is http://cygwin.mirrors.pair.com/
--2011-05-16 09:31:51--  http://cygwin.mirrors.pair.com//setup.bz2
Resolving cygwin.mirrors.pair.com (cygwin.mirrors.pair.com)... 216.92.2.149
Connecting to cygwin.mirrors.pair.com (cygwin.mirrors.pair.com)|216.92.2.149|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 268764 (262K) [text/plain]
Saving to: `setup.bz2'

100%[======================================================================================================================>] 268,764     --.-K/s   in 0.02s

2011-05-16 09:31:51 (11.5 MB/s) - `setup.bz2' saved [268764/268764]

Updated setup.ini

Searching for installed packages matching setuptools:

Searching for installable packages matching setuptools:
You can see that there is no binary for setuptools in the Cygwin binary repository but low and behold the gods of the ports community have saved us the work of doing this ourselves.
$ apt-cygports find setuptools
Working directory is /setup
Mirror is ftp://sourceware.org/pub/cygwinports/
--2011-05-16 09:31:41--  ftp://sourceware.org/pub/cygwinports//setup.bz2
           => `.listing'
Resolving sourceware.org (sourceware.org)... 209.132.180.131
Connecting to sourceware.org (sourceware.org)|209.132.180.131|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /pub/cygwinports/ ... done.
==> PASV ... done.    ==> LIST ... done.

    [ <=>                                                                                                                   ] 728         --.-K/s   in 0.06s

2011-05-16 09:31:42 (12.9 KB/s) - `.listing' saved [728]

Removed `.listing'.
--2011-05-16 09:31:42--  ftp://sourceware.org/pub/cygwinports//setup.bz2
           => `setup.bz2'
==> CWD not required.
==> PASV ... done.    ==> RETR setup.bz2 ... done.
Length: 456183 (445K)

100%[======================================================================================================================>] 456,183      434K/s   in 1.0s

2011-05-16 09:31:43 (434 KB/s) - `setup.bz2' saved [456183]

Updated setup.ini

Searching for installed packages matching setuptools:

Searching for installable packages matching setuptools:
python-setuptools
python3-setuptools

After running apt-cygports install python-setuptools I’m all set. A quick

$ python -c 'import setuptools'

 

shows me that setuptools is installed so now I can just install my python packages the way I like to.

$ easy_install pip virtualenv
Searching for pip
Reading http://pypi.python.org/simple/pip/
Reading http://pip.openplans.org
Reading http://www.pip-installer.org
Best match: pip 1.0.1
Downloading http://pypi.python.org/packages/source/p/pip/pip-1.0.1.tar.gz#md5=28dcc70225e5bf925532abc5b087a94b
Processing pip-1.0.1.tar.gz
Running pip-1.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-AmKs1m/pip-1.0.1/egg-dist-tmp-eM10iI
warning: no files found matching '*.html' under directory 'docs'
warning: no previously-included files matching '*.txt' found under directory 'docs/_build'
no previously-included directories found matching 'docs/_build/_sources'
Adding pip 1.0.1 to easy-install.pth file
Installing pip script to /usr/bin
Installing pip-2.6 script to /usr/bin

Installed /usr/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg
Processing dependencies for pip
Finished processing dependencies for pip
Searching for virtualenv
Reading http://pypi.python.org/simple/virtualenv/
Reading http://virtualenv.openplans.org
Reading http://www.virtualenv.org
Best match: virtualenv 1.6.1
Downloading http://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.6.1.tar.gz#md5=1a475df2219457b6b4febb9fe595d915
Processing virtualenv-1.6.1.tar.gz
Running virtualenv-1.6.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-azd5Po/virtualenv-1.6.1/egg-dist-tmp-L5w7AD
warning: no previously-included files matching '*.*' found under directory 'docs/_templates'
Adding virtualenv 1.6.1 to easy-install.pth file
Installing virtualenv script to /usr/bin

Installed /usr/lib/python2.6/site-packages/virtualenv-1.6.1-py2.6.egg
Processing dependencies for virtualenv
Finished processing dependencies for virtualenv

Now a word of warning. I don’t like to manually install things on my system in a way that I can’t keep them managable and distict from what the system is managing. The packages that easy_install just installed were put into /usr/ namespace. I could have had them installed into /usr/local but in my laziness I have let this one slide. To prevent any further cruft I have installed virtualenv. Virtualenv and the python virtual toolset could have a post all their own. In any case if you want to use virtualenv with cygwin you can see how I use it below.

$ pip install virtualenvwrapper
Downloading/unpacking virtualenvwrapper
  Downloading virtualenvwrapper-2.7.1.tar.gz (690Kb): 690Kb downloaded
  Running setup.py egg_info for package virtualenvwrapper
Requirement already satisfied (use --upgrade to upgrade): virtualenv in /usr/lib/python2.6/site-packages/virtualenv-1.6.1-py2.6.egg (from virtualenvwrapper)
Installing collected packages: virtualenvwrapper
  Running setup.py install for virtualenvwrapper
      2 [main] python 644 C:\cygwin\bin\python.exe: *** fatal error - unable to remap \\?\C:\cygwin\lib\python2.6\lib-dynload\time.dll to same address as parent: 0x18B40000 != 0x1A230000
Stack trace:
Frame     Function  Args
00289F08  6102796B  (00289F08, 00000000, 00000000, 00000000)
0028A1F8  6102796B  (6117EC60, 00008000, 00000000, 61180977)
0028B228  61004F1B  (611A7FAC, 6124355C, 18B40000, 1A230000)
End of stack trace
      2 [main] python 7940 fork: child 644 - died waiting for dll loading, errno 11
    Error [Errno 11] Resource temporarily unavailable while executing command /usr/bin/python -c "import setuptools;__file__='/home/jrizzo/build/virtualenvwrapper/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/pip-vDBPql-record/install-record.txt
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg/pip/basecommand.py", line 126, in main
    self.run(options, args)
  File "/usr/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg/pip/commands/install.py", line 228, in run
    requirement_set.install(install_options, global_options)
  File "/usr/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg/pip/req.py", line 1100, in install
    requirement.install(install_options, global_options)
  File "/usr/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg/pip/req.py", line 572, in install
    cwd=self.source_dir, filter_stdout=self._filter_install, show_stdout=False)
  File "/usr/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg/pip/__init__.py", line 220, in call_subprocess
    cwd=cwd, env=env)
  File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
    errread, errwrite)
  File "/usr/lib/python2.6/subprocess.py", line 1049, in _execute_child
    self.pid = os.fork()
OSError: [Errno 11] Resource temporarily unavailable

a forking mess

Before continuing with my installation of python I want to point out something you may run into from time to time.  This happened to me when setting up python so I point it out now, but this could happen whenever you update the Cygwin binaries.

This error is due to difficulty implementing POSIX standard fork on Windows. In order to work around some issues there is some voodoo magic going on that requires a very special memory layout in assemblies such as the cygwin1.dll. To fix this problem you will probably need to run a rebaseall and possibly peflagsall. Just remember that you can’t run those commands while you have any cygwin processes running. Even if you are careful about not having any cygwin processes running you should probably to reboot after you run those commands. If you don’t reboot don’t yell at me ;).  Others have written about this in more detail so take a look before blindly running rebaseall.

Continuing with your normally scheduled broadcast… back to setting up python

So now after running rebaseall you can see that pip installed virtualenvwrapper nicely.

$ pip install virtualenvwrapper
Downloading/unpacking virtualenvwrapper
  Running setup.py egg_info for package virtualenvwrapper
Requirement already satisfied (use --upgrade to upgrade): virtualenv in /usr/lib/python2.6/site-packages/virtualenv-1.6.1-py2.6.egg (from virtualenvwrapper)
Installing collected packages: virtualenvwrapper
  Running setup.py install for virtualenvwrapper
    changing mode of build/scripts-2.6/virtualenvwrapper.sh from 644 to 755
    Skipping installation of /usr/lib/python2.6/site-packages/virtualenvwrapper/__init__.py (namespace package)
    Installing /usr/lib/python2.6/site-packages/virtualenvwrapper-2.7.1-py2.6-nspkg.pth
    changing mode of /usr/bin/virtualenvwrapper.sh to 755
Successfully installed virtualenvwrapper
Cleaning up...

Now I can setup my python virtual environment and install the things I use on an every day basis. The following steps were taken directly from the virtualenvwrapper.sh script

  1. Create a directory to hold the virtual environments. (mkdir $HOME/.virtualenvs).
  2. Add a line like “export WORKON_HOME=$HOME/.virtualenvs” to your .bashrc.
  3. Add a line like “source /path/to/this/file/virtualenvwrapper.sh” to your .bashrc.
  4. Run: source ~/.bashrc
  5. Run: workon
  6. A list of environments, empty, is printed.
  7. Run: mkvirtualenv temp
  8. Run: workon
  9. This time, the “temp” environment is included.
  10. Run: workon temp
  11. The virtual environment is activated.

Generally speaking the standard steps for using virtualenvwrapper are good however a couple of notes.  I usually use the following command when creating virtual environments so that I don’t have any conflict or dependency on the system installed packages.

me@cygwin ~/.virtualenvs
$ mkvirtualenv --distribute --no-site-packages --python=python2.6.exe py26
Running virtualenv with interpreter /usr/bin/python2.6.exe
New python executable in py26/bin/python2.6
Also creating executable in py26/bin/python.exe
Installing distribute....................................................................................................................................................................................done.
Installing pip...............done.
virtualenvwrapper.user_scripts creating /home/me/.virtualenvs/py26/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/me/.virtualenvs/py26/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/me/.virtualenvs/py26/bin/preactivate
virtualenvwrapper.user_scripts creating /home/me/.virtualenvs/py26/bin/postactivate
virtualenvwrapper.user_scripts creating /home/me/.virtualenvs/py26/bin/get_env_details
(py26)

So now I can install the other programs I want without worrying about any cygwin specific packaging.

me@cygwin ~/.virtualenvs
$ which pip
/home/me/.virtualenvs/py26/bin/pip
(py26)
me@cygwin ~/.virtualenvs
$ pip install ipython
Downloading/unpacking ipython
 Downloading ipython-0.10.2.zip (6.4Mb): 6.4Mb downloaded
 Running setup.py egg_info for package ipython
 running egg_info
Installing collected packages: ipython
 Running setup.py install for ipython
 running install
 Installing iptest script to /home/me/.virtualenvs/py26/bin
 Installing ipythonx script to /home/me/.virtualenvs/py26/bin
 Installing ipcluster script to /home/me/.virtualenvs/py26/bin
 Installing ipython script to /home/me/.virtualenvs/py26/bin
 Installing pycolor script to /home/me/.virtualenvs/py26/bin
 Installing ipcontroller script to /home/me/.virtualenvs/py26/bin
 Installing ipengine script to /home/me/.virtualenvs/py26/bin
Successfully installed ipython
Cleaning up...
(py26)

A lot more can be done here. In fact I usually use virtualenvwrapper on top of virtualenv and several other things but I think this is enough for now.  To read more take a look at the following links.

  • Virtualenv: http://pypi.python.org/pypi/virtualenv
  • Virtualenvwrapper: http://www.doughellmann.com/docs/virtualenvwrapper/

Summary

So now you can see some of the things that I do with Cygwin to make working in a Windows world more comfortable but what about you?  Do you have any tips or techniques that give you that cozy Unix feeling?  Let me know what you think and maybe I’ll follow up with other things I’ve found.
References
  1. http://www.cygwin.com/install.html
  2. http://lifehacker.com/179514/geek-to-live–introduction-to-cygwin-part-i
  3. http://www.e-texteditor.com/forum/viewtopic.php?p=1823
  4. http://sourceware.org/cygwinports/
  5. http://www.gentoo-wiki.info/HOWTO_Gentoo_on_Cygwin
  6. http://code.google.com/p/apt-cyg/
  7. http://mah.everybody.org/docs/ssh

John Rizzo

Technology, problem solving, learning, business, society and where those things intersect is what I am always thinking about. From my hobbies to my profession I attempt to combine those interests in a way that makes the sum greater than the whole. Find out more about me at linkedin, http://www.linkedin.com/in/johnrizzo1.

You may also like...

Comments

  1. Alex says:

    Thanks mate, this was very helpful for me.

  2. greg says:

    This was freakin’ great.
    1. All you cmds, installs, etc. worked. no hassles.
    2. I didn’t know about virtualenv or virtualenvwarpper — thks!
    3. An elegant way to get ipython installed in comparison to other advice out there.

    A hearty thanks.

  3. Matthew says:

    Great article. However, I recommend for this and all your posts that you use a MUCH more readable font — i.e. much darker (bold won’t cut it). I really had to strain to read this because there’s no contrast between the background and the font. Just some basic UI knowledge.
    Thanks

    1. johnrizzo1 says:

      Thanks for your comment. This is my personal site so I occasionally use it to experiment with new things. I was trying out a new theme and I agree with you that it had some very bad UI issues. I decided not to use it for other reasons so now I’m back to my old standby 😉

  4. Shelby says:

    Greetings I am so grateful I found your website, I really found you by
    accident, while I was looking on Askjeeve for something else, Anyhow
    I am here now and would just like to say thanks
    a lot for a fantastic post and a all round entertaining blog (I
    also love the theme/design), I don’t have time to
    read it all at the minute but I have saved it and also added in your RSS feeds, so when I
    have time I will be back to read much more, Please do keep up the fantastic work.