Wednesday, April 13, 2016

Fun experiences using Wine in Docker


I sometimes work with a legacy codebase that targets both Windows and Linux; the build system is GNU Make-based, and builds on Linux. For the Windows components, the build system invokes NMAKE, using Wine. Yes, it's messy; yes I want to replace it; but no there's no time budgeted right now.

Lately, I've been moving more and more of our build infrastructure to Docker. It makes keeping the build environments up-to-date for developers easier, and simplifies the setup for Continuous Integration. Check out my tool, Scuba for using Docker to perform local builds, and GitLab CI.

You can see where this is going. I decided to convert our legacy build VM into a Docker image; Wine and NMAKE included. I didn't know what I was getting myself into.

VM to Docker Image

Of course, the right way to create a Docker image is to use a Dockerfile. However, this current VM had experienced years of tweaks, potentially relying on subtle toolchain-version-specific quirks. I wasn't about to re-build it from scratch, so I decided to convert the VM filesystem directly to a Docker image.

The initial conversion turned out to be straightforward. First, I cloned the VM, so I could work destructively. Next, I uninstalled everything that wasn't necessary for a Docker image (including KDE, X11, firewall, etc.) Then, I powered down the cloned VM, and mounted its virtual disk under another VM, running Docker. From there, it's as simple as using Tar to create the Docker image:

# cd /mnt/buildvm; tar -c * | docker import --change='CMD /bin/bash' - buildsys:1

This adds all of the directories from the mounted build VM disk, and creates a tar stream which is piped into docker import - (where - means standard input). Note that I'm also setting the `CMD` to be `/bin/bash`; this way, the image can be run by simply using docker run -it buildsys:1, without having to specify /bin/bash every run.

After the initial conversion was done and I no longer needed to "boot" in the conventional way, I continued to run the image, removing more stuff, like:

  • rpm -e --nodeps kernel-xxx (You don't need a kernel when running under Docker, but don't want to remove other things that "depend" on it.)
  • yum remove dracut grub plymouth
  • yum clean all && rm -rf /var/cache/yum
  • rm -rf /var/log/* /tmp/*
I definitely had to be careful not to remove things that Wine unexpectedly relied upon. As I did this, I occasionally ran the image through a docker export / docker import cycle to actually reduce the virtual size of the image.

Wine without X11

The first time I tried to run wine in a Docker container, I was met with the following warnings/errors:

Application tried to create a window, but no driver could be loaded.
Make sure that your X server is running and that $DISPLAY is set correctly.
Googling for the error yielded some results from some other guys crazy enough to try using Wine in Docker also, like this SuperUser post and this GitHub project. It seemed that I would need some sort of X server after all, and that Xvfb (X Virtual FrameBuffer) was the solution.

You can simply run xvfb-run wine whatever.exe, and this will avoid the "no $DISPLAY" problems. Great. However, I didn't want to change any of our code to have to run under Docker. Specifically, I didn't want to track down every invocation of wine and prefix it with xvfb-run; what if we are running on native X?

Instead, I came up with what I believe is a novel solution: ENTRYPOINT. This essentially prefixes the user's command with whatever is specified in ENTRYPOINT - just what we want to do with xvfb-run. So the last time I re-imported the tarball, I added --change='ENTRYPOINT xvfb-run'. There's probably a way to do this after it's been imported, but this was the most convenient at the time.

Now, when I run docker run --rm -it buildsys:1 /bin/bash, I can verify that $DISPLAY is set, and Wine is happy. For now.

More to come...

Friday, November 20, 2015

Installing ESXi in a QEMU-KVM virtual machine, under libvirt / virt-manager

For a test setup, it may be useful to install VMware ESXi in a QEMU-KVM guest. If, like me, you're using libvirt (using virt-manager) to manage your VMs, here's some information to get this set up. I'm using Fedora 22, and ESXi 5.5.0.

There are other posts explaining how to set this up, but I wanted to share my experience, which is specific to virt-manager, and the newer QEMU.

Here's a step-by-step procedure for getting this working.

Add required KVM kernel module parameters:

  1. Edit (or create) /etc/modprobe.d/kvm-intel.conf to look like this:
    options kvm ignore_msrs=1
    options kvm-intel nested=y ept=y
  2. Remove the KVM module and re-load it with the new parameters:
    # modprobe -r kvm-intel kvm; modprobe kvm kvm-intel

Setup ESXi VM guest configuration:

  1. Create your ESXi VM using virt-manager.
  2. Change the NIC to vmxnet3. You'll have to manually type this in; it won't be in the drop-down.
  3. You'll need at least 2 GiB of RAM. (During install it actually came back with:
    <MEMORY_SIZE ERROR: This host has 2.00 GiB of RAM. 3.97 GiB are needed>
  4. Edit the config for this VM (named "esxi-test" here):
    # virsh -c 'qemu:///system' edit esxi-test
  5. Edit the first line of the XML file to be:
    <domain type='kvm' xmlns:qemu=''>
  6. Change the CPU type:
    <cpu mode='host-passthrough'/>
  7. Add this block anywhere inside of <domain>...</domain>:
        <qemu:arg value='-machine'/>
        <qemu:arg value='vmport=off'/>
  8. Save and quit
Boot into the ESXi installer, and enjoy!

In dmesg, I see kvm spewing these messages, which probably have to do with ignroe_msrs:

kvm [3864]: vcpu0 ignored rdmsr: 0x34
kvm [3864]: vcpu0 ignored rdmsr: 0x34

ESXi 6.0.0 Notes:

I tried to use ESXi 6.0.0, but it didn't seem to find a network card, even though I specified vmnet3. These notes apply to 6.0.0:

  • Note that the installer appears to hang at "user loaded successfully." for about 110 seconds. "Running nfcd start" also takes a while. I have no idea why.


Thursday, July 9, 2015


Every geek has his/her favorite set of tools for accomplishing various tasks. Here are mine.

I tend to split my time between Linux and Windows so where possible there will be solutions for both. Preference is of course given to cross-platform FOSS projects.

This will be updated as I run across new tools or inventory the ones I use.

Wednesday, January 28, 2015

"Installing" Zotero

I've been playing with a new research tool called Zotero, which helps you keep track of research papers, etc. as you come across them.

I'm using their standalone version, which Chrome can push to via an extension. So far it seems really nice.

Zotero doesn't come with an installer on Linux, and I wanted to put it somewhere more permanent than my Downloads directory. So I did the following which makes Zotero feel very at home on my Centos 7 machine.

  1. Download the Linux tar.bz2 file
  2. Switch to root, and move the tar.bz2 file to /opt and extract it. Then rename the output directory:
    $ sudo su -
    # mv Zotero- /opt
    # cd /opt
    # tar xf Zotero-
    # rm Zotero-
    # mv Zotero_linux-x86_64 zotero
  3. Retrieve the Zotero icon and add it to the icons/ directory:
    # wget -O zotero/icons/zotero-new-z-48px.png
  4. Now as your user, create the desktop shortcut, using this .desktop file I put together:
    # exit
    $ wget -O ~/.local/share/applications/zotero.desktop
Note that the standalone version of Zotero keeps its local data in ~/.zotero. That's it! Enjoy!

Sunday, January 25, 2015

Installing rdesktop on Centos 7

I'll briefly summarize this blog post on installing rdesktop on Centos 7.

First, we'll set up the RPM build environment (as your local user):

$ sudo yum install rpm-build make gcc
$ mkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
$ echo '%_topdir %(echo $HOME)/rpmbuild' > ~/.rpmmacros

Now, fetch and install the source package:

$ wget
$ rpm -i rdesktop-1.8.2-0.1.rfx.src.rpm

Install devel dependencies and build:

$ sudo yum install openssl-devel libXt-devel libsamplerate-devel pcsc-lite-devel
$ rpmbuild -ba ~/rpmbuild/SPECS/rdesktop.spec 
$ sudo yum localinstall ~rpmbuild/RPMS/x86_64/rdesktop-1.8.2-0.1.el7.centos.x86_64.rpm

Saturday, January 24, 2015

Connecting to a Cisco ASA VPN with DoD CAC on CentOS 7

Update: I've created scripts to automate much of this process. You can find them on GitHub.

I often need to connect to a VPN with a Cisco ASA box at the head-end, using a DoD CAC (smart card) for authentication.

On Windows, this is often accomplished using Cisco's AnyConnect VPN client software. On Linux however, that option would never work for me. I tried to download it from the VPN https site, but it wouldn't load.

On Linux, we have an open-source alternative, called openconnect. The difficult part is getting it to use our smart card, and present the correct certificate to the VPN.

I found the following pages very useful in trying to get this all to work:

openconnect uses p11-kit to interact with PKCS #11 modules. (PKCS #11 is the standard for interfacing with cryptographic tokens, like smart cards.) The first thing we need to do is tell p11-kit to use the libcoolkey pkcs11 module. Do this by creating a new file named /etc/pkcs11/modules/libcoolkey.module, and adding the following line to it:


Next, we'll use p11tool --list-tokens to list all of the tokens on our system. You should see your smart card in this list. Mine showed up like this (along with others):

$ p11tool --list-tokens
Token 6:
 URL: pkcs11:model=;manufacturer=;serial=;token=REINHART.JONATHON.RICHARD.xxxxxxxx

Now, we want to look at all of the certificates available on our smart card. We'll use p11tool --list-all-certs [url], where [url] is the URL of our smart card token from the previous step:

$ p11tool --list-all-certs pkcs11:model=;manufacturer=;serial=;token=REINHART.JONATHON.RICHARD.xxxx
Object 0:
 URL: pkcs11:model=;manufacturer=;serial=;token=REINHART.JONATHON.RICHARD.xxxxxx;id=%01;object=CAC%20ID%20Certificate;object-type=cert
 Type: X.509 Certificate
 Label: CAC ID Certificate
 ID: 00:01

Object 1:
 URL: pkcs11:model=;manufacturer=;serial=;token=REINHART.JONATHON.RICHARD.xxxxxx;id=%02;object=CAC%20Email%20Signature%20Certificate;object-type=cert
 Type: X.509 Certificate
 Label: CAC Email Signature Certificate
 ID: 00:02

Object 2:
 URL: pkcs11:model=;manufacturer=;serial=;token=REINHART.JONATHON.RICHARD.xxxxxx;id=%03;object=CAC%20Email%20Encryption%20Certificate;object-type=cert
 Type: X.509 Certificate
 Label: CAC Email Encryption Certificate
 ID: 00:03
So we can see the three certificates available on our smart card.

The Windows AnyConnect software will pop-up a dialog asking you to select the certificate for authentication when the server asks for a client certificate. openconnect currently has no such functionality, so we need to explicitly tell openconnect which certificate to use. In my case, I already knew it was the certificate with ID: 00:02, the "CAC Email Signature Certificate". So I pass the -c option, with the minimal URL to unambiguously refer to that certificate:

$ sudo openconnect -c 'pkcs11:token=REINHART.JONATHON.RICHARD.xxxxxx;id=%02'

Note that I had to use sudo because openconnect will invoke some scripts to set up the tun device and routing.

At this point, openconnect should ask for your PIN, and then successfully connect to the VPN! If not, you may need to try the other certificates, by changing the id= part of the certificate URL.

Finally, there are still a few outstanding warnings that occur during this process:

  • Certificate from VPN server "" failed verification. Reason: signer not found - I need to determine which certificate this is exactly, and how to add it to my trusted certificate store.

Note: I've had to install various packages and make various changes in playing with my smart card, so if something isn't working for you, or I've skipped a step, please leave a comment so I can make this post more accurate. Thanks!

Update: Additional steps - I'll work these in above at some point:

  • yum install coolkey
  • service pcscd start (on Fedora 21)