Connecting to a Cisco ASA VPN with DoD CAC on CentOS 7

Update: I've created scripts to automate much of this process. You can find them on GitHub.

---

I often need to connect to a VPN with a Cisco ASA box at the head-end, using a DoD CAC (smart card) for authentication.

On Windows, this is often accomplished using Cisco's AnyConnect VPN client software. On Linux however, that option would never work for me. I tried to download it from the VPN https site, but it wouldn't load.

On Linux, we have an open-source alternative, called openconnect. The difficult part is getting it to use our smart card, and present the correct certificate to the VPN.

I found the following pages very useful in trying to get this all to work:

• Howto: Connect to Cisco AnyConnect VPN using OpenConnect and PKI Token
• CAC modules - openconnect-devel mailing list
• p11tool Invocation

openconnect uses p11-kit to interact with PKCS #11 modules. (PKCS #11 is the standard for interfacing with cryptographic tokens, like smart cards.) The first thing we need to do is tell p11-kit to use the libcoolkey pkcs11 module. Do this by creating a new file named /etc/pkcs11/modules/libcoolkey.module, and adding the following line to it:

module:/usr/lib64/pkcs11/libcoolkeypk11.so

---

Next, we'll use p11tool --list-tokens to list all of the tokens on our system. You should see your smart card in this list. Mine showed up like this (along with others):

$ p11tool --list-tokens
...
Token 6:
 URL: pkcs11:model=;manufacturer=;serial=;token=REINHART.JONATHON.RICHARD.xxxxxxxx
 Label: REINHART.JONATHON.RICHARD.xxxxxxxx
 Manufacturer: 
 Model: 
 Serial:

---

Now, we want to look at all of the certificates available on our smart card. We'll use p11tool --list-all-certs [url], where [url] is the URL of our smart card token from the previous step:

$ p11tool --list-all-certs pkcs11:model=;manufacturer=;serial=;token=REINHART.JONATHON.RICHARD.xxxx
Object 0:
 URL: pkcs11:model=;manufacturer=;serial=;token=REINHART.JONATHON.RICHARD.xxxxxx;id=%01;object=CAC%20ID%20Certificate;object-type=cert
 Type: X.509 Certificate
 Label: CAC ID Certificate
 ID: 00:01

Object 1:
 URL: pkcs11:model=;manufacturer=;serial=;token=REINHART.JONATHON.RICHARD.xxxxxx;id=%02;object=CAC%20Email%20Signature%20Certificate;object-type=cert
 Type: X.509 Certificate
 Label: CAC Email Signature Certificate
 ID: 00:02

Object 2:
 URL: pkcs11:model=;manufacturer=;serial=;token=REINHART.JONATHON.RICHARD.xxxxxx;id=%03;object=CAC%20Email%20Encryption%20Certificate;object-type=cert
 Type: X.509 Certificate
 Label: CAC Email Encryption Certificate
 ID: 00:03

So we can see the three certificates available on our smart card.

---

The Windows AnyConnect software will pop-up a dialog asking you to select the certificate for authentication when the server asks for a client certificate. openconnect currently has no such functionality, so we need to explicitly tell openconnect which certificate to use. In my case, I already knew it was the certificate with ID: 00:02, the "CAC Email Signature Certificate". So I pass the -c option, with the minimal URL to unambiguously refer to that certificate:

$ sudo openconnect -c 'pkcs11:token=REINHART.JONATHON.RICHARD.xxxxxx;id=%02' vpn.example.com

Note that I had to use sudo because openconnect will invoke some scripts to set up the tun device and routing.

At this point, openconnect should ask for your PIN, and then successfully connect to the VPN! If not, you may need to try the other certificates, by changing the id= part of the certificate URL.

Finally, there are still a few outstanding warnings that occur during this process:

• Certificate from VPN server "vpn.example.com" failed verification. Reason: signer not found - I need to determine which certificate this is exactly, and how to add it to my trusted certificate store.

Note: I've had to install various packages and make various changes in playing with my smart card, so if something isn't working for you, or I've skipped a step, please leave a comment so I can make this post more accurate. Thanks!

Update: Additional steps - I'll work these in above at some point:

• yum install coolkey
• service pcscd start (on Fedora 21)

Adding storage to Proxmox VE

The 1 TB drives for my HP server came today. Scott at All Computer Parts was very helpful and quick to reply. I quickly went to hot-plugging them into my server running Proxmox VE.

After the drives spun-up, I logged into the HP System Management Homepage (download) and opened the HP Array Configuration Utility. From there, I selected 3 of the unassigned drives and created an array. I then crated a RAID 5 logical drive using all of the space available on the array (1.8 TB). Finally, I added the remaining drive as a hot-spare, by selecting the array, and clicking "Spare Management". This way, if one drive goes offline, the hot-spare will immediate take its place, and the array will be rebuilt.

The logical disk is immediately detected by the Linux kernel, evidenced by /var/log/messages:

Jan 23 22:38:40 dragster kernel: hpsa 0000:04:00.0: Direct-Access     device c0b0t0l1 added.
Jan 23 22:38:40 dragster kernel: scsi 0:0:0:1: Direct-Access     HP       LOGICAL VOLUME   6.40 PQ: 0 ANSI: 5
Jan 23 22:38:40 dragster kernel: sd 0:0:0:1: Attached scsi generic sg3 type 0
Jan 23 22:38:40 dragster kernel: sd 0:0:0:1: [sdb] 3906918832 512-byte logical blocks: (2.00 TB/1.81 TiB)
Jan 23 22:38:40 dragster kernel: sd 0:0:0:1: [sdb] Write Protect is off
Jan 23 22:38:40 dragster kernel: sd 0:0:0:1: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 23 22:38:40 dragster kernel: sdb: unknown partition table
Jan 23 22:38:40 dragster kernel: sd 0:0:0:1: [sdb] Attached SCSI disk

Proxmox works well with LVM. It actually works directly with LVM volume groups, creating logical volumes on-the-fly for new VMs, etc. You can find detailed information on the Proxmox wiki, but the procedure was quite simple:

1. Create the LVM physical volume on the physical disk: (In this case, the "physical disk" was a RAID logical disk)
   

   # pvcreate /dev/sdb
   Physical volume "/dev/sdb" successfully created

   
   Note that I created the physical volume directly on the block device, without partitioning the drive. LVM does not require a partition table, and I'm not booting to the disk, so there was no need.
2. Create a LVM volume group from that single physical volume:
   

   # vgcreate raid5vg /dev/sdb
     Volume group "raid5vg" successfully created

And we're ready to go! List the LVM volume groups with the vgs command:

# vgs
  VG      #PV #LV #SN Attr   VSize  VFree
  pve       1   3   0 wz--n- 67.83g 8.50g
  raid5vg   1   1   0 wz--n-  1.82t 1.80t

Now it's time to tell Proxmox about the new storage. Log into the Proxmox web UI, and select the "Datacenter" node in the tree. On the Storage tab, select Add > LVM. On that dialog, we select the new volume group and given it a name (I made it the same as the vg).

And your storage is made available to Proxmox!

I went ahead and moved the couple VMs I had from the old main storage to the new array. You can do this by highlighting the Hard Disk on the VM's Hardware tab, and clicking "Move disk".

Add Suspend and Hibernate to GNOME 3 shell status menu in CentOS 7

I spent days searching for the Suspend option in my new CentOS 7 installation.

Well, I finally found it, but it wasn't as easy as you'd expect.

First you need to install the gnome-shell-extension-alternative-status-menu extension, as well as gnome-tweak-tool:

    $ sudo yum install gnome-shell-extension-alternative-status-menu gnome-tweak-tool

Next, log-out and log back in.

Then, use gnome-tweak-tool to enable the fancy new extension:

And there you have it; a menu that Windows has had by default for a decade.

Dual Booting with GRUB2 (CentOS 7) and Windows 7

TL;DR:You need to install the ntfs-3g package, in order for os-prober to detect Windows installations. This allows grub2-mkconfig to automatically generate an entry for dual-booting into Windows.

---

Doing a lot more hardware hacking these days, I've felt constrained running Linux in a VM all the time. I was especially disappointed that VirtualBox doesn't expose nested Intel VT-x features to its guests. So I've decided to try dual-booting again, going with the very stable CentOS 7.

Not willing to sacrifice any space on my Windows SSD, I put another Crucial SSD in my machine - this time the 256 GB version of their newer MX100 series. Downloading the NetInstall ISO and pointing at a relatively close mirror gave a very satisfying install experience. Having the whole drive made things quite easy as well - except for the actual Dual-Booting part.

I wasn't terribly surprised that the setup process didn't automatically add a GRUB 2 entry for booting to my Windows 7 drive. Everything I read indicated that simply running grub2-mkconfig should set up the GRUB config script to include Windows. Yet, it wasn't working for me. Supposedly GRUB 2 uses os-prober to automatically detect other OSes and generate boot entries for them. However, running os-prober showed no Windows install, even though my drive was clearly visible.

After stumbling across this post on LinuxQustions.org, it turns out that the NTFS-3g package (for mounting NTFS volumes) isn't installed by default, and os-prober needs this the mount the drive and detect the installed OS. After installing ntfs-3g (from the EPEL repository), I was able to run grub2-mkconfig -o /boot/grub2/grub.cfg and successfully add an entry for Windows 7.

Installing Nemiver on Centos 7

I recently heard about Nemiver, a standalone C/C++ debugger for GNOME, and wanted to give it a try.

My current Linux development box is running CentOS 7. While Nemiver packages exist for CentOS 6, the same cannot be said for CentOS 7. So I proceeded to build it from source.

First, I went ahead and cloned the Git repository: git clone git://git.gnome.org/nemiver

After running the ./autogen.sh script, it was clear that I was in for several iterations of dependency installation. I'll save you the trouble:

sudo yum install gnome-common intltool yelp-devel yelp-tools boost-devel sqlite-devel GConf2-devel libgtop2-devel glibmm24-devel gtkmm30-devel gtk3-devel gtksourceview3-devel vte3 vte3-devel

Note that I do have the EPEL repository enabled, so I'm not sure if some of those packages came from EPEL or not.

Unfortunately, there seems to be no gtksourceviewmm-3.0 package for CentOS 7 either! Sure, why not install that one from source, too?

First, download gtksourceviewmm-3.2.0.tar.xz (or later). By default, configure defaults to PREFIX=/usr/local. If you don't change this, pkg-config won't know where do find it. So ./configure --prefix=/usr --libdir=/usr/lib64 seems to be what we want. Then make, sudo make install as usual.

Now, you should be able to finish configuring and making Nemiver!

Note that there also seemed to be a build issue in src/confmgr/nmv-gconf-mgr.cc. The following patch took care of it for me - not sure how this went unnoticed.

--- a/src/confmgr/nmv-gconf-mgr.cc
+++ b/src/confmgr/nmv-gconf-mgr.cc
@@ -32,6 +32,7 @@
 NEMIVER_BEGIN_NAMESPACE (nemiver)
 
 using nemiver::common::GCharSafePtr;
+using nemiver::common::GErrorSafePtr;
 
 class GConfMgr : public IConfMgr {
     GConfMgr (const GConfMgr &);

Until this is fixed, you can use my fork at git@github.com:JonathonReinhart/nemiver.git

GitLab time zone issues

GitLab is a great open source GitHub clone, which I've started using for tracking my personal Git repos.

One frustrating thing I've found with GitLab however, is its handling of time zones.

Read:

• Wrong time on gitlab events #7049
• Allow user to select their timezone in profile
• Add timezone configuration to gitlab.yml

Apparently GitLab doesn't detect the user's timezone (or store it in their profile) and display times accordingly.  Everything outside of the Git timestamps appears to be tracked in UTC.

To fix this for my local server, I edited
    /opt/gitlab/embedded/service/gitlab-rails/config/application.rb
and specified
    config.time_zone = 'Eastern Time (US & Canada)'



Update:

In version 7.5.0, this configuration option was moved to gitlab.yml. Now you don't have to re-set this option after every upgrade. Instructions for how to set it for an Omnibus install (in /etc/gitlab/gitlab.rb) are here.

TrueCrypt-end

Today, the TrueCrypt website and SourceForge project page suddenly changed, indicating the end of TrueCrypt development. truecrypt.org now redirects to their SourceForge project page, and the content has been replaced with a surprising message:

Not only has development officially ceased, but TrueCrypt is being declared "not secure", and the official webpage is suggesting that people migrate to BitLocker!  (BitLocker is the drive encryption solution built in to some versions of Windows Vista and later.)  Furthermore, a new version 7.2 had been released, which warns users that TrueCrypt is insecure. The repository had been scrubbed, and all previous binaries had been deleted.

I'm sure you could almost hear the collective WTF?! from everyone in the InfoSec community.

A series of edits indicating that the software had been discontinued were even posted to the TrueCrypt Wikipedia page by a user with the handle Truecrypt-end.

At first, it seemed like some pranksters had managed to take over the TrueCrypt website, and poke fun by suggesting users migrate to their inferior commercial competitor, BitLocker.  Well, the DNS records had not changed, so everything was good there. And SourceForge indicated that there was no suspicious behavior on the account (ya know, aside from closing everything down!)

Of course there are rumors abound at all of the tech watering holes, from Slashdot to the /r/sysadmin subreddit  to the InfoSec Stack Exchange site and of course Twitter.  While many still believe that the project was hacked, others are pondering the possibility that the devs were asked to insert a back-door, and subject to a gag-order preventing them from disclosing the requirement.  The TrueCrypt development team has remained behind the big black curtain for most of its development which makes the situation even more curious. Perhaps a vulnerability had been discovered and the developers simply didn't want to be involved with the product any more. There is certainly no shortage of opinions on the matter. The most interesting theory I've heard is that this is a sort of warrant canary.

So what about this new version 7.2?

The binaries were signed with the same GPG key as all previous releases, indicating that this release was "official", or at least produced by someone with access to the private key.

Internally, the TrueCrypt.exe executable and the truecrypt(-x64).sys drivers were signed (a la Microsoft Authenticode) with a different certificate than 7.1.1, but that certificate expired shortly after the last release. This new certificate was issued (to the same named entity) shortly before the previous certificate expired. It's very unlikely that someone was able to spoof a new certificate in this manner, and had planned it two years ago.  [Screenshots tomorrow.]

The changes to the latest version's source code were posted to GitHub.  This paints probably the most confusing picture of all.  

First, the code has been littered with warning messages and error codes indicating that "Using TrueCrypt is not secure".  Next, we see that pretty much all of the code related to creating encrypted volumes has been removed, and replaced with AbortProcess ("INSECURE_APP");. We also notice that all code related to updates, error reporting, user's guide/help, or anything pointing back to the TrueCrypt website has been removed.  Clearly, the developers consciously made the decision to burn all bridges, and carefully executed a plan to do so.

What's very bizarre however, is that while there were 4112 deletions, there were also 1760 additions to the code. Along with other minor bugfixes, it appears that in-place decryption was newly implemented.  It looks as if this was code that was part of an upcoming release that brought some improvements after a two-year break. Unfortunately, it also came with mass deletions that rendered the software useless for anyone seeking to create encrypted content.

What are you thoughts?