Cross Release APT Managment aka How to Watch Netflix on Debian 7 Wheezy

2015-03-11T00:00:00+00:00

Native Netflix video streaming has come to GnuLinux! …if you have the correct library versions.

I am currently running GNU-Linux Debian 7 Wheezy with OpenBox. I really enjoy this lightweight, speedy and easily customized window manager (OpenBox uses simple XML configuration files). So I was also pretty excited when Netflix added HTML5 streaming support and read that folks were proclaiming success in Google Chrome browsers without necessitating any agent masking workarounds.

However, I found I was still getting errors when attempting to stream video in Chrome. The forums I was reading were reporting that when using the Chrome 36+ browser, Netflix would allow Linux streaming. Most all of these forums were based in a Ubuntu 14.04+ environment. Nevertheless, I found a hint as to how to proceed in Debian after reading this article regarding libnss:

“Netflix streams its video in HTML5, but uses a technology called Encrypted Media Extensions to prevent piracy. These extensions in turn require a set of libraries called Network Security Services that the browser can access.”

Debian Wheezy’s repo list maxed out at libnss3==2:3.14 and I would need libnss3==2:3.16+ in order to pass the DRM tests and securely stream with Netflix’s HTML5 option enabled. In order to allow this libnss upgrade, I would first need to provide APT with instructions to pull from the Debian “jessie” development branch.

This is accomplished by setting repo priorities. I created a “jessie” specific APT sources list and added the Debian repo url’s for jessie:

$ cat /etc/apt/sources.list.d/jessie.list
## DEBIAN JESSIE
deb ftp://ftp.debian.org/debian/ jessie main
deb-src ftp://ftp.debian.org/debian/ jessie main

And set pin priorities for libnss3 to fetch jessie libraries over wheezy while defining a lower priority of all other jessie packages:

$ cat /etc/apt/preferences
Package: *
Pin: release a=waldorf
Pin-Priority: 1001

Package: *
Pin: release a=wheezy
Pin-Priority: 500

Package: *
Pin: release a=jessie
Pin-Priority: 110

Package: libnss3
Pin: release n=jessie
Pin-Priority: 510

Now, update apt and confirm higher libnss3 installation candidates:

$ sudo apt-get update && sudo apt-cache policy libnss3
libnss3:
  Installed: 2:3.14.5-1+deb7u3
  Candidate: 2:3.17.2-1.1
  Package pin: 2:3.17.2-1.1
  Version table:
    2:3.17.2-1.1 510
        500 ftp://ftp.debian.org/debian/ jessie/main amd64 Packages
*** 2:3.14.5-1+deb7u3 510
        500 http://http.debian.net/debian/ wheezy/main amd64 Packages
        500 http://security.debian.org/ wheezy/updates/main amd64 Packages
        100 /var/lib/dpkg/status

Install new libnss3 candidate:

$ sudo apt-get install libnss3=2:3.17.2-1.1

Restart any Chrome instances (and upgrade to 36+ if you haven’t yet) and enjoy Netflix streaming on Debian Linux!

Use Ansible/Jinja2 templates to change file content based on target OS

2013-12-19T00:00:00+00:00

In the End Point hosting team we really love automating repetitive tasks, especially when it involves remembering many little details which can over time be forgotten, like differences of coreutils location between some versions of Ubuntu (Debian), CentOS (Red Hat) and OpenBSD variants.

In our environment we bind the backup SSH user authorized_keys entry to a custom command in order to have it secured by being, among other aspects, tied to a specific rsync call.

So in our case the content of our CentOS authorized_keys would be something like:

command="/bin/nice -15 /usr/bin/rsync --server --daemon .",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAB3[...]Q== endpoint-backup

Sadly that’s only true for CentOS systems so that if you want to automate the distribution of authorized_keys (as we’ll show in another post) to different Linux distributions (like Ubuntu) you may need to tweak it to comply to the new standard “/usr/bin” location, which will be eventually adopted by all new Linux versions overtime.. RHEL 7.x onward included.

To do the OS version detection we decided to use an Ansible/Jinja2 template by placing the following line in the Ansible task:

- name: Deploy /root/.ssh/authorized_keys
  template: src=all/root/.ssh/authorized_keys.j2
            dest=/root/.ssh/authorized_keys
            owner=root
            group=root
            mode=0600

And inside the actual file place a slightly modified version of the line above:

command="{% if ansible_os_family != "RedHat" %}/usr{% endif %}/bin/nice -15 /usr/bin/rsync --server --daemon .",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAB3[...]Q== endpoint-backup"

So that if the target OS is not part of the “RedHat” family it will add the “/usr” in front of the “/bin/nice” absolute path.

Easy peasy, ain’t it?

Now go out there and exploit this feature to all your needs.

PostgreSQL Debian apt repository GnuPG key update

2013-10-15T00:00:00+00:00

The excellent PGDG (PostgreSQL Global Development Group) apt repositories provide current point releases of supported PostgreSQL versions for Debian and Ubuntu LTS Linux. If you’d like to use a newer version of PostgreSQL than ships with your Linux distribution, or need to use an older Postgres release, you should take a look at http://wiki.postgresql.org/wiki/Apt.

A minor housekeeping matter arose just a few days ago: The GnuPG key used to sign the PostgreSQL packages expired on October 13. During a run of apt-get update && apt-get upgrade that leads to errors such as those seen here:

Get:1 http://security.debian.org wheezy/updates Release.gpg [836 B]
Get:2 http://security.debian.org wheezy/updates Release [102 kB]
Get:3 http://apt.postgresql.org wheezy-pgdg Release.gpg [836 B]
Get:4 http://ftp.de.debian.org wheezy Release.gpg [1,672 B]
Get:5 http://apt.postgresql.org wheezy-pgdg Release [29.2 kB]
Get:6 http://ftp.de.debian.org wheezy-updates Release.gpg [836 B]
Get:7 http://ftp.de.debian.org wheezy Release [168 kB]
Err http://apt.postgresql.org wheezy-pgdg Release

Get:8 http://security.debian.org wheezy/updates/main Sources [61.2 kB]
Get:9 http://security.debian.org wheezy/updates/main amd64 Packages [113 kB]
Get:10 http://security.debian.org wheezy/updates/main Translation-en [66.5 kB]
Get:11 http://ftp.de.debian.org wheezy-updates Release [124 kB]
Get:12 http://ftp.de.debian.org wheezy/main Sources [5,959 kB]
Get:13 http://ftp.de.debian.org wheezy/main amd64 Packages [5,848 kB]
Get:14 http://ftp.de.debian.org wheezy/main Translation-en [3,851 kB]
Get:15 http://ftp.de.debian.org wheezy-updates/main Sources [1,995 B]
Hit http://ftp.de.debian.org wheezy-updates/main amd64 Packages/DiffIndex
Hit http://ftp.de.debian.org wheezy-updates/main Translation-en/DiffIndex
Fetched 16.3 MB in 5s (3,094 kB/s)
Reading package lists... Done
W: A error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error:
http://apt.postgresql.org wheezy-pgdg Release: The following signatures were invalid: KEYEXPIRED 1381654177

W: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/wheezy-pgdg/Release

W: Some index files failed to download. They have been ignored, or old ones used instead.

And, fair enough, the PostgreSQL apt wiki page mentions this in its news section:

2013-10-10: New pgdg-keyring version extending the key expiration date. The old expiration date is 2013-10-13.

Updating the key to quell the error may not be quite as simple as expected, though. If you follow the instructions on the page, you would do this:

wget --quiet -O - http://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc | sudo apt-key add -

Indeed that imports the new key. However, you may still see errors from apt. Why? It’s possible to have apt trusted keys installed in more than one place:

# apt-key list
/etc/apt/trusted.gpg
--------------------
pub   4096R/ACCC4CF8 2011-10-13 [expires: 2016-02-24]
uid                  PostgreSQL Debian Repository

/etc/apt/trusted.gpg.d//apt.postgresql.org.gpg
----------------------------------------------
pub   4096R/ACCC4CF8 2011-10-13 [expired: 2013-10-13]
uid                  PostgreSQL Debian Repository

A script is referenced from the apt setup page but it doesn’t do the exact same thing as the instructions show. Instead, it installs the GPG key in /etc/apt/sources.list.d/pgdg.list, and unless we update or remove that file, apt will continue to see the expired key and complain about it.

In our example above, we have already imported the new key into the main /etc/apt/trusted.gpg keystore, so let’s just remove the original key that was in its own file in /etc/apt/trusted.gpg.d/ like this:

rm -f /etc/apt/trusted.gpg.d/apt.postgresql.org.gpg

Afterward, apt happily proceeds with its work.

Proxmox and the fun maze of IPv6 connectivity

2013-07-08T00:00:00+00:00

While working on the Proxmox machine setup and specifically on the IPv6 connectivity I found a problem where after a reboot I always kept getting the *** net.ipv6.conf.all.forwarding*** and all related variable set to 0, thus giving lots of IPv6 network connectivity issues on the guests.

While brainstorming with a colleague on this, we discovered in the boot logs these few messages which are quite indicative of something horrible happening at boot:

# less /var/log/boot.0
[..]
Mon Jul  8 18:38:59 2013: Setting kernel variables ...sysctl: cannot stat /proc/sys/net/ipv6/conf/all/forwarding: No such file or directory
Mon Jul  8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/forwarding: No such file or directory
Mon Jul  8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/autoconf: No such file or directory
Mon Jul  8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/accept_dad: No such file or directory
Mon Jul  8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/accept_ra: No such file or directory
Mon Jul  8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/accept_ra_defrtr: No such file or directory
Mon Jul  8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/accept_ra_rtr_pref: No such file or directory
Mon Jul  8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/accept_ra_pinfo: No such file or directory
Mon Jul  8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/accept_source_route: No such file or directory
Mon Jul  8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/accept_redirects: No such file or directory
Mon Jul  8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/all/autoconf: No such file or directory
Mon Jul  8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/all/accept_dad: No such file or directory
Mon Jul  8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/all/accept_ra: No such file or directory
Mon Jul  8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/all/accept_ra_defrtr: No such file or directory
Mon Jul  8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/all/accept_ra_rtr_pref: No such file or directory
Mon Jul  8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/all/accept_ra_pinfo: No such file or directory
Mon Jul  8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/all/accept_source_route: No such file or directory
Mon Jul  8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/all/accept_redirects: No such file or directory
Mon Jul  8 18:38:59 2013: done.
[..]

The following steps would be to either crawl through the “inextricable” maze of the ProxMox (PVE) boot initrd image and probably came up with the solution or find a quick way to deal with this in a clean way without touching the boot process.

Since it was all due to sysctl being called too early in the boot process and then not finding proper IPv6 module already loaded calling it again later would suffice. So I simply added the following line to /etc/network/interfaces

iface eth0 inet6 static
address    YOUR:IPV6:IS:HERE
netmask    64
up ip -6 route add default via fe80::1 dev eth0
up sysctl -p # <------ ADDED THIS LINE TO FIX IPv6 CONNECTIVITY ISSUES

And there it goes. Reboot once again to verify and you should be all set.

IPv6 Tunnels with Debian/Ubuntu behind NAT

2012-03-01T00:00:00+00:00

As part of End Point’s preparation for World IPv6 Launch Day, I was asked to get my IPv6 certification from Hurricane Electric. It’s a fun little game-based learning program which had me setup a IPv6 tunnel. IPv6 tunnels are used to provide IPv6 for those whose folks whose ISP or hosting provider don’t currently support IPv6, by “tunneling” it over IPv4. The process for creating a tunnel is straight forward enough, but there were a few configuration steps I felt could be better explained.

After creating a tunnel, Hurricane Electric kindly provides a summary of your configuration and offers example configurations for several different operating systems and routers. Below is my configuration summary and the example generated by Hurricane Electric.

However, entering these commands change won’t survive a restart. For Debian/Ubuntu users an update in /etc/network/interfaces does the trick.

#/etc/network/interfaces
auto he-ipv6
iface he-ipv6 inet6 v4tunnel
  address 2001:470:4:9ae::2
  netmask 64
  endpoint 209.51.161.58
  local 204.8.67.188
  ttl 225 
  gateway 2001:470:4:9ae::1

Firewall Configuration

If you’re running UFW the updates to /etc/default/ufw are very straightforward. Simply change the IPV6 directive to yes. Restart the firewall and your network interfaces and you should be able to ping6 ipv6.google.com. I also recommend hitting http://test-ipv6.com/ for a detailed configuration test.

Behind NAT

If you’re behind a NAT, the configuration needs to be tweaked a bit. First, you’ll want to setup a static IP address behind your router. If you’re router supports configuration of forwarding more than just TCP/UDP, you’ll want to forward protocol 41 (aka IPv6) (NOT PORT 41), which is responsible for IPv6 tunneling over IPv4, to your static address. If you’ve got a consumer grade router that doesn’t support this, you’ll just have to put your machine in the DMZ, thus putting your computer “in front” of your router’s firewall. Please make sure you are running a local software firewall if you chose this option.

After handling the routing of protocol 41, there is one small configuration change to /etc/network/interfaces. You must change your tunnel’s local address from your public IP address, to your private NATed address. Here is an example configuration including both the static IP configuration and the updated tunnel configuration.

#/etc/network/interfaces
auto eth0
iface eth0 inet static
  address 192.168.0.50
  netmask 255.255.255.0
  gateway 192.168.0.1 

auto he-ipv6
iface he-ipv6 inet6 v4tunnel
  address 2001:470:4:9ae::2
  netmask 64
  endpoint 209.51.161.58
  <b>local 192.168.0.50</b>
  ttl 225 
  gateway 2001:470:4:9ae::1

Don’t forget to restart your networking interfaces after these changes. I found a good ol’ restart was helpful as well, but of course, we don’t have this luxury in production, so be careful!

Checking IPv6

If you’re reading this article, you’re probably responsible for several hosts. For a gentle reminder which of your sites you’ve not yet setup IPv6, I recommend checking out IPvFoo for Chrome or 4or6 for Firefox. These tools make it easy for you to see which of your sites are ready for World IPv6 Launch Day!

Getting Help

Hurricane Electric provides really great support for their IPv6 tunnel services (which is completely free). Simply email ipv6@he.net and provide them with some useful information such as:

cat /etc/network/interfaces
cat netstat -nrA inet6  (these are your IPv6 routing tables)
cat /etc/default/ufw
relevant router configurations

I was very impressed to get a response from a competent person in 15 minutes! Sadly, there is one downside to using this tunnel; IRC is not an allowed.

Due to an increase in IRC abuse, new non-BGP tunnels now have IRC blocked by default. If you are a Sage, you can re-enable IRC by visiting the tunnel details page for that specific tunnel and selecting the ‘Unblock IRC’ option. Existing tunnels have not been filtered.

I guess ya gotta earn it to use IRC over your tunnel. Good luck!

Linux unshare -m for per-process private filesystem mount points

2012-01-27T00:00:00+00:00

Private mount points with unshare

Linux offers some pretty interesting features that are either new, borrowed, obscure, experimental, or any combination of those qualities. One such feature that is interesting is the unshare() function, which the unshare(2) man page says “allows a process to disassociate parts of its execution context that are currently being shared with other processes. Part of the execution context, such as the mount namespace, is shared implicitly when a new process is created using fork(2) or vfork(2)”.

I’m going to talk here about one option to unshare: per-process private filesystem mount points, also described as mount namespaces. This Linux kernel feature has been around for a few years and is easily accessible in the userland command unshare(1) in util-linux-ng 2.17 or newer (which is now simply util-linux again without the “ng” distinction because the fork took over mainline development).

Running unshare -m gives the calling process a private copy of its mount namespace, and also unshares file system attributes so that it no longer shares its root directory, current directory, or umask attributes with any other process.

Yes, completely private mount points for each process. Isn’t that interesting and strange?

A demonstration

Here’s a demonstration on an Ubuntu 11.04 system. In one terminal:

% su -
Password:
# unshare -m /bin/bash
# secret_dir=`mktemp -d --tmpdir=/tmp`
# echo $secret_dir
/tmp/tmp.75xu4BfiCw
# mount -n -o size=1m -t tmpfs tmpfs $secret_dir
# df -hT
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/mapper/auge-root
              ext4    451G  355G   74G  83% /

There’s no system-wide sign of /tmp/tmp.* there thanks to mount -n which hides it. But it can be seen process-private here:

# grep /tmp /proc/mounts
tmpfs /tmp/tmp.75xu4BfiCw tmpfs rw,relatime,size=1024k 0 0
# cd $secret_dir
# ls -lFa
total 36
drwxrwxrwt  2 root root    40 2011-11-03 22:10 ./
drwxrwxrwt 21 root root 36864 2011-11-03 22:10 ../
# touch play-file
# mkdir play-dir
# ls -lFa
total 36
drwxrwxrwt  3 root root    80 2011-11-03 22:10 ./
drwxrwxrwt 21 root root 36864 2011-11-03 22:10 ../
drwxr-xr-x  2 root root    40 2011-11-03 22:10 play-dir/
-rw-r--r--  1 root root     0 2011-11-03 22:10 play-file

Afterward, in another terminal, and thus a separate process with no visibility into the above-shown terminal process’s private mount points:

% su -
Password:
# grep /tmp /proc/mounts
# cd /tmp/tmp.75xu4BfiCw
# ls -lFa
total 40
drwx------  2 root root  4096 2011-11-03 22:10 ./
drwxrwxrwt 21 root root 36864 2011-11-03 22:18 ../

It’s all secret!

Use cases

This feature makes it possible for us to create a private temporary filesystem that even other root-owned processes cannot see or browse through, raising the bar considerably for a naive attacker to get access to sensitive files or even see that they exist, at least when they’re not currently open and visible to e.g. lsof.

Of course a sophisticated attacker would presumably have a tool to troll through kernel memory looking for what they need. As always, assume that a sophisticated attacker who has access to the machine will sooner or later have anything they really want from it. But we’d might as well make it a challenge.

Another possible use of this feature is to have a process unmount a filesystem privately, perhaps to reduce the exposure of other files on a system to a running daemon if it is compromised.

/etc/mtab vs. /proc/mounts

Experimenting with this feature also drew my attention to differences in how popular Linux distributions expose mount points. There are actually traditionally two places that the list of mounts is stored on a Linux system.

First, the classic Unix /etc/mtab, which is in essence a materialized view. It is the reason that on the Ubuntu 11.04 example above we see the private mount point everywhere on the system, but it reported different disk sizes. The existence of the mount point was global in /etc/mtab but the sizes are determined dynamically and differ based on process’s view into the mount points themselves. The mount -n option tells mount to not put the new mount point into /etc/mtab. And this is what the df(1) command refers to. How repulsive that a file in the normally read-only /etc is written to so nonchalantly!

Second, the Linux-specific /proc/mounts, which is real-time, exact, and accurate, and can appear differently to each process. The mount invocation can’t hide anything from /proc/mounts. This is what you would think is the only place to look for mounts, but /etc/mtab is still used some places.

Ubuntu 11.04 still has both, with a separate /etc/mtab. Fedora 16 has done away with /etc/mtab entirely and made it merely a symlink to /proc/mounts, which makes sense, but that is a newer convention and leads to the surprising difference here.

Linux distributions and unshare

The unshare userland command in util-linux(-ng) comes with RHEL 6, Debian 6, Ubuntu 11.04, and Fedora 16, but not on the very common RHEL 5 or CentOS 5. Because we needed it on RHEL 5, I made a simple package that contains only the unshare(1) command and peacefully coexists with the older stock RHEL 5 util-linux. It’s called util-linux-unshare and here are the RPM downloads for RHEL 5:

x86_64: util-linux-unshare-2.20.1-3.ep.x86_64.rpm
i386: util-linux-unshare-2.20.1-3.ep.i386.rpm
SRPM: util-linux-unshare-2.20.1-3.ep.src.rpm

I hope you’ve found this as interesting as I did!