Immutable servers. Also called Phoenix servers, it's hot and hip in the world of cloud services and… actually not all that bad an idea. I used to have a server called Phoenix, so one could say I was ahead of this curve. One would be wrong, but one could say it. 

In reality, I read about this concept a while ago but kind of dismissed it back then as being complicated and difficult to work with. Since then though, I found and became rather interested in an Illumos distribution called SmartOS, in my quest to find a replacement for my aging Solaris server. And it turns out that the immutable server concept and SmartOS are perfect companions.

As I was experimenting with SmartOS datasets I noticed that they were kind of hard to upgrade. Taking a dataset, pointing its pkgsrc repository at a newer version and running pkgin full-upgrade works sometimes but is more likely to cause all sorts of mayhem and chaos instead. By contrast, what I'm used to in Debian is you change your apt sources, run apt-get dist-upgrade and it mostly just works. 

This of course is a bit of a problem: Joyent, the company behind SmartOS if you weren't following, won't maintain all versions of pkgsrc forever and at some point things start getting insecure; you want to keep up-to-date with packages for software that is internet-facing, like your MTA or web server.

So as I was figuring out how to deal with this, there were two important things I ran into: first, Joyent's Machine Image Build Environment (MIBE for short), which is a set of scripts that Joyent uses to build its own SmartOS images. Secondly, a zone configuration setting called "delegated datasets" which is a Solaris Zones feature that allows a zone to manage its own ZFS datasets and in SmartOS gives you a zfs filesystem separate from the image filesystem. 

At that point, things clicked into place and everything made sense: this is how you do immutable infrastructure. And not just that: this is how you do it elegantly.

The benefits are great: you don't edit the configuration of the machine directly so there's never drift (and the machine is never in an unknown state). With traditional configuration management (like Puppet or CFEngine), it's always tempting to make local changes that are neither reflected in the configuration management software nor reset by it, which can cause problems down the line. This approach prevents those problems because upon reprovisioning, all local modifications will with 100% certainty be lost so you know not do that. Upgrades are relatively easy because the images are built from scratch every time, so you always start with a clean slate. And you have a fully-functioning image that you can test properly before you deploy it, because you then deploy that exact image. If you've had a security problem and someone compromised your server, just reprovision and it's good as new; meanwhile you can fix the problem and build a new image that fixes the vulnerability. 

TL;DR: I am now convinced immutable server are the way to go.

It's also the way I've gone, so currently have five custom MIBE images and a fork of another:

• mi-poop-mail: an image running Exim, Dovecot, Clamav and Spamassassin
• mi-poop-webmail: an image that presents a Roundcube webmail interface
• mi-poop-unifi: an image that runs the UniFi controller software from Ubiquiti
• mi-poop-zotonic: an image that runs Zotonic (and serves up this exact website, in fact.)
• mi-poop-irc: an image that runs UnrealIRCd and powers irc.poop.nl, a GIMPnet server.
• mi-dsapid: an image that runs a datasets server that I've modified a bit to use SSL.

They are based on either the base-64 or minimal-64 images from Joyent. I'll explain briefly how this works.

The MIBE environment consists of three parts insofar as what goes into the zone image:

• A directory tree to be copied into the image 
• A list of packages to be installed ('packages' file)
• A script to be executed after the packages are installed ('customize' script)

First, the directory tree is copied into the image filesystem. This is where you can add things like directories that should exist for your software, configuration files, SMF manifests, etc. Also here is where you add scripts that are to be executed by 'zoneinit'. I'll get to that in a bit.

Second, packages are installed. They are read from the 'packages' file and installed one-by-one as if you were pkgin installing them all. Just a shortcut to doing exactly that in the customize script, really.

Last, the 'customize' script is run. This script can do anything really. In my current images, it's almost empty for mi-poop-unifi (only adding a user for it) but for mi-poop-zotonic it's where Zotonic is downloaded and compiled. 

When that is done, MIBE will pack up the image and generate a manifest which you can then use to upload it to a datasets server or install directly with imgadm. 

When you create a zone, it is first provisioned. This is where the zoneinit scripts I referred to earlier come in: they are run upon the first boot of a zone. In these scripts you can enable services, set-up the delegated dataset, etc. I use them to set the timezone, disable ssh, enable services and install the unifi software (to avoid distributing it).

Some of these things can be done from the customize script too, but since these scripts are executed in the running zone, they have access to zone-related information such as its hostname, IP info and the customer/internal metadata. There is a time-out to provisioning zones, so you want to try and do things that take a lot of time in the 'customize' script rather than in zoneinit scripts. That said, anything done before the image is packed up will incur a cost with regards to the image file size. Take that into account and do what makes the most sense. 

I have put my mibe images on github so you can take a look at how you could do immutable infrastructure with SmartOS. There are more mibe images on github but most don't take advantage of delegated datasets which -- if you ask me -- is one of SmartOS's most useful features. The links to the repos are above. They are set up to be as generic as possible with all private data on a delegated dataset, so others can use them too. I also have the prebuilt images available on my datasets server, smartos.poop.nl.

 

If you have an LDAP set-up with unix/linux machines and you want to add OSX into this mix, having users log in with their LDAP credentials but with local home directories on each machine you might run into an interesting problem: your user home directories will likely be under /home but while on a standard OSX install that directory exists, it is the territory of the automounter.

So unless you configure the automounter to do something, the home directory won't be created and things will be generally strange and non-function because OSX really likes to assume the availability of a home directory. Which I think is actually quite reasonable.

This problem can be fixed easily by disabling the automounter for /home in the automounter's configuration file, /etc/auto_master. But, that requires modifying a local file on every machine which may not be ideal. Or at least I think it isn't.

So I asked the internet. And came up with nothing. There are many pages that tell you how to configure it to mount NFS or Samba or AFP shares, but none to tell you how it is disabled. 

The answer was hidden away somewhere at the bottom of the man page for auto_master:

-null This map has no entries. It is used to disable entries that occur later in the auto_master file.

That works, because the auto_master and auto_home default configurations are to check in the directory service first and if there are multiple entries for the same thing, the first match is used. So if you set a mount to "-null" in there, anything with the same mount point after it is ignored. So here's what I stuck in my LDAP server to tell the automounter to leave /home alone:

dn: automountMapName=auto_master,dc=poop,dc=nl
objectClass: top
objectClass: automountMap
automountMapName: auto_master

dn: automountKey=/home,automountMapName=auto_master,dc=poop,dc=nl
objectClass: top
objectClass: automount
automountKey: /home
automountInformation: -null

That is, one auto_master container, containing one key for /home which is set to "-null". 

I greatly dislike using DSL modems as routers. They tend to be limited in almost every way imaginable. Unfortunately, getting a proper router with a DSL interface is very much not worth it for at home. They tend to be pricy. And then some.

So in the past I've set my DSL modems to get as far out of the way as possible. That has involved using my Alcatel Speedtouch Home for as long as I could (one would dial-in to that using PPTP), then getting a real Speedtouch 780wl and using Static IP Spoofing and finally getting a real Thomson TG789vn instead of the piece of shit KPN sends you and doing the exact same thing until VDSL2 came along.

For the Speedtouch 780wl (which is an ADSL2 modem) and the 789vn operating on an ADSL2 network with PPPoA, various pre-fab configs to get it out of the way are available, from Thomson no less. Just download the utility, pick a config and it'll set-up the modem to do what you want. Works fine.

Now, I got this 789vn because I would soon be upgraded to VDSL2. Unlike ADSL2, KPN runs its VDSL2 network based on Ethernet rather than ATM. So to go with that, rather than having the modem do PPPoA, it does PPPoE. There are no pre-fab configs for the combination of VDSL + PPPoE + SIP_SPOOF. So I took the PPPoA spoof config and a VDSL config and mashed them up and a night of tinkering later I had a working config. Sort-of.

The config I landed on required setting the MAC-address of the router as a static entry in the modem's ARP table, or otherwise the modem would point the public IP address at itself. It also required serious trickery on the router end. The OpenBSD box I used couldn't really support this way of working it, so I had to use the modem's rfc1918 addressing and have it NAT the secondary address with a static config even for locally generated packets. The shiny new Juniper router I use now either can't do that or I haven't figured out how, so instead I had to let it use DHCP and tell it the default route was somewhere random in the /8 (yes, it tells you your address is a /8 address and proxy-arps that entire subnet) the modem advertises. Before both of those, I used a Linux box that required yet another, completely different approach.

"Not ideal" doesn't really cover it.

I have a Juniper SRX100 now to do my routing work that I bought second-hand recently. It made me revisit this entire situation and how non-ideal it was, exactly.

Before, I said that KPN's VDSL-network is Ethernet-based, with PPPoE to do the actual link set-up. This is actually quite useful, because PPPoE can be done natively by most routers on their Ethernet interfaces. That includes the SRX100. Of course, the Thomson won't let you by default.

Now, these Thomsons, while limited in many ways, are actually quite capable in a bunch of others. They're just hard to configure, and good documentation is beyond sparse. But I found documentation on how to configure the important part of what I wanted to do here: its built-in ethernet bridge.

In this post I will demonstrate how the goal of using the Thomson TG789vn (though this should apply to other models too, at least so long as PPPoE is used) as if it were really just a modem can be achieved. Or, more accurately, how I achieved it. My sample-size here is one, so all I can do is write down what worked for me.

These modems all have an internal logical ethernet switch (bridge) to which multiple logical interfaces can be connected. The logical interfaces can then be connected (if so desired) to physical or virtual (internal) interfaces, they can be assigned VLANs (with tagging), priorities and all that jazz. It's like the thing has a fully managed switch entirely contained within its ASICs.

This bridge is what we need to get a real bridged connection, so that the actual router can talk PPPoE to the ISP and the modem really is just that: a modem. The modem has 5 ethernet ports; 4 are Fast Ethernet and another Gigabit Ethernet port is unused by default, the idea being that the FE ports are for the LAN and the GigE port is a WAN port for non-DSL deployments.

We will turn that GigE port into the port we use to talk PPPoE to the ISP. The other ports can remain as-is; so we can continue to manage the modem through its usual IP address. The device supports VLAN tagging so it can technically all be done over a single ethernet connection, but I chose to use two connections instead.

First of all, the Thomson modems' web interfaces are so useless it's hard to describe. It can be likened to having to chop down a tree with only a butter knife to use as a tool. So we must use the telnet interface and log in as Administrator.

Before you do anything, you should back-up your current configuration so you can revert to it in case it all goes wrong. Surprisingly, this is something the web interface actually is useful for, offering an easy way to download the configuration to your computer and upload it again should the need arise. The option is under Thomson Gateway -> Configuration.

Second, it's a good idea to delete the current PPPoE settings from the modem. I made the mistake of not doing this, and when I got to the point where the VDSL-interface worked again the modem actually set up its PPPoE connection again and left me wondering why my router wouldn't negotiate an IP address.

Now, if your modem is doing PPPoE, it'll already be set-up to use a virtual internal interface to connect. You will need some information, though, which you may get as follows:

{Administrator}=>:ppp iflist
Internet: dest  ETH_HSIA_vdsl [local disconnect]   [02:20:17]  retry : 10

Deleting the interface is done using the ppp ifdelete command:

:ppp ifdelete intf=Internet

The ETH_HSIA_vdsl interface is a virtual interface that exists only within the modem. It'll likely be assigned to some VLAN:

{Administrator}=>:eth iflist
ETH_HSIA_vdsl   : Dest: bridge
                  Connection State: connected  Retry: 10
                  WAN: Enabled
                  Priority Tagging: Disabled
                  PortNr: 1
                  VLAN: HSIA_vdsl

Indeed, it is assigned to the vlan HSIA_vdsl. This, then, is the vlan we need to add the fifth ethernet port to, because that will make the PPPoE packets appear to the bridge in the same way as they did when the modem still did PPPoE itself. The virtual ethernet interface to the PPPoE process is conceptually the same as having a device that does PPPoE physically connected with an ethernet cable to this vlan, except it is handled entirely within the modem's hard- and software.

We will now add our fifth, unused, ethernet interface to that VLAN. First, it must be added to the bridge. Then we can set a few options on it and lastly it much be attached to the physical interface.

:eth bridge ifadd brname=bridge intf=ethport5 dest=ethif5
:eth bridge ifconfig brname=bridge intf=ethport5 wan=disabled
:eth bridge ifattach brname=bridge intf=ethport5

Then, we can add it to the vlan we want it in, and remove it from the default vlan.

:eth bridge vlan ifdelete name=default intf=ethport5
:eth bridge vlan ifadd name=HSIA_vdsl intf=ethport5
:eth bridge ifconfig brname=bridge intf=ethport5 vlan=HSIA_vdsl

This should be enough to get our 5th ethernet port bridged with the VDSL. Save the config using saveall.

On the router side, you should now be able to connect an interface (typically labeled WAN) to the Thomson's fifth Ethernet port (the red one on the TG789vn) and set it up to use PPPoE. For completeness, here's how I make JunOS do that:

pp0 {
    unit 0 {
        point-to-point;
        ppp-options {
            pap {
                local-name "kpn@direct-adsl";
                local-password "$9$IW/crKvWXbsg"; ## SECRET-DATA
                passive;
            }
        }
        pppoe-options {
            underlying-interface fe-0/0/0.0;
            idle-timeout 0;
            auto-reconnect 5;
            client;
        }
        family inet {
            negotiate-address;
        }
    }
}

Or in commands:

set interfaces pp0 unit 0 point-to-point
set interfaces pp0 unit 0 ppp-options pap local-name "kpn@direct-adsl"
set interfaces pp0 unit 0 ppp-options pap local-password "$9$IW/crKvWXbsg"
set interfaces pp0 unit 0 ppp-options pap passive
set interfaces pp0 unit 0 pppoe-options underlying-interface fe-0/0/0.0
set interfaces pp0 unit 0 pppoe-options idle-timeout 0
set interfaces pp0 unit 0 pppoe-options auto-reconnect 5
set interfaces pp0 unit 0 pppoe-options client
set interfaces pp0 unit 0 family inet negotiate-address

The password, of course, is the incredibly secret word "KPN". I don't think the details actually matter here as KPN figures out who you are by what physical interface you connect to on their end rather than a username/password combo.

If you happen to use the same combo, I found that I also needed to force the TCP Maximum Segment Size to 1452 (the maximum for a 1492 mtu connection), or I wouldn't be able to talk to half the internet:

set security flow tcp-mss all-tcp mss 1452

All of the above was written down well after an entire night's worth of tinkering, distilled down into the only changes that actually matter. In fact, mostly I wrote it down for myself because I couldn't find anything already written down about it and I didn't want to forget. I hope it is of some use to you.

Coinciding closely with Apple's release of Safari 5, touting full-screen playback of HTML5 video, YouTube have disabled their use of the WebKit-API that makes this possible: webkitEnterFullScreen(). They now believe that the browser itself should provide a full-screen button. I do not. But that leaves me with full-browser-window YouTube, or Flash.

Luckily Safari as of version 5 supports extensions! So I made one that turns the Full Screen button on YouTube back into a real Full Screen button. This extension only works on the HTML 5 player (flash does full-screen on its own) and there is one known issue: to prevent stretching 4:3 videos to 16:9 screens, it has to explicitly set the size of the video element. But this size then sticks and I don't think exiting full-screen is detectable, so after using full-screen the button next to it, changing the player size from smaller to larger and vice versa, is broken. If I ever find out how to fix that, I'll update the extension.

Meanwhile; download it here: FullScreenYouTube

Edit: It appears that MacOS X 10.6 or higher is needed to support full-screen on the Mac.

Second edit: I've just uploaded a new version that fixes a few issues, chief among which that YouTube broke it entirely.

Third edit: YouTube changed all the class names in the player. I've updated the extension accordingly.

Fourth edit: I don't know why, but YouTube's scripts starting running before mine, thus already setting up their event listeners before I got the chance to prevent that. I now add another button for the really full-screen option. The original button can be removed or left as is, using the preferences.

Fifth edit: Youtube switched from their white player to a black one, which made the button look shitty with the old designed-for-white icon. I've updated (v 1.4) so it looks more in-place again.

This has been tricky, but not wanting cleartext passwords to go over the wire I figured it out. Mac OS X, as of version 10.5 (Leopard), the clients have been very picky about using LDAP over SSL. If anything is 'wrong' (even though it may not be considered as such by unsuspecting system administrators), the process will fail silently. I've tested this on Snow Leopard (10.6.2) and Leopard Server (10.5) with OpenLDAP.

First, you must use SSL, not TLS with starttls. This means running your LDAP server on port 636 (by default). In OpenLDAP this is achieved by using the -h option to start it, telling it "-h 'ldap:// ldaps://'", for instance. So far this is 'normal', after all not all clients always support TLS.

From here on it gets specific.

Second, the certificate must not be self-signed. It must be signed by a CA (but it can be your own CA if you want). As of Snow Leopard, the CA's certificate must be in the System keychain, or in the System Roots. This means a CA recognised by the OS will work out-of-the-box. In Leopard this is different, as the keychain and OpenSSL don't know about eachother. In this case you must configure the certificate in /etc/openldap/ldap.conf instead.

Third, the reverse DNS must match. This is not normal procedure: normally the certificate is checked according only to the host name you enter (i.e., forward DNS). Again if this is off, it will fail silently. Interestingly, the system will accept simply putting the host in /etc/hosts if this works for you.

I have tried countless variations of this scenario. The only thing that I found would work is what is described above.