Goodbye, Forsterstraße. Hello Saltlabs!

After more than 10 years we left our beloved office with its beautiful garden a few weeks ago and moved into “Saltlabs” in the center of Halle – which was created as a workplace for IT companies, featuring a coffee bar, shared infrastructure (kitchen, fast network, meeting rooms with large screens) and will also get a large coworking space next year.

Our new address:

gocept gmbh & co. kg
Leipziger Straße 70
06108 Halle (Saale)
Germany

If you are in Halle, be sure to stop by and have a coffee with us. 🙂

haproxy load-balancing for PHP applications with sticky sessions

We like applications that are written with a shared-nothing approach: it greatly simplifies running multiple instances on multiple hosts and allows for simple, robust load-balancer configuration.

Recently, we had to deploy a PHP application that – in the last minute – turned out to use PHP sessions and thus required sticky sessions.

We haven’t used sticky sessions in a while and the amount of reading required to find the specific working setup was substantial, so we’ll repeat here what a post at networkinghowtos.com already figured out:

backend default appsession PHPSESSID len 64 timeout 3h request-learn prefix

As you can see there isn’t much magic to it – the haproxy manual has a good in-detail explanation of the appsession option. The biggest point of this option is that you do not have haproxy injecting another session identifier but simply piggybacks on the existing one that PHP determines. Also, this option combines nicely with “leastconn” balancing if your application only uses cookies on a few selected pages and many users do not trigger getting a session cookie.

September, 18th–20th: DevOps Sprint

Since we have a strong history in web development, but also were involved in operating web applications we developed, the DevOps movement hit our nerves.

Under the brand name “Flying Circus” we are establishing a platform respecting the DevOps principles.

A large portion of our day-to-day work is dedicated to DevOps related topics. We like to collaborate by sharing ideas and work on tools we all need to make operations and development of web applications a smooth experience. A guiding question: how can we improve the operability of web applications?

A large field of sprintable topics comes to our mind:

Logging

Enable web application developers to integrate logging mechanisms into their apps easily. By using modern tools like Logstash for collecting and analyzing of the data, operators are able to find causes performance or other problems efficiently.

Live-Debugging and Monitoring

Monitoring is a must when operation software. At least for some people (including ourselves), Nagios is not the best fit for DevOps teams.

Deploying

We always wanted to have reproducable automated deployments. Coming from the Zope world, started with zc.buildout, we developed our own deployment tool batou. More recently upcoming projects, such as ansible, and tools (more or less) bound to cloud services like heroku.

Backup

After using bacula for a while, we started to work on backy, which aims to work directly on volume files of virtual machines.

and more…

Join us to work on these things and help to make DevOps better! The sprint will take place at our office, Forsterstraße 29, Halle (Saale), Germany. On September, 20th we will have a great party in the evening.

If you want to attend, please sign up on http://www.meetup.com/DevOps-Sprint/events/191582682/.

Accomodation

For your stay in Halle, we can recommend the following Hotels: “City Hotel am Wasserturm”, “Dorint Hotel Charlottenhof”, “Dormero Hotel Rotes Ross”. For those on budget, there is the youth hostel Halle (http://halle.djh-sachsen-anhalt.de/). Everything is in walking distance from our office.

Reproducable automated deployments on RaspberryPi with batou

For continuous integration during development, we use Jenkins to automatically run tests for all projects we maintain. Some time ago we wanted to increase visibility of the results, so we set up a Raspberry Pi driving a few meters of LPD8806-based LED strip on which we can address single LEDs to represent the status of individual or aggregated builds.

Automating deployments is a good idea…

After an SD Card failure we were painfully remembered how hard it can be to set up a service where all parts were deployed manually. Fortunately we wrote at least some minimal documentation on how to set everything up, so after a few days we were presented with many broken builds. Of course nobody cared about the build status with all LEDs being dark. 🙁

Let’s automate!

Today we wondered if we can use our deployment-tool batou to make reproduceable deployments to a raspberry pi, and did some tests on a vanilla raspbian image (2013-07-26 “Wheezy”).

Preparing your Raspberry Pi

Of course, you can not deploy to it without some simple preparations. First thing is, batou needs to be able to log on the target host with a public ssh key, so we copied our public key to the raspi which has the address 192.168.0.5 in this example:

local> ssh-copy-id pi@192.168.0.5
pi@192.168.0.5's password:
Type password of user pi, default: "raspberry"

(If you don’t have the ssh-copy-id, you have to manually append your ssh public key to /home/pi/.ssh/authorized_keys, which you will need to create on a plain installation)

Manually install minimal requirements

Batou does also have a few requirements which are needed to bootstrap the environment:

mercurial – to pull the buildout which sets up batou
python-virtualenv – to create a clean python environment for the buildout
python-dev – to compile libcrypto against

Note: We are currently working on batou 1.0 which most likely will no longer need any of these.

You can install all the requirements at once with the following command on your raspi:

pi> sudo aptitude install mercurial python-virtualenv python-dev

Prepare batou

Now you are ready to do your first batou deployment to a raspberry pi. For our experiments we created a small hello-world batou deployment, containing a test component which deploys a file /tmp/test which contains “foo” to a raspberry pi specified by an IP address.

To begin, clone the repository on your local machine:

local> hg clone https://bitbucket.org/gocept/batou-on-raspberrypi
local> cd batou-on-raspberrypi

Now, edit environments/pi.cfg and set the IP-address of your Raspi.

To create the nessecary scripts to do the deployment, run buildout to create a sandbox containing all dependencies of batou and the scripts you can use to deploy:

local> python bootstrap.py
…
local> bin/buildout

Deploy!

After some minutes, your batou deployment sandbox will be ready for use. You most likely modified environments/pi.cfg so you need to check in that change first, because batou refuses to deploy a dirty working copy.

local> hg ci -m 'change ip of my raspi'

To run the deployment, call batou-remote with the name of the environment (“pi”, which corresponds to environments/pi.cfg). Because the ssh user you use to connect with the target host differs from your local user, you have to specify it with --ssh-user.

local> bin/batou-remote pi --ssh-user=pi

Batou will now set up itself on the remote side and deploys all components specified in pi.cfg. To show it worked, check if the deployed file contains the correct content:

pi> cat /tmp/test
foo

TL;DR

Create reproducable automated deployments for your software is great fun.
Preparing a raspi to be a target host for batou 0.2.12 based deployments is easy:
- Install python-virtualenv, mercurial and python-dev.
- Put your ssh public key on the raspi.
Example deployment can be found on bitbucket.

#monitoringlove sprint takeaway

A few weeks ago I co-organised and participated in a #monitoringlove sprint in Berlin.

My personal plan was to play with more modern utilities that can potentially replace our existing Nagios monitoring chain. The result of what I think would be a good setup would probably look like this:

Most of those parts already exist. The new thing in there is what I called “riemann-actual” – something that generates new events based on existing events from the index. I call this “higher order” monitoring – in Nagios these would be known as “business processes”.

The word “business processes” is a bit misleading as nothing is really about processes there: it means taking previously taken monitoring data and subsuming it into a more dense expression. Ideally you can recombine any of your metrics to make an overall statement of “everything is good if more than 80% of the appservers are up and we have less than 5% of error response rate and the frontpage is reachable from at least 3 outside systems”.

Data gathering

First, I tried to setup something for data gathering. I already got the recommendation to look at scales for in-app metrics and found it easy to get started. I like the notion that metrics in your app behave a little bit like logging: you don’t care where they go and you expect the user of your system to configure an actual target. The built-in webserver is nice to get started and graphite as a protocol seams fair enough nowadays to forward data.

To gather system-level metrics I guess both collectd and statsd are fine points to start from. I used collectd to begin with as it actually had a riemann output plugin.

Central processing

We want to be able to take all of the data we acquire into account on making decisions quickly. Riemann seems to be the most suitable tool for this task. After playing around for a while trying to implement “business process” monitoring in clojure I found it easier to provide a Python-environment that can talk to riemann and do those decisions. I made this available as “riemann-actual” on bitbucket.

I noticed that this setup would require only a very generic riemann configuration and could perform on a per-customer or per-project basis by just adding more of those loops on top of riemann.

Performance-wise I was extremely happy. I could have a 10Hz monitoring loop resulting in about 1k events per second on my computer. With that resolution all business processes would notice an outage with no visible delay.

A nice feat is that Riemann can generate events when old events reach their TTL. This way you can make sure that you notice when a system you are monitoring “goes dark”.

Also, it seems that Riemann configuration can be unit-tested easily: feed events in, watch the index, or see events coming out. It doesn’t get much simpler than that.

The configurable dashboard in Riemann 0.2 is already very helpful: responsive, flexible, and fast – until you try to display 10k metrics at once. 😉 It needs a little more finishing but it’s on a good way.

Distributed consumers

My understand of Riemann is that it wants to be a nexus for “central, volatile, shared state”. This means you get a lot of updates going through and that it needs to be good with I/O. OTOH it means that it shouldn’t do much and just make it easy for you to router your data somewhere else.

Actually looking at further consumers didn’t happen as 3 days aren’t that long. 🙂 I see graphite on the horizon (with the setup becoming easier over time) as well as more custom tooling to turn events into notifications, etc.

A look at OpenTSDB seemed promising at first but it turns out to have an even more complex setup requirement than graphite. I got it running but it seemed extremly hard to control, so I dropped it after a few hours.

Overall it seems that since the outcry of #monitoringsucks a lot has happened and I’m faithful that there’s a way out of Nagiosland in the near future.

More notes from our sprint are available at pysprints.de. (Although in German.)

August, 15th–17th: Sprinting on Pyramid

After Zope “-the-Framework” reaching the end of its lifecycle during the last few years, we did a bunch of new projects with Pyramid, a nice web framework primarily authored by long-term Zope developer Chris McDonough.

We think it’s about time to give something back to the community, and become more involved in Pyramid development. We therefore happily announce to host a large Pyramid sprint organised in cooperation with the pysprints.de team. You find more information and sprint topics at GitHub.

The sprint starts on Thu, August 15th 10:00h CEST, and ends on Sat, August 17th with a garden party in the evening! Expect BBQ, beer and (most likely) live music!

If you would like to attend, please sign up on lanyrd.

PyCon 2013 report

PyCon 2013 was an excellent conference bringing together Python’s vast, diverse, and technically excellent community. I had the opportunity to visit the whole conference including the sprint days.

Magnitude

The size of the community seems well reflected by the number of attendees that PyCon US attracts: the limit of 2,500 attendees was reached on 2013-02-02, about 1 month prior to the conference. This should be about 500 attendees more than in 2012 when they exceeded their planned capacity of 1,500 ending up with 2,000 IIRC.

It was very nice to see that the organization is growing along with the task: everything ran very smoothly, a lot of detailed changes over last year, some for better, some for worse (Remember: if you want to improve, you need to change, and the means you need to accept set backs to learn.)

Diversity

Yes, there was this FUBAR situation regarding a “code of conduct” violation. I think too many people who have not been at PyCon have contributed to the turmoil already so I’ll refrain from commenting.

I was happy to hear that PyLadies (and everybody else working on the diversity of the community) could see their efforts showing excellent results: around 20% of all participants were women (or girls). I had the impression on the first day of the conference that more women were around than usually on tech conferences.

But not only that, we also had:

a wide range of ages: from kids, to students, to way more senior people
very business and very relaxed, alternative people (Plone RV, anyone?)
visitors from all over the world

I had to ponder a bit why this actually makes me happy: the diversity shows me that what we do is important to everyone and does not need to be either obscure and geeky or shirt-and-tie business.

We can have a community where you can be geeky and nerdy, do business, and feel like a human being. How great is that? Conferences always tend to be very intense environments, somewhat “from outer space”. Combined with travelling overseas for almost two weeks, having a human environment just makes it so worthwhile and a bit more sustainable.

To everybody who did not have an absolutely great experience personally: I’m empathatic and I hope next PyCon will be better for you. A lot has been said about the code of conduct and the organizers definitely pay a lot of attention to it. Nevertheless: 2,500 people stuffed into a few rooms over almost a week will cause friction here and there. If I should encounter a similar situation myself I will hopefully be able to apply some of the experience as a bystander and: stay calm, be friendly, and help defusing situations.

Technical excellence

There isn’t much you can do to get more sophisticated technical people talking about programming into the same spot compared to PyCon. Maybe DEFCON, or USENIX, or other more orthogonally oriented spots. But for practicality this is just it.

I recommend you visit pyvideo.org and go through the recorded videos of all sessions. It’s always a good idea to listen to what Raymond Hettinger has to say. And Guido, of course.

Sprinting

I felt very productive during the sprints: I started out sitting in a room with Nate Aune, Jeff Forcier, and some others, talking about deployment things. I worked a bit on our deployment utility batou trying to soften some rough edges and gather feedback from others.

However, I also had the PyPI mirror client software on my radar. As we are operating one of the official mirrors (the F mirror) I was fed up by the constant breakage that the existing pep381client experienced everywhere. I sat down, refactored, and lo and behold! a bandersnatch appeared. This is a full rewrite that can be used with the existing mirror data and is much more reliable and – in the case of error – easier debug and recover.

Sponsoring

gocept also was a silver sponsor for PyCon. We already sponsored PyCon in 2012, but this year we:

did not insert more stuff into the attendee bag (it’s way too heavy already anyways)
did set up a booth to become approachable and get to talk to people

Our product (the Flying Circus) is in use for consulting clients but still on its way to become a product that you can just use by registering and providing payment details. Operations as a service is a very dynamic space today and we had some good opportunities to try to explain what we envision and where we think existing IaaS and PaaS models are aiming at the wrong thing. If you’re interested in this kind of thing, then visit our homepage and sign up to our newsletter and we’ll keep you updated.

PyCon has been a very sponsor-friendly place, especially for small businesses. It’s always a hassle to bring a lot of stuff half around the globe, but the environment was perfect to just bring some banner and flyers and talk to people strolling around.

2014

So next year, PyCon US will actually be PyCon North America, as the conferences moves to Montreal. Besides making this a much shorter trip I’m also looking forward to some new cultural impressions.

How we organize large-scale roll-outs

In the coming week we will deploy an extensive OS update to our production environment which (right now) currently consists of 41 physical hosts running 195 virtual machines.

Updates like this are prepared very carefully in many small steps using our development and staging setups that reflect the exactly same environment as our production systems in the data center.

Nevertheless, we learned to expect the unexpected when deploying to our production environment. This is why we established the one/few/many paradigm for large updates. The remainder of this post talks about our scheduling mechanism to determine which machines are updated at what point in time.

Automated maintenance scheduling

The Flying Circus configuration management database (CMDB) keeps track of times that are acceptable to each customer for scheduled maintenance. When a machine determines that a particular automated activity will be disruptive (e.g. because it makes the system temporarily unstable or reboots) then it requests a maintenance period from the CMDB based on the customers’ preferences and the estimated duration of the downtime Customers are then automatically notified what will happen at what time.

This alone is too little to make a large update that affects all machines (and thus all customers) but it’s the mechanical foundation of the next step.

Maintenance weeks

When we roll out a large update we rather add additional padding for errors and thus we invented the “maintenance week”. For this we can ask the CMDB to proactively schedule relatively large maintenance windows for all machines in a given pattern.

Here’s a short version of how this schedule is built when an administrator pushes the “Schedule maintenance week” button in our CMDB (all times in UTC):

Monday 09:00 – automation management, monitoring, and binary package compilation get updated
Monday 13:00 – the first router and one storage server are updated
Monday 17:00 – internal test machines (our litmus machines) and a small but representative set of customer machines that are marked as test environments get updated
Tuesday 17:00 – the remainder of customer test machines, up to 5% of untested production VMs, and 20% of the storage servers are updated
Wednesday 17:00 – 30% of the production VMs get updated and 30% of the storage servers are updated
Thursday 17:00 – the remaining production VMs and storage servers get updated
Saturday 09:00 – KVM hosts are updated and rebooted
Saturday 13:00 – the second router is updated

Once the schedule has been established, customers are informed by email about the assigned slots. An internal cross-check ensures that all machines in the affected location do have a window assigned for this week.

This procedure causes the number of machines that get updated rise from Monday (22 machines) to Thursday (about 100 machines). Any problems we find on Monday we can fix on a small number of machines and provide a bugfix to avoid the issue on later days completely.

However, if you read the list carefully you are probably asking yourself: Why are customer VMs without tests updated early? Doesn’t this force customers without tests to experience outages more heavily?

Yes. And in our opinion this is a good thing: First, in earlier phases we have smaller numbers of machines to deal with. Any breakage that occurs on Monday or Tuesday can be dealt with more timely than if unexpected breakage occurs on Wednesday or Thursday where many machines are updated at onces. Second, if your service is critical then you should feel the pain of not having tests (similar to pain that you experience if you don’t write unit tests and try to refactor). We believe that “herd immunity” will give you a false sense of security and rather have unexpected errors occur early and clearly visible so they can be approached with a good fix instead of hiding them as long as possible.

We’re looking forward to our updates next week. Obviously we’re preparing for unexpected surprises, but what will they have in stock for us this time?

We also appreciate feedback: How do you prepare updates for many machines? Is there anything we’re missing? Anything that sounds like a good idea to you? Let us know – leave a comment!

developer & admin BBQ IV

Zum vierten Mal wird am 30. April 2013 um 14:00 Uhr das „developer & admin BBQ“ stattfinden. Die Veranstaltung bietet Software-Entwicklern und Administratoren ein Forum um Ideen, Probleme und deren Lösungen auszutauschen. In einem an „Open Space“ angelehnten Format hat jeder Teilnehmer die Möglichkeit eigene Themen einzubringen, die dann in kleineren Runden bearbeitet werden können.

In der Vergangenheit wurden sowohl konkrete technische Problemstellungen, wie zum Beispiel „Plattformübergreifende Entwicklung für Mobilgeräte“ oder „Pymp your (vim|emacs) – sinnvolle Editor-Erweiterungen für Python-Entwicklung“ thematisiert. Aber auch theoretische Themen rund um agile Entwicklungsprozesse („Test Driven Development“) oder Anwendungsbetrieb („Deploying applications and the 12-factor app“) wurden behandelt.

Wie bereits für die letzten Veranstaltungen gibt es auch dieses Mal eine Reihe von Themenvorschlägen:

Raspberry Pi – Möglichkeiten, Grenzen, Alternativen?
Übung macht den Meister: Code Katas.
Der „CMS-Zoo“ – Auf der Suche nach einem vernünftigen CMS.
Erst den Test, dann den Code – Test Driven Development
Ceph – performantes, stabiles und skalierbares verteiltes Dateisystem im Produktiveinsatz

Ab ca. 19 Uhr gibt es beim gemeinsamen Abendessen (je nach Wetterlage auch am Lagerfeuer mit Grill), die Möglichkeit sich in angenehmer Atmosphäre weiter auszutauschen und kennen zu lernen.

Wie bisher findet das BBQ wieder bei der Firma gocept gmbh & co. kg, Forsterstraße 29 in Halle statt.

Anmeldungen zur Teilnahme sowie Vorschläge für weitere Themen können hier eingetragen werden. Weiterhin ist die Veranstaltung bei meetup gelistet. Auch der Facebook-Event darf gerne zur Anmeldung benutzt werden.

P.S.: Since we are addressing local audience we are keeping this post in German. Basically we want developers and admins in our area to meet up, exchange ideas, and enjoy BBQ.

Happy new year – cleaning up the server room!

Welcome to 2013!

Alex and I are using this time of the year when most of our colleagues are still on holidays to perform maintenance on our office infrastructure.

To prepare for all the goodness we have planned for the Flying Circus in 2013 we decided to upgrade our internet connectivity (switching from painful consumer-grade DSL/SDSL connections to fibre, yeah!) and also clean up our act in our small private server room. For that we decided to buy a better UPS and PDUs, a new rack to get some space between the servers and clean up the wiring.

Yesterday we prepared the parts we can do ourselves in preparation of the electricians coming in on Friday to start installing that nice Eaton 9355 8kVA UPS.

So, while the office was almost empty two of us managed to use our experience with the data center setups we do to turn a single rack (pictures of which we’re too ashamed to post) into this:

Although the office was almost abandoned, those servers to serve a real purpose and we had to be careful to avoid too massive interruptions as they do handle:

our phone system and office door bell
secondary DNS for our infrastructure and customer domains
chat and support systems
monitoring with business-critical alerting

Here’s how we did it:

Power down all the components that are not production-related and move them from the existing rack (right one on the front picture) to the new one. For that we already had our rack logically split between “infrastructure development” and “office business” machines.
Move the development components (1 switch, 7 servers, 1 UPS) to the new rack. Wire everything again (nicely!) and power it up. Use the power-up cycle to verify that IPMI remote control works. Also notice which machines don’t boot cleanly (which we only found on machines that are under development regarding kernels anyway, yay).
Notice that the old UPS isn’t able to actually run all those servers’ load and keep one turned off until we get the new UPS installed.
Now that we had space in the existing rack we re-distributed the servers there as well to make the arrangement more logical (routers, switches, and other telco-related stuff at the top). Turn off single servers one-by-one and keep everyone in the office informed about short outages.
Install new PDUs in the space we got after removing superfluous cables. Get lots of scratches while taking stuff out and in.
Update our inventory databases, take pictures, write blog post. 🙂

As the existing setup was quite old and grew over time we were pretty happy to be able to apply the lessons we learned in those years in between and get everything cleaned up in less than 10 hours. We notice the following things that we did differently this time (and have been doing so in the data center for a while already):

Create bundles of network cables for one server (we use 4) and put them in a systematic pattern into the switch, label them once with the servername at each end. Colors indicate VLAN.
Use real PDUs both for IEC and Schuko equipment. Avoid consumer-grade power-distribution.
Leave a rack unit between each component to allow operating without hurting yourself, the flexibility to pass wires (KVM) to the front, and to avoid temperature peaks within the rack.
Having over-capacity makes it easier to keep things clean which in turn makes you more flexible and brings ease to your mind to focus on the important stuff.

As the pictures indicate we aren’t completely done installing all the shiny new things, so here’s what’s left for the next days and weeks:

Wait for the electricians and Eaton to install and activate our new UPS.
Wire up the new PDUs with the new UPS and clean up the power wiring for each server.
Wait for the telco to finish digging and putting fibre into the street and get their equipment installed so we can enjoy a “real” internet connection.

All in all, we had a very productive and happy first working day in 2013. If this pace keeps up then we should encounter the singularity sometime in April.

Author: Daniel Havlik