Gentoo in a production environment

Image: 

Here is an interesting interview with Ramon van Alteren, team leader of thesystem engineering department at Hyves. that discuses the advantages of using Gentoo on servers, which I really like. Hyves is a European social networking site akin to MySpace, FaceBook or Orkut.

My opinion? I've been a Gentoo Linux fan for many years now. I like it and the fact that you can really learn something from it, unlike Debian, Redhat, or their clones. I useit for both server and desktop machines alike, however, always receivednegative comments on both sides, especially servers. I remember manyarguments at work on why CentOS and Debian are far better... Let me sayfrom personal experience, what a load of bull.

Interview

GMN: Please give us a brief introduction of yourself and your role at Hyves.

Ramon: I'm a long time Gentoo user both personally and professionally andstarted using Gentoo somewhere in 2002 coming from Mandrake. Before that I'dbeen using Linux for about 5 years.

Hyves is a social network site akin to MySpace, FaceBook, Orkut etc. Thewebsite was originally started by Koen, Floris and Raymond, I joined the companyin November 2005 as the first paid sysadmin to help with managing the 40+ Gentoolinux servers.

The System Engineering department currently is a team of 9 people who manage allservers from bare-metal provisioning up to the application software includingthe network infrastructure. The website itself is created by the developers (ateam of 14 people).

GMN: What is the scale of your website?

Ramon: We currently have over 6 million registered users and counting,peak performance is currently at 150+ million page views per day (13 million perhour). These are filtered page views, raw http requests on our web farm are wellover 20.000 req/sec.

On our mysql database backend, we serve roughly 100.000 actions/sec at peakperformance. Aside from the web frontend and the database backend we have a verylarge file-storage for media that our users upload. The total used storagetotals around roughly about 280 Terabytes. All in all with all hosts accountedfor we have 1800 servers most of them with 2 or 4 cores each. All of these runGentoo Linux. :-)

GMN: Why did you choose Gentoo Linux to power your infrastructure? Whatfeatures in particular made you choose Gentoo over all the other distributionsout there?

Ramon: We've been running on Gentoo Linux since the inception of thewebsite. There has been a very short period at the very beginning when thecompany owned two colocated servers which came with Fedora but that was a verylong time ago. Our main reasons to switch to Gentoo were:

• USE flags (the ability to easily switch additional functionality on or off)
• Extreme customizability (custom ebuilds, install locations etc.)
• Close tracking of upstream (speed of updates)
• Good community support
• Excellent documentation

We're using some very bleeding edge software in some cases to run our websitewith fairly extensive customizations in general to adjust for scale and/orperformance. Portage, overlays and the ebuild system in general go a long way toachieve that while staying within the Gentoo distribution framework for all theother stuff.

The fact that a bare-bones Gentoo install doesn't come with all kinds of cruftand distribution-centric customizations, helps to keep our systems mean andlean.

In upgrading from php-4 to php-5 we've found slotting to be an invaluablefeature which eased the migration a great deal. Additionally we had to stay onphp-4 for some time after Gentoo deprecated support for it, pulling in securitypatches with an ebuild turned out to be fairly simple, enabling us to migrate atour own pace.

GMN: Please describe how you used the tools Gentoo offers to makemaintaining such a huge pool of servers easier.

Ramon: Most of our system administration is not done using Gentoo toolsbut by using an automated configuration management system for Unix hosts calledpuppet. It uses eix to build a package database.

Among the tools Gentoo offers:

• We use catalyst to build our own stage3 and stage4 tarballs.
• We use quickinstall by Andrew Gaffney as part of our provisioning framework
• Catalyst relies on genkernel to build our kernel
• We use Portage's binary package support for critical fixes

GMN: Have you been through any major "stumbling blocks" while setting upyour infrastructure? If so, how did you go about resolving them? Did you findthe various support avenues that Gentoo offers to be helpful at the time?

Ramon: Several of course! Most of them had absolutely nothing to do withGentoo but more with bugs and performance issues in various software packages.

With respect to community support, we actively search and query IRC, bugzillaand sometimes the forums as well. Additionally we track the gentoo-dev list forchanges. The forums tend to be a valuable source of information for desktopissues we might have had (most of the system engineering team runs Gentoo asdesktop OS). However for large-scale server issues it has a bit too highnoise/content ratio because of its desktop oriented nature.

We try to give stuff that we learned back to the community by filing bugs, ifpossible with patches. However we find that most of the stuff we customize is offairly limited interest to the larger community because of its focus on our ownenvironment.

Running a server park this large is usually eats away most of the timeavailable, so there is little left to clean up patches and/or ebuilds to makethem more generally usable. We do try to file amd64 keyword stabling reports onstuff we deploy in our server park that hasn't been marked stable (yet).

GMN: Is there a feature that you wish Gentoo had?

Ramon: Let's see [rumbles around desk for the long list of features we'dwish gentoo had]. More seriously, the list is actually surprisingly short:

• Currently I'm working on getting stackless python working in an agreeable fashion with the rest of the python framework in Gentoo. We've rolled out a preliminary stackless ebuild which basically installs it side by side with the normal python setup in /opt but I consider this to be sub-optimal at the least. We haven't finished it yet, but I think it will be doable.
• I'd really really like to see a release-based Portage snapshot released together with the actual release. However, that is not of much use without a snapshot from the distfiles mirrors at the same time. This would make Gentoo less of a moving target and a more reliable base to build servers on. There has been talk about this on several mailing lists where people invariably object that this would create a false sense of stability because no one would port security patches and/or issues but as far as we are concerned that would not be necessary. I understand the constraints placed on the mirror system by such a setup. We are doing this internally for our stage4 files at the moment and that's pretty doable, but then, we only mirror the distfiles part (that we actually use in our stage4 builds) and not the entire Gentoo release with all the related desktop software.
• We depend fairly heavily on the binary package support in Gentoo, and we need to separate packages over multiple ServerOS images (stage4 builds) if we need them with different USE flags. I would like proper USE flag support for binary packages.
• A little less aggressive pruning of ebuilds in the Portage tree would help. A lot of people are probably still running apache-2.0.x versions, as we are. They've left the Portage tree which is kind of sad, I miss them!

GMN: Any concluding comments?

Ramon: The beauty of Gentoo is that it offers a flexible framework forbuilding a highly customizable linux base to run your application on withoutgetting in the way. That flexibility and customizability brings its owncomplexity but as far as I'm concerned it has been worth the trouble.

Running this kind of infrastructure with just 9 people is a form of highlyorganized madness! If there are any developers or sysadmins out there with ahealthy interest in high-performance large-scale infrastructures and AmsterdamI'd like to talk to them. We have open positions in both development and systemengineering and would prefer to hire people from the Gentoo community.

We welcome feedback from the community, you can reach me personally byemail or poke me on IRC(My nick is Innocenti on the Freenode network). If you're interested inworking with us, please don't contact me directly but send mail to:jobs.sysadmin@hyves.nl.

And last but not least I'm really pleased to make the following announcement:

Based on a recent discussion with Robin H. Johnson and Alex Howells from gentoo-infra, Hyves will sponsor the Gentoocommunity by helping out with new servers for Bugzilla. We are putting uptwo large AMD64, 16Gb servers with fast SCSI disks for the database backendand 2 beefy webservers to improve the current bugzilla situation. We'recurrently working with infra to get the stuff up and running and I hopethe servers to go live soon.

GMN: Thank you, Ramon, for your time and for speaking with us!