Wed, 05 May 2010 (1759 EST)
So, I had an issue with my poor overworked web servers. Average traffic would use at least 6 gigs of RAM, with each Apache prefork child using an average of 16 megs of ram. So during heavy traffic days, one or more of the web servers would actually start to swap unless I put a less then desirable cap on max connections. I'd heard for years that Apache Worker was the new hotness, but PHP wasn't considered stable enough to rely on in a threaded environment. There's been sporadic reports of success with "only using a limited set of known stable modules", but not a whole lot of actual Howtos beyond using Fastcgi.

Regarding Fastcgi, I did get it to work to some extend, but had a few issues with race conditions around how fcgi and apache itself reaps and respawns child threads, along with having to restructure a signficant amount of our code to be compatible with fcgi's method of doing things. One such example is we were no longer able to use php ini overrides in htaccess files (because fcgi doesn't get called for htaccess). In the end, it's workable, but a giant pain in the ass, and one more piece of duct tape to have to troubleshoot. It's always nice to have another option in reserve though.

Regarding resource usage, the jury's still out on how much less processor. Although I do know that non php stuff is MUCH faster under worker. But for RAM, which was the main incentive for making this change (more RAM, or more web servers), under load that would previously consume about 6 gigs, I'm currently at 1.6 gigs of usage. Yay!

Anyway, here's a quick rundown on what I had to do to get mod_php working with Apache's Worker MPM. I may write this up in a prettier fashion later, probably after I do it a couple more times.

1.) Grab PHP sources
- mkdir ~/build/;cd ~/build
- wget
- tar xjvf php-5.2.13.tar.bz2

2.) Grab Suhosin patch (prevents some nasty buffer overruns etc)
- wget
- cd php-5.2.13/
- patch -p 1 -i ../suhosin-patch-5.2.13-0.9.7.patch

3.) Compile PHP (modify options as appropriate, make sure --enable-maintainer-zts is in your configure for worker)
-- Note that you may need to install various otehr utilities and libraries for the compile to work properly, depending on extensions used.
-- One thing that may help get build dependencies on Debian is 'apt-get build-dep php5', which should download all the packages needed to build the debian php source package.

- ./configure --prefix=/usr/local --with-apxs2=/usr/bin/apxs2 --disable-cgi --with-layout=GNU --with-config-file-path=/etc/php5/apache2 --with-config-file-scan-dir=/etc/php5/apache2/conf.d --disable-ipv6 --with-openssl --without-kerberos --with-pcre-regex=/usr --with-zlib --with-zlib-dir=/usr --enable-bcmath --with-bz2 --enable-calendar --enable-ctype --with-curl=shared,/usr --without-qdbm --without-gdbm --with-db4 --with-libxml-dir=/usr --enable-exif --disable-ftp --with-gd=shared,/usr --enable-gd-native-ttf --with-gmp=shared,/usr --with-jpeg-dir=shared,/usr --with-xpm-dir=shared,/usr/X11R6 --with-png-dir=shared,/usr --with-freetype-dir=shared,/usr --with-gettext --with-mhash=shared,/usr --with-ldap=shared,/usr --with-ldap-sasl=/usr --with-mcrypt=shared,/usr --enable-mbstring --without-msql --without-mssql --with-mysql=shared,/usr --disable-pdo --with-pspell=shared,/usr --without-mm --disable-shmop --enable-soap --enable-sockets --with-sqlite=shared,/usr --with-regex=php --disable-sysvshm --disable-wddx --with-xmlrpc=shared --with-iconv --with-xsl=shared,/usr --enable-zip --with-pear=/usr/share/php --with-tsrm-pthreads --enable-maintainer-zts

- make
- make test
- apt-get install checkinstall (checkinstall will monitor a 'make install' type of command and generate a .deb from it, which will let you satisfy dependencies)
- checkinstall -D make install (It will give you a chance to edit the control variables, I suggest setting "provides" to 'php5, phpapi-20060613, php5-common', and the "version" to something like 5.2.13.threaded for packages that require a specific version of php)

-- I had some issues with packages depending on php5-common, but not recognizing the above "provides" line. To fix that do the following:
- apt-get install equivs
- equivs-control php5-common
- Edit the generated php5-common file in your current directory, specifically to set the needed version and provides lines.
- equivs-build php5-common
- dpkg -i php5-common_*.deb

4.) Make sure debian links your php binaries in the normal places (assumes no other php alternatives are there, and there may be a better way to "group" these with update-alternatives):

- update-alternatives --install "/usr/bin/php" "php" "/usr/local/bin/php" 1
- update-alternatives --install "/usr/bin/php5" "php5" "/usr/local/bin/php" 1
- update-alternatives --install "/usr/bin/pear" "pear" "/usr/local/bin/pear" 1
- update-alternatives --install "/usr/bin/pecl" "pecl" "/usr/local/bin/pecl" 1

4.) Grab and compile non standard php modules (specifically memcache and apc)
- pecl install memcache
- pecl install apc

5.) Install apache worker
- apt-get install apache2-mpm-worker
- If it yells at you about missing dependencies that are LIES because you compiled php and it isn't recognizing your stuff, then you can probably trick it with 'apt-get --ignore-missing install apache2-mpm-worker'

6.) Check that php is working
- Run the php command line by just typing "php". It should hang waiting for input, if things are wrong, it will die with a segmentation fault
- Make sure all your php modules are loaded with 'php -m'
- If you are segfaulting, you can usually track down the issue with 'strace php', which will show you all the system calls (like OPEN(, ...)) that php tried to run. The last one will usually be the module that crashed php, and you can try unloading it. You will probably need to be able to compile ALL php modules you need for them to be able to talk to threaded php (zts).
Wed, 23 Dec 2009 (1650 EST)
We recently were having an issue where one of the php servers would hit a CPU bottleneck due to a web crawler sitting on a KeepAlive connection and churning through several thousand pages. This was discovered by looking at the highest CPU children of Apache and noticing they were almost universally Yahoo or Google crawlers. Below is a perl subroutine that I whipped up for another monitoring script that will pull CPU and PID info from the output of a 'top -bn1' invocation, and will then renice any non root apache processes using more then 20% of a CPU.
sub renice_apache
        my @top = `/usr/bin/top -bn1`;
        my %header;
        my $command = 'apache2'; # Name of apache process

        my @renice;
        my @unnice;

        foreach (@top)
                my @line = split(/\s+/, $_);

                if(defined $line[8] && exists $header{'%cpu'} && $line[0] =~ /^\d+$/)
                        next unless (defined $line[$header{command}] && ($line[$header{user}] !~ /^\s*root\s*$/i) && ($line[$header{command}] =~ /^\s*\Q$command\E\s*$/i));

                        next unless ($line[$header{'%cpu'}] =~ /^\s*(\d+)/);

                        push(@renice, $line[$header{pid}]) if (($line[$header{ni}] < 5) && $1 > 20); # More then 20% cpu on one process
                        push(@unnice, $line[$header{pid}]) if (($line[$header{ni}] > 5) && $1 < 1); # Raise priority on children no longer under load
                } elsif (!exists $header{'%cpu'})
                        my %tempheader;
                        for (0 .. $#line)
                                $tempheader{lc($line[$_])} = $_;

                        if(defined $tempheader{pid} && defined $tempheader{'%cpu'} && defined $tempheader{command})
                                %header = %tempheader;

        system("/usr/bin/renice 10 " . join(' ', @renice)) if ($#renice > -1);
        system("/usr/bin/renice 3 " . join(' ', @unnice)) if ($#unnice > -1);
So, we have these Dell PowerEdge servers at work. Great machines, and they all come with these Broadcom Network cards that use a driver called 'bnx2' under the Linux kernel. As some things do, this requires special firmware to be made available to the network driver in order to work correctly. As you might imagine, this can become very annoying when you're trying to do a network install and your network card is greedily asking for files that aren't on the CD.

So, over the course of running into this same issue for almost 2 years now, I've tried a few things to streamline the process:
  • Grab the latest Weekly/Testing Installer - Some will ask to load firmware off USB, some won't. None have done it right as of yet.
  • Grab special backported images - No luck at first, but the latest version has the files available. Unfortunately they were in the wrong directory, which led me to the solution below.
  • Using the first full Debian CD to do a network-less install, and then loading firmware-bnx2 on the finished install, followed by manual network config.
  • And I don't remember, I've tried so many different "recommended" ways of loading firmware into the installer that I've forgotten several.

Anyway, here's the easiest way I've found:
  1. Grab a Network Install image. Real men use the Testing installer because it's pretty obvious if you have a bad image.
  2. Get one of those fancy USB memory sticks. Or if you're cheap like me, find a coworker with one and "borrow" it.
  3. Extract the firmware from the bnx2 firmware package (.fw files) into a firmware/ directory on the USB key. Or use mine here.
  4. Run through the network install until you are just BEFORE the "detect network" steps.
  5. Hit alt-F2 to switch to a console, and plug the USB key in.
  6. The kernel should auto-mount the key, and tell you the partition in dmesg: dmesg | tail -30 should show you something like "mounted on sdb1"
  7. Mount the partition and copy the firmware files to where the kernel needs them: mount -t vfat /dev/sdb1 /mnt;cp -a /mnt/firmware/ /lib/firmware/;umount /mnt
  8. Hit Alt-F1 to switch back to the Installer, and run the Network detect. If it fails to detect, switch back to the F2 console and look at dmesg. It is likely looking for a different version of one of the firmware files. You can usually get away with renaming another file with the same version to the one it is looking for. The worst that happens is it still won't load the firmware (well, for the bnx2 stuff, I don't know if other firmware will cause your dog to explode for different hardware).
  9. Once it tries to detect the card and fails to load the firmware, it won't look for new firmware unless you unload the module first. To force it to reload the driver, remove it in the F2 console with: modprobe -r bnx2
  10. With the firmware there, the network detection should go smoothly, and the Installer will be on its merry way. Once its finished and you're able to run apt in your shiny new shell, install the firmware-bnx2 package, or whatever firmware package you may have used this for if they exist. This should hopefully help any debian kernels keep things in place as you upgrade.

I hope this saves someone a few hours of frustration. At the very least, I have some notes I can look at when this is still happening a year from now.
Fri, 18 Jul 2008 (1052 EST)
After a day of trying to figure out why one of my web servers was locking up, I found that it was using a bit too much memory. But I had no idea how much, and Linux memory reporting is a bit arcane at best. Especially with something like apache + PHP using shared memory pools. So after some analysis, I came up with the following script:

It had the following output:

Total memory available: 3.21G
Total used by apache2 (451 instances): 3.80G
Total used by other processes: 0.12G

Average memory used per apache2 process: 8.63M
Recommended number of processes based on Average: 381
Needed memory for 500 processes based on Average: 4.21G

Max memory used for apache2 process: 17.61M
Recommended number of processes based on Max: 186
Needed memory for 500 processes based on Max: 8.60G

Mean plus two Standard Deviations (bulk of usage under max): 11.28M
Recommended number of processes based on Mean + 2*Stdev: 291
Needed memory for 500 processes based on Mean + 2*Stdev: 5.51G
At the time I ran it, I had 450 instances out of a configured max of 700 active. Come to find out I only had the ram to support about 380 of those at best. :(

Hopefully some other people find this useful, as it's pretty hard to get a straight answer on "How many processes can I support?" Now if only php and the standard modules would officially work with the worker MPM.
Wed, 30 Apr 2008 (1450 EST)
So, for the web servers at work, we had disabled Apache's KeepAlive settings due to possible denial of service. As part of some performance tuning I had turned it back on with a low timeout (4 seconds) to see how it held up. Here's the results:

Concurrent Connections

The drop on the right is where KeepAlive was turned on. We did need to increase our MaxServers a bit for when load got high, but overall it's resulted in a huge speedup for the servers and less thrashing in general.
Wed, 20 Dec 2006 (1517 EST)
Xkcd is amazing. Best webcomic ever.
Tue, 28 Nov 2006 (1801 EST)
So, I'm poking around some programming related documentation and come across an article about using Perl for statistics.

I start reading it, and almost immediately, Pandora started playing a song by a band called Statistics.

Tue, 03 Oct 2006 (1534 EST)
I snagged a bag of chocolate covered Espresso beans fro Starbucks today. The serving size is 28 beans. I think I'm going to eat them all and see if I can either explode, or start vibrating so fast I fall through the floor.
Sat, 22 Jul 2006 (0216 EST)
Click this link. (nothing bad will happen)
Fri, 16 Dec 2005 (2308 EST)
Just came across this rather old news article concerning IGE CEO Brock Pierce.

I thought promoting Chinese gold farming sweatshops and being a festering boil on the side of the MMO gaming community was bad enough(Not to mention advertisements on the back cover of every major gaming magazines for over a year now). Turns out he was (ALLEGEDLY) a pedophile as well.

From Vnunet:

The founders of flopcom Digital Entertainment Network (DEN) are still languishing in a Spanish jail, and investigators in the US are stepping up efforts to bring them home to face sex offence charges.

DEN co-founders Marc Collins-Rector, Chad Shackley and Brock Pierce were arrested in June on an international warrant after being indicted in New Jersey on five counts of transporting a minor across state lines for the purpose of engaging in sexual acts.

The company raised $75m through an Initial Public Offer, but collapsed as allegations of its founders' conduct hit the media. DEN's assets were sold at auction for $105,000 this summer.

The men fled to Spain where they were arrested by Spanish police who found "enormous amounts of child porn" at their villa.

All three are likely to face Spanish charges, but US police have stated that Spain will let the US charge them first, according to the New York Post.

Collins-Rector, Shackley and Pierce already face a $4.5m default judgement over offences with teenaged boys in the Beverly Hills and west Los Angeles area dating back to the early 1990s, after losing a civil court case brought by victims.

Tue, 19 Jul 2005 (1341 EST)
I probably should have mentioned it when it launched last week, but The Escapist launched last Tuesday. It's a magazine put out by the people I work with that deals with the Gaming Industry via a different topic each week. There's a new issue every Tuesday, with a supplement every Friday.

I personally was never much for writing, so mainly I just babysit and maintain the server and software behind the site. Note to self: Apache does NOT like being restarted when there's a few thousand connections to the web server. Trying to rotate logs to address an internal issue caused a chain of events simultaneously with a double Slashdotting. But the actual traffic it handled well once it was back up. *whistles innocently*

This week deals with mobile gaming. Check it out.
Wed, 01 Jun 2005 (0036 EST)
This marks the day when I finally got off my ass and finished moving stuff to the new server up in the Canadian datacenter. Same Host, just on an MCI net and at a slightyl more convienient location for them. Also, more RAM! Woo! Now just to compile a shiny new kernel...
Mon, 09 May 2005 (1324 EST)
Heading out to E3 in LA for my first time. Not sure exactly what I'll be doing yet, but should be an interesting experience. Especially since I'm going as Press. Free Loot!
Sun, 20 Mar 2005 (0912 EST)
This is what you get when you're trapped in an office with a Canadian and a Mac user for too long.
End of days.
Thu, 17 Mar 2005 (1239 EST)
Mal'Ganis more like Mal'GOONis am i rite
Wow. Just wow.

A co-worker brought in a compilation of the Transmetropolitan comic series. It's not exactly new, but I never really read comics seriously, and never any of the newer more serious ones. Just old comics like Fantastic Four, Green Lantern, Ghost Rider, etc. Not only is it gloriously violent, and in a future worthy of Gibson, but the story just sucks you in.

For the uninitiated, it centers in the fairly distant future around a Journalist named Spider Jerusalem. He's antisocial, xenophobic, hates dogs, and has probably done every drug he could get his hands on. He gets pulled back into the "City" due to a prior overdue book-deal contract made when years before the main storyline. It centers around him doing investigative journalism on Police corruption, corrupt Presidents, and just going around and pissing off people in general.

The real fun comes in the world this all takes place in, with cameras everywhere (microscopic even), hyper violent kid's shows (Sex Puppets being a popular one as well), new religions being founded hourly, people genetically blending themselves with "Grey" style aliens, machines getting high on drug attachments, park benches that turn red hot or release deadly virii between 10pm and 6am, and more. Oh yeah, the art is really well done as well, there's quite a few extremely detailed panorama shots.

Now I need to go and pick up all the issues for myself to read it again.
Fri, 11 Feb 2005 (1054 EST)

This guy could be an Olympic-calibre Gymnast if he wanted I'm sure.
Fri, 11 Feb 2005 (0038 EST)
This is the new Truffle Shuffle.

I need to find a way to make this my screensaver.
Sat, 22 Jan 2005 (0425 EST)
Bunny Suicides!

I especially enjoy the Noah's Ark and Terminator ones.
Wed, 08 Dec 2004 (2103 EST)
I got 13 santas, hows about you?
All content/ramblings/etc here are copyright (c) 2004 Erik Jacobson (except for the stuff that isn't). Not to be read before swimming, or taken with alchohol. Anything bad is your fault, anything good is mine.