Just thought I'd post this here to keep folks informed.
The short version:
Some Chinese dude downloading files in a peculiar way tickled a bug in some software on our webserver, so to avoid that, we've made some changes.
The long version:
Over the past couple of weeks, I've seen a series of Chinese IP addresses downloading specific files on nesdev.com (mainly large ZIP files). Nothing surprising there, but there was an inadvertent "trickle-down" effect as a result of such.
The client software they're using is some sort of multipart downloader (and it spoofs its User-Agent, so I have no idea what it's actually called). Multipart downloaders download only certain sections of the file, then put it all together once all the sections are done. The software is configured for something like 8 concurrent HTTP connections at once. 31457280 / 8 = 3932160 bytes per HTTP connection.
There's really nothing "wrong" with this, and lots of people use them, mainly to get around bandwidth limit rules imposed by their ISP (rather than just rate-limit the port entirely, the ISP chooses to rate-limit each individual TCP connection; by using multiples at once, you can essentially get around the bandwidth limit. I know Telia (Sweden) uses such things...).
Anyway, the above situation was causing an Apache module we use (called mod_cband) for limiting individual websites' speed (per second) to break horribly, resulting in all the other sites locking up until some of the TCP sockets from the multipart downloader had closed. I know, it sounds like a system or Apache tuning problem, but trust me, it isn't: disable mod_cband and the problem goes away entirely. Hmm... ;)
Then I found a couple Debian PRs documenting similar problems with the speed limiting function of mod_cband. We don't use Linux, but the issue isn't Linux-specific. That, combined with the fact that the only home page for mod_cband has been gone/offline for months, caused the Debian folks to pull the software from Debian's stable and unstable branches entirely:
http://bugs.debian.org/375016
http://bugs.debian.org/418645
That pretty much convinced me right there. I took the liberty of saying "fuck this", removed mod_cband entirely, and resorted to using kernel-level bandwidth limiting features (for those who care, pf ALTQ).
This required me to give nesdev.com it's own IP address, rather than use name-based virtualhosts like I've been doing for many years. The IP of the site has moved from 72.20.106.5 to 72.20.106.40, in case you notice DNS changes. I also updated the DNS records for nesdev.com and www.nesdev.com to point to the new IP.
Since DNS takes time to propagate, I've left support in our webserver for answering nesdev.com on both the old and new IP. In a few days, I'll be adjusting Apache to no longer respond for nesdev.com on the .5 address. (Our DNS TTLs are fairly short, so a few days is quite generous)
That's all. :-)
Oh, and one other thing: yes, the bandwidth limit is the same as it was before (384kbit/sec). :P
The short version:
Some Chinese dude downloading files in a peculiar way tickled a bug in some software on our webserver, so to avoid that, we've made some changes.
The long version:
Over the past couple of weeks, I've seen a series of Chinese IP addresses downloading specific files on nesdev.com (mainly large ZIP files). Nothing surprising there, but there was an inadvertent "trickle-down" effect as a result of such.
The client software they're using is some sort of multipart downloader (and it spoofs its User-Agent, so I have no idea what it's actually called). Multipart downloaders download only certain sections of the file, then put it all together once all the sections are done. The software is configured for something like 8 concurrent HTTP connections at once. 31457280 / 8 = 3932160 bytes per HTTP connection.
There's really nothing "wrong" with this, and lots of people use them, mainly to get around bandwidth limit rules imposed by their ISP (rather than just rate-limit the port entirely, the ISP chooses to rate-limit each individual TCP connection; by using multiples at once, you can essentially get around the bandwidth limit. I know Telia (Sweden) uses such things...).
Anyway, the above situation was causing an Apache module we use (called mod_cband) for limiting individual websites' speed (per second) to break horribly, resulting in all the other sites locking up until some of the TCP sockets from the multipart downloader had closed. I know, it sounds like a system or Apache tuning problem, but trust me, it isn't: disable mod_cband and the problem goes away entirely. Hmm... ;)
Then I found a couple Debian PRs documenting similar problems with the speed limiting function of mod_cband. We don't use Linux, but the issue isn't Linux-specific. That, combined with the fact that the only home page for mod_cband has been gone/offline for months, caused the Debian folks to pull the software from Debian's stable and unstable branches entirely:
http://bugs.debian.org/375016
http://bugs.debian.org/418645
That pretty much convinced me right there. I took the liberty of saying "fuck this", removed mod_cband entirely, and resorted to using kernel-level bandwidth limiting features (for those who care, pf ALTQ).
This required me to give nesdev.com it's own IP address, rather than use name-based virtualhosts like I've been doing for many years. The IP of the site has moved from 72.20.106.5 to 72.20.106.40, in case you notice DNS changes. I also updated the DNS records for nesdev.com and www.nesdev.com to point to the new IP.
Since DNS takes time to propagate, I've left support in our webserver for answering nesdev.com on both the old and new IP. In a few days, I'll be adjusting Apache to no longer respond for nesdev.com on the .5 address. (Our DNS TTLs are fairly short, so a few days is quite generous)
That's all. :-)
Oh, and one other thing: yes, the bandwidth limit is the same as it was before (384kbit/sec). :P
