Life in denial (of service)
In late October, I started experiencing difficulty hitting my own web site. I would see odd errors from the browser (Document contains no data) and every now and again, one of my anxious clicks on the refresh button would yield a page. Email accounts were inaccessible, even SSH was refusing me most of the time.
I thought it was just another case of a cheap home router dying before its time. And I posted to that effect. So I swapped in my backup router, a known-good device, albeit of the same, home user variety.
That appeared to work for about an evening. But it locked up completely several hours after I brought it online. A reset restored connectivity, but it locked up again. I went back to my production router. At least that gave me the occasional connectivity.
I went out and got another router. This time, I figured, I’d get the Apple Airport base station. They’re expensive (almost twice the price of the Linksys or Netgear ones I’d decided were irretrievably unreliable) so I thought maybe that’s because they actually do quality control.
I set up the router and within a few minutes, it had crawled to a halt, unable, even to broadcast news of its existence to the Airport admin utility. I put my production router back onto my DSL line and hooked up the computers to it once more. Same deal. It was time to look for another explanation.
I decided to look at the router’s (quite limited) logging and statistical functions to see if I could get any clues. Netgear’s logging is limited to telling you if little Billy has been trying to surf to bignarstypornsite.com, but nothing else. It does, however, tell you about the overall flow of data.
I was getting one point five million bytes per second. That’s about a meg and a half every seconds. In the words of one of my networking genius acquaintances, that’s more than several DSLs can handle.
I phoned Magma, my upstream provider. They couldn’t see anything wrong. They said if I suspected a DOS, then I needed to send them the logs to show who was doing what. “You’re funny,” I said.
My net genius acquaintances told me there’s probably not much Magma could do anyway and that the real solution was to use a “real system” for my router and firewall.
So on Thursday, I took a deep breath, put a second network card in a disused PC and installed an open source linux-based firewall package on it. Magma was very helpful in helping me diagnose my initial connection problems, and by 1 am Friday, it was running.
The firewall machine immediately shot up to a load rate of 4.5 (anything over 1.0 means it’s getting more than it can handle). But it kept on plugging, and while my services were still unreachable for the most part, I learned I was being flooded with a series of DNS requests from tens of thousands of different IP addresses, organized in groups of subnets.
I played with the settings in iptables, adjusted a few things but still the bombardment continued. Magma called. They wanted to sell me a Cisco PIX 501 or and/or sell me their managed firewall service. But the sales guy really couldn’t give me a serious answer about the prospects for success if I bought their $700 box. Indeed he sounded really information-light. I told him I’d see what was happening on Monday and make a decision.
So again I consult my net genius acquaintances and one said, most wisely I think, “This isn’t a hardware/software issue. The issue is knowing what to
filter and what not to.” He said I’d be better advised to pay someone to configure my existing gadgets than to buy another gadget or learn to do it myself.
Then suddenly, miraculously and possibly temporarily, it stopped. My firewall’s load levels are down to 0.15 now. Everything is working just dandy.
I appreciate the relief, though I know it may be temporary. But I would still love to know why my server was targeted and why it suddenly stopped. I host some political sites. My own posts are often about things political. Is it possible that some rightwing cracker put me on his shitlist for a pre/post US election squish-the-left/pro-war DDOS attack? I don’t know.
I’ve picked up some light reading and am planning on making the most of my good luck to do my utmost to stay online from here on in.
If it comes back, or if, in the future, my server ever suffers some other prolonged down time, I will use my web space at magma to post emergency updates.
