That’s a fine server, Anthony, a fine, fine server!

Twighlight Zone Good Life Anthony

You’re a bad bot. I’m tired of playing with you. I’m going to make you dead now.

July, 2014.  The internet is a secluded village, all controlled and terrorized by one boy…

Meet Anthony, from the ‘It’s a Good Life’ episode of The Twilight Zone (Nov. 1961).  Details over at imdb, or watch the whole episode (with modernized commercial/ad inserts) at hulu.

We’ve hired Anthony, now in his 50s to do away with spam, Zombie DNS DDoS Bots, and other such pests buzzing around and annoying or destroying everything and everyone in the internet play ground.  We should have thought of it earlier… Just making bad things dead or wishing them into the cornfield.

OK, not quite that easy, here’s what’s up in the fight against Spam-Nados and Zombie Bots…

[Thermal] Inversion Layers.  In the undersea world of submariners these separations between temperatures of water mask objects from sonar and/or, of course, thermal imaging.  What does that have to do with servers and DDoS attacks? Nothing.  Except the association is how we remember:

Inverse Tactics.  At some point around the Happy Holiday Season 2013-2014 (aka Christmas to normal people) a server manager woke from a horrible holiday dream – a dream of fail2ban memory leaks, dead sockets, out of memory errors and massive log files – with a vision of a less nightmarish future.  Simplicity was unmasked.  The fortune-cookie like note that comes from the bobble-head-devil fortune machine in that other episode of Twilight Zone:

Invert your tactics.

Instead of trying to use fail2ban (perl/python/loosely interpreted regexs) to sift through everything and lock out the bad guys after they had been bad, and possibly then wrong because UDP sources are easily spoofed; use the ‘firewall’ (compiled, close to the kernel, strict iptables) as a firewall.  Seems simple, use the firewall as the firewall. But that’s hind-sight.

So, simple, use the firewall, invert the tactics.  What does that mean?

It means: allow only certain things along the ‘port 53 chain’ ~ only the domains that we truly are the authoritative host for.  And, since our little web-server is also a recursive-caching DNS server for itself and others; only do this filtration on the ‘net-facing’ (public) adapter.  Confused? Good. Welcome to the Twilight Zone.

Inverted logic:
– Old way: allow everything, find anything ‘known bad’ and block that; ever growing list of ‘known bad’ (because the bad-bot guys change what they query you to death with)
– New way: allow only the domains we NS host to be allowed passed iptables to get to bind/named and furthermore to fail2ban; drop all other (not known good) DNS queries.  Yep; drop; no tarpit, no ‘channel[s] closed’, no loopback, no reply, no nothing, DEEeee-ROP. (Wish bad queries into the corn field.)

Simple. Smart. Effective.  Can’t find the notes to give credit to some very helpful websites/pages that helped us through this “seems simple (everything computer is supposed to be simple)” but was very, very complicated set of iptables rules. TODO for later: find the notes, give the credit.

Here’s the summary:
– iptables does string matching (the * here is that iptables string matching on UDP sockets is very complicated, based on hex conversions of the strings, very strict, but very fast)
– set up a new ‘table (chain)’ and set of ‘RETURN’ rules based on the domains we actually want to give answers to DNS queries for
– set up the new ‘table (chain)’ to default to DROP (not reject, not return) all others

Here’s the implementation, using the plain text matching of iptables instead of the complete domain.tld (the dots don’t translate so you have to use hex to match domain(special)tld):

# create the chain to drop everything we don't specifically know
if ! iptables -L netdnswash -n  >/dev/null 2>&1 ; then
        #echo "debugging make the chain netdnswash"
        iptables -N netdnswash
# flush it either way
iptables -F netdnswash
# default rule = DROP.  This is to stop trying to block badbots, only allow good stuff.
# This should also drop the '.' (single dot) query
iptables -A netdnswash -j DROP
### during build/testing return everything, watch the counters on the one-two test domains
### iptables -A netdnswash -j RETURN
# put the most common ones up top (e.g. since that's the name server, then down to the least busy)
# reverse order because inserts stack on top
iptables -I netdnswash -i eth1 -p udp -m udp --dport 53 -m string --algo bm --icase --to 255 --string 'mldragon' -m comment --comment "" -j RETURN
iptables -I netdnswash -i eth1 -p udp -m udp --dport 53 -m string --algo bm --icase --to 255 --string 'computermedic' -m comment --comment " always first" -j RETURN
# make sure this is first in the INPUT chain - returns to INPUT will allow fail2ban to catch crap.gooddomain.tld
# make sure that fail2ban (actions) build their rules at/after 2. More manual thinking for fail2ban (admins) but way less work
# make sure it doesn't already exists
if ! iptables -L INPUT -n | grep 'netdnswash' -c 2>&1 ; then
        #echo "debugging make the rule in INPUT"
        iptables -I INPUT 1 -i eth1 -p udp -m udp --dport 53 -j netdnswash

Note 1: all chain creation prior to adding it to the INPUT chain rules.
Note 2: only traffic on the ‘net-facing/public’ adapter goes through here (at the bottom, …eth1…)
Note 3: used inserts ( -I ) so what you want first you have to put last in the script (invert again)
Note 4: once fully tested and put into operation: iptables-save >/path/to/conf_file so that if the server restart iptables doesn’t have to be ‘re-made’.
Note 5: change the fail2ban jail create/destroy commands to insert at 2, so fail2ban can be restarted without messing up the new iptables stuff.

Now, important, the lack of the complete domain.tld business.  If we had hundreds or thousands of Ns in our NS we would go through the trouble to use ‘hex string matching’ to get the whole thing in there.  But the “less than hundreds” number of domains on this server lets us deal with the occasional bad bot playing computermedic.tld [other than org] games.  How do we deal with that?

fail2ban of course.  Now, much relaxed and under-loaded because no more rolling/spoofed-source DNS/DDoS querries.  Again, don’t want to tell the bad-guys how to defeat our newly created defensive systems, but suffice it to say: if you run 3 or more or www999, www998, www997 (the rolling ones we encountered) queries against the server fail2ban is going to shut you down (1st time for 2 hours, 2nd time for… drumroll… Evuh!).  First the cornfield for a reasonable timeout, then fail2bAnthony makes you dead.

Less Spammy Inboxes version 14.7 (year.month versioning because there have been way too many versions)

TZ Anthony 2

I hate anybody that doesn’t like me!

Now that fail2bAnthony has less DNS problems on his hands, lots of free memory and CPU cycles, has not gone to swap in at least 3 months it’s time to put fail2bAnthony on task of wishing spambots into the cornfield, and those dreaded three-headed-gophers (the guess your password bots that then use your email to send spam everywhere ‘authenticated’): Dead.

fail2ban was already working the ‘mail servers logfiles’ to identify bad auth attempts and ban bad guys there.  However! We utilize an ‘Anti-Spam Relay/Filter Server’ (or several) and a great many of the bad auth attempts were going to those.  A couple of new fail2ban filters and actions, done.  Send legit email if you want copies of the fail2ban files, not publishing those (x-th time: don’t give the bad-guys a ‘how we defend ourselves’ map).  We’ll suffice it to say in this regard: had to get a few users new passwords, but ‘rejects’, bad server reputation points, and dictionary/rolling login attempts, junk in the mailq, against all mail systems are significantly decreased.  Will be even more decreased as time goes by because after certain re-bans, ips and whole ranges are banned for 1 year (all ports).  Cornfield, cornfield, dead.

A little over a year ago we said ‘Finally…’ ( ).  Doing all of this iptables, fail2ban, named (forgot to mention above, no more spoofed/loop-back entries in named conf files because no more bad domains/tlds allowed through), mail/spam server tuning revealed a major problem in the spam filter/relay setup: SenderBase did not install last year.  ‘Hypothetically speaking’ we could pretend that we use a certain Anti-Spam SMTP Proxy ( ASSP ) and we ‘hypothetically’ used their install/update scripts to get the thing moving.  Well, in those 12-18 days of extreme server configs and installs and the months of Zombie-Bot and DDoS-Nado wars after, the one little message in a log file seldom read about Net::SenderBase not being available was missed. Then there is the fact that ASSP ‘hypothetically’ does not log (even if debug level logging is selected) anything about SenderBase if it was not available at startup.  Where the problem lies: in some nix distros the install/update scripts, and the howto pages, that say use MCPAN to install Net::SenderBase don’t warn you to watch the output carefully to make sure there wasn’t any error[s].  Here’s the deal, and a warning: Watch the output of install Net::SenderBase carefully!!  If there are errors (any error means no SenderBase, no filters, regex matching, country matching, scoring, based on SenderBase in ASSP, and nothing to let you know): force install Net::SenderBase.

Bam. Done, had to go back an re-tweak/un-tweak about 12,000 ASSP (hypothetically) settings, logs, regex-es, etc., that had been tweaked trying to figure out why so much wasn’t working, gave ASSP a full stop (/etc/init.d/assp stop), let it rest a minute, fired it back up (replace stop with start) and wah-lah! 79% blocked instead of 45-51%.  Less spammy inboxes.

Updated wordpress to 3.9.1 today.
Not important but it works *great* in IE11.  Makes links correctly, which it did not.  Looks a little better.  Good job wordpress.