You’re a bad bot. I’m tired of playing with you. I’m going to make you dead now.

July, 2014.  The internet is a secluded village, all controlled and terrorized by one boy…

Meet Anthony, from the ‘It’s a Good Life’ episode of The Twilight Zone (Nov. 1961).  Details over at imdb, or watch the whole episode (with modernized commercial/ad inserts) at hulu.

We’ve hired Anthony, now in his 50s to do away with spam, Zombie DNS DDoS Bots, and other such pests buzzing around and annoying or destroying everything and everyone in the internet play ground.  We should have thought of it earlier… Just making bad things dead or wishing them into the cornfield.

OK, not quite that easy, here’s what’s up in the fight against Spam-Nados and Zombie Bots…

[Thermal] Inversion Layers.  In the undersea world of submariners these separations between temperatures of water mask objects from sonar and/or, of course, thermal imaging.  What does that have to do with servers and DDoS attacks? Nothing.  Except the association is how we remember:

Inverse Tactics.  At some point around the Happy Holiday Season 2013-2014 (aka Christmas to normal people) a server manager woke from a horrible holiday dream – a dream of fail2ban memory leaks, dead sockets, out of memory errors and massive log files – with a vision of a less nightmarish future.  Simplicity was unmasked.  The fortune-cookie like note that comes from the bobble-head-devil fortune machine in that other episode of Twilight Zone:

Invert your tactics.

Instead of trying to use fail2ban (perl/python/loosely interpreted regexs) to sift through everything and lock out the bad guys after they had been bad, and possibly then wrong because UDP sources are easily spoofed; use the ‘firewall’ (compiled, close to the kernel, strict iptables) as a firewall.  Seems simple, use the firewall as the firewall. But that’s hind-sight.

So, simple, use the firewall, invert the tactics.  What does that mean?

It means: allow only certain things along the ‘port 53 chain’ ~ only the domains that we truly are the authoritative host for.  And, since our little web-server is also a recursive-caching DNS server for itself and others; only do this filtration on the ‘net-facing’ (public) adapter.  Confused? Good. Welcome to the Twilight Zone.

Inverted logic:
– Old way: allow everything, find anything ‘known bad’ and block that; ever growing list of ‘known bad’ (because the bad-bot guys change what they query you to death with)
– New way: allow only the domains we NS host to be allowed passed iptables to get to bind/named and furthermore to fail2ban; drop all other (not known good) DNS queries.  Yep; drop; no tarpit, no ‘channel[s] closed’, no loopback, no reply, no nothing, DEEeee-ROP. (Wish bad queries into the corn field.)

Simple. Smart. Effective.  Can’t find the notes to give credit to some very helpful websites/pages that helped us through this “seems simple (everything computer is supposed to be simple)” but was very, very complicated set of iptables rules. TODO for later: find the notes, give the credit.

Here’s the summary:
– iptables does string matching (the * here is that iptables string matching on UDP sockets is very complicated, based on hex conversions of the strings, very strict, but very fast)
– set up a new ‘table (chain)’ and set of ‘RETURN’ rules based on the domains we actually want to give answers to DNS queries for
– set up the new ‘table (chain)’ to default to DROP (not reject, not return) all others

Here’s the implementation, using the plain text matching of iptables instead of the complete domain.tld (the dots don’t translate so you have to use hex to match domain(special)tld):

#!/bin/sh
#
# create the chain to drop everything we don't specifically know
if ! iptables -L netdnswash -n  >/dev/null 2>&1 ; then
        #echo "debugging make the chain netdnswash"
        iptables -N netdnswash
fi
# flush it either way
iptables -F netdnswash
# default rule = DROP.  This is to stop trying to block badbots, only allow good stuff.
# This should also drop the '.' (single dot) query
iptables -A netdnswash -j DROP
### during build/testing return everything, watch the counters on the one-two test domains
### iptables -A netdnswash -j RETURN
# put the most common ones up top (e.g. computermedic.org since that's the name server, then down to the least busy)
# reverse order because inserts stack on top
iptables -I netdnswash -i eth1 -p udp -m udp --dport 53 -m string --algo bm --icase --to 255 --string 'mldragon' -m comment --comment "mldragon.com" -j RETURN
iptables -I netdnswash -i eth1 -p udp -m udp --dport 53 -m string --algo bm --icase --to 255 --string 'computermedic' -m comment --comment "computermedic.org always first" -j RETURN
# make sure this is first in the INPUT chain - returns to INPUT will allow fail2ban to catch crap.gooddomain.tld
# make sure that fail2ban (actions) build their rules at/after 2. More manual thinking for fail2ban (admins) but way less work
# make sure it doesn't already exists
if ! iptables -L INPUT -n | grep 'netdnswash' -c 2>&1 ; then
        #echo "debugging make the rule in INPUT"
        iptables -I INPUT 1 -i eth1 -p udp -m udp --dport 53 -j netdnswash
fi

Note 1: all chain creation prior to adding it to the INPUT chain rules.
Note 2: only traffic on the ‘net-facing/public’ adapter goes through here (at the bottom, …eth1…)
Note 3: used inserts ( -I ) so what you want first you have to put last in the script (invert again)
Note 4: once fully tested and put into operation: iptables-save >/path/to/conf_file so that if the server restart iptables doesn’t have to be ‘re-made’.
Note 5: change the fail2ban jail create/destroy commands to insert at 2, so fail2ban can be restarted without messing up the new iptables stuff.

Now, important, the lack of the complete domain.tld business.  If we had hundreds or thousands of Ns in our NS we would go through the trouble to use ‘hex string matching’ to get the whole computermedic.org thing in there.  But the “less than hundreds” number of domains on this server lets us deal with the occasional bad bot playing computermedic.tld [other than org] games.  How do we deal with that?

fail2ban of course.  Now, much relaxed and under-loaded because no more rolling/spoofed-source DNS/DDoS querries.  Again, don’t want to tell the bad-guys how to defeat our newly created defensive systems, but suffice it to say: if you run 3 or more ddosasia.computermedic.org or www999, www998, www997 (the rolling ones we encountered) queries against the server fail2ban is going to shut you down (1st time for 2 hours, 2nd time for… drumroll… Evuh!).  First the cornfield for a reasonable timeout, then fail2bAnthony makes you dead.

Less Spammy Inboxes version 14.7 (year.month versioning because there have been way too many versions)

I hate anybody that doesn’t like me!

Now that fail2bAnthony has less DNS problems on his hands, lots of free memory and CPU cycles, has not gone to swap in at least 3 months it’s time to put fail2bAnthony on task of wishing spambots into the cornfield, and those dreaded three-headed-gophers (the guess your password bots that then use your email to send spam everywhere ‘authenticated’): Dead.

fail2ban was already working the ‘mail servers logfiles’ to identify bad auth attempts and ban bad guys there.  However! We utilize an ‘Anti-Spam Relay/Filter Server’ (or several) and a great many of the bad auth attempts were going to those.  A couple of new fail2ban filters and actions, done.  Send legit email if you want copies of the fail2ban files, not publishing those (x-th time: don’t give the bad-guys a ‘how we defend ourselves’ map).  We’ll suffice it to say in this regard: had to get a few users new passwords, but ‘rejects’, bad server reputation points, and dictionary/rolling login attempts, junk in the mailq, against all mail systems are significantly decreased.  Will be even more decreased as time goes by because after certain re-bans, ips and whole ranges are banned for 1 year (all ports).  Cornfield, cornfield, dead.

A little over a year ago we said ‘Finally…’ ( http://www.computermedic.org/?p=9 ).  Doing all of this iptables, fail2ban, named (forgot to mention above, no more spoofed/loop-back entries in named conf files because no more bad domains/tlds allowed through), mail/spam server tuning revealed a major problem in the spam filter/relay setup: SenderBase did not install last year.  ‘Hypothetically speaking’ we could pretend that we use a certain Anti-Spam SMTP Proxy ( ASSP ) and we ‘hypothetically’ used their install/update scripts to get the thing moving.  Well, in those 12-18 days of extreme server configs and installs and the months of Zombie-Bot and DDoS-Nado wars after, the one little message in a log file seldom read about Net::SenderBase not being available was missed. Then there is the fact that ASSP ‘hypothetically’ does not log (even if debug level logging is selected) anything about SenderBase if it was not available at startup.  Where the problem lies: in some nix distros the install/update scripts, and the howto pages, that say use MCPAN to install Net::SenderBase don’t warn you to watch the output carefully to make sure there wasn’t any error[s].  Here’s the deal, and a warning: Watch the output of install Net::SenderBase carefully!!  If there are errors (any error means no SenderBase, no filters, regex matching, country matching, scoring, based on SenderBase in ASSP, and nothing to let you know): force install Net::SenderBase.

Bam. Done, had to go back an re-tweak/un-tweak about 12,000 ASSP (hypothetically) settings, logs, regex-es, etc., that had been tweaked trying to figure out why so much wasn’t working, gave ASSP a full stop (/etc/init.d/assp stop), let it rest a minute, fired it back up (replace stop with start) and wah-lah! 79% blocked instead of 45-51%.  Less spammy inboxes.

Updated wordpress to 3.9.1 today.
Not important but it works *great* in IE11.  Makes links correctly, which it did not.  Looks a little better.  Good job wordpress.

Equally exciting, terrifying, low-budget and prone to sequels.

So bad it’s good movie lovers, click the link above and see if you can survive that whirlwind of bites.

Server admins, stay right here and get ready for DNS-Zombie-Bots Two: More Tech-Talk and .configs Than You Can Stand!  (Or, “Bored To Death!” Or, “You can have the whole seat, but you only need the edge!”)

Or, I had to document it so I can take it from server to server without trusting my memory, so I thought I would share.

It started with a ‘Hay Bay-Bay’ – or a ‘clients-per-query’ message.

Lots of tweaks, tunes, service this restart, /etc/init.d/that restart later: ‘clients-per-query’ (increased/decreased) messages, lots of them.  (Somehow, sync’d between servers, trying to figure that BIND9 magic out would be like trying to reach into the mouth of one of those sharknado sharks and pull its heart out.  It is because it is, do the fixes you can do and worry about enigmatic synchronicity later.)

Here’s the setup again so when you try these things on a server with a point-zero-zero-one version difference you’ll know why it doesn’t work:
~ CentOS x86_64 6.4 (Installed, updated [yum update] June, 2013)
~ bind / named
* # rndc status: version: 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.4
* # yum list bind: bind.x86_64 32:9.8.2-0.17.rc1.el6_4.4
~ fail2ban
* # yum list fail2ban: fail2ban.noarch 0.8.8-3.el6 ( 0.8.10-1.el6 available )

Since last I wrote about it ( Killin Zombie Bots ) some seemingly minor, but very important changes mainly to the bind/named and related conf files.
/etc/resolv.conf: nameserver 127.0.0.1
~ all ‘in-server’ services should ask ‘self’ for DNS, when self doesn’t know it “recurses” (goes upstream) and caches so that for a time (cache TTLs and expirys) ‘self’ does know the answer.
/etc/named.conf (in the options { } block): querylog yes;
~ log at boot (that semi-colon ‘;’ is very-necessary)
/etc/fail2ban/jail.conf: ignoreip = 127.0.0.1/8
~ ‘confirmed’ (it is the default in the [default] section) – Self: don’t ban we.

clients-per-query groans the almost healed zombie server

Ask the modern zombie-to-English interpreter ( google ) what that means and the interpreter says:
About 2,440,000 results  (0.33 seconds)

Go on an Injun vision quest and consult the shaman: add more RAM.

Don’t run down that 2.44-Million results rabbit hole.  Smack your head and think I should have thought of that when the light bulb turns on in there behind the sign that reads:

The cache of a caching DNS server on a moderate-to-heavy-load server can get quite large.

That’s RAM bay-bays, nothing else.  Don’t believe yourself, # free. @125MB of 2GB left and look lower: @105648[k] used Swap:.  This is on the ‘dedicated’ DNS server.  Pop-Nerd-Quiz: Going swap happens when?  Right, when ‘real memory’ (RAM) is full.

Jump over to the we-have-it-no-matter-what-it-is-and-cheap shop of the new millennium ( eBay ) and [Buy It Now] on 8GB of the wrong RAM for your server.  Smack your head again because everything is ‘the hard way’ (like being the only seal in a Sharknado), put the wrong RAM up for sale and [Buy It Now-Now] on 8GB of hopefully the right RAM for your server.  When it shows up (it will be right this time!) hope that 8GBs is enough for a caching DNS server.

Summary: clients-per-query = add more RAM.

There is a nightmare-nado of other things you can try to tweak or tune or limit and shutdown -r now… or you can RAM-up and see all those Zombie-Language /var/log/ messages in your logs vanish.  If you are ‘flush with GB’ and nothing in swap: it’s 2.44 million curtains for you, tough guy (in 0.33 seconds)!

Z0mb13 t4Lk (or ID-10-T bot-writer cOdEs) and fail2ban

 1rip, 1Rip, 1rIP, and so-on.  case-insensitive, or ignore-case. Sounds so easy.  So you try a little (?i) and a little \/\.IhateRegEx (interpreted through .py and other things depending on revision or build number) and pretty soon you are standing in the eye of a shark-icane with your Mary Poppins umbrella waiting for the winds to take you to a hopefully quick and not too painful shark-shutdown (in the air).

begin here (/etc/fail2ban/filter.d dir):
[root@server filter.d]# cp named-flood.conf named-ignoretest.conf
[root@server filter.d]# vim named-ignoretest.conf

It has now (sorry for the wordpress word-wrap):

failregex = .* named\[.*\]: client <HOST>\#.*: query: (1rip\.com|isc\.org|\.) (IN|ANY) *

Based on the only search result that made sense ( https://github.com/fail2ban/fail2ban/issues/48 ) and ( http://www.tutorialspoint.com/python/python_reg_expressions.htm ) [and about 100 trial-with-error failures] change it to:

 failregex = .* named\[.*\]: client <HOST>\#.*: query: ((?i)1rip|1rip\.com|isc\.org|\.) (IN|ANY) *

The important thing here (besides this is not tested against any other versions): ((?i)[pipe separated list]).  The ‘ignore-case’ (?i) toggle is working on all of the entries in the [pipe separated list].  Another thing: I didn’t test and don’t care if the case-insensitive compare carries over to the (IN|ANY).

Because the only ‘spoof’ in there is 1rip.com (now case does not matter) some of those isc.org queries are still getting answered, and the (space)1rip(space) [1rip without a domain extension] are still doing something (as yet unknown) to the cache and the upstream.  What is known about those is that they are now successfully triggering fail2ban to shut those servers/ips down after a couple of hits and send the rest of their millions of attempts to >dev/null.

Doing packet/byte count watches ( #iptables -n -L -v –line-numbers ) reveals that once ‘dumped’ into the ‘fail2ban filter table’ the bad-zombie-bots (flooding w/requrests) are ‘dropping’ many hundreds of thousands of requests (packets) and GBs of data per hour.

2    3000K  864M fail2ban-dnsflood    all  -- *  *   0.0.0.0/0   0.0.0.0/0
3    1829K  792M fail2ban-maillogins  all  -- *  *   0.0.0.0/0   0.0.0.0/0

“It’s only” 72MB (@10% by bytes), but fully 39% of all packet-traffic being killed by this fail2ban zombie-net – on this one particular server.  Not sure how to ‘math it out’ but it is also a server-unload because that many (1171K = 1.2-Million) queries/requests are not being cache-pulled or sent upstream – on this one particular server.  (iptables numbers above were reset 60 minutes previous)

Slowly I turned, step by step, inch by inch
(shark by shark twisting in the wind)

I ran off on a statistics tangent and never completed the fail2ban new-regex howto.

The new-est /etc/fail2ban/filter.d/named-flood.conf needs to be up-to-dated:
# vim /etc/fail2ban/filter.d/named-flood.conf

[Definition]
failregex = .* named\[.*\]: client <HOST>\#.*: query: ((?i)1rip|1rip\.com|isc\.org|\.) (IN|ANY) *
ignoreregex =

:wq (write and quit)

Make a test file:
#vim /tmp/testfile.txt (press insert when it ‘loads: New File’)

Jul 11 05:40:22 server named[1301]: client 1.1.1.1#1: query: 1rip IN ANY +E ([ip of server])
Jul 11 05:40:22 server named[1301]: client 2.2.2.2#2: query: 1rip IN ANY +E ([ip of server])
Jul 11 05:40:23 server named[1301]: client 3.3.3.3#3: query: 1Rip IN ANY +E ([ip of server])
Jul 11 05:40:23 server named[1301]: client 4.4.4.4#4: query: 1rIp IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 5.5.5.5#5: query: 1riP IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 6.6.6.6#6: query: 1rip IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 7.7.7.7#7: query: 1rip.com IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 8.8.8.8#8: query: 1rIp.com IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 9.9.9.9#9: query: linenine.com IN ANY +E ([ip of server])

:wq (write and quit)

[root@server filter.d]# fail2ban-regex /tmp/testfile.txt named-flood.conf

Should get 8 “number of match”

Compare to grep-ing (note the spaces and escaped .s inside the single quotes)
grep -c -i ‘ 1rip ‘ /tmp/testfile.txt : 6
grep -c -i ‘ 1rip\.com ‘  /tmp/testfile.txt : 2
grep -c -i ‘ \. ‘  /tmp/testfile.txt : 0
grep -c -i ‘ isc\.org ‘  /tmp/testfile.txt : 0

Test and compare against the real thing:

Make a working copy:
[root@server filter.d]# cp /var/log/messages /tmp/x.txt

[root@server filter.d]# fail2ban-regex /tmp/x.txt named-flood.conf
Takes a while on this server, then:
Success, the total number of match is 106225

Compare to grep-ing (note the spaces and escaped .s inside the single quotes)
grep -c -i ‘ 1rip ‘ /tmp/x.txt : 81034
grep -c -i ‘ 1rip\.com ‘  /tmp/x.txt : 2977
grep -c -i ‘ \. ‘  /tmp/x.txt : 15917
grep -c -i ‘ isc\.org ‘  /tmp/x.txt : 6297
——————– 106225 all added together

Sure looks like this is working, rm all those test files in /tmp/, then:

# /etc/init.d/fail2ban restart

Because of the amount of sharks in this nado, you might (we have to) manually block some ip’s while fail2ban gets back in the race.  Once fail2ban is all caught up and ready to go up against the whirlwind of feeding-frenzied zombie-shark-bots, manually release those and let fail2ban do its thing.

One last piece of housekeeping in this bad movie: reset the counters.
#iptables -Z

About one minute later, late one Saturday afternoon:

num   pkts   bytes    target (rest snipped)
1     2059    175K    fail2ban-dnsflood
2      352   66583    fail2ban-maillogins

A whopping 17% of packet-traffic is NOT a DNS-DDoS-Flood packet.

June 6, 2013: need a new website for a customer.  “Let’s do wordpress!”  Boom #1: wordpress needs newer php.

June 6-8, 2013: Thinking, discussing, planning regarding: in-place updates never work.  “Just setup a new server all up to date and ready for wordpress.”

June 9, 2013: Smart I.T. peoples backup on Sunday, run a full system backup… Boom #2: backup hard drives marked “read only” (usually means damage or imminent failure).

June 9-16, 2013: 2 new (rebuild one, fire up and build the “backup equipment” been sitting idle since 2008 waiting for catastrophe) servers to dish DNS, one will be “live” server, one is a fall-back/backup.

June 9-16, 2013: The endless cascade of computer junk.  One thing leads to one more thing that reveals that other problem and so-on and so-on.

June 16, 2013 11:00PM: everything transferred from 3 servers onto/into the new pair.  Shutdown (shutdown -h now) “ninesix” (online since 1.2010) and “isp1100″ (since 2006?? 2005?? earlier??).  Unplug this, plug in that, check this check that… “Tomorrow, let’s do wordpress!”

June 17, 2013: Boom #3.  bind (DNS Server) SERVFAIL.  Zone file for MLD Computers ( mldragon.com, how cool is that ) marked with .err extension, all ‘other’ domains working (sort of), but DNS errors for one cause DNS errors for many.  Finally got passed bind/DNS problems (delete the zones and re-create is the hind-sight how-to).  ns3 DNS ports lost in the shuffle (they say forwarding and accepted, poof no reply).

June 17, 2013: Boom #4.  Lots of “little fixes” needed to make sites work/look like they used to.  All new server os and hosting software should not be mixed with all old web design.

June 18, 2013: Finally…
All of this and that and the others settled down enough: “Let’s do wordpress!”

So, testing it out here before trying on the customer site that started this ‘mess’ 12 days ago.