You’re a bad bot. I’m tired of playing with you. I’m going to make you dead now.

July, 2014.  The internet is a secluded village, all controlled and terrorized by one boy…

Meet Anthony, from the ‘It’s a Good Life’ episode of The Twilight Zone (Nov. 1961).  Details over at imdb, or watch the whole episode (with modernized commercial/ad inserts) at hulu.

We’ve hired Anthony, now in his 50s to do away with spam, Zombie DNS DDoS Bots, and other such pests buzzing around and annoying or destroying everything and everyone in the internet play ground.  We should have thought of it earlier… Just making bad things dead or wishing them into the cornfield.

OK, not quite that easy, here’s what’s up in the fight against Spam-Nados and Zombie Bots…

[Thermal] Inversion Layers.  In the undersea world of submariners these separations between temperatures of water mask objects from sonar and/or, of course, thermal imaging.  What does that have to do with servers and DDoS attacks? Nothing.  Except the association is how we remember:

Inverse Tactics.  At some point around the Happy Holiday Season 2013-2014 (aka Christmas to normal people) a server manager woke from a horrible holiday dream – a dream of fail2ban memory leaks, dead sockets, out of memory errors and massive log files – with a vision of a less nightmarish future.  Simplicity was unmasked.  The fortune-cookie like note that comes from the bobble-head-devil fortune machine in that other episode of Twilight Zone:

Invert your tactics.

Instead of trying to use fail2ban (perl/python/loosely interpreted regexs) to sift through everything and lock out the bad guys after they had been bad, and possibly then wrong because UDP sources are easily spoofed; use the ‘firewall’ (compiled, close to the kernel, strict iptables) as a firewall.  Seems simple, use the firewall as the firewall. But that’s hind-sight.

So, simple, use the firewall, invert the tactics.  What does that mean?

It means: allow only certain things along the ‘port 53 chain’ ~ only the domains that we truly are the authoritative host for.  And, since our little web-server is also a recursive-caching DNS server for itself and others; only do this filtration on the ‘net-facing’ (public) adapter.  Confused? Good. Welcome to the Twilight Zone.

Inverted logic:
– Old way: allow everything, find anything ‘known bad’ and block that; ever growing list of ‘known bad’ (because the bad-bot guys change what they query you to death with)
– New way: allow only the domains we NS host to be allowed passed iptables to get to bind/named and furthermore to fail2ban; drop all other (not known good) DNS queries.  Yep; drop; no tarpit, no ‘channel[s] closed’, no loopback, no reply, no nothing, DEEeee-ROP. (Wish bad queries into the corn field.)

Simple. Smart. Effective.  Can’t find the notes to give credit to some very helpful websites/pages that helped us through this “seems simple (everything computer is supposed to be simple)” but was very, very complicated set of iptables rules. TODO for later: find the notes, give the credit.

Here’s the summary:
– iptables does string matching (the * here is that iptables string matching on UDP sockets is very complicated, based on hex conversions of the strings, very strict, but very fast)
– set up a new ‘table (chain)’ and set of ‘RETURN’ rules based on the domains we actually want to give answers to DNS queries for
– set up the new ‘table (chain)’ to default to DROP (not reject, not return) all others

Here’s the implementation, using the plain text matching of iptables instead of the complete domain.tld (the dots don’t translate so you have to use hex to match domain(special)tld):

#!/bin/sh
#
# create the chain to drop everything we don't specifically know
if ! iptables -L netdnswash -n  >/dev/null 2>&1 ; then
        #echo "debugging make the chain netdnswash"
        iptables -N netdnswash
fi
# flush it either way
iptables -F netdnswash
# default rule = DROP.  This is to stop trying to block badbots, only allow good stuff.
# This should also drop the '.' (single dot) query
iptables -A netdnswash -j DROP
### during build/testing return everything, watch the counters on the one-two test domains
### iptables -A netdnswash -j RETURN
# put the most common ones up top (e.g. computermedic.org since that's the name server, then down to the least busy)
# reverse order because inserts stack on top
iptables -I netdnswash -i eth1 -p udp -m udp --dport 53 -m string --algo bm --icase --to 255 --string 'mldragon' -m comment --comment "mldragon.com" -j RETURN
iptables -I netdnswash -i eth1 -p udp -m udp --dport 53 -m string --algo bm --icase --to 255 --string 'computermedic' -m comment --comment "computermedic.org always first" -j RETURN
# make sure this is first in the INPUT chain - returns to INPUT will allow fail2ban to catch crap.gooddomain.tld
# make sure that fail2ban (actions) build their rules at/after 2. More manual thinking for fail2ban (admins) but way less work
# make sure it doesn't already exists
if ! iptables -L INPUT -n | grep 'netdnswash' -c 2>&1 ; then
        #echo "debugging make the rule in INPUT"
        iptables -I INPUT 1 -i eth1 -p udp -m udp --dport 53 -j netdnswash
fi

Note 1: all chain creation prior to adding it to the INPUT chain rules.
Note 2: only traffic on the ‘net-facing/public’ adapter goes through here (at the bottom, …eth1…)
Note 3: used inserts ( -I ) so what you want first you have to put last in the script (invert again)
Note 4: once fully tested and put into operation: iptables-save >/path/to/conf_file so that if the server restart iptables doesn’t have to be ‘re-made’.
Note 5: change the fail2ban jail create/destroy commands to insert at 2, so fail2ban can be restarted without messing up the new iptables stuff.

Now, important, the lack of the complete domain.tld business.  If we had hundreds or thousands of Ns in our NS we would go through the trouble to use ‘hex string matching’ to get the whole computermedic.org thing in there.  But the “less than hundreds” number of domains on this server lets us deal with the occasional bad bot playing computermedic.tld [other than org] games.  How do we deal with that?

fail2ban of course.  Now, much relaxed and under-loaded because no more rolling/spoofed-source DNS/DDoS querries.  Again, don’t want to tell the bad-guys how to defeat our newly created defensive systems, but suffice it to say: if you run 3 or more ddosasia.computermedic.org or www999, www998, www997 (the rolling ones we encountered) queries against the server fail2ban is going to shut you down (1st time for 2 hours, 2nd time for… drumroll… Evuh!).  First the cornfield for a reasonable timeout, then fail2bAnthony makes you dead.

Less Spammy Inboxes version 14.7 (year.month versioning because there have been way too many versions)

I hate anybody that doesn’t like me!

Now that fail2bAnthony has less DNS problems on his hands, lots of free memory and CPU cycles, has not gone to swap in at least 3 months it’s time to put fail2bAnthony on task of wishing spambots into the cornfield, and those dreaded three-headed-gophers (the guess your password bots that then use your email to send spam everywhere ‘authenticated’): Dead.

fail2ban was already working the ‘mail servers logfiles’ to identify bad auth attempts and ban bad guys there.  However! We utilize an ‘Anti-Spam Relay/Filter Server’ (or several) and a great many of the bad auth attempts were going to those.  A couple of new fail2ban filters and actions, done.  Send legit email if you want copies of the fail2ban files, not publishing those (x-th time: don’t give the bad-guys a ‘how we defend ourselves’ map).  We’ll suffice it to say in this regard: had to get a few users new passwords, but ‘rejects’, bad server reputation points, and dictionary/rolling login attempts, junk in the mailq, against all mail systems are significantly decreased.  Will be even more decreased as time goes by because after certain re-bans, ips and whole ranges are banned for 1 year (all ports).  Cornfield, cornfield, dead.

A little over a year ago we said ‘Finally…’ ( http://www.computermedic.org/?p=9 ).  Doing all of this iptables, fail2ban, named (forgot to mention above, no more spoofed/loop-back entries in named conf files because no more bad domains/tlds allowed through), mail/spam server tuning revealed a major problem in the spam filter/relay setup: SenderBase did not install last year.  ‘Hypothetically speaking’ we could pretend that we use a certain Anti-Spam SMTP Proxy ( ASSP ) and we ‘hypothetically’ used their install/update scripts to get the thing moving.  Well, in those 12-18 days of extreme server configs and installs and the months of Zombie-Bot and DDoS-Nado wars after, the one little message in a log file seldom read about Net::SenderBase not being available was missed. Then there is the fact that ASSP ‘hypothetically’ does not log (even if debug level logging is selected) anything about SenderBase if it was not available at startup.  Where the problem lies: in some nix distros the install/update scripts, and the howto pages, that say use MCPAN to install Net::SenderBase don’t warn you to watch the output carefully to make sure there wasn’t any error[s].  Here’s the deal, and a warning: Watch the output of install Net::SenderBase carefully!!  If there are errors (any error means no SenderBase, no filters, regex matching, country matching, scoring, based on SenderBase in ASSP, and nothing to let you know): force install Net::SenderBase.

Bam. Done, had to go back an re-tweak/un-tweak about 12,000 ASSP (hypothetically) settings, logs, regex-es, etc., that had been tweaked trying to figure out why so much wasn’t working, gave ASSP a full stop (/etc/init.d/assp stop), let it rest a minute, fired it back up (replace stop with start) and wah-lah! 79% blocked instead of 45-51%.  Less spammy inboxes.

Updated wordpress to 3.9.1 today.
Not important but it works *great* in IE11.  Makes links correctly, which it did not.  Looks a little better.  Good job wordpress.

The digital world has out-run squirrel:

• Does not work fully with Internet Explorer 11
• Does not work fully with mac and iDevices (pod, phone, pad, OSX (**ok with Mozilla or firefox on OSX but no one runs that)
• Does not work fully with [An]Droid devices
• Does not work fully with Surface (trimmed IE 10/11)

So, 1.11.14 we installed and tested version 0.9.5 of roundcube.

Hosted at MLD/CMI? Check it out at [www.yourdomain.xyz]/rcmail or, for example:

http://www.computermedic.org/rcmail

Log in using your full email address as username (just like squirrel).

After your first log in you should go to the settings screen and turn on HTML composition and viewing and set your signature options (if you keep using roundcube).

Both work fine side-by-side, but you should not be logged in with both at the same time.

Link at the top/right soon.

We have had to turn back on/up as many of the filters and blocks as possible because the “spam load” on some of the users (including the admin) is overwhelming.

Some of your contacts may get ‘rejections’ or ‘bounces’ if their servers are identified as “blacklisted” (DNSBL).  If you have any reports from your friends about problems email support@computermedic.org and we’ll get them whitelisted.

It was a backdraft. Or the eye of the Zombie-Nado-Cane. When the bad-bots got some air around August 5th – hak4umz.net DDoS or DNS Amplification – fail2ban (and the servers) got burned.

Even the “eye-dee-keff-kuh-may” (TammyBelle’s God Mode Code for DOOM][ ) cheat didn’t help.  fail2ban got clobbered… ‘already banned’ every one second in the log and no more bans happening because 100s or 1000s of times per second from 100s or thousands of bots: bad requests.

Here is the Serious! problem when you put the fail2ban -vs- the entire globe death match together:

2013-08-13 12:40:37,730 fail2ban.actions: INFO   [named-flood] <ip> already banned
2013-08-13 12:40:38,732 fail2ban.actions: INFO   [named-flood] <ip> already banned

almost exactly one second apart, hundreds of times, and no new banning going on.
Q: Why?
A: fail2ban appears to have a “one second pulse/parse” clock built in to it.

Q: So?
A: So, when 4,000 log entries appear in a log that fail2ban is reading within that one second, fail2ban ‘queues’ (or spools or fifo’s) those 4,000 entries into an internal list and tries to de-queue them one-per-second.

Easier math: (“let’s say”) there are 10 ‘fail regex’ entries pouring into your log per second. Trying to de-queue the messages from the first second takes fail2ban 9 seconds.  By the time it gets done, there are 90 more messages/fails waiting.  So every second that goes by (in this low number scenario) the problem gets 10-to-the-10th-power worse.  The problem being fail2ban over-run by those headless bomb-toting zombies.  The “real world” explanation: fail2ban lags out and becomes combat ineffective.  In cop-talk the radio call from Officer fail2ban would be: “Extended”

Now, a “server admin” must consider – besides ‘shutdown -h now’ – is there a solution to the problem? First part of that: what – exactly – is the problem.  More Q/A (logic/reasoning):
Q: Problem?
A: fail2ban says ‘already banned’ and is ‘lagged out'; can’t fight the good fight.

Q: Why?
A: Too many log entries per second.  fail2ban reads logs and ‘actions’ based on log entries.

Q: So, why don’t you server admins just limit the number of log entries? (Instead of trying to hyper-tune fail2ban, just give it less to do? Remember the old-old server used to say ‘…the previous message repeated ### times…’)
A: Why didn’t I think of that.

The old-old server was a Gentoo box dragged across the millennium boundary by makes and make-installs.  It finally wore out (it still runs, it was just retired because it had done it’s duty) this year.  A little searching about ‘the previous message repeated’ and was reminded that that is called: rate-limit-ing.  A modern Centos-6-x86_64 install (not a bunch of custom compiled stuff on a 32-bit Gentoo) uses an ‘out of the box’ rsyslog and doesn’t say things like ‘…the previous message…’  The new stuff says:
imuxsock begins to drop messages from pid 1228 due to rate-limiting

Very little more searching finds:
http://www.rsyslog.com/tag/rate-limiting/

The docs are a ‘little dated’ (2010) but the essentials are there to solve the problem (problem being ‘too many log entries for poor old fail2ban’).

vim /etc/rsyslog.conf (and add as the 2nd and 3rd uncommented lines):

#### 8.12.13 - try to slow the message floods so fail2ban won't die so much ####
$SystemLogRateLimitInterval 1
$SystemLogRateLimitBurst 5

[Esc]:wq (write and quit)

Now do a /etc/init.d/rsyslog restart or service rsyslog restart (reload does not work, I tried it) and…

Tah-dah!  fail2ban can keep up with the log.  Some of the abusers (firey screaming zombies with tater-bombs) get by for a few seconds until the rate-limit/fail2ban get Serious!; but, real-world they were getting by by the hundres-of-thousands before this fix (while poor old fail2ban was over-run or lag-back-buffered).

It may not be ‘iddqd’ (god/degreelessness mode in ‘that other great fps’), but $SystemLogRateLimitInterval/$SystemLogRateLimitBurst is very close to TammyBelle’s “eye-dee-keff-kuh-may” (megaarmor, weapons and keys) for fail2ban.

Almost as good as Tangy-Bells for break-shishst.
Very happy, ammo added.

*** A minor success/victory ***
49 hours later… fail2ban chugging along ban/unban-ing, much smaller log files, no other services lagged out because of the packet attacks on port 53….

Suddenly, in sync with Aug 1 (00:00:00) rolling in around the globe (asia, Europe, etc):
It stopped.

All of the ‘watch words’ we’ve been jamming in fail2ban vanished.  By 00:00:00 Eastern US: 1 ip in the 2 hour block box.  At 12:00:00 (pm) July 31: 1000+.

Whatever time-bomb went off and killed all the zombie bots – hoozah.

There is something very creepy about it, though.

When (in the zombie apocalypse wars) you get a ‘tides have gone out’ moment: fortify defenses.

Updated fail2ban from 8.8 to 8.10.  Lots of features and fixes noted at their site – hope so, have had some trouble with fail2ban 8.8 and this recent assault.

In the last 2 weeks we have put some smarter rules in effect: empty zones with ‘deny’ default in /etc/named.conf

// sema.cz 7.29.13 - 7.30.13 100s
zone "sema.cz" {
        type master;
        allow-query { none; };
        allow-transfer { none; };
        file "named.empty";
};

This, of course, adds some bloat to the logs ( /IN/ANY denied line ). But it doesn’t go upstream.

More later, if the DNS ddos returns, or if time permits.

******** 8.1.13 (later in the day) update *********

Still too quiet after 100s-of-millions of attempts over the last 2 months.

The couple that have appeared reveal that the updated fail2ban tries to be nicey-nice ( reject ) instead of dumping zombies into the pit of oblivion (drop).

The new fail2ban default is:
blocktype = REJECT –reject-with icmp-port-unreachable

So, I am trying in the jail.conf file:

action  = iptables-allports[name=dnsflood, blocktype=DROP]

Took about 1 minute for the last of the zombie-hee-cans to get DROP-ped.  Something went right today.

Equally exciting, terrifying, low-budget and prone to sequels.

So bad it’s good movie lovers, click the link above and see if you can survive that whirlwind of bites.

Server admins, stay right here and get ready for DNS-Zombie-Bots Two: More Tech-Talk and .configs Than You Can Stand!  (Or, “Bored To Death!” Or, “You can have the whole seat, but you only need the edge!”)

Or, I had to document it so I can take it from server to server without trusting my memory, so I thought I would share.

It started with a ‘Hay Bay-Bay’ – or a ‘clients-per-query’ message.

Lots of tweaks, tunes, service this restart, /etc/init.d/that restart later: ‘clients-per-query’ (increased/decreased) messages, lots of them.  (Somehow, sync’d between servers, trying to figure that BIND9 magic out would be like trying to reach into the mouth of one of those sharknado sharks and pull its heart out.  It is because it is, do the fixes you can do and worry about enigmatic synchronicity later.)

Here’s the setup again so when you try these things on a server with a point-zero-zero-one version difference you’ll know why it doesn’t work:
~ CentOS x86_64 6.4 (Installed, updated [yum update] June, 2013)
~ bind / named
* # rndc status: version: 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.4
* # yum list bind: bind.x86_64 32:9.8.2-0.17.rc1.el6_4.4
~ fail2ban
* # yum list fail2ban: fail2ban.noarch 0.8.8-3.el6 ( 0.8.10-1.el6 available )

Since last I wrote about it ( Killin Zombie Bots ) some seemingly minor, but very important changes mainly to the bind/named and related conf files.
/etc/resolv.conf: nameserver 127.0.0.1
~ all ‘in-server’ services should ask ‘self’ for DNS, when self doesn’t know it “recurses” (goes upstream) and caches so that for a time (cache TTLs and expirys) ‘self’ does know the answer.
/etc/named.conf (in the options { } block): querylog yes;
~ log at boot (that semi-colon ‘;’ is very-necessary)
/etc/fail2ban/jail.conf: ignoreip = 127.0.0.1/8
~ ‘confirmed’ (it is the default in the [default] section) – Self: don’t ban we.

clients-per-query groans the almost healed zombie server

Ask the modern zombie-to-English interpreter ( google ) what that means and the interpreter says:
About 2,440,000 results  (0.33 seconds)

Go on an Injun vision quest and consult the shaman: add more RAM.

Don’t run down that 2.44-Million results rabbit hole.  Smack your head and think I should have thought of that when the light bulb turns on in there behind the sign that reads:

The cache of a caching DNS server on a moderate-to-heavy-load server can get quite large.

That’s RAM bay-bays, nothing else.  Don’t believe yourself, # free. @125MB of 2GB left and look lower: @105648[k] used Swap:.  This is on the ‘dedicated’ DNS server.  Pop-Nerd-Quiz: Going swap happens when?  Right, when ‘real memory’ (RAM) is full.

Jump over to the we-have-it-no-matter-what-it-is-and-cheap shop of the new millennium ( eBay ) and [Buy It Now] on 8GB of the wrong RAM for your server.  Smack your head again because everything is ‘the hard way’ (like being the only seal in a Sharknado), put the wrong RAM up for sale and [Buy It Now-Now] on 8GB of hopefully the right RAM for your server.  When it shows up (it will be right this time!) hope that 8GBs is enough for a caching DNS server.

Summary: clients-per-query = add more RAM.

There is a nightmare-nado of other things you can try to tweak or tune or limit and shutdown -r now… or you can RAM-up and see all those Zombie-Language /var/log/ messages in your logs vanish.  If you are ‘flush with GB’ and nothing in swap: it’s 2.44 million curtains for you, tough guy (in 0.33 seconds)!

Z0mb13 t4Lk (or ID-10-T bot-writer cOdEs) and fail2ban

 1rip, 1Rip, 1rIP, and so-on.  case-insensitive, or ignore-case. Sounds so easy.  So you try a little (?i) and a little \/\.IhateRegEx (interpreted through .py and other things depending on revision or build number) and pretty soon you are standing in the eye of a shark-icane with your Mary Poppins umbrella waiting for the winds to take you to a hopefully quick and not too painful shark-shutdown (in the air).

begin here (/etc/fail2ban/filter.d dir):
[root@server filter.d]# cp named-flood.conf named-ignoretest.conf
[root@server filter.d]# vim named-ignoretest.conf

It has now (sorry for the wordpress word-wrap):

failregex = .* named\[.*\]: client <HOST>\#.*: query: (1rip\.com|isc\.org|\.) (IN|ANY) *

Based on the only search result that made sense ( https://github.com/fail2ban/fail2ban/issues/48 ) and ( http://www.tutorialspoint.com/python/python_reg_expressions.htm ) [and about 100 trial-with-error failures] change it to:

 failregex = .* named\[.*\]: client <HOST>\#.*: query: ((?i)1rip|1rip\.com|isc\.org|\.) (IN|ANY) *

The important thing here (besides this is not tested against any other versions): ((?i)[pipe separated list]).  The ‘ignore-case’ (?i) toggle is working on all of the entries in the [pipe separated list].  Another thing: I didn’t test and don’t care if the case-insensitive compare carries over to the (IN|ANY).

Because the only ‘spoof’ in there is 1rip.com (now case does not matter) some of those isc.org queries are still getting answered, and the (space)1rip(space) [1rip without a domain extension] are still doing something (as yet unknown) to the cache and the upstream.  What is known about those is that they are now successfully triggering fail2ban to shut those servers/ips down after a couple of hits and send the rest of their millions of attempts to >dev/null.

Doing packet/byte count watches ( #iptables -n -L -v –line-numbers ) reveals that once ‘dumped’ into the ‘fail2ban filter table’ the bad-zombie-bots (flooding w/requrests) are ‘dropping’ many hundreds of thousands of requests (packets) and GBs of data per hour.

2    3000K  864M fail2ban-dnsflood    all  -- *  *   0.0.0.0/0   0.0.0.0/0
3    1829K  792M fail2ban-maillogins  all  -- *  *   0.0.0.0/0   0.0.0.0/0

“It’s only” 72MB (@10% by bytes), but fully 39% of all packet-traffic being killed by this fail2ban zombie-net – on this one particular server.  Not sure how to ‘math it out’ but it is also a server-unload because that many (1171K = 1.2-Million) queries/requests are not being cache-pulled or sent upstream – on this one particular server.  (iptables numbers above were reset 60 minutes previous)

Slowly I turned, step by step, inch by inch
(shark by shark twisting in the wind)

I ran off on a statistics tangent and never completed the fail2ban new-regex howto.

The new-est /etc/fail2ban/filter.d/named-flood.conf needs to be up-to-dated:
# vim /etc/fail2ban/filter.d/named-flood.conf

[Definition]
failregex = .* named\[.*\]: client <HOST>\#.*: query: ((?i)1rip|1rip\.com|isc\.org|\.) (IN|ANY) *
ignoreregex =

:wq (write and quit)

Make a test file:
#vim /tmp/testfile.txt (press insert when it ‘loads: New File’)

Jul 11 05:40:22 server named[1301]: client 1.1.1.1#1: query: 1rip IN ANY +E ([ip of server])
Jul 11 05:40:22 server named[1301]: client 2.2.2.2#2: query: 1rip IN ANY +E ([ip of server])
Jul 11 05:40:23 server named[1301]: client 3.3.3.3#3: query: 1Rip IN ANY +E ([ip of server])
Jul 11 05:40:23 server named[1301]: client 4.4.4.4#4: query: 1rIp IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 5.5.5.5#5: query: 1riP IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 6.6.6.6#6: query: 1rip IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 7.7.7.7#7: query: 1rip.com IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 8.8.8.8#8: query: 1rIp.com IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 9.9.9.9#9: query: linenine.com IN ANY +E ([ip of server])

:wq (write and quit)

[root@server filter.d]# fail2ban-regex /tmp/testfile.txt named-flood.conf

Should get 8 “number of match”

Compare to grep-ing (note the spaces and escaped .s inside the single quotes)
grep -c -i ‘ 1rip ‘ /tmp/testfile.txt : 6
grep -c -i ‘ 1rip\.com ‘  /tmp/testfile.txt : 2
grep -c -i ‘ \. ‘  /tmp/testfile.txt : 0
grep -c -i ‘ isc\.org ‘  /tmp/testfile.txt : 0

Test and compare against the real thing:

Make a working copy:
[root@server filter.d]# cp /var/log/messages /tmp/x.txt

[root@server filter.d]# fail2ban-regex /tmp/x.txt named-flood.conf
Takes a while on this server, then:
Success, the total number of match is 106225

Compare to grep-ing (note the spaces and escaped .s inside the single quotes)
grep -c -i ‘ 1rip ‘ /tmp/x.txt : 81034
grep -c -i ‘ 1rip\.com ‘  /tmp/x.txt : 2977
grep -c -i ‘ \. ‘  /tmp/x.txt : 15917
grep -c -i ‘ isc\.org ‘  /tmp/x.txt : 6297
——————– 106225 all added together

Sure looks like this is working, rm all those test files in /tmp/, then:

# /etc/init.d/fail2ban restart

Because of the amount of sharks in this nado, you might (we have to) manually block some ip’s while fail2ban gets back in the race.  Once fail2ban is all caught up and ready to go up against the whirlwind of feeding-frenzied zombie-shark-bots, manually release those and let fail2ban do its thing.

One last piece of housekeeping in this bad movie: reset the counters.
#iptables -Z

About one minute later, late one Saturday afternoon:

num   pkts   bytes    target (rest snipped)
1     2059    175K    fail2ban-dnsflood
2      352   66583    fail2ban-maillogins

A whopping 17% of packet-traffic is NOT a DNS-DDoS-Flood packet.

Q: Why?
A: Because gmail, yahoo and aol have managed to get their servers listed in Uniform Resource Identifier Blacklist (URIBL) databases. **

The Q/A above leads to the P/S below (as distasteful as it may be).

Problem: gmail (google), yahoo and aol [and others] are URIBL blocked by the spam filters/servers, we all want email from our friends at gmail, yahoo and (yes, still) aol.
Solution: turn off URIBL blocking at the spam filters/servers. (Throw IN the bad bathwater with the good?)

Someday, maybe, the internet will be a better more friendly place to live.  Until then, a few more *Junk* emails every day.

** gmail, yahoo and aol do not provide ‘easy platforms’ for spammers, phishers and junk/bulk email proliferators; they get listed on these ‘block lists’ by their competitors, odd-balls that hate big companies and goofy, immature people that have nothing better to do.

June 6, 2013: need a new website for a customer.  “Let’s do wordpress!”  Boom #1: wordpress needs newer php.

June 6-8, 2013: Thinking, discussing, planning regarding: in-place updates never work.  “Just setup a new server all up to date and ready for wordpress.”

June 9, 2013: Smart I.T. peoples backup on Sunday, run a full system backup… Boom #2: backup hard drives marked “read only” (usually means damage or imminent failure).

June 9-16, 2013: 2 new (rebuild one, fire up and build the “backup equipment” been sitting idle since 2008 waiting for catastrophe) servers to dish DNS, one will be “live” server, one is a fall-back/backup.

June 9-16, 2013: The endless cascade of computer junk.  One thing leads to one more thing that reveals that other problem and so-on and so-on.

June 16, 2013 11:00PM: everything transferred from 3 servers onto/into the new pair.  Shutdown (shutdown -h now) “ninesix” (online since 1.2010) and “isp1100″ (since 2006?? 2005?? earlier??).  Unplug this, plug in that, check this check that… “Tomorrow, let’s do wordpress!”

June 17, 2013: Boom #3.  bind (DNS Server) SERVFAIL.  Zone file for MLD Computers ( mldragon.com, how cool is that ) marked with .err extension, all ‘other’ domains working (sort of), but DNS errors for one cause DNS errors for many.  Finally got passed bind/DNS problems (delete the zones and re-create is the hind-sight how-to).  ns3 DNS ports lost in the shuffle (they say forwarding and accepted, poof no reply).

June 17, 2013: Boom #4.  Lots of “little fixes” needed to make sites work/look like they used to.  All new server os and hosting software should not be mixed with all old web design.

June 18, 2013: Finally…
All of this and that and the others settled down enough: “Let’s do wordpress!”

So, testing it out here before trying on the customer site that started this ‘mess’ 12 days ago.