What happened: clients could not connect.  It would start to, prompt for credentials, show the welcome screen, then a disconnect with a message:

Your Remote Desktop session has ended.

Another user connected to the remote computer, so your connection was lost. Try connecting again, or contact your network administrator or technical support group.

The only way around it was to physically “Sign out” at the hardware.

The fix: add a password to the account (see the notes below) and enable the “Allow only…NLA” (picture) option in the remote desktop settings.

How this all happened…

First: not a very well used Windows 8 “test” computer.  When we set it up originally we didn’t put a password on it.  We must have had some old Macs or XP boxes that we tried to connect RDP clients from, so had un-checked the “Allow connections only…” option in the Remote tab.

2nd: No “pre-release” Windows X stuff.  On July 29, 2015 we had to update this computer to 8.1 before upgrading to X.  The 8.1 and the X (Ten/10) installers did not mention exactly how important a password is to these newest systems.  A little online searching tells us that it’s not Microsoft’s fault, not too many people seem to have this exact series of mistakes/user errors.

Finally: we ended up with a Windows X (Pro) computer with RDC (the new name for RDP) enabled, old un-secure clients allowed, and no “only user” account password.

How this all got fixed…

First: After reading too many online posts and pages about RDP (RDC) in general we saw “a couple” of mentions about that “Allow connections only…” checkbox.  Went to see what was up: un-checked.  Put a check-mark in there and tried again…

Second: Secure connections require minimally secure computers (user account needs a password).  When we tried to connect this time we got a “Cannot connect” error stating that “…the password is expired… …or…” (about 3 total possibilities that we didn’t screenshot).  We knew there was no password so it wasn’t “expired.”  Back to the hardware, set a password on the user account, Sign out, Sign in…

Finally:  Add the new password to the RDP (RDC) client connection dialogue and everything works fine.  RD Client connected, console user was Signed out.

This was a very “anomalous behavior incident” but if we could do it someone else probably could as well.  So, now it’s documented.

You’re a bad bot. I’m tired of playing with you. I’m going to make you dead now.

July, 2014.  The internet is a secluded village, all controlled and terrorized by one boy…

Meet Anthony, from the ‘It’s a Good Life’ episode of The Twilight Zone (Nov. 1961).  Details over at imdb, or watch the whole episode (with modernized commercial/ad inserts) at hulu.

We’ve hired Anthony, now in his 50s to do away with spam, Zombie DNS DDoS Bots, and other such pests buzzing around and annoying or destroying everything and everyone in the internet play ground.  We should have thought of it earlier… Just making bad things dead or wishing them into the cornfield.

OK, not quite that easy, here’s what’s up in the fight against Spam-Nados and Zombie Bots…

[Thermal] Inversion Layers.  In the undersea world of submariners these separations between temperatures of water mask objects from sonar and/or, of course, thermal imaging.  What does that have to do with servers and DDoS attacks? Nothing.  Except the association is how we remember:

Inverse Tactics.  At some point around the Happy Holiday Season 2013-2014 (aka Christmas to normal people) a server manager woke from a horrible holiday dream – a dream of fail2ban memory leaks, dead sockets, out of memory errors and massive log files – with a vision of a less nightmarish future.  Simplicity was unmasked.  The fortune-cookie like note that comes from the bobble-head-devil fortune machine in that other episode of Twilight Zone:

Invert your tactics.

Instead of trying to use fail2ban (perl/python/loosely interpreted regexs) to sift through everything and lock out the bad guys after they had been bad, and possibly then wrong because UDP sources are easily spoofed; use the ‘firewall’ (compiled, close to the kernel, strict iptables) as a firewall.  Seems simple, use the firewall as the firewall. But that’s hind-sight.

So, simple, use the firewall, invert the tactics.  What does that mean?

It means: allow only certain things along the ‘port 53 chain’ ~ only the domains that we truly are the authoritative host for.  And, since our little web-server is also a recursive-caching DNS server for itself and others; only do this filtration on the ‘net-facing’ (public) adapter.  Confused? Good. Welcome to the Twilight Zone.

Inverted logic:
– Old way: allow everything, find anything ‘known bad’ and block that; ever growing list of ‘known bad’ (because the bad-bot guys change what they query you to death with)
– New way: allow only the domains we NS host to be allowed passed iptables to get to bind/named and furthermore to fail2ban; drop all other (not known good) DNS queries.  Yep; drop; no tarpit, no ‘channel[s] closed’, no loopback, no reply, no nothing, DEEeee-ROP. (Wish bad queries into the corn field.)

Simple. Smart. Effective.  Can’t find the notes to give credit to some very helpful websites/pages that helped us through this “seems simple (everything computer is supposed to be simple)” but was very, very complicated set of iptables rules. TODO for later: find the notes, give the credit.

Here’s the summary:
– iptables does string matching (the * here is that iptables string matching on UDP sockets is very complicated, based on hex conversions of the strings, very strict, but very fast)
– set up a new ‘table (chain)’ and set of ‘RETURN’ rules based on the domains we actually want to give answers to DNS queries for
– set up the new ‘table (chain)’ to default to DROP (not reject, not return) all others

Here’s the implementation, using the plain text matching of iptables instead of the complete domain.tld (the dots don’t translate so you have to use hex to match domain(special)tld):

#!/bin/sh
#
# create the chain to drop everything we don't specifically know
if ! iptables -L netdnswash -n  >/dev/null 2>&1 ; then
        #echo "debugging make the chain netdnswash"
        iptables -N netdnswash
fi
# flush it either way
iptables -F netdnswash
# default rule = DROP.  This is to stop trying to block badbots, only allow good stuff.
# This should also drop the '.' (single dot) query
iptables -A netdnswash -j DROP
### during build/testing return everything, watch the counters on the one-two test domains
### iptables -A netdnswash -j RETURN
# put the most common ones up top (e.g. computermedic.org since that's the name server, then down to the least busy)
# reverse order because inserts stack on top
iptables -I netdnswash -i eth1 -p udp -m udp --dport 53 -m string --algo bm --icase --to 255 --string 'mldragon' -m comment --comment "mldragon.com" -j RETURN
iptables -I netdnswash -i eth1 -p udp -m udp --dport 53 -m string --algo bm --icase --to 255 --string 'computermedic' -m comment --comment "computermedic.org always first" -j RETURN
# make sure this is first in the INPUT chain - returns to INPUT will allow fail2ban to catch crap.gooddomain.tld
# make sure that fail2ban (actions) build their rules at/after 2. More manual thinking for fail2ban (admins) but way less work
# make sure it doesn't already exists
if ! iptables -L INPUT -n | grep 'netdnswash' -c 2>&1 ; then
        #echo "debugging make the rule in INPUT"
        iptables -I INPUT 1 -i eth1 -p udp -m udp --dport 53 -j netdnswash
fi

Note 1: all chain creation prior to adding it to the INPUT chain rules.
Note 2: only traffic on the ‘net-facing/public’ adapter goes through here (at the bottom, …eth1…)
Note 3: used inserts ( -I ) so what you want first you have to put last in the script (invert again)
Note 4: once fully tested and put into operation: iptables-save >/path/to/conf_file so that if the server restart iptables doesn’t have to be ‘re-made’.
Note 5: change the fail2ban jail create/destroy commands to insert at 2, so fail2ban can be restarted without messing up the new iptables stuff.

Now, important, the lack of the complete domain.tld business.  If we had hundreds or thousands of Ns in our NS we would go through the trouble to use ‘hex string matching’ to get the whole computermedic.org thing in there.  But the “less than hundreds” number of domains on this server lets us deal with the occasional bad bot playing computermedic.tld [other than org] games.  How do we deal with that?

fail2ban of course.  Now, much relaxed and under-loaded because no more rolling/spoofed-source DNS/DDoS querries.  Again, don’t want to tell the bad-guys how to defeat our newly created defensive systems, but suffice it to say: if you run 3 or more ddosasia.computermedic.org or www999, www998, www997 (the rolling ones we encountered) queries against the server fail2ban is going to shut you down (1st time for 2 hours, 2nd time for… drumroll… Evuh!).  First the cornfield for a reasonable timeout, then fail2bAnthony makes you dead.

Less Spammy Inboxes version 14.7 (year.month versioning because there have been way too many versions)

I hate anybody that doesn’t like me!

Now that fail2bAnthony has less DNS problems on his hands, lots of free memory and CPU cycles, has not gone to swap in at least 3 months it’s time to put fail2bAnthony on task of wishing spambots into the cornfield, and those dreaded three-headed-gophers (the guess your password bots that then use your email to send spam everywhere ‘authenticated’): Dead.

fail2ban was already working the ‘mail servers logfiles’ to identify bad auth attempts and ban bad guys there.  However! We utilize an ‘Anti-Spam Relay/Filter Server’ (or several) and a great many of the bad auth attempts were going to those.  A couple of new fail2ban filters and actions, done.  Send legit email if you want copies of the fail2ban files, not publishing those (x-th time: don’t give the bad-guys a ‘how we defend ourselves’ map).  We’ll suffice it to say in this regard: had to get a few users new passwords, but ‘rejects’, bad server reputation points, and dictionary/rolling login attempts, junk in the mailq, against all mail systems are significantly decreased.  Will be even more decreased as time goes by because after certain re-bans, ips and whole ranges are banned for 1 year (all ports).  Cornfield, cornfield, dead.

A little over a year ago we said ‘Finally…’ ( http://www.computermedic.org/?p=9 ).  Doing all of this iptables, fail2ban, named (forgot to mention above, no more spoofed/loop-back entries in named conf files because no more bad domains/tlds allowed through), mail/spam server tuning revealed a major problem in the spam filter/relay setup: SenderBase did not install last year.  ‘Hypothetically speaking’ we could pretend that we use a certain Anti-Spam SMTP Proxy ( ASSP ) and we ‘hypothetically’ used their install/update scripts to get the thing moving.  Well, in those 12-18 days of extreme server configs and installs and the months of Zombie-Bot and DDoS-Nado wars after, the one little message in a log file seldom read about Net::SenderBase not being available was missed. Then there is the fact that ASSP ‘hypothetically’ does not log (even if debug level logging is selected) anything about SenderBase if it was not available at startup.  Where the problem lies: in some nix distros the install/update scripts, and the howto pages, that say use MCPAN to install Net::SenderBase don’t warn you to watch the output carefully to make sure there wasn’t any error[s].  Here’s the deal, and a warning: Watch the output of install Net::SenderBase carefully!!  If there are errors (any error means no SenderBase, no filters, regex matching, country matching, scoring, based on SenderBase in ASSP, and nothing to let you know): force install Net::SenderBase.

Bam. Done, had to go back an re-tweak/un-tweak about 12,000 ASSP (hypothetically) settings, logs, regex-es, etc., that had been tweaked trying to figure out why so much wasn’t working, gave ASSP a full stop (/etc/init.d/assp stop), let it rest a minute, fired it back up (replace stop with start) and wah-lah! 79% blocked instead of 45-51%.  Less spammy inboxes.

Updated wordpress to 3.9.1 today.
Not important but it works *great* in IE11.  Makes links correctly, which it did not.  Looks a little better.  Good job wordpress.

It was a backdraft. Or the eye of the Zombie-Nado-Cane. When the bad-bots got some air around August 5th – hak4umz.net DDoS or DNS Amplification – fail2ban (and the servers) got burned.

Even the “eye-dee-keff-kuh-may” (TammyBelle’s God Mode Code for DOOM][ ) cheat didn’t help.  fail2ban got clobbered… ‘already banned’ every one second in the log and no more bans happening because 100s or 1000s of times per second from 100s or thousands of bots: bad requests.

Here is the Serious! problem when you put the fail2ban -vs- the entire globe death match together:

2013-08-13 12:40:37,730 fail2ban.actions: INFO   [named-flood] <ip> already banned
2013-08-13 12:40:38,732 fail2ban.actions: INFO   [named-flood] <ip> already banned

almost exactly one second apart, hundreds of times, and no new banning going on.
Q: Why?
A: fail2ban appears to have a “one second pulse/parse” clock built in to it.

Q: So?
A: So, when 4,000 log entries appear in a log that fail2ban is reading within that one second, fail2ban ‘queues’ (or spools or fifo’s) those 4,000 entries into an internal list and tries to de-queue them one-per-second.

Easier math: (“let’s say”) there are 10 ‘fail regex’ entries pouring into your log per second. Trying to de-queue the messages from the first second takes fail2ban 9 seconds.  By the time it gets done, there are 90 more messages/fails waiting.  So every second that goes by (in this low number scenario) the problem gets 10-to-the-10th-power worse.  The problem being fail2ban over-run by those headless bomb-toting zombies.  The “real world” explanation: fail2ban lags out and becomes combat ineffective.  In cop-talk the radio call from Officer fail2ban would be: “Extended”

Now, a “server admin” must consider – besides ‘shutdown -h now’ – is there a solution to the problem? First part of that: what – exactly – is the problem.  More Q/A (logic/reasoning):
Q: Problem?
A: fail2ban says ‘already banned’ and is ‘lagged out'; can’t fight the good fight.

Q: Why?
A: Too many log entries per second.  fail2ban reads logs and ‘actions’ based on log entries.

Q: So, why don’t you server admins just limit the number of log entries? (Instead of trying to hyper-tune fail2ban, just give it less to do? Remember the old-old server used to say ‘…the previous message repeated ### times…’)
A: Why didn’t I think of that.

The old-old server was a Gentoo box dragged across the millennium boundary by makes and make-installs.  It finally wore out (it still runs, it was just retired because it had done it’s duty) this year.  A little searching about ‘the previous message repeated’ and was reminded that that is called: rate-limit-ing.  A modern Centos-6-x86_64 install (not a bunch of custom compiled stuff on a 32-bit Gentoo) uses an ‘out of the box’ rsyslog and doesn’t say things like ‘…the previous message…’  The new stuff says:
imuxsock begins to drop messages from pid 1228 due to rate-limiting

Very little more searching finds:
http://www.rsyslog.com/tag/rate-limiting/

The docs are a ‘little dated’ (2010) but the essentials are there to solve the problem (problem being ‘too many log entries for poor old fail2ban’).

vim /etc/rsyslog.conf (and add as the 2nd and 3rd uncommented lines):

#### 8.12.13 - try to slow the message floods so fail2ban won't die so much ####
$SystemLogRateLimitInterval 1
$SystemLogRateLimitBurst 5

[Esc]:wq (write and quit)

Now do a /etc/init.d/rsyslog restart or service rsyslog restart (reload does not work, I tried it) and…

Tah-dah!  fail2ban can keep up with the log.  Some of the abusers (firey screaming zombies with tater-bombs) get by for a few seconds until the rate-limit/fail2ban get Serious!; but, real-world they were getting by by the hundres-of-thousands before this fix (while poor old fail2ban was over-run or lag-back-buffered).

It may not be ‘iddqd’ (god/degreelessness mode in ‘that other great fps’), but $SystemLogRateLimitInterval/$SystemLogRateLimitBurst is very close to TammyBelle’s “eye-dee-keff-kuh-may” (megaarmor, weapons and keys) for fail2ban.

Almost as good as Tangy-Bells for break-shishst.
Very happy, ammo added.

*** A minor success/victory ***
49 hours later… fail2ban chugging along ban/unban-ing, much smaller log files, no other services lagged out because of the packet attacks on port 53….

Suddenly, in sync with Aug 1 (00:00:00) rolling in around the globe (asia, Europe, etc):
It stopped.

All of the ‘watch words’ we’ve been jamming in fail2ban vanished.  By 00:00:00 Eastern US: 1 ip in the 2 hour block box.  At 12:00:00 (pm) July 31: 1000+.

Whatever time-bomb went off and killed all the zombie bots – hoozah.

There is something very creepy about it, though.

When (in the zombie apocalypse wars) you get a ‘tides have gone out’ moment: fortify defenses.

Updated fail2ban from 8.8 to 8.10.  Lots of features and fixes noted at their site – hope so, have had some trouble with fail2ban 8.8 and this recent assault.

In the last 2 weeks we have put some smarter rules in effect: empty zones with ‘deny’ default in /etc/named.conf

// sema.cz 7.29.13 - 7.30.13 100s
zone "sema.cz" {
        type master;
        allow-query { none; };
        allow-transfer { none; };
        file "named.empty";
};

This, of course, adds some bloat to the logs ( /IN/ANY denied line ). But it doesn’t go upstream.

More later, if the DNS ddos returns, or if time permits.

******** 8.1.13 (later in the day) update *********

Still too quiet after 100s-of-millions of attempts over the last 2 months.

The couple that have appeared reveal that the updated fail2ban tries to be nicey-nice ( reject ) instead of dumping zombies into the pit of oblivion (drop).

The new fail2ban default is:
blocktype = REJECT –reject-with icmp-port-unreachable

So, I am trying in the jail.conf file:

action  = iptables-allports[name=dnsflood, blocktype=DROP]

Took about 1 minute for the last of the zombie-hee-cans to get DROP-ped.  Something went right today.

Equally exciting, terrifying, low-budget and prone to sequels.

So bad it’s good movie lovers, click the link above and see if you can survive that whirlwind of bites.

Server admins, stay right here and get ready for DNS-Zombie-Bots Two: More Tech-Talk and .configs Than You Can Stand!  (Or, “Bored To Death!” Or, “You can have the whole seat, but you only need the edge!”)

Or, I had to document it so I can take it from server to server without trusting my memory, so I thought I would share.

It started with a ‘Hay Bay-Bay’ – or a ‘clients-per-query’ message.

Lots of tweaks, tunes, service this restart, /etc/init.d/that restart later: ‘clients-per-query’ (increased/decreased) messages, lots of them.  (Somehow, sync’d between servers, trying to figure that BIND9 magic out would be like trying to reach into the mouth of one of those sharknado sharks and pull its heart out.  It is because it is, do the fixes you can do and worry about enigmatic synchronicity later.)

Here’s the setup again so when you try these things on a server with a point-zero-zero-one version difference you’ll know why it doesn’t work:
~ CentOS x86_64 6.4 (Installed, updated [yum update] June, 2013)
~ bind / named
* # rndc status: version: 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.4
* # yum list bind: bind.x86_64 32:9.8.2-0.17.rc1.el6_4.4
~ fail2ban
* # yum list fail2ban: fail2ban.noarch 0.8.8-3.el6 ( 0.8.10-1.el6 available )

Since last I wrote about it ( Killin Zombie Bots ) some seemingly minor, but very important changes mainly to the bind/named and related conf files.
/etc/resolv.conf: nameserver 127.0.0.1
~ all ‘in-server’ services should ask ‘self’ for DNS, when self doesn’t know it “recurses” (goes upstream) and caches so that for a time (cache TTLs and expirys) ‘self’ does know the answer.
/etc/named.conf (in the options { } block): querylog yes;
~ log at boot (that semi-colon ‘;’ is very-necessary)
/etc/fail2ban/jail.conf: ignoreip = 127.0.0.1/8
~ ‘confirmed’ (it is the default in the [default] section) – Self: don’t ban we.

clients-per-query groans the almost healed zombie server

Ask the modern zombie-to-English interpreter ( google ) what that means and the interpreter says:
About 2,440,000 results  (0.33 seconds)

Go on an Injun vision quest and consult the shaman: add more RAM.

Don’t run down that 2.44-Million results rabbit hole.  Smack your head and think I should have thought of that when the light bulb turns on in there behind the sign that reads:

The cache of a caching DNS server on a moderate-to-heavy-load server can get quite large.

That’s RAM bay-bays, nothing else.  Don’t believe yourself, # free. @125MB of 2GB left and look lower: @105648[k] used Swap:.  This is on the ‘dedicated’ DNS server.  Pop-Nerd-Quiz: Going swap happens when?  Right, when ‘real memory’ (RAM) is full.

Jump over to the we-have-it-no-matter-what-it-is-and-cheap shop of the new millennium ( eBay ) and [Buy It Now] on 8GB of the wrong RAM for your server.  Smack your head again because everything is ‘the hard way’ (like being the only seal in a Sharknado), put the wrong RAM up for sale and [Buy It Now-Now] on 8GB of hopefully the right RAM for your server.  When it shows up (it will be right this time!) hope that 8GBs is enough for a caching DNS server.

Summary: clients-per-query = add more RAM.

There is a nightmare-nado of other things you can try to tweak or tune or limit and shutdown -r now… or you can RAM-up and see all those Zombie-Language /var/log/ messages in your logs vanish.  If you are ‘flush with GB’ and nothing in swap: it’s 2.44 million curtains for you, tough guy (in 0.33 seconds)!

Z0mb13 t4Lk (or ID-10-T bot-writer cOdEs) and fail2ban

 1rip, 1Rip, 1rIP, and so-on.  case-insensitive, or ignore-case. Sounds so easy.  So you try a little (?i) and a little \/\.IhateRegEx (interpreted through .py and other things depending on revision or build number) and pretty soon you are standing in the eye of a shark-icane with your Mary Poppins umbrella waiting for the winds to take you to a hopefully quick and not too painful shark-shutdown (in the air).

begin here (/etc/fail2ban/filter.d dir):
[root@server filter.d]# cp named-flood.conf named-ignoretest.conf
[root@server filter.d]# vim named-ignoretest.conf

It has now (sorry for the wordpress word-wrap):

failregex = .* named\[.*\]: client <HOST>\#.*: query: (1rip\.com|isc\.org|\.) (IN|ANY) *

Based on the only search result that made sense ( https://github.com/fail2ban/fail2ban/issues/48 ) and ( http://www.tutorialspoint.com/python/python_reg_expressions.htm ) [and about 100 trial-with-error failures] change it to:

 failregex = .* named\[.*\]: client <HOST>\#.*: query: ((?i)1rip|1rip\.com|isc\.org|\.) (IN|ANY) *

The important thing here (besides this is not tested against any other versions): ((?i)[pipe separated list]).  The ‘ignore-case’ (?i) toggle is working on all of the entries in the [pipe separated list].  Another thing: I didn’t test and don’t care if the case-insensitive compare carries over to the (IN|ANY).

Because the only ‘spoof’ in there is 1rip.com (now case does not matter) some of those isc.org queries are still getting answered, and the (space)1rip(space) [1rip without a domain extension] are still doing something (as yet unknown) to the cache and the upstream.  What is known about those is that they are now successfully triggering fail2ban to shut those servers/ips down after a couple of hits and send the rest of their millions of attempts to >dev/null.

Doing packet/byte count watches ( #iptables -n -L -v –line-numbers ) reveals that once ‘dumped’ into the ‘fail2ban filter table’ the bad-zombie-bots (flooding w/requrests) are ‘dropping’ many hundreds of thousands of requests (packets) and GBs of data per hour.

2    3000K  864M fail2ban-dnsflood    all  -- *  *   0.0.0.0/0   0.0.0.0/0
3    1829K  792M fail2ban-maillogins  all  -- *  *   0.0.0.0/0   0.0.0.0/0

“It’s only” 72MB (@10% by bytes), but fully 39% of all packet-traffic being killed by this fail2ban zombie-net – on this one particular server.  Not sure how to ‘math it out’ but it is also a server-unload because that many (1171K = 1.2-Million) queries/requests are not being cache-pulled or sent upstream – on this one particular server.  (iptables numbers above were reset 60 minutes previous)

Slowly I turned, step by step, inch by inch
(shark by shark twisting in the wind)

I ran off on a statistics tangent and never completed the fail2ban new-regex howto.

The new-est /etc/fail2ban/filter.d/named-flood.conf needs to be up-to-dated:
# vim /etc/fail2ban/filter.d/named-flood.conf

[Definition]
failregex = .* named\[.*\]: client <HOST>\#.*: query: ((?i)1rip|1rip\.com|isc\.org|\.) (IN|ANY) *
ignoreregex =

:wq (write and quit)

Make a test file:
#vim /tmp/testfile.txt (press insert when it ‘loads: New File’)

Jul 11 05:40:22 server named[1301]: client 1.1.1.1#1: query: 1rip IN ANY +E ([ip of server])
Jul 11 05:40:22 server named[1301]: client 2.2.2.2#2: query: 1rip IN ANY +E ([ip of server])
Jul 11 05:40:23 server named[1301]: client 3.3.3.3#3: query: 1Rip IN ANY +E ([ip of server])
Jul 11 05:40:23 server named[1301]: client 4.4.4.4#4: query: 1rIp IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 5.5.5.5#5: query: 1riP IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 6.6.6.6#6: query: 1rip IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 7.7.7.7#7: query: 1rip.com IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 8.8.8.8#8: query: 1rIp.com IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 9.9.9.9#9: query: linenine.com IN ANY +E ([ip of server])

:wq (write and quit)

[root@server filter.d]# fail2ban-regex /tmp/testfile.txt named-flood.conf

Should get 8 “number of match”

Compare to grep-ing (note the spaces and escaped .s inside the single quotes)
grep -c -i ‘ 1rip ‘ /tmp/testfile.txt : 6
grep -c -i ‘ 1rip\.com ‘  /tmp/testfile.txt : 2
grep -c -i ‘ \. ‘  /tmp/testfile.txt : 0
grep -c -i ‘ isc\.org ‘  /tmp/testfile.txt : 0

Test and compare against the real thing:

Make a working copy:
[root@server filter.d]# cp /var/log/messages /tmp/x.txt

[root@server filter.d]# fail2ban-regex /tmp/x.txt named-flood.conf
Takes a while on this server, then:
Success, the total number of match is 106225

Compare to grep-ing (note the spaces and escaped .s inside the single quotes)
grep -c -i ‘ 1rip ‘ /tmp/x.txt : 81034
grep -c -i ‘ 1rip\.com ‘  /tmp/x.txt : 2977
grep -c -i ‘ \. ‘  /tmp/x.txt : 15917
grep -c -i ‘ isc\.org ‘  /tmp/x.txt : 6297
——————– 106225 all added together

Sure looks like this is working, rm all those test files in /tmp/, then:

# /etc/init.d/fail2ban restart

Because of the amount of sharks in this nado, you might (we have to) manually block some ip’s while fail2ban gets back in the race.  Once fail2ban is all caught up and ready to go up against the whirlwind of feeding-frenzied zombie-shark-bots, manually release those and let fail2ban do its thing.

One last piece of housekeeping in this bad movie: reset the counters.
#iptables -Z

About one minute later, late one Saturday afternoon:

num   pkts   bytes    target (rest snipped)
1     2059    175K    fail2ban-dnsflood
2      352   66583    fail2ban-maillogins

A whopping 17% of packet-traffic is NOT a DNS-DDoS-Flood packet.

First [is first ~LK]: an apology to any “upstream” DNS servers that our pet zombie bots may have passed bad requests to.  Like us, you probably didn’t know it was happening.

“What’s happening?!”
~ Dana (Dominique Dunne, Poltergeist, 1982)

Thousands Hundreds of thousands Millions Hundreds of millions of bad DNS requests.

Let me start at the beginning.  All “new” servers at MLD (computermedic.org) – weblogger here.

Checking on things (scanning logs) on the new equipment to see how it was doing: a new message in /var/log/messages: named[1584]: clients-per-query decreased to 61 (as low as the 20s). Never heard of it (the old servers had bind/named logging disabled).  A little search finds: # rndc querylog (turn on logging for bind).

# tail -f /var/log/messages
Boom.  Screens scrolling by too fast to see what’s going on.
Jun 27 06:55:58 nineseven named[1584]: client 183.178.216.210#12673: query: 1rip IN A + (70.63.178.157)
Jun 27 07:29:28 nineseven named[1584]: client 213.128.75.196#41712: query: 1rip.com IN ANY +E (70.63.178.157)
…and a bunch of query: DDoS.asia (how inconspicuous)

The immediate problems (why these goofy old internet attacks work):
1. Nobody watch-dogs their name servers (if they – or we – did, we would have known this was happening).  By default, bind/named do not log.  Most people running a name server are running secondary to a web, email or app server; it’s just part of the setup and quickly forgotten.
2. Name Servers (mostly) pass queries “upstream” when they do not have an answer themselves. This is the big deal: Bot->to->ns1.computermedic.org “what is the ip address of 1rip?” ns1 does not know, tries to be overly helpful and “I’ll find out and get back to you” asks an upstream server “can you tell me the address of 1rip so I can tell bot?”; Well, what if upstream server doesn’t have an answer?  This continues (upstream processing / passing of queries) until either a timeout occurs or some server gives an answer – even if the answer is: 1rip doesn’t have any DNS, thank you.
3. #2 thousands of times per second.  The only ‘symptom’ on a new/modern server with a new/modern OS: clients-per-query decreased

#4: How do ‘we’ know #1 is true – we didn’t catch this for all these years – and our upstreams have never called or sent email to say “did you know that your servers are bombing ours with upstream queries?”  We are unwitting zombies in the zombie-internet-apocalypse.

The decided upon solution: Stop this madness. Call the internet police and have them write a ticket and issue a “strongest terms” warning to all of the other did-not-know-they-were-zombies-zombies.  They can just slap the old bracelets on those rogue bots, they need to be jailed anyway.

There is no such thing as internet police – call the server admins, work out what you can do with the tools that you have.  Solution: fail2ban, iptables firewall, spoof zones.

The problems with the solution (.plan): fail2ban, iptables, and bind (real zones, let alone spoofs) are extremely complicated to setup and use; and, equally frustrating in their levels of “should work, but doesn’t work.”  The ‘another’ problem(s) with the solution (.plan): we use a “web interface server manager” that wants to control everything about the server and is prone to overwriting changes you make directly to the machine. The server manager has its own firewall rules and implements them using iptables.  That is “sort of” good, because iptables is already up and running and known to be working.

.plan in action part 1 – fail2ban

 See who, what, where, why and how about fail2ban here: http://www.fail2ban.org

Then search with your favorite flavor of search engine about fail2ban howto, and those evil-incarnate regexs (regex->py(thon) style->fail2ban interpreted). Then come back here for the step-by-step.

Our test this .plan server: 2-Core Intel processor (old), CentOS x86_64 6-point-something, bind and fail2ban installed with yum.  Will the steps work with your (other than CentOS 6 basically OOB)? Probably not, every OS distributor likes to move files around or change some trivial word (directories, folders, libraries?) to make their OS “better”.  The ‘paths’ and ‘stuff’ below are for our server, they “should be” nearly the same on yours.

First [is FIRST, Lilly Kate says], make sure bind/named are logging.  We turned it on “by hand” and have not yet configured the .conf ( /etc/named.conf or /etc/named.conf.local ) to do the logging on boot.

# rndc querylog

Now, tail -f /var/log/messages and make sure you start to see some [named] entries.

If you do not see any bad guy messages or floods, stop, don’t sweat it, check your logs next month.

The first-worst zombie-bot-flood was/is queries for: 1rip.com
After 2 hours of logging: #grep -c ‘1rip’ /var/log/messages
100025

Now, once you start to try to figure out fail2ban you’re going to find about a-million weblogger (moron) pages that tell you: RTFM.  The Linux nerds love that, translated from nix-nerd to English RTFM means “I don’t know either but I like writing that on boards and web pages.”

Here’s TFM for this particular problem and this particular fail2ban .plan (of course, with the particular setup/OS mentioned previously):

# vim /etc/fail2ban/filter.d/named-flood.conf
(new file… press insert to get into insert mode)
[Definition]
failregex = .* named\[.*\]: client <HOST>\#.*: query: 1rip.com IN *
ignoreregex =
:wq

I spent better than 2 hours testing, trying, RTFM-ing, playing with, etc. that failregex and fail2ban-regex (their testing tool). If your log lines look any-at-all different than ours did (scroll way up) then this regex won’t work for you – you have to tweak it and test it.  Copy a few lines from your log to a file ( if you don’t know how to do that, you really should not be messing with servers and firewalls ); make sure you have some ‘good’ DNS responses in the file.
# vim /tmp/testfail2regex.txt (new file, insert/paste some lines, :wq).
# fail2ban-regex /tmp/testfail2regex.txt /etc/fail2ban/filter.d/named-flood.conf

OK, I’ll assume if you are still reading your test said: some number of hits / matches.

( add the following at the bottom of the file )
( edit 7.1.13 – use the example at the bottom of this page )
# vim /etc/fail2ban/jail.conf
[named-flood]
enabled = true
filter = named-flood
logpath = /var/log/messages
action = iptables-allports[name=dnsflood]
bantime = 600 ; start with 10 mins
findtime = 1
maxretry = 1
( :wq to write and quit )

Do NOT miss or skip this step, fail2ban’s default iptables rules default to tcp – DNS is udp and if you don’t fix the action file fail2ban bans all tcp, DNS flood will continue.

# vim /etc/fail2ban/action.d/iptables-allports
#protocol = tcp (this # is not the nix command prompt, it’s a comment)
protocol = all
( :wq to write and quit )

# /etc/init.d/fail2ban restart

That’s (almost) it. You should start seeing ( tail -f /var/log/fail2ban.log ) bans and unbans.

Why the 10-minute ( bantime = 600 ) wall?  Why not forever, or a month? Because, reminder: our servers were sending these requests to upstream DNS resolvers for years.  We never got locked/blocked from anywhere.  Ten minutes is a “good/fair start” – hopefully if these are requests from real servers (not all zombie bots) they’ll end up fixed.  And, another “to remember” is that if my “upstream providers” ban me forever, on all ports, how am I going to get legitimate DNS requests resolved?  It’s a very complex problem/solution scenario (in the real world).

The last step of part 1 (fail2ban) is to fix a mis-configuration “out of the box” with fail2ban’s own logs.  The “default” conf (configuration file) has logging set to /var/log/fail2ban.log.  That’s AOK with us.  The default /etc/logrotate.d/fail2ban (installed by/with fail2ban) has a line that tells fail2ban to use SYSLOG (/var/log/messages) after rotation.  Don’t want that so:

# vim /etc/logrotate.d/fail2ban
(change the size from 30k to a bigger/better number – don’t want 100s of fail2ban-date.log files)
size 500k
(delete [dd] these two lines 6.29.13: correction-change the postrotate command)
postrotate
/usr/local/bin/fail2ban-client set logtarget /var/log/fail2ban.log 2>/dev/null || true
( :wq )

( edit 7.1.13 – use the example at the bottom of the page )
Your finished /etc/logrotate.d/fail2ban file should be:
/var/log/fail2ban.log {
missingok
notifempty
size 500k
create 0600 root root
postrotate
/usr/local/bin/fail2ban-client set logtarget /var/log/fail2ban.log 2>/dev/null || true
endscript
}

.plan part 2: spoofed zone – stop sending upstream

The only “hitch” with the fail2ban solution (using fail2ban for anything) is that it is an “earn our dis-trust” practice.  Meaning: you let anyone and everyone in until they do something (accidentally or not) that gets them thrown out.  Running internet servers is a lot like running a bar – just more drunks.

At our “digital Cheers” we run a clicky-easy administration program.  Clicky-add a DNS Zone.  Here, again, problems.  Clicky-easy does spell checking and IP validation and all kinds of neat “don’t let someone make server-death mistakes” because they think clicking a mouse makes them an RTFM-posting Certified Server AdMInIsTraTOR.  If we write our own zone files or mod the .conf files of bind/named – Clicky-easy will overwrite our “mastery” the next time we make changes.

So.  Work around our own watch dogs.

Why, again?  Because we want our servers to answer the zombie-bots or “downstream” requests ( for 1rip.com in particular ) with “I know the answer to that!!!” (but it’s a CIA dis-information campaign) rather than becoming an in-stream zombie ourselves.  Let me try to rephrase that:
Zombie-Bot asks us: 1rip.com?
Right now, since fail2ban is up and only suffers a few seconds lag time:
we (named) finds no local ‘zone file’ (“I dunno”) so passes those few 1rip.com?s upstream.
Right now, what “we” want to happen:
we (named) finds a local ‘zone file’ and answer the bad-zombie-bot (or less-informed-down-stream):
1rip.com? I know them – look deep inside your self Clarisse! ( spoof the reply, 1rip.com = 127.0.0.1 )

So. Work around our own watch dogs: Clicky-easy will not let you enter (lo, loopback, localhost, 127.0.0.1) or any other “bad” stuff into a zone file.  Reformulate the .plan, write it out in advance (WTFM so you can RTFM), try it on a non-production server, then go at it, fast, because you’re messing with a production server now.

Take down the named, you don’t want to give ‘good answers’ to those ‘bad requests’.
# /etc/init.d/named stop

Clicky-Easy-Server-Admin->
Add a Zone (Wizard), all real information to pass ‘muster’ (spell check and validation)
Save Zone, logout of Clicky-Easy, exit Clicky-Easy
Wait for it… there it is…
/var/named/1rip.com.zone
*** not the real file name, don’t want to give away too much

# vim /var/named/1rip.com.zone (use your real file name)
$TTL        3600
@       IN      SOA     ns1.1rip.com. null.1rip.com. (
2013062706       ; serial, todays date + todays serial #
7200              ; refresh, seconds
540              ; retry, seconds
604800              ; expire, seconds
86400 )            ; minimum, seconds
;
1rip.com. 3600 A        127.0.0.1
1rip.com. 3600      NS        ns1.1rip.com.
ns1 3600 A        127.0.0.1
( :wq to write and quit )

# /etc/init.d/named start

Using a very different clicky-easy Data-Base admin program, go into the db for clicky-easy-server-admin and change/remove it’s version of the ‘bad zone’ data.
Note to self: DO NOT use clicky-easy to open/view/edit/save Spoofed-Zone Domains/DNS.

******** postrotate ********

 6.28.13: The .plan in .effect is working.  Instead of ~100,000 1rips per hour (2 ns servers) it has fallen off to about ~100/hour at the primary.  Yes, there’s a lot of bloated log files (so what, 3TB drives are in the US$100 range right now); yes, there is some cpu overhead to fail2ban (0.0,0.1,0.0 last check); yes, the 10 minute kick is probably not enough (90 % of the bans in the fail2ban.log happen one second after that zombie-bot-ip was Unban-ed) – but, hay, 10% went away forever in one day.

The “way up side”: absolutely no more “passed upstream requests” for 1rip from us to our upstreams.  No more “our side” delays waiting for responses or timeouts from upstream.  Last night clients-per-query reduced to 99.

The ‘couple of other things’ that need to be done (or could be), jail a couple more of the offensive requests (no need to spoof dns for DDoS.asia – only about 1000 attempts in the last 24 hours).

The simple-est solution is to copy/paste/edit the fail2ban rule file:
# vim /etc/fail2ban/filter.d/named-flood.conf
(yy to yank the failregex line, p to paste a copy, change the badguy name)
[Definition]
failregex = .* named\[.*\]: client <HOST>\#.*: query: 1rip.com IN *
.* named\[.*\]: client <HOST>\#.*: query: DDoS.asia IN *
ignoreregex =
:wq

# /etc/init.d/fail2ban restart -or- # fail2ban-client restart

There are ways to ignore case and macro-expand the regex’s – (?this|that) – not too vastly interested in it.  There are really not that many different variations coming in right now (absolutely zero ddos.asia; all DDoS.asia) [ grep -c ‘DDoS.asia’ /var/log/messages = 1070 and grep -i -c ‘DDoS.asia’ /var/log/messages = 1070 ].  If you want to “get all fancy” then see fail2bans apache-badbots.conf file.

– – – – – 6.28.13 Late Update – – – – –

A 2nd one appears today (been in the logs a couple of (hundred thousand) times): isc.org

Found: http://my.opera.com/jlouisbiz/blog/2013/05/14/blocking-amplified-dns-attack-by-the-ip-address-and-using-linux-firewall-softwar

# vim /etc/fail2ban/filter.d/named-flood.conf
(yy to yank the failregex line, p to paste a copy, change the badguy name)
[Definition]
failregex = .* named\[.*\]: client <HOST>\#.*: query: 1rip.com IN *
.* named\[.*\]: client <HOST>\#.*: query: DDoS.asia IN *
.* named\[.*\]: client <HOST>\#.*: query: isc.org IN *
ignoreregex =
:wq

# /etc/init.d/fail2ban restart -or- # fail2ban-client restart

Stretched the ban time out to 20 minutes.

July 1, 2013: Some notes about how this has continued:
– it has only gotten worse (another post coming)
– had ‘timing’ issues with fail2ban (blahblah already banned 1000s of times)
– stretched the jail time out to 2 hours (sorry infected-good-guys id’d as zombie-bad-guys)
– got errors trying to use apache-bad-bots style ‘expansion’ so ended up with:

/etc/fail2ban/jail.conf

# DLW 6.27.13 - should use .local file? in a hurry to stop flood attack, sync/ddos
[named-flood]
enabled = true
filter  = named-flood
logpath = /var/log/messages
action  = iptables-allports[name=dnsflood]
bantime = 7200  ; start with 10 mins, bumped to 20, bumped to 2 hours 6.30
findtime = 2
maxretry = 1

The findtime of “2” stopped fail2ban from tripping over its own feet (stuck in an “…already banned” loop).

/etc/fail2ban/filter.d/named-flood.conf

[Definition]
failregex = .* named\[.*\]: client <HOST>\#.*: query: (1rip\.com|isc\.org) IN *
ignoreregex =

B-b-b-buh-but wait, there’s more! fail2ban’s default install and documentation have another ‘glitch':
the logs don’t postrotate because they put the wrong path to the right file in their script.

[root@servername~]# /usr/local/bin/fail2ban-client set logtarget /var/log/fail2ban.log
-bash: /usr/local/bin/fail2ban-client: No such file or directory
[root@servername~]# /usr/bin/fail2ban-client set logtarget /var/log/fail2ban.log
Current logging target is:
`- /var/log/fail2ban.log
[root@servername ~]# tail -f /var/log/fail2ban.log
2013-07-01 23:19:28,262 fail2ban.server : INFO
------------>   Changed logging target to /var/log/fail2ban.log for Fail2ban v0.8.8

Notice the absence of /local/ in the path.  One more tidbit fixed.
/etc/logrotate.d/fail2ban

/var/log/fail2ban.log {
    missingok
    notifempty
    size 500k
    create 0600 root root
    postrotate
      /usr/bin/fail2ban-client set logtarget /var/log/fail2ban.log 2>/dev/null || true
    endscript
}

/etc/logrotate.conf has the defaults for rotate and other settings

!!! Note: isc.org is a real deal – not some left over bad-bot junk.  They make bind (DNS Name Server) and all kinds of good stuff so that we can have these internets without typing IP addresses for everything.  Because they are ‘real’ (unlike 1rip.com) we did not attempt to ‘spoof’ their zone – the floods were amazing, now were passing up a few 1000 ‘snuck by fail2ban’ requests.  But, sorry bad-zombie-bots – we don’t server your kind here (aka we don’t run a free-for-all DNS server, you zombies will have to flood someone else).

There is still another post coming about this madness, but it’s a different wopper all together.