This is not a new thing, it has been going on for years since WordPress versions in the middle 3.xx-es.  It’s not a WordPress “bug” – it’s because PHP v5.3.xx went End Of Life almost 2 years ago (on 14 Aug 2014 – see: http://php.net/eol.php ).  And, see: https://wordpress.org/support/topic/wp-35-is-not-compatible-with-php-53-please-read

Now, here’s the deal.  Computers are [supposedly] cheap these days.  Supposedly.  So, [supposedly] you just buy some awesome new quad-this-or-that x64 with at least 128GB RAM and some number of TBs of storage to… host your 20-30 MB WordPress sites.  Because images, that’s why.  Oh yeah, then manually migrate 100 or so websites, databases, all the associated “stuff” and spend “a few minutes each” changing all of the myriad of things you had to do to those sites and config files to make it work a couple-few years ago.

Or: keep what you’ve got and deal with it until the new V-Lab-Super-Box is online and your “team” starts pushing things around in their spare time.  Right.

If you did google-bar the above image issue you’ll have discovered that it has gotten progressively worse the more versions away from good olde EOL PHP 5.3.xx WordPress has gotten.  We experienced this.  This page is being written in WordPress 4.2.8, the site that went to pot today auto-magic-ly updated itself to 4.5.2.  That site was having dreaded the “http Error” problem with images for over a year.  Now, today, kaput!  No images, no PDFs, no nothing through the [Add Media] [Upload] section.  We finally got a PDF to upload and link in a page by angrily clicking refresh and doing the same thing over and over (Add media, library, upload… select… repeat).  Once upon a time we could size images to “web friendly” sizes and do the Angry-Repeat click-ery – no more.  Kaput!

“Well,” says google-bar, “didn’t you know about the Media From FTP plugin?”  https://wordpress.org/plugins/media-from-ftp/  “No,” says us.  So, we give it a try and when you click [Activate] it happily tells you:  Only works with PHP version 5.4 or higher.

Then it happened, just before give-up-time, because: (1) had already uploaded via ftp the problem image; (2) had looked at the [Insert From URL] button hundreds of times today.  [Insert From Url] clicked, http://domainname.com/wp-content/uploads/2016/05/PictureName.jpg…  BAM! Done.

Now it doesn’t show up in the Library but it shows up in the page – and in Edit mode there’s the alignment stuff, room for a title and a caption, and the ability to make it a link to itself (full sized).  The only hang-up: it inserts the image with width and height set to it’s “real” values (giant in this case).  Last trick, get some server-side resizing done (on the fly stuff) and…  BAM! Done.

So, at long last, here is the long way around work-around for Great New WordPress on Great Olde Server[s] with incompatible PHP (and other things) versions.

• ftp upload your image (keep the WordPress method of folder naming in case you ever “migrate” the site to Great New Server[s])
• In the page or post: [Add Media] and [Insert From URL]
• In the page or post: [Edit] the image, make it a link to itself, add a caption, title and alignment if you wish.
• [Preview changes] and use whatever you use to see the on-screen height-width numbers you need to stop the server from sending the full-sized image to that thumbnail-sized box.  (We did Print-Screen, Paint, Select on a Windows computer.)
• Switch your WordPress Editor to [Text] (underlying HTML) and change the width and height properties of your IMG tag.  (We changed – width=”2592″ height=”1944″ – to – width=”662″ height=”496″.)

Finally, [Save] the page or post, and go shake that piggy-bank and see about that new high-speed quad or oct thing with all the latest of everything (including PHP).  Test runs on many devices and browsers: invisible server-side image resize and send (thanks, Apache) and post complete.  Not very “reflexive” but “good enough” for now.

Now that that is taken care of… a test… of WordPress 4.2.8

This file would not upload.

Above: try again, pig headed try again, try again… error, error, long slow progress bar – worked !  Not worth the trouble.

Last notes and link:  Thanks to WordPress ( http://www.wordpress.org ) for not updating us to death.  The new Quad is actually in place and in QA Testing mode – but, migration is not going to happen until a lot more testing is done.  Known bugs are better than unknown ones.

The digital world has out-run squirrel:

• Does not work fully with Internet Explorer 11
• Does not work fully with mac and iDevices (pod, phone, pad, OSX (**ok with Mozilla or firefox on OSX but no one runs that)
• Does not work fully with [An]Droid devices
• Does not work fully with Surface (trimmed IE 10/11)

So, 1.11.14 we installed and tested version 0.9.5 of roundcube.

Hosted at MLD/CMI? Check it out at [www.yourdomain.xyz]/rcmail or, for example:

http://www.computermedic.org/rcmail

Log in using your full email address as username (just like squirrel).

After your first log in you should go to the settings screen and turn on HTML composition and viewing and set your signature options (if you keep using roundcube).

Both work fine side-by-side, but you should not be logged in with both at the same time.

Link at the top/right soon.

It was a backdraft. Or the eye of the Zombie-Nado-Cane. When the bad-bots got some air around August 5th – hak4umz.net DDoS or DNS Amplification – fail2ban (and the servers) got burned.

Even the “eye-dee-keff-kuh-may” (TammyBelle’s God Mode Code for DOOM][ ) cheat didn’t help.  fail2ban got clobbered… ‘already banned’ every one second in the log and no more bans happening because 100s or 1000s of times per second from 100s or thousands of bots: bad requests.

Here is the Serious! problem when you put the fail2ban -vs- the entire globe death match together:

2013-08-13 12:40:37,730 fail2ban.actions: INFO   [named-flood] <ip> already banned
2013-08-13 12:40:38,732 fail2ban.actions: INFO   [named-flood] <ip> already banned

almost exactly one second apart, hundreds of times, and no new banning going on.
Q: Why?
A: fail2ban appears to have a “one second pulse/parse” clock built in to it.

Q: So?
A: So, when 4,000 log entries appear in a log that fail2ban is reading within that one second, fail2ban ‘queues’ (or spools or fifo’s) those 4,000 entries into an internal list and tries to de-queue them one-per-second.

Easier math: (“let’s say”) there are 10 ‘fail regex’ entries pouring into your log per second. Trying to de-queue the messages from the first second takes fail2ban 9 seconds.  By the time it gets done, there are 90 more messages/fails waiting.  So every second that goes by (in this low number scenario) the problem gets 10-to-the-10th-power worse.  The problem being fail2ban over-run by those headless bomb-toting zombies.  The “real world” explanation: fail2ban lags out and becomes combat ineffective.  In cop-talk the radio call from Officer fail2ban would be: “Extended”

Now, a “server admin” must consider – besides ‘shutdown -h now’ – is there a solution to the problem? First part of that: what – exactly – is the problem.  More Q/A (logic/reasoning):
Q: Problem?
A: fail2ban says ‘already banned’ and is ‘lagged out'; can’t fight the good fight.

Q: Why?
A: Too many log entries per second.  fail2ban reads logs and ‘actions’ based on log entries.

Q: So, why don’t you server admins just limit the number of log entries? (Instead of trying to hyper-tune fail2ban, just give it less to do? Remember the old-old server used to say ‘…the previous message repeated ### times…’)
A: Why didn’t I think of that.

The old-old server was a Gentoo box dragged across the millennium boundary by makes and make-installs.  It finally wore out (it still runs, it was just retired because it had done it’s duty) this year.  A little searching about ‘the previous message repeated’ and was reminded that that is called: rate-limit-ing.  A modern Centos-6-x86_64 install (not a bunch of custom compiled stuff on a 32-bit Gentoo) uses an ‘out of the box’ rsyslog and doesn’t say things like ‘…the previous message…’  The new stuff says:
imuxsock begins to drop messages from pid 1228 due to rate-limiting

Very little more searching finds:
http://www.rsyslog.com/tag/rate-limiting/

The docs are a ‘little dated’ (2010) but the essentials are there to solve the problem (problem being ‘too many log entries for poor old fail2ban’).

vim /etc/rsyslog.conf (and add as the 2nd and 3rd uncommented lines):

#### 8.12.13 - try to slow the message floods so fail2ban won't die so much ####
$SystemLogRateLimitInterval 1
$SystemLogRateLimitBurst 5

[Esc]:wq (write and quit)

Now do a /etc/init.d/rsyslog restart or service rsyslog restart (reload does not work, I tried it) and…

Tah-dah!  fail2ban can keep up with the log.  Some of the abusers (firey screaming zombies with tater-bombs) get by for a few seconds until the rate-limit/fail2ban get Serious!; but, real-world they were getting by by the hundres-of-thousands before this fix (while poor old fail2ban was over-run or lag-back-buffered).

It may not be ‘iddqd’ (god/degreelessness mode in ‘that other great fps’), but $SystemLogRateLimitInterval/$SystemLogRateLimitBurst is very close to TammyBelle’s “eye-dee-keff-kuh-may” (megaarmor, weapons and keys) for fail2ban.

Almost as good as Tangy-Bells for break-shishst.
Very happy, ammo added.

*** A minor success/victory ***
49 hours later… fail2ban chugging along ban/unban-ing, much smaller log files, no other services lagged out because of the packet attacks on port 53….

Equally exciting, terrifying, low-budget and prone to sequels.

So bad it’s good movie lovers, click the link above and see if you can survive that whirlwind of bites.

Server admins, stay right here and get ready for DNS-Zombie-Bots Two: More Tech-Talk and .configs Than You Can Stand!  (Or, “Bored To Death!” Or, “You can have the whole seat, but you only need the edge!”)

Or, I had to document it so I can take it from server to server without trusting my memory, so I thought I would share.

It started with a ‘Hay Bay-Bay’ – or a ‘clients-per-query’ message.

Lots of tweaks, tunes, service this restart, /etc/init.d/that restart later: ‘clients-per-query’ (increased/decreased) messages, lots of them.  (Somehow, sync’d between servers, trying to figure that BIND9 magic out would be like trying to reach into the mouth of one of those sharknado sharks and pull its heart out.  It is because it is, do the fixes you can do and worry about enigmatic synchronicity later.)

Here’s the setup again so when you try these things on a server with a point-zero-zero-one version difference you’ll know why it doesn’t work:
~ CentOS x86_64 6.4 (Installed, updated [yum update] June, 2013)
~ bind / named
* # rndc status: version: 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.4
* # yum list bind: bind.x86_64 32:9.8.2-0.17.rc1.el6_4.4
~ fail2ban
* # yum list fail2ban: fail2ban.noarch 0.8.8-3.el6 ( 0.8.10-1.el6 available )

Since last I wrote about it ( Killin Zombie Bots ) some seemingly minor, but very important changes mainly to the bind/named and related conf files.
/etc/resolv.conf: nameserver 127.0.0.1
~ all ‘in-server’ services should ask ‘self’ for DNS, when self doesn’t know it “recurses” (goes upstream) and caches so that for a time (cache TTLs and expirys) ‘self’ does know the answer.
/etc/named.conf (in the options { } block): querylog yes;
~ log at boot (that semi-colon ‘;’ is very-necessary)
/etc/fail2ban/jail.conf: ignoreip = 127.0.0.1/8
~ ‘confirmed’ (it is the default in the [default] section) – Self: don’t ban we.

clients-per-query groans the almost healed zombie server

Ask the modern zombie-to-English interpreter ( google ) what that means and the interpreter says:
About 2,440,000 results  (0.33 seconds)

Go on an Injun vision quest and consult the shaman: add more RAM.

Don’t run down that 2.44-Million results rabbit hole.  Smack your head and think I should have thought of that when the light bulb turns on in there behind the sign that reads:

The cache of a caching DNS server on a moderate-to-heavy-load server can get quite large.

That’s RAM bay-bays, nothing else.  Don’t believe yourself, # free. @125MB of 2GB left and look lower: @105648[k] used Swap:.  This is on the ‘dedicated’ DNS server.  Pop-Nerd-Quiz: Going swap happens when?  Right, when ‘real memory’ (RAM) is full.

Jump over to the we-have-it-no-matter-what-it-is-and-cheap shop of the new millennium ( eBay ) and [Buy It Now] on 8GB of the wrong RAM for your server.  Smack your head again because everything is ‘the hard way’ (like being the only seal in a Sharknado), put the wrong RAM up for sale and [Buy It Now-Now] on 8GB of hopefully the right RAM for your server.  When it shows up (it will be right this time!) hope that 8GBs is enough for a caching DNS server.

Summary: clients-per-query = add more RAM.

There is a nightmare-nado of other things you can try to tweak or tune or limit and shutdown -r now… or you can RAM-up and see all those Zombie-Language /var/log/ messages in your logs vanish.  If you are ‘flush with GB’ and nothing in swap: it’s 2.44 million curtains for you, tough guy (in 0.33 seconds)!

Z0mb13 t4Lk (or ID-10-T bot-writer cOdEs) and fail2ban

 1rip, 1Rip, 1rIP, and so-on.  case-insensitive, or ignore-case. Sounds so easy.  So you try a little (?i) and a little \/\.IhateRegEx (interpreted through .py and other things depending on revision or build number) and pretty soon you are standing in the eye of a shark-icane with your Mary Poppins umbrella waiting for the winds to take you to a hopefully quick and not too painful shark-shutdown (in the air).

begin here (/etc/fail2ban/filter.d dir):
[root@server filter.d]# cp named-flood.conf named-ignoretest.conf
[root@server filter.d]# vim named-ignoretest.conf

It has now (sorry for the wordpress word-wrap):

failregex = .* named\[.*\]: client <HOST>\#.*: query: (1rip\.com|isc\.org|\.) (IN|ANY) *

Based on the only search result that made sense ( https://github.com/fail2ban/fail2ban/issues/48 ) and ( http://www.tutorialspoint.com/python/python_reg_expressions.htm ) [and about 100 trial-with-error failures] change it to:

 failregex = .* named\[.*\]: client <HOST>\#.*: query: ((?i)1rip|1rip\.com|isc\.org|\.) (IN|ANY) *

The important thing here (besides this is not tested against any other versions): ((?i)[pipe separated list]).  The ‘ignore-case’ (?i) toggle is working on all of the entries in the [pipe separated list].  Another thing: I didn’t test and don’t care if the case-insensitive compare carries over to the (IN|ANY).

Because the only ‘spoof’ in there is 1rip.com (now case does not matter) some of those isc.org queries are still getting answered, and the (space)1rip(space) [1rip without a domain extension] are still doing something (as yet unknown) to the cache and the upstream.  What is known about those is that they are now successfully triggering fail2ban to shut those servers/ips down after a couple of hits and send the rest of their millions of attempts to >dev/null.

Doing packet/byte count watches ( #iptables -n -L -v –line-numbers ) reveals that once ‘dumped’ into the ‘fail2ban filter table’ the bad-zombie-bots (flooding w/requrests) are ‘dropping’ many hundreds of thousands of requests (packets) and GBs of data per hour.

2    3000K  864M fail2ban-dnsflood    all  -- *  *   0.0.0.0/0   0.0.0.0/0
3    1829K  792M fail2ban-maillogins  all  -- *  *   0.0.0.0/0   0.0.0.0/0

“It’s only” 72MB (@10% by bytes), but fully 39% of all packet-traffic being killed by this fail2ban zombie-net – on this one particular server.  Not sure how to ‘math it out’ but it is also a server-unload because that many (1171K = 1.2-Million) queries/requests are not being cache-pulled or sent upstream – on this one particular server.  (iptables numbers above were reset 60 minutes previous)

Slowly I turned, step by step, inch by inch
(shark by shark twisting in the wind)

I ran off on a statistics tangent and never completed the fail2ban new-regex howto.

The new-est /etc/fail2ban/filter.d/named-flood.conf needs to be up-to-dated:
# vim /etc/fail2ban/filter.d/named-flood.conf

[Definition]
failregex = .* named\[.*\]: client <HOST>\#.*: query: ((?i)1rip|1rip\.com|isc\.org|\.) (IN|ANY) *
ignoreregex =

:wq (write and quit)

Make a test file:
#vim /tmp/testfile.txt (press insert when it ‘loads: New File’)

Jul 11 05:40:22 server named[1301]: client 1.1.1.1#1: query: 1rip IN ANY +E ([ip of server])
Jul 11 05:40:22 server named[1301]: client 2.2.2.2#2: query: 1rip IN ANY +E ([ip of server])
Jul 11 05:40:23 server named[1301]: client 3.3.3.3#3: query: 1Rip IN ANY +E ([ip of server])
Jul 11 05:40:23 server named[1301]: client 4.4.4.4#4: query: 1rIp IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 5.5.5.5#5: query: 1riP IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 6.6.6.6#6: query: 1rip IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 7.7.7.7#7: query: 1rip.com IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 8.8.8.8#8: query: 1rIp.com IN ANY +E ([ip of server])
Jul 11 05:40:24 server named[1301]: client 9.9.9.9#9: query: linenine.com IN ANY +E ([ip of server])

:wq (write and quit)

[root@server filter.d]# fail2ban-regex /tmp/testfile.txt named-flood.conf

Should get 8 “number of match”

Compare to grep-ing (note the spaces and escaped .s inside the single quotes)
grep -c -i ‘ 1rip ‘ /tmp/testfile.txt : 6
grep -c -i ‘ 1rip\.com ‘  /tmp/testfile.txt : 2
grep -c -i ‘ \. ‘  /tmp/testfile.txt : 0
grep -c -i ‘ isc\.org ‘  /tmp/testfile.txt : 0

Test and compare against the real thing:

Make a working copy:
[root@server filter.d]# cp /var/log/messages /tmp/x.txt

[root@server filter.d]# fail2ban-regex /tmp/x.txt named-flood.conf
Takes a while on this server, then:
Success, the total number of match is 106225

Compare to grep-ing (note the spaces and escaped .s inside the single quotes)
grep -c -i ‘ 1rip ‘ /tmp/x.txt : 81034
grep -c -i ‘ 1rip\.com ‘  /tmp/x.txt : 2977
grep -c -i ‘ \. ‘  /tmp/x.txt : 15917
grep -c -i ‘ isc\.org ‘  /tmp/x.txt : 6297
——————– 106225 all added together

Sure looks like this is working, rm all those test files in /tmp/, then:

# /etc/init.d/fail2ban restart

Because of the amount of sharks in this nado, you might (we have to) manually block some ip’s while fail2ban gets back in the race.  Once fail2ban is all caught up and ready to go up against the whirlwind of feeding-frenzied zombie-shark-bots, manually release those and let fail2ban do its thing.

One last piece of housekeeping in this bad movie: reset the counters.
#iptables -Z

About one minute later, late one Saturday afternoon:

num   pkts   bytes    target (rest snipped)
1     2059    175K    fail2ban-dnsflood
2      352   66583    fail2ban-maillogins

A whopping 17% of packet-traffic is NOT a DNS-DDoS-Flood packet.

June 6, 2013: need a new website for a customer.  “Let’s do wordpress!”  Boom #1: wordpress needs newer php.

June 6-8, 2013: Thinking, discussing, planning regarding: in-place updates never work.  “Just setup a new server all up to date and ready for wordpress.”

June 9, 2013: Smart I.T. peoples backup on Sunday, run a full system backup… Boom #2: backup hard drives marked “read only” (usually means damage or imminent failure).

June 9-16, 2013: 2 new (rebuild one, fire up and build the “backup equipment” been sitting idle since 2008 waiting for catastrophe) servers to dish DNS, one will be “live” server, one is a fall-back/backup.

June 9-16, 2013: The endless cascade of computer junk.  One thing leads to one more thing that reveals that other problem and so-on and so-on.

June 16, 2013 11:00PM: everything transferred from 3 servers onto/into the new pair.  Shutdown (shutdown -h now) “ninesix” (online since 1.2010) and “isp1100″ (since 2006?? 2005?? earlier??).  Unplug this, plug in that, check this check that… “Tomorrow, let’s do wordpress!”

June 17, 2013: Boom #3.  bind (DNS Server) SERVFAIL.  Zone file for MLD Computers ( mldragon.com, how cool is that ) marked with .err extension, all ‘other’ domains working (sort of), but DNS errors for one cause DNS errors for many.  Finally got passed bind/DNS problems (delete the zones and re-create is the hind-sight how-to).  ns3 DNS ports lost in the shuffle (they say forwarding and accepted, poof no reply).

June 17, 2013: Boom #4.  Lots of “little fixes” needed to make sites work/look like they used to.  All new server os and hosting software should not be mixed with all old web design.

June 18, 2013: Finally…
All of this and that and the others settled down enough: “Let’s do wordpress!”

So, testing it out here before trying on the customer site that started this ‘mess’ 12 days ago.