There are things you should not do as a system administrator, one of which is not to copy, but we all do it as it saves time – this blog entry has the story when it went wrong, no badly but I lost many, many hours to debug and correct the (small) error.
The Story of Thou Shalt Not Copy
A few years back I had to create a new CentOS based server. As many of the tasks where the same as one of the machines I was looking after I rsync’ed it from an (rsync based) back up I do very regularly – see this entry about my cheap snapshot system. The source system had many of the task similar (e.g. Samba, Mail server, internal webserver) but it also had all of the latest patches and updates for all of the RPM’s – so copying it was a huge time saver.
All I had to do after the rsync had finished
- find all source ip-addresses in the /etc tree and replace it with the target ip-address
- find all source hostnames in /etc tree and replace it with the target hostname
- adjust some files to suit the new needs (sendmail, apache, named, ntp etc)
- adjust the firewall to suit the new LAN
- delete users and add new users (unix and samba)
- etc
The entire copy and adjust operation went smoothly, no problem of moving the target machine to it’s new location and firing it up – from memory there was no problem at all. I just added new users and disks for the /samba and /home directories, that was it.
Zoom a few years later … Monday late afternoon this week I was starting receiving
SOURCE_HOST: Host denied: sshd from 43.255.189.72(unknown)-43.255.189.72 (0)
Additionally to the firewall rules I have tcp-wrappers setup to allow certain networks only.
My hosts.deny contains
ALL : ALL : spawn ( /usr/sbin/safe_finger -l @%h | /usr/bin/Mail -s "SOURCE_HOST\: Host denied\: %d from %c(%u)-%h" email@some.place.com) &
A quick search revealed
- it was coming from somewhere in Hongkong
- the address was in the bad books everywhere
- it was keeping knocking at a slow rate (I have throttling setup)
so I had to do something.
Since the machine “SOURCE_HOST” is located behind a DMZ and Firewall I checked the traffic on the gateways, but no traffic from that network was coming in through the gateways, even using deep packet inspection – which made me a little uncomfortable as you can imagine … I checked eth0 on the SOURCE_HOST – nothing. I also turned off a VPN, still not change but I kept the VPN turned off.
The first thing I did was to disable all WIFI modems, NAS drives, solar power data collector, etc. – anything that had some OS build in to make sure it wasn’t coming from an internal, hacked device – but the packets were still coming.
Then I checked when everybody had left and I was alone in the company and everything (and I mean everything) was turned off, only servers were still running – the packets were still coming – so the only place could be the SOURCE_HOST itself as at that stage I had turned off everything, even the phone system, alarm system – the lot.
I asked for help on a few mailing lists (e.g. luv.asn.au) and a few people made some suggestions.
Amongst the suggestions I employed a range of utilities on the SOURCE_HOST including chkrootkit, rkhunter, unhide, file comparisons etc.
What made the thing a little tricky was the slow rate the packets were coming in, once every 10..15 minutes or so and not steady, sometime 2 packets in 5 minutes, sometime 2 packets in 20 minutes – but I was not prepared to kill the throttle in the firewall for ssh packets, considering that the machine was in the bad books.
But NOTHING was revealed.
I ran all the checking utils on the other server just to be sure, NOTHING was revealed.
The Cause (or Better the Source) of the Problem
On Sunday last week one of my sons wanted to install his latest game on our PS3 and it did no work. We first thought it was the game servers that had the problem as on the day there were loads of messages on the boards. As it turns out it was the firewall of the gateway on our home network that did not let required packets through.
On Monday early morning last week I checked the game servers and what ports were required, so I added those missing ports to the rules … the game starting loading and updating, a couple of hours later he was playing the game.
Because I could not figure out my problem above and I remembered the change I made to the rules of the firewall I thought I check the incoming traffic on that gateway … and to my surprise the traffic was coming in from 43.255.0.0/16.
After a little digging I realized my problem, I had forgotten to change the name of the host in the hosts.deny file from the original SOURCE_HOST to the TARGET_HOST after the copying – after I changed the name the logs were finally reporting the correct machine and debugging took 5 minutes – fixing took a little longer, see below.
The reason why the ssh packets got in (and other packets) was a comma “,” that I placed incorrectly:
/sbin/iptables -v -m multiport -p tcp -A input --dport 7500:10000,20000:25000,40000:45000 -j ACCEPT
which means if EITHER incoming or outgoing is any of the ports (and range) mentioned, let it through. So if anyone connects to the gateway connecting to SSHD with a source address of the port ranges specified it was LET THROUGH.
Whenever I add/change/delete firewall rules I always use an external server I have access to (not related to our domains) or my laptop with (now 4G) modem and nmap/wireshark etc to check whether everything is fine – not that Monday morning.
I completly rewrote the iptables rules, off course!
Moral of the Story
- Don’t (re-)write firewall rules when you have to rush to work even if it’s for the most important people in your life.
- Thou shalt not copy – if you have to do it you need to be really, REALLY, REALLY cautious!
- Check!
- Check!
Two things that helped me
- Besides firewall rules I have other preventions installed, e.g. tcp-wrappers
- SSHD is throttled
Let good coding be with you.