On Sun, 2012-01-01 at 14:23 -0800, Bennett Haselton wrote:
> (Sorry, third time -- last one, promise, just giving it a subject line!)
>
> OK, a second machine hosted at the same hosting company has also apparently
> been hacked. Since 2 of out of 3 machines hosted at that company have now
> been hacked, but this hasn't happened to any of the other 37 dedicated
> servers that I've got hosted at other hosting companies (also CentOS, same
> version or almost), this makes me wonder if there's a security breach at
> this company, like if they store customers' passwords in a place that's
> been hacked. (Of course it could also be that whatever attacker found an
> exploit, was just scanning that company's address space for hackable
> machines, and didn't happen to scan the address space of the other hosting
> companies.)
>
> So, following people's suggestions, the machine is disconnected and hooked
> up to a KVM so I can still examine the files. I've found this file:
> -rw-r--r-- 1 root root 1358 Oct 21 17:40 /home/file.pl
> which appears to be a copy of this exploit script:
> http://archive.cert.uni-stuttgart.de/bugtraq/2006/11/msg00302.html
> Note the last-mod date of October 21.
>
> No other files on the system were last modified on October 21st. However
> there was a security advisory dated October 20th which affected httpd:
> http://mailinglist-archive.com/centos-announce/2011-10/00035-CentOSannounce+CESA20111392+Moderate+CentOS+5+i386+httpd+Update
> https://rhn.redhat.com/errata/RHSA-2011-1392.html
>
> and a large number of files on the machine, including lots of files in */
> usr/lib64/httpd/modules/* and */lib/modules/2.6.18-274.7.1.el5/kernel/* ,
> have a last-mod date of October 20th. So I assume that these are files
> which were updated automatically by yum as a result of the patch that goes
> with this advisory -- does that sound right?
>
> So a couple of questions that I could use some help with:
>
> 1) The last patch affecting httpd was released on October 20th, and the
> earliest evidence I can find of the machine being hacked is a file dated
> October 21st. This could be just a coincidence, but could it also suggest
> that the patch on October 20th introduced a new exploit, which the attacker
> then used to get in on October 21st?
> (Another possibility: I think that when yum installs updates, it
> doesn't actually restart httpd. So maybe even after the patch was
> installed, my old httpd instance kept running and was still vulnerable? As
> for why it got hacked the very next day, maybe the attacker looked at the
> newly released patch and reverse-engineered it to figure out where the
> vulnerabilities were, that the patch fixed?)
>
> 2) Since the */var/log/httpd/* and /var/log/secure* logs only go back 4-5
> weeks by default, it looks like any log entries related to how the attacker
> would have gotten in on or before October 21st, are gone. (The secure*
> logs do show multiple successful logins as "root" within the last 4 weeks,
> mostly from IP addresses in Asia, but that's to be expected once the
> machine was compromised -- it doesn't help track down how they originally
> got in.) Anywhere else that the logs would contain useful data?
----
the particular issue which was patched by this httpd (apache) update was
to fix a problem with reverse proxy so the first question is did this
server actually have a reverse proxy configured?
My next thought is that since this particular hacker managed to get
access to more than one of your machines, is it possible that there is a
mechanism (ie a pre-shared public key) that would allow them access to
the second server from the first server they managed to crack? The point
being that this computer may not have been the one that they originally
cracked and there may not be evidence of cracking on this computer.
The script you identified would seem to be a script for attacking other
systems and by the time it landed on your system, it was already broken
into.
There are some tools to identify a hackers access though they are often
obscured by the hacker...
last # reads /var/log/wtmp and provides a list of users, login date/time
login duration, etc. Read the man page for last to get other options on
its usage including the '-f' option to read older wtmp log files if
needed.
lastb # reads /var/log/btmp much as above but list 'failed' logins
though this requires pro-active configuration and if you didn't do that,
you probably will do that going forward.
looking at /etc/passwd to see what users are on your system and then
search their $HOME directories carefully for any evidence that their
account was the first one compromised. Very often, a single user with a
weak password has his account cracked and then a hacker can get a copy
of /etc/shadow and brute force the root password.
Consider that this type of activity is often done with 'hidden' files &
directories. This hacker was apparently brazen enough to operate openly
in /home so it's likely that he wasn't very concerned about his cracking
being discovered.
The most important thing to do at this point is to figure out HOW they
got into your systems in the first place and discussions of SELinux and
yum updates are not useful to that end. Yes, you should always update
and always run SELinux but not useful in determining what actually
happened.
Make a list of all open ports on this system, check the directories,
files, data from all daemons/applications that were exposed (Apache?
PHP?, MySQL?, etc.) and be especially vigilant to any directories where
user apache had write access.
Again though, I am concerned that your first action on your first
discovered hacked server was to wipe it out and of a notion that it's
entirely possible that the actual cracking occurred on that system and
this (and perhaps other servers) are simply free gifts to the hacker
because they had pre-shared keys or the same root password.
Craig
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
