Hello,
just if it helps, please find below these lines the steps I have used to
analyze several suspicious machines in some customers, to check if they
have been compromised or not:
* Chrootkit && rkhunter -> To search for known trojans and common linux
malware.
* unhide (http://www.unhide-forensics.info/) -> to check for hidden
processes and tcp sockets
* rpm -Va -> To check binary integrity against installed rpms
* If netstat binary looks to be sane, check listening sockets
* If ps binary looks to be sane, check shown running processes
* Check console connections with "last" and "lastb" commands
* Tcpdump on network interfaces avoiding traffic for known running
services (80, 25, 21, etc... depending on the role of the machine) to
check for the weird traffic
* grep -i segfault /var/log/* -> to check for buffer overflows in logs
* grep -i auth /var/log/* |grep -i failed -> to check authentication
failed tries.
* lsmod -> to check loaded kernel modules (it is ver difficult to find
out something wrong here, but just to be sure nothing weird appears).
* lsof -> to check opened current files
* Check xinetd -> to find out if someone has added some new "service"
* have a look to /tmp, /opt, /usr/bin, /usr/local/bin, /usr/sbin and
.bash_history...
* check /etc/passwd and verify created users are licit to be there.
* check crontab for every user to avoid any process to be programmed
Hope the checklist helps...
Regards,
El 02/01/12 09:04, Craig White escribió:
> On Sun, 2012-01-01 at 14:23 -0800, Bennett Haselton wrote:
>> (Sorry, third time -- last one, promise, just giving it a subject line!)
>>
>> OK, a second machine hosted at the same hosting company has also apparently
>> been hacked. Since 2 of out of 3 machines hosted at that company have now
>> been hacked, but this hasn't happened to any of the other 37 dedicated
>> servers that I've got hosted at other hosting companies (also CentOS, same
>> version or almost), this makes me wonder if there's a security breach at
>> this company, like if they store customers' passwords in a place that's
>> been hacked. (Of course it could also be that whatever attacker found an
>> exploit, was just scanning that company's address space for hackable
>> machines, and didn't happen to scan the address space of the other hosting
>> companies.)
>>
>> So, following people's suggestions, the machine is disconnected and hooked
>> up to a KVM so I can still examine the files. I've found this file:
>> -rw-r--r-- 1 root root 1358 Oct 21 17:40 /home/file.pl
>> which appears to be a copy of this exploit script:
>> http://archive.cert.uni-stuttgart.de/bugtraq/2006/11/msg00302.html
>> Note the last-mod date of October 21.
>>
>> No other files on the system were last modified on October 21st. However
>> there was a security advisory dated October 20th which affected httpd:
>> http://mailinglist-archive.com/centos-announce/2011-10/00035-CentOSannounce+CESA20111392+Moderate+CentOS+5+i386+httpd+Update
>> https://rhn.redhat.com/errata/RHSA-2011-1392.html
>>
>> and a large number of files on the machine, including lots of files in */
>> usr/lib64/httpd/modules/* and */lib/modules/2.6.18-274.7.1.el5/kernel/* ,
>> have a last-mod date of October 20th. So I assume that these are files
>> which were updated automatically by yum as a result of the patch that goes
>> with this advisory -- does that sound right?
>>
>> So a couple of questions that I could use some help with:
>>
>> 1) The last patch affecting httpd was released on October 20th, and the
>> earliest evidence I can find of the machine being hacked is a file dated
>> October 21st. This could be just a coincidence, but could it also suggest
>> that the patch on October 20th introduced a new exploit, which the attacker
>> then used to get in on October 21st?
>> (Another possibility: I think that when yum installs updates, it
>> doesn't actually restart httpd. So maybe even after the patch was
>> installed, my old httpd instance kept running and was still vulnerable? As
>> for why it got hacked the very next day, maybe the attacker looked at the
>> newly released patch and reverse-engineered it to figure out where the
>> vulnerabilities were, that the patch fixed?)
>>
>> 2) Since the */var/log/httpd/* and /var/log/secure* logs only go back 4-5
>> weeks by default, it looks like any log entries related to how the attacker
>> would have gotten in on or before October 21st, are gone. (The secure*
>> logs do show multiple successful logins as "root" within the last 4 weeks,
>> mostly from IP addresses in Asia, but that's to be expected once the
>> machine was compromised -- it doesn't help track down how they originally
>> got in.) Anywhere else that the logs would contain useful data?
> ----
> the particular issue which was patched by this httpd (apache) update was
> to fix a problem with reverse proxy so the first question is did this
> server actually have a reverse proxy configured?
>
> My next thought is that since this particular hacker managed to get
> access to more than one of your machines, is it possible that there is a
> mechanism (ie a pre-shared public key) that would allow them access to
> the second server from the first server they managed to crack? The point
> being that this computer may not have been the one that they originally
> cracked and there may not be evidence of cracking on this computer.
>
> The script you identified would seem to be a script for attacking other
> systems and by the time it landed on your system, it was already broken
> into.
>
> There are some tools to identify a hackers access though they are often
> obscured by the hacker...
>
> last # reads /var/log/wtmp and provides a list of users, login date/time
> login duration, etc. Read the man page for last to get other options on
> its usage including the '-f' option to read older wtmp log files if
> needed.
>
> lastb # reads /var/log/btmp much as above but list 'failed' logins
> though this requires pro-active configuration and if you didn't do that,
> you probably will do that going forward.
>
> looking at /etc/passwd to see what users are on your system and then
> search their $HOME directories carefully for any evidence that their
> account was the first one compromised. Very often, a single user with a
> weak password has his account cracked and then a hacker can get a copy
> of /etc/shadow and brute force the root password.
>
> Consider that this type of activity is often done with 'hidden' files&
> directories. This hacker was apparently brazen enough to operate openly
> in /home so it's likely that he wasn't very concerned about his cracking
> being discovered.
>
> The most important thing to do at this point is to figure out HOW they
> got into your systems in the first place and discussions of SELinux and
> yum updates are not useful to that end. Yes, you should always update
> and always run SELinux but not useful in determining what actually
> happened.
>
> Make a list of all open ports on this system, check the directories,
> files, data from all daemons/applications that were exposed (Apache?
> PHP?, MySQL?, etc.) and be especially vigilant to any directories where
> user apache had write access.
>
> Again though, I am concerned that your first action on your first
> discovered hacked server was to wipe it out and of a notion that it's
> entirely possible that the actual cracking occurred on that system and
> this (and perhaps other servers) are simply free gifts to the hacker
> because they had pre-shared keys or the same root password.
>
> Craig
>
>
--
Lorenzo Martinez Rodriguez
Visit me: http://www.lorenzomartinez.es
Mail me to: lorenzo@xxxxxxxxxxxxxxxxxx
My blog: http://www.securitybydefault.com
My twitter: @lawwait
PGP Fingerprint: 97CC 2584 7A04 B2BA 00F1 76C9 0D76 83A2 9BBC BDE2
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
