On Jan 1, 2012, at 8:50 PM, Bennett Haselton wrote:
> On Sun, Jan 1, 2012 at 5:33 PM, RILINDO FOSTER <rilindo@xxxxxx> wrote:
>
>> ≈On Jan 1, 2012, at 8:24 PM, Bennett Haselton wrote:
>>
>>> On Sun, Jan 1, 2012 at 4:57 PM, Rilindo Foster <rilindo@xxxxxx> wrote:
>>>
>>>>
>>>>
>>>> On Jan 1, 2012, at 5:23 PM, Bennett Haselton <bennett@xxxxxxxxxxxxx>
>>>> wrote:
>>>>
>>>>> (Sorry, third time -- last one, promise, just giving it a subject
>> line!)
>>>>>
>>>>> OK, a second machine hosted at the same hosting company has also
>>>> apparently
>>>>> been hacked. Since 2 of out of 3 machines hosted at that company have
>>>> now
>>>>> been hacked, but this hasn't happened to any of the other 37 dedicated
>>>>> servers that I've got hosted at other hosting companies (also CentOS,
>>>> same
>>>>> version or almost), this makes me wonder if there's a security breach
>> at
>>>>> this company, like if they store customers' passwords in a place that's
>>>>> been hacked. (Of course it could also be that whatever attacker found
>> an
>>>>> exploit, was just scanning that company's address space for hackable
>>>>> machines, and didn't happen to scan the address space of the other
>>>> hosting
>>>>> companies.)
>>>>>
>>>>> So, following people's suggestions, the machine is disconnected and
>>>> hooked
>>>>> up to a KVM so I can still examine the files. I've found this file:
>>>>> -rw-r--r-- 1 root root 1358 Oct 21 17:40 /home/file.pl
>>>>> which appears to be a copy of this exploit script:
>>>>> http://archive.cert.uni-stuttgart.de/bugtraq/2006/11/msg00302.html
>>>>> Note the last-mod date of October 21.
>>>>>
>>>>> No other files on the system were last modified on October 21st.
>> However
>>>>> there was a security advisory dated October 20th which affected httpd:
>>>>>
>>>>
>> http://mailinglist-archive.com/centos-announce/2011-10/00035-CentOSannounce+CESA20111392+Moderate+CentOS+5+i386+httpd+Update
>>>>> https://rhn.redhat.com/errata/RHSA-2011-1392.html
>>>>>
>>>>> and a large number of files on the machine, including lots of files in
>> */
>>>>> usr/lib64/httpd/modules/* and
>> */lib/modules/2.6.18-274.7.1.el5/kernel/* ,
>>>>> have a last-mod date of October 20th. So I assume that these are files
>>>>> which were updated automatically by yum as a result of the patch that
>>>> goes
>>>>> with this advisory -- does that sound right?
>>>>>
>>>>> So a couple of questions that I could use some help with:
>>>>>
>>>>> 1) The last patch affecting httpd was released on October 20th, and the
>>>>> earliest evidence I can find of the machine being hacked is a file
>> dated
>>>>> October 21st. This could be just a coincidence, but could it also
>>>> suggest
>>>>> that the patch on October 20th introduced a new exploit, which the
>>>> attacker
>>>>> then used to get in on October 21st?
>>>>> (Another possibility: I think that when yum installs updates, it
>>>>> doesn't actually restart httpd. So maybe even after the patch was
>>>>> installed, my old httpd instance kept running and was still vulnerable?
>>>> As
>>>>> for why it got hacked the very next day, maybe the attacker looked at
>> the
>>>>> newly released patch and reverse-engineered it to figure out where the
>>>>> vulnerabilities were, that the patch fixed?)
>>>>>
>>>>> 2) Since the */var/log/httpd/* and /var/log/secure* logs only go back
>> 4-5
>>>>> weeks by default, it looks like any log entries related to how the
>>>> attacker
>>>>> would have gotten in on or before October 21st, are gone. (The secure*
>>>>> logs do show multiple successful logins as "root" within the last 4
>>>> weeks,
>>>>> mostly from IP addresses in Asia, but that's to be expected once the
>>>>> machine was compromised -- it doesn't help track down how they
>> originally
>>>>> got in.) Anywhere else that the logs would contain useful data?
>>>>> _______________________________________________
>>>>> CentOS mailing list
>>>>> CentOS@xxxxxxxxxx
>>>>> http://lists.centos.org/mailman/listinfo/centos
>>>>
>>>> Do you have SELinux enabled? If not, you might need to turn that on, as
>>>> that would have prevented that exploit.
>>>>
>>>>
>>>
>>> I don't, but I'm not sure what you mean by "that would have prevented
>> that
>>> exploit". How do you know what exploit they used, much less that SELinux
>>> would have prevented it?
>>>
>>> Or are you assuming that my guess was correct that they got in because
>> of a
>>> vulnerability that was opened up by the patch being auto-installed on
>>> 10/20, and that if *that* is the case, that SELinux would have prevented
>>> that? How does that work, how would SELinux have prevented a
>> vulnerability
>>> being opened up by the patch install?
>>
>> The script in question is an exploit from a web board which is apparently
>> designed to pull outside traffic. If you had SELinux, it would put httpd in
>> its own context and by default, it will NOT allow connections from that
>> context to another. You have to enable it with:
>>
>> setsebool -P httpd_can_network_connect on
>>
>
> I'm not sure what you mean by "an exploit from a web board which is
> apparently designed to pull outside traffic". Like Ljubomir said, it looks
> like a script that is used from machine X to DOS attack machine Y, if
> machine Y has the VBulletin bulletin board software installed on it. Is
> that what you meant? :)
>
> But anyway, since the file was located at /home/file.pl (and since attacker
> had console access), presumably it wasn't being invoked by the web server,
> only from the command line. So how would it have made any difference if
> httpd was running in its own context, if that script was not being invoked
> by httpd?
How do you know that the attack has console access? If you mean ssh access, there may not be a need for the attack use that - the attack can simply use an exploit in a known web application and install their own copy of ssh or some other remote access daemon - and do it in way that it is not detectable even in the system logs.
Which brings up an important point - most intrusions on web hosts is through unpatched web sites that attacks used to attempt further privilege escalations. That is why you have mechanisms like SELinux to lock down the app, so that the worse they can do is to break the site, but not become root.
- Rilindo
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
