On Mon, January 25, 2016 12:04 pm, Benjamin Smith wrote:
> No, we haven't been hacked. ;)
> We have a prospective client who is asking us what our policy is in the
> event
> of unauthorized access. Obviously you fix the system(s) that have been
> compromised, but what steps do you take to mitigate the effects of a
> breach?
> What is industry best practice? So far, searches haven't produced anything
> that looks consistent, except maybe identity monitoring for financial
> data.
> (EG: Target breach)
> We host a significant amount of educational data, but no financial
> information.
> How would we even respond to this question?
> I've also posted this question at
> https://www.reddit.com/r/linuxadmin/comments/42mi1r/what_to_do_when_youve_been_hacked/
Of course, mine will by no means be comprehensive or detailed. The order
may change when you start investigatinge it. But just in general:
1. It already happened, do not rush to do anything, change anything,
reboot, kill processes. If you can keep it for some time connected to the
network, you have chance to have more leak or deeper damage, but
simultaneously you may loose some of the tracks if you disconnect (thing
self sweeping up malicious things when network connection is lost). They
usually suggest yank from network, but it is your decision. Rebooting will
wipe content of /dev/shm and similar places used by bad guys to an
advantage of disappearing tracks
2. You should have backups, but you may need to make a copy of stuff that
might have changed after last backup
3. damage check
4. finding out level of compromise and the way compromise happened. Was it
root compromise and they got the whole system? Or some regular user
account was compromised (password or secret key was stolen). In last case:
were then attempts of elevation of privileges (they usually are)? Were
these attempts successful? (this is why you keep your system patched - it
is rare that there is hole on fully patched system known to bad guys, but
not software developers/system vendors). Was it through some service?
5. what have they done to come back or use your system otherwise? This
usually is parallel to investigating potential system level compromise.
Namely: scan open ports internally, and compare to scan from external box
(though some malicious may listen to their master box only, so this may
not necessarily reveal everything). Check if any of binaries or libraries
were modified (you should be prepared for that, look for integrity tools
like afick, eics, I used tripwire before it went commercial).
6. Preserve hacked system drive(s) (or their images) for further
forensics, which may take two weeks and longer... Build new system, patch,
enable the services you need and restore, make sure you don't open the
hole they came through. Restore files.
7. If it was system level compromise (root) sysadmin is obliged to contact
all users about it, stressing that their secret keys and passwords to
machines they were logging from compromised machine should be considered
compromised.
8. After you feel you recovered from compromise, and have freshly build
system with all necessary services running, do not drop forensics, it is
tedious but necessary thing.
Process accounting may be awfully helpful (you do have process accounting
enabled, right?)
SANS is one of great resources everybody will probably mention.
But the best is to avoid system level compromise, so: patch, patch,
patch... (or: update, update, update...).
As far as user level compromises are concerned: with 100+ users it is just
a question of time when someone's account will be compromised (often in a
system compromise elsewhere where user has an account). The best you can
do is to run your systems with an assumption that bad guys are already
inside. Make sure they can not do damage (even DOS, not to mention
elevation of privileges).
I hope, this helps.
Valeri
++++++++++++++++++++++++++++++++++++++++
Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247
++++++++++++++++++++++++++++++++++++++++
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
https://lists.centos.org/mailman/listinfo/centos
