5 thoughts on - What To Do When You’ve Been Hacked?

• Of course, mine will by no means be comprehensive or detailed. The order may change when you start investigatinge it. But just in general:

  1. It already happened, do not rush to do anything, change anything, reboot, kill processes. If you can keep it for some time connected to the network, you have chance to have more leak or deeper damage, but simultaneously you may loose some of the tracks if you disconnect (thing self sweeping up malicious things when network connection is lost). They usually suggest yank from network, but it is your decision. Rebooting will wipe content of /dev/shm and similar places used by bad guys to an advantage of disappearing tracks

  2. You should have backups, but you may need to make a copy of stuff that might have changed after last backup

  3. damage check

  4. finding out level of compromise and the way compromise happened. Was it root compromise and they got the whole system? Or some regular user account was compromised (password or secret key was stolen). In last case:
  were then attempts of elevation of privileges (they usually are)? Were these attempts successful? (this is why you keep your system patched – it is rare that there is hole on fully patched system known to bad guys, but not software developers/system vendors). Was it through some service?

  5. what have they done to come back or use your system otherwise? This usually is parallel to investigating potential system level compromise. Namely: scan open ports internally, and compare to scan from external box
  (though some malicious may listen to their master box only, so this may not necessarily reveal everything). Check if any of binaries or libraries were modified (you should be prepared for that, look for integrity tools like afick, eics, I used tripwire before it went commercial).

  6. Preserve hacked system drive(s) (or their images) for further forensics, which may take two weeks and longer… Build new system, patch, enable the services you need and restore, make sure you don’t open the hole they came through. Restore files.

  7. If it was system level compromise (root) sysadmin is obliged to contact all users about it, stressing that their secret keys and passwords to machines they were logging from compromised machine should be considered compromised.

  8. After you feel you recovered from compromise, and have freshly build system with all necessary services running, do not drop forensics, it is tedious but necessary thing.

  Process accounting may be awfully helpful (you do have process accounting enabled, right?)

  SANS is one of great resources everybody will probably mention.

  But the best is to avoid system level compromise, so: patch, patch, patch… (or: update, update, update…).

  As far as user level compromises are concerned: with 100+ users it is just a question of time when someone’s account will be compromised (often in a system compromise elsewhere where user has an account). The best you can do is to run your systems with an assumption that bad guys are already inside. Make sure they can not do damage (even DOS, not to mention elevation of privileges).

  I hope, this helps.

  Valeri

  ++++++++++++++++++++++++++++++++++++++++
  Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247
  ++++++++++++++++++++++++++++++++++++++++

• Tell them you use the Mr. Miyagi defense: “Don’t get hit.”

  Your prospective client sounds like they’re expecting someone to have established procedures to deal with breaches. You know who has established procedures? Organizations that see the same problems again and again.

  Selecting an information service provider based on which one is best at recovering from a hack attack is like hiring a football coach based on how skilled he is at setting bones or selecting a cargo ship captain based on how good he is at patching hull breaches.

  Why is “We’ve been at this for 20 years and have never *had* to clean up after a hacking incident” not an excellent rejoinder?

  You should not have to ask this. You should know it, because you are a professional and have been in this industry long enough.

  Since you don’t, maybe you shouldn’t be bidding on this job.

  I don’t mean to make this sound cabalistic, where only insiders know the secret handshakes, but rather exactly the opposite: this is information you should have been slowly absorbing for years:

  – SSH instead of telnet and FTP
  – HTTPS wherever possible over HTTP
  – Always enable SELinux
  – Prefer to surf default SELinux policies rather than override or custom-craft
  – Know in your heart that deny-by-default firewalls are a good thing
  – Turn off unnecessary services…
  – …then run “netstat -na | grep LISTEN” and justify each output line
  – Understand chown and chmod effects implicitly
  – Be able to read ls -l output at a blink

  And much more.

  All of this will be covered in any decent text on Unix/Linux security. Sorry, I can’t give recommendations since I got past the book learnin’ stage long ago, and have been accreting such things ever since.

  Coming back to martial arts, at some point you get past the point of conscious action and react implicitly. The equivalent in security is recognizing risks and mitigating against them before they become NY Times headlines.

• Agreed! (although for us it has been 15 years.

  Which I’d consider “best practices” and we do them. They are specifically asking about what to do *after* a breach. Despite all the best practices in place, there’s *still* some risk.

• Start looking for new job, maybe ;-)

  Valeri

  ++++++++++++++++++++++++++++++++++++++++
  Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247
  ++++++++++++++++++++++++++++++++++++++++

• If someone wants in to your network then they will get in. There is no point in deluding yourself or your clients on that point.

  The first thing that you must do after a breach is detected, or even suspected, is to notify all affected parties. There is an institutional bias against revelation of security incidents because of the fear of embarrassment. This is often couched in terms using the word ‘premature’. Failure to disclose at the earliest opportunity is unethical and ultimately self-defeating. You will never regain trust thereafter.

  The second thing to do, concurrently with the first, is to isolate the affected systems from the rest of your network. If that means physically pulling wires and putting the things on their own switch and LAN segment blocked from the rest of your networks then do it. If it means shutting down the affected hosts then do it. If if means disconnecting from the network at your gateway then do it. They are in and they are looking for ways to expand their foothold. Delaying containment is pointless.

  The third thing to do is to involve the authorities. Unauthorised computer access is an indictable offence in Canada and the UK. It is a federal felony in the U.S.A. If you have an incident then report it. That means you should have computer emergency response contact information and reporting protocols already in place.

  Now, with your clients and the authorities notified and the suspect systems isolated, you begin to map out your recovery strategy. The basic bones of which you have already written down and implemented in your backup and disaster recovery plan. A security breach is a disaster. You need to start with that point clearly in mind and proceed on that basis.

  Once corporate and client services are restored on clean hosts and reconnected to the Internet then begin your investigation. Use your AIDE and syslog records to determine the point of entry, the length of compromise and the extent of penetration. If possible identify the nature of the attackers and their target. Where possible keep the compromised hosts’ disk drives unaltered for further technical analysis. Where warranted bring in forensic investigators to examine them.

  It will likely prove impossible to positively identify them but you should be able to glean some inkling if this was a targeted breach or an opportunistic one. If the former then they will be back and you will need to consider how to deal with the next assault.