Auditing Linux Security

File/Folder Integrity Monitoring With AIDE (WIP)

Steps

  1. Install AIDE.

    On Debian based systems:

     sudo apt install aide aide-common
    
  2. Make a backup of AIDE’s defaults file:

     sudo cp -p /etc/default/aide /etc/default/aide-COPY-$(date +"%Y%m%d%H%M%S")
    
  3. Go through /etc/default/aide and set AIDE’s defaults per your requirements. If you want AIDE to run daily and e-mail you, be sure to set CRON_DAILY_RUN to yes.

  4. Make a backup of AIDE’s configuration files:

     sudo cp -pr /etc/aide /etc/aide-COPY-$(date +"%Y%m%d%H%M%S")
    
  5. On Debian based systems:

    • AIDE’s configuration files are in /etc/aide/aide.conf.d/.
    • You’ll want to go through AIDE’s documentation and the configuration files in to set them per your requirements.
    • If you want new settings, to monitor a new folder for example, you’ll want to add them to /etc/aide/aide.conf or /etc/aide/aide.conf.d/.
    • Take a backup of the stock configuration files: sudo cp -pr /etc/aide /etc/aide-COPY-$(date +"%Y%m%d%H%M%S").
  6. Create a new database, and install it.

    On Debian based systems:

     sudo aideinit
    
    Running aide --init...
    Start timestamp: 2019-04-01 21:23:37 -0400 (AIDE 0.16)
    AIDE initialized database at /var/lib/aide/aide.db.new
    Verbose level: 6
    
    Number of entries:      25973
    
    ---------------------------------------------------
    The attributes of the (uncompressed) database(s):
    ---------------------------------------------------
    
    /var/lib/aide/aide.db.new
      RMD160   : moyQ1YskQQbidX+Lusv3g2wf1gQ=
      TIGER    : 7WoOgCrXzSpDrlO6I3PyXPj1gRiaMSeo
      SHA256   : gVx8Fp7r3800WF2aeXl+/KHCzfGsNi7O
                 g16VTPpIfYQ=
      SHA512   : GYfa0DJwWgMLl4Goo5VFVOhu4BphXCo3
                 rZnk49PYztwu50XjaAvsVuTjJY5uIYrG
                 tV+jt3ELvwFzGefq4ZBNMg==
      CRC32    : /cusZw==
      HAVAL    : E/i5ceF3YTjwenBfyxHEsy9Kzu35VTf7
                 CPGQSW4tl14=
      GOST     : n5Ityzxey9/1jIs7LMc08SULF1sLBFUc
                 aMv7Oby604A=
    
    
    End timestamp: 2019-04-01 21:24:45 -0400 (run time: 1m 8s)
    
  7. Test everything works with no changes.

    On Debian based systems:

     sudo aide.wrapper --check
    
    Start timestamp: 2019-04-01 21:24:45 -0400 (AIDE 0.16)
    AIDE found NO differences between database and filesystem. Looks okay!!
    Verbose level: 6
    
    Number of entries:      25973
    
    ---------------------------------------------------
    The attributes of the (uncompressed) database(s):
    ---------------------------------------------------
    
    /var/lib/aide/aide.db
      RMD160   : moyQ1YskQQbidX+Lusv3g2wf1gQ=
      TIGER    : 7WoOgCrXzSpDrlO6I3PyXPj1gRiaMSeo
      SHA256   : gVx8Fp7r3800WF2aeXl+/KHCzfGsNi7O
                 g16VTPpIfYQ=
      SHA512   : GYfa0DJwWgMLl4Goo5VFVOhu4BphXCo3
                 rZnk49PYztwu50XjaAvsVuTjJY5uIYrG
                 tV+jt3ELvwFzGefq4ZBNMg==
      CRC32    : /cusZw==
      HAVAL    : E/i5ceF3YTjwenBfyxHEsy9Kzu35VTf7
                 CPGQSW4tl14=
      GOST     : n5Ityzxey9/1jIs7LMc08SULF1sLBFUc
                 aMv7Oby604A=
    
    
    End timestamp: 2019-04-01 21:26:03 -0400 (run time: 1m 18s)
    
  8. Test everything works after making some changes.

    On Debian based systems:

     sudo touch /etc/test.sh
     sudo touch /root/test.sh
        
     sudo aide.wrapper --check
        
     sudo rm /etc/test.sh
     sudo rm /root/test.sh
        
     sudo aideinit -y -f
    
    Start timestamp: 2019-04-01 21:37:37 -0400 (AIDE 0.16)
    AIDE found differences between database and filesystem!!
    Verbose level: 6
    
    Summary:
      Total number of entries:      25972
      Added entries:                2
      Removed entries:              0
      Changed entries:              1
    
    ---------------------------------------------------
    Added entries:
    ---------------------------------------------------
    
    f++++++++++++++++: /etc/test.sh
    f++++++++++++++++: /root/test.sh
    
    ---------------------------------------------------
    Changed entries:
    ---------------------------------------------------
    
    d =.... mc.. .. .: /root
    
    ---------------------------------------------------
    Detailed information about changes:
    ---------------------------------------------------
    
    Directory: /root
      Mtime    : 2019-04-01 21:35:07 -0400        | 2019-04-01 21:37:36 -0400
      Ctime    : 2019-04-01 21:35:07 -0400        | 2019-04-01 21:37:36 -0400
    
    
    ---------------------------------------------------
    The attributes of the (uncompressed) database(s):
    ---------------------------------------------------
    
    /var/lib/aide/aide.db
      RMD160   : qF9WmKaf2PptjKnhcr9z4ueCPTY=
      TIGER    : zMo7MvvYJcq1hzvTQLPMW7ALeFiyEqv+
      SHA256   : LSLLVjjV6r8vlSxlbAbbEsPcQUB48SgP
                 pdVqEn6ZNbQ=
      SHA512   : Qc4U7+ZAWCcitapGhJ1IrXCLGCf1IKZl
                 02KYL1gaZ0Fm4dc7xLqjiquWDMSEbwzW
                 oz49NCquqGz5jpMIUy7UxA==
      CRC32    : z8ChEA==
      HAVAL    : YapzS+/cdDwLj3kHJEq8fufLp3DPKZDg
                 U12KCSkrO7Y=
      GOST     : 74sLV4HkTig+GJhokvxZQm7CJD/NR0mG
                 6jV7zdt5AXQ=
    
    
    End timestamp: 2019-04-01 21:38:50 -0400 (run time: 1m 13s)
    
  9. That’s it. If you set CRON_DAILY_RUN to yes in /etc/default/aide then cron will execute /etc/cron.daily/aide every day and e-mail you the output.

Updating The Database

Every time you make changes to files/folders that AIDE monitors, you will need to update the database to capture those changes. To do that on Debian based systems:

sudo aideinit -y -f

Anti-Virus Scanning With ClamAV (WIP)

How It Works

  • ClamAV is a virus scanner
  • ClamAV-Freshclam is a service that keeps the virus definitions updated
  • ClamAV-Daemon keeps the clamd process running to make scanning faster

Notes

  • These instructions do not tell you how to enable the ClamAV daemon service to ensure clamd is running all the time. clamd is only if you’re running a mail server and does not provide real-time monitoring of files. Instead, you’d want to scan files manually or on a schedule.

Steps - Antivirus scan using ClamAV

  1. Install ClamAV.

    On Debian based systems:

     sudo apt install clamav clamav-freshclam clamav-daemon
    
  2. Make a backup of clamav-freshclam’s configuration file /etc/clamav/freshclam.conf:

     sudo cp --archive /etc/clamav/freshclam.conf /etc/clamav/freshclam.conf-COPY-$(date +"%Y%m%d%H%M%S")
    
  3. clamav-freshclam’s default settings are probably good enough but if you want to change them, you can either edit the file /etc/clamav/freshclam.conf or use dpkg-reconfigure:

     sudo dpkg-reconfigure clamav-freshclam
    

    Note: The default settings will update the definitions 24 times in a day. To change the interval, check the Checks setting in /etc/clamav/freshclam.conf or use dpkg-reconfigure.

  4. Start the clamav-freshclam service:

     sudo service clamav-freshclam start
    
  5. You can make sure clamav-freshclam running:

     sudo service clamav-freshclam status
    
    ● clamav-freshclam.service - ClamAV virus database updater
       Loaded: loaded (/lib/systemd/system/clamav-freshclam.service; enabled; vendor preset: enabled)   Active: active (running) since Sat 2019-03-16 22:57:07 EDT; 2min 13s ago
         Docs: man:freshclam(1)
               man:freshclam.conf(5)
               https://www.clamav.net/documents
     Main PID: 1288 (freshclam)
       CGroup: /system.slice/clamav-freshclam.service
               └─1288 /usr/bin/freshclam -d --foreground=true
    
    Mar 16 22:57:08 host freshclam[1288]: Sat Mar 16 22:57:08 2019 -> ^Local version: 0.100.2 Recommended version: 0.101.1
    Mar 16 22:57:08 host freshclam[1288]: Sat Mar 16 22:57:08 2019 -> DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav
    Mar 16 22:57:15 host freshclam[1288]: Sat Mar 16 22:57:15 2019 -> Downloading main.cvd [100%]
    Mar 16 22:57:38 host freshclam[1288]: Sat Mar 16 22:57:38 2019 -> main.cvd updated (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
    Mar 16 22:57:40 host freshclam[1288]: Sat Mar 16 22:57:40 2019 -> Downloading daily.cvd [100%]
    Mar 16 22:58:13 host freshclam[1288]: Sat Mar 16 22:58:13 2019 -> daily.cvd updated (version: 25390, sigs: 1520006, f-level: 63, builder: raynman)
    Mar 16 22:58:14 host freshclam[1288]: Sat Mar 16 22:58:14 2019 -> Downloading bytecode.cvd [100%]
    Mar 16 22:58:16 host freshclam[1288]: Sat Mar 16 22:58:16 2019 -> bytecode.cvd updated (version: 328, sigs: 94, f-level: 63, builder: neo)
    Mar 16 22:58:24 host freshclam[1288]: Sat Mar 16 22:58:24 2019 -> Database updated (6086349 signatures) from db.local.clamav.net (IP: 104.16.219.84)
    Mar 16 22:58:24 host freshclam[1288]: Sat Mar 16 22:58:24 2019 -> ^Clamd was NOT notified: Can't connect to clamd through /var/run/clamav/clamd.ctl: No such file or directory
    

    Note: Don’t worry about that Local version line. Check https://serverfault.com/questions/741299/is-there-a-way-to-keep-clamav-updated-on-debian-8 for more details.

  6. Make a backup of clamav-daemon’s configuration file /etc/clamav/clamd.conf:

     sudo cp --archive /etc/clamav/clamd.conf /etc/clamav/clamd.conf-COPY-$(date +"%Y%m%d%H%M%S")
    
  7. You can change clamav-daemon’s settings by editing the file /etc/clamav/clamd.conf or useing dpkg-reconfigure:

     sudo dpkg-reconfigure clamav-daemon
    

Scanning Files/Folders

  • To scan files/folders use the clamscan program.
  • clamscan runs as the user it is executed as so it needs read permissions to the files/folders it is scanning.
  • Using clamscan as root is dangerous because if a file is in fact a virus there is risk that it could use the root privileges.
  • To scan a file: clamscan /path/to/file.
  • To scan a directory: clamscan -r /path/to/folder.
  • You can use the -i switch to only print infected files.
  • Check clamscan’s man pages for other switches/options.

Rootkit Detection With Rkhunter (WIP)

Steps - How to detect Rootkits with Rkhunter

  1. Install Rkhunter.

    On Debian based systems:

     sudo apt install rkhunter
    
  2. Make a backup of rkhunter’ defaults file:

     sudo cp -p /etc/default/rkhunter /etc/default/rkhunter-COPY-$(date +"%Y%m%d%H%M%S")
    
  3. rkhunter’s configuration file is /etc/rkhunter.conf. Instead of making changes to it, create and use the file /etc/rkhunter.conf.local instead:

     sudo cp -p /etc/rkhunter.conf /etc/rkhunter.conf.local
    
  4. Go through the configuration file /etc/rkhunter.conf.local and set to your requirements. My recommendations:

    Setting Note
    UPDATE_MIRRORS=1  
    MIRRORS_MODE=0  
    MAIL-ON-WARNING=root  
    COPY_LOG_ON_ERROR=1 to save a copy of the log if there is an error
    PKGMGR=... set to the appropriate value per the documentation
    PHALANX2_DIRTEST=1 read the documentation for why
    WEB_CMD="" this is to address an issue with the Debian package that disables the ability for rkhunter to self-update.
    USE_LOCKING=1 to prevent issues with rkhunter running multiple times
    SHOW_SUMMARY_WARNINGS_NUMBER=1 to see the actual number of warnings found
  5. You want rkhunter to run every day and e-mail you the result. You can write your own script or check https://www.tecmint.com/install-rootkit-hunter-scan-for-rootkits-backdoors-in-linux/ for a sample cron script you can use.

    On Debian based system, rkhunter comes with cron scripts. To enable them check /etc/default/rkhunter or use dpkg-reconfigure and say Yes to all of the questions:

     sudo dpkg-reconfigure rkhunter
    
  6. After you’ve finished with all of the changes, make sure all the settings are valid:

     sudo rkhunter -C
    
  7. Update rkhunter and its database:

     sudo rkhunter --versioncheck
     sudo rkhunter --update
     sudo rkhunter --propupd
    
  8. If you want to do a manual scan and see the output:

     sudo rkhunter --check
    

Rootkit Detection With chrootkit (WIP)

Steps - How to detect Rootkits with chrootkit

  1. Install chkrootkit.

    On Debian based systems:

     sudo apt install chkrootkit
    
  2. Do a manual scan:

     sudo chkrootkit
    
    ROOTDIR is `/'
    Checking `amd'...                                           not found
    Checking `basename'...                                      not infected
    Checking `biff'...                                          not found
    Checking `chfn'...                                          not infected
    Checking `chsh'...                                          not infected
    ...
    Checking `scalper'...                                       not infected
    Checking `slapper'...                                       not infected
    Checking `z2'...                                            chklastlog: nothing deleted
    Checking `chkutmp'...                                       chkutmp: nothing deleted
    Checking `OSX_RSPLUG'...                                    not infected
    
  3. Make a backup of chkrootkit’s configuration file /etc/chkrootkit.conf:

     sudo cp --archive /etc/chkrootkit.conf /etc/chkrootkit.conf-COPY-$(date +"%Y%m%d%H%M%S")
    
  4. You want chkrootkit to run every day and e-mail you the result.

    On Debian based system, chkrootkit comes with cron scripts. To enable them check /etc/chkrootkit.conf or use dpkg-reconfigure and say Yes to the first question:

     sudo dpkg-reconfigure chkrootkit
    

logwatch - system log analyzer and reporter

Your server will be generating a lot of logs that may contain important information. Unless you plan on checking your server everyday, you’ll want a way to get e-mail summary of your server’s logs. To accomplish this we’ll use logwatch.

How It Works

logwatch scans system log files and summarizes them. You can run it directly from the command line or schedule it to run on a recurring schedule. logwatch uses service files to know how to read/summarize a log file. You can see all of the stock service files in /usr/share/logwatch/scripts/services.

logwatch’s configuration file /usr/share/logwatch/default.conf/logwatch.conf specifies default options. You can override them via command line arguments.

Goals

  • Logwatch configured to send a daily e-mail summary of all of the server’s status and logs

Notes

  • Your server will need to be able to send e-mails for this to work
  • The below steps will result in logwatch running every day. If you want to change the schedule, modify the cronjob to your liking. You’ll also want to change the range option to cover your recurrence window. See https://www.badpenguin.org/configure-logwatch-for-weekly-email-and-html-output-format for an example.
  • If logwatch fails to deliver mail due to the e-mail having long lines please check https://blog.dhampir.no/content/exim4-line-length-in-debian-stretch-mail-delivery-failed-returning-message-to-sender as documented in issue #29. If you you followed Gmail and Exim4 As MTA With Implicit TLS then we already took care of this in step #7.

Steps

  1. Install logwatch.

    On Debian based systems:

     sudo apt install logwatch
    
  2. To see a sample of what logwatch collects you can run it directly:

     sudo /usr/sbin/logwatch --output stdout --format text --range yesterday --service all
    
    
     ################### Logwatch 7.4.3 (12/07/16) ####################
            Processing Initiated: Mon Mar  4 00:05:50 2019
            Date Range Processed: yesterday
                                  ( 2019-Mar-03 )
                                  Period is day.
            Detail Level of Output: 5
            Type of Output/Format: stdout / text
            Logfiles for Host: host
     ##################################################################
    
     --------------------- Cron Begin ------------------------
    ...
    ...
     ---------------------- Disk Space End -------------------------
    
    
     ###################### Logwatch End #########################
    
  3. Go through logwatch’s self-documented configuration file /usr/share/logwatch/default.conf/logwatch.conf before continuing. There is no need to change anything here but pay special attention to the Output, Format, MailTo, Range, and Service as those are the ones we’ll be using. For our purposes, instead of specifying our options in the configuration file, we will pass them as command line arguments in the daily cron job that executes logwatch. That way, if the configuration file is ever modified (e.g. during an update), our options will still be there.

  4. Make a backup of logwatch’s daily cron file /etc/cron.daily/00logwatch and unset the execute bit:

     sudo cp --archive /etc/cron.daily/00logwatch /etc/cron.daily/00logwatch-COPY-$(date +"%Y%m%d%H%M%S")
     sudo chmod -x /etc/cron.daily/00logwatch-COPY*
    
  5. By default, logwatch outputs to stdout. Since the goal is to get a daily e-mail, we need to change the output type that logwatch uses to send e-mail instead. We could do this through the configuration file above, but that would apply to every time it is run – even when we run it manually and want to see the output to the screen. Instead, we’ll change the cron job that executes logwatch to send e-mail. This way, when run manually, we’ll still get output to stdout and when run by cron, it’ll send an e-mail. We’ll also make sure it checks for all services, and change the output format to html so it’s easier to read regardless of what the configuration file says. In the file /etc/cron.daily/00logwatch find the execute line and change it to:

     /usr/sbin/logwatch --output mail --format html --mailto root --range yesterday --service all
    
    #!/bin/bash
    
    #Check if removed-but-not-purged
    test -x /usr/share/logwatch/scripts/logwatch.pl || exit 0
    
    #execute
    /usr/sbin/logwatch --output mail --format html --mailto root --range yesterday --service all
    
    #Note: It's possible to force the recipient in above command
    #Just pass --mailto [email protected] instead of --output mail
    
     sudo sed -i -r -e "s,^($(sudo which logwatch).*?),# \1         # commented by $(whoami) on $(date +"%Y-%m-%d @ %H:%M:%S")\n$(sudo which logwatch) --output mail --format html --mailto root --range yesterday --service all         # added by $(whoami) on $(date +"%Y-%m-%d @ %H:%M:%S")," /etc/cron.daily/00logwatch
    
  6. You can test the cron job by executing it:

     sudo /etc/cron.daily/00logwatch
    

    Note: If logwatch fails to deliver mail due to the e-mail having long lines please check https://blog.dhampir.no/content/exim4-line-length-in-debian-stretch-mail-delivery-failed-returning-message-to-sender as documented in issue #29. If you you followed Gmail and Exim4 As MTA With Implicit TLS then we already took care of this in step #7.

ss - Seeing Ports Your Server Is Listening On

Ports are how applications, services, and processes communicate with each other – either locally within your server or with other devices on the network. When you have an application or service (like SSH or Apache) running on your server, they listen for requests on specific ports.

Obviously we don’t want your server listening on ports we don’t know about. We’ll use ss to see all the ports that services are listening on. This will help us track down and stop rogue, potentially dangerous, services.

Goals

  • find out non-localhost what ports are open and listening for connections

Steps

  1. To see the all the ports listening for traffic:

     sudo ss -lntup
    
    Netid  State      Recv-Q Send-Q     Local Address:Port     Peer Address:Port
    udp    UNCONN     0      0                      *:68                  *:*        users:(("dhclient",pid=389,fd=6))
    tcp    LISTEN     0      128                    *:22                  *:*        users:(("sshd",pid=4390,fd=3))
    tcp    LISTEN     0      128                   :::22                 :::*        users:(("sshd",pid=4390,fd=4))
    

    Switch Explanations:

    • l = display listening sockets
    • n = do now try to resolve service names
    • t = display TCP sockets
    • u = display UDP sockets
    • p = show process information
  2. If you see anything suspicious, like a port you’re not aware of or a process you don’t know, investigate and remediate as necessary.

Lynis - Linux Security Auditing

From https://cisofy.com/lynis/:

Lynis is a battle-tested security tool for systems running Linux, macOS, or Unix-based operating system. It performs an extensive health scan of your systems to support system hardening and compliance testing.

Notes

  • CISOFY offers packages for many distributions. Check https://packages.cisofy.com/ for distribution specific installation instructions.

Steps

  1. Install lynis. https://cisofy.com/lynis/#installation has detailed instructions on how to install it for your distribution.

    On Debian based systems, using CISOFY’s community software repository:

     sudo apt install apt-transport-https ca-certificates host
     sudo wget -O - https://packages.cisofy.com/keys/cisofy-software-public.key | sudo apt-key add -
     sudo echo "deb https://packages.cisofy.com/community/lynis/deb/ stable main" | sudo tee /etc/apt/sources.list.d/cisofy-lynis.list
     sudo apt update
     sudo apt install lynis host
    
  2. Update it:

     sudo lynis update info
    
  3. Run a security audit:

     sudo lynis audit system
    

    This will scan your server, report its audit findings, and at the end it will give you suggestions. Spend some time going through the output and address gaps as necessary.

OSSEC - Host Intrusion Detection

From https://github.com/ossec/ossec-hids

OSSEC is a full platform to monitor and control your systems. It mixes together all the aspects of HIDS (host-based intrusion detection), log monitoring and SIM/SIEM together in a simple, powerful and open source solution.

Steps

  1. Install OSSEC-HIDS from sources
     sudo apt install -y libz-dev libssl-dev libpcre2-dev build-essential libsystemd-dev
     wget https://github.com/ossec/ossec-hids/archive/3.7.0.tar.gz
     tar xzf 3.7.0.tar.gz
     cd ossec-hids-3.7.0/
     sudo ./install.sh
    
  2. Useful commands:

Agent information

    sudo /var/ossec/bin/agent_control -i <AGENT_ID>

AGENT_ID by default is 000, to be sure the command sudo /var/ossec/bin/agent_control -l can be used.

Run integrity/rootkit checking

OSSEC by default run rootkit check each 2 hours.

    sudo /var/ossec/bin/agent_control -u <AGENT_ID> -r 

Alerts

  • All:
      tail -f /var/ossec/logs/alerts/alerts.log
    
  • Integrity check:
      sudo cat /var/ossec/logs/alerts/alerts.log | grep -A4  -i integrity
    
  • Rootkit check:
       sudo cat /var/ossec/logs/alerts/alerts.log | grep -A4  "rootcheck,"
    

Licenses and Attributions


Speak Your Mind