Books / How To Secure A Linux Server / Chapter 5

Auditing Linux Security

In this chapter

File/Folder Integrity Monitoring With AIDE (WIP)
- Steps
Anti-Virus Scanning With ClamAV (WIP)
- Steps - Antivirus scan using ClamAV
Rootkit Detection With Rkhunter (WIP)
- Steps - How to detect Rootkits with Rkhunter
Rootkit Detection With chrootkit (WIP)
- Steps - How to detect Rootkits with chrootkit
logwatch - system log analyzer and reporter
- Steps
ss - Seeing Ports Your Server Is Listening On
- Steps
Lynis - Linux Security Auditing
- Steps
OSSEC - Host Intrusion Detection
- Steps

File/Folder Integrity Monitoring With AIDE (WIP)

Steps

Install AIDE.

On Debian based systems:
```
 sudo apt install aide aide-common
```

Make a backup of AIDE’s defaults file:

 sudo cp -p /etc/default/aide /etc/default/aide-COPY-$(date +"%Y%m%d%H%M%S")

Go through /etc/default/aide and set AIDE’s defaults per your requirements. If you want AIDE to run daily and e-mail you, be sure to set CRON_DAILY_RUN to yes.

Make a backup of AIDE’s configuration files:

 sudo cp -pr /etc/aide /etc/aide-COPY-$(date +"%Y%m%d%H%M%S")

On Debian based systems:
- AIDE’s configuration files are in /etc/aide/aide.conf.d/.
- You’ll want to go through AIDE’s documentation and the configuration files in to set them per your requirements.
- If you want new settings, to monitor a new folder for example, you’ll want to add them to /etc/aide/aide.conf or /etc/aide/aide.conf.d/.
- Take a backup of the stock configuration files: sudo cp -pr /etc/aide /etc/aide-COPY-$(date +"%Y%m%d%H%M%S").

Create a new database, and install it.

On Debian based systems:

 sudo aideinit

Running aide --init...
Start timestamp: 2019-04-01 21:23:37 -0400 (AIDE 0.16)
AIDE initialized database at /var/lib/aide/aide.db.new
Verbose level: 6

Number of entries:      25973

---------------------------------------------------
The attributes of the (uncompressed) database(s):
---------------------------------------------------

/var/lib/aide/aide.db.new
  RMD160   : moyQ1YskQQbidX+Lusv3g2wf1gQ=
  TIGER    : 7WoOgCrXzSpDrlO6I3PyXPj1gRiaMSeo
  SHA256   : gVx8Fp7r3800WF2aeXl+/KHCzfGsNi7O
             g16VTPpIfYQ=
  SHA512   : GYfa0DJwWgMLl4Goo5VFVOhu4BphXCo3
             rZnk49PYztwu50XjaAvsVuTjJY5uIYrG
             tV+jt3ELvwFzGefq4ZBNMg==
  CRC32    : /cusZw==
  HAVAL    : E/i5ceF3YTjwenBfyxHEsy9Kzu35VTf7
             CPGQSW4tl14=
  GOST     : n5Ityzxey9/1jIs7LMc08SULF1sLBFUc
             aMv7Oby604A=


End timestamp: 2019-04-01 21:24:45 -0400 (run time: 1m 8s)

Test everything works with no changes.

On Debian based systems:

 sudo aide.wrapper --check

Start timestamp: 2019-04-01 21:24:45 -0400 (AIDE 0.16)
AIDE found NO differences between database and filesystem. Looks okay!!
Verbose level: 6

Number of entries:      25973

---------------------------------------------------
The attributes of the (uncompressed) database(s):
---------------------------------------------------

/var/lib/aide/aide.db
  RMD160   : moyQ1YskQQbidX+Lusv3g2wf1gQ=
  TIGER    : 7WoOgCrXzSpDrlO6I3PyXPj1gRiaMSeo
  SHA256   : gVx8Fp7r3800WF2aeXl+/KHCzfGsNi7O
             g16VTPpIfYQ=
  SHA512   : GYfa0DJwWgMLl4Goo5VFVOhu4BphXCo3
             rZnk49PYztwu50XjaAvsVuTjJY5uIYrG
             tV+jt3ELvwFzGefq4ZBNMg==
  CRC32    : /cusZw==
  HAVAL    : E/i5ceF3YTjwenBfyxHEsy9Kzu35VTf7
             CPGQSW4tl14=
  GOST     : n5Ityzxey9/1jIs7LMc08SULF1sLBFUc
             aMv7Oby604A=


End timestamp: 2019-04-01 21:26:03 -0400 (run time: 1m 18s)

Test everything works after making some changes.

On Debian based systems:

 sudo touch /etc/test.sh
 sudo touch /root/test.sh
    
 sudo aide.wrapper --check
    
 sudo rm /etc/test.sh
 sudo rm /root/test.sh
    
 sudo aideinit -y -f

Start timestamp: 2019-04-01 21:37:37 -0400 (AIDE 0.16)
AIDE found differences between database and filesystem!!
Verbose level: 6

Summary:
  Total number of entries:      25972
  Added entries:                2
  Removed entries:              0
  Changed entries:              1

---------------------------------------------------
Added entries:
---------------------------------------------------

f++++++++++++++++: /etc/test.sh
f++++++++++++++++: /root/test.sh

---------------------------------------------------
Changed entries:
---------------------------------------------------

d =.... mc.. .. .: /root

---------------------------------------------------
Detailed information about changes:
---------------------------------------------------

Directory: /root
  Mtime    : 2019-04-01 21:35:07 -0400        | 2019-04-01 21:37:36 -0400
  Ctime    : 2019-04-01 21:35:07 -0400        | 2019-04-01 21:37:36 -0400


---------------------------------------------------
The attributes of the (uncompressed) database(s):
---------------------------------------------------

/var/lib/aide/aide.db
  RMD160   : qF9WmKaf2PptjKnhcr9z4ueCPTY=
  TIGER    : zMo7MvvYJcq1hzvTQLPMW7ALeFiyEqv+
  SHA256   : LSLLVjjV6r8vlSxlbAbbEsPcQUB48SgP
             pdVqEn6ZNbQ=
  SHA512   : Qc4U7+ZAWCcitapGhJ1IrXCLGCf1IKZl
             02KYL1gaZ0Fm4dc7xLqjiquWDMSEbwzW
             oz49NCquqGz5jpMIUy7UxA==
  CRC32    : z8ChEA==
  HAVAL    : YapzS+/cdDwLj3kHJEq8fufLp3DPKZDg
             U12KCSkrO7Y=
  GOST     : 74sLV4HkTig+GJhokvxZQm7CJD/NR0mG
             6jV7zdt5AXQ=


End timestamp: 2019-04-01 21:38:50 -0400 (run time: 1m 13s)

That’s it. If you set CRON_DAILY_RUN to yes in /etc/default/aide then cron will execute /etc/cron.daily/aide every day and e-mail you the output.

Updating The Database

Every time you make changes to files/folders that AIDE monitors, you will need to update the database to capture those changes. To do that on Debian based systems:

sudo aideinit -y -f

Anti-Virus Scanning With ClamAV (WIP)

How It Works

ClamAV is a virus scanner
ClamAV-Freshclam is a service that keeps the virus definitions updated
ClamAV-Daemon keeps the clamd process running to make scanning faster

Notes

These instructions do not tell you how to enable the ClamAV daemon service to ensure clamd is running all the time. clamd is only if you’re running a mail server and does not provide real-time monitoring of files. Instead, you’d want to scan files manually or on a schedule.

Steps - Antivirus scan using ClamAV

Install ClamAV.

On Debian based systems:

 sudo apt install clamav clamav-freshclam clamav-daemon

Make a backup of clamav-freshclam’s configuration file /etc/clamav/freshclam.conf:

 sudo cp --archive /etc/clamav/freshclam.conf /etc/clamav/freshclam.conf-COPY-$(date +"%Y%m%d%H%M%S")

clamav-freshclam’s default settings are probably good enough but if you want to change them, you can either edit the file /etc/clamav/freshclam.conf or use dpkg-reconfigure:
```
 sudo dpkg-reconfigure clamav-freshclam
```
Note: The default settings will update the definitions 24 times in a day. To change the interval, check the Checks setting in /etc/clamav/freshclam.conf or use dpkg-reconfigure.
Start the clamav-freshclam service:
```
 sudo service clamav-freshclam start
```

You can make sure clamav-freshclam running:

 sudo service clamav-freshclam status

● clamav-freshclam.service - ClamAV virus database updater
   Loaded: loaded (/lib/systemd/system/clamav-freshclam.service; enabled; vendor preset: enabled)   Active: active (running) since Sat 2019-03-16 22:57:07 EDT; 2min 13s ago
     Docs: man:freshclam(1)
           man:freshclam.conf(5)
           https://www.clamav.net/documents
 Main PID: 1288 (freshclam)
   CGroup: /system.slice/clamav-freshclam.service
           └─1288 /usr/bin/freshclam -d --foreground=true

Mar 16 22:57:08 host freshclam[1288]: Sat Mar 16 22:57:08 2019 -> ^Local version: 0.100.2 Recommended version: 0.101.1
Mar 16 22:57:08 host freshclam[1288]: Sat Mar 16 22:57:08 2019 -> DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav
Mar 16 22:57:15 host freshclam[1288]: Sat Mar 16 22:57:15 2019 -> Downloading main.cvd [100%]
Mar 16 22:57:38 host freshclam[1288]: Sat Mar 16 22:57:38 2019 -> main.cvd updated (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
Mar 16 22:57:40 host freshclam[1288]: Sat Mar 16 22:57:40 2019 -> Downloading daily.cvd [100%]
Mar 16 22:58:13 host freshclam[1288]: Sat Mar 16 22:58:13 2019 -> daily.cvd updated (version: 25390, sigs: 1520006, f-level: 63, builder: raynman)
Mar 16 22:58:14 host freshclam[1288]: Sat Mar 16 22:58:14 2019 -> Downloading bytecode.cvd [100%]
Mar 16 22:58:16 host freshclam[1288]: Sat Mar 16 22:58:16 2019 -> bytecode.cvd updated (version: 328, sigs: 94, f-level: 63, builder: neo)
Mar 16 22:58:24 host freshclam[1288]: Sat Mar 16 22:58:24 2019 -> Database updated (6086349 signatures) from db.local.clamav.net (IP: 104.16.219.84)
Mar 16 22:58:24 host freshclam[1288]: Sat Mar 16 22:58:24 2019 -> ^Clamd was NOT notified: Can't connect to clamd through /var/run/clamav/clamd.ctl: No such file or directory

Note: Don’t worry about that Local version line. Check https://serverfault.com/questions/741299/is-there-a-way-to-keep-clamav-updated-on-debian-8 for more details.

Make a backup of clamav-daemon’s configuration file /etc/clamav/clamd.conf:

 sudo cp --archive /etc/clamav/clamd.conf /etc/clamav/clamd.conf-COPY-$(date +"%Y%m%d%H%M%S")

You can change clamav-daemon’s settings by editing the file /etc/clamav/clamd.conf or useing dpkg-reconfigure:
```
 sudo dpkg-reconfigure clamav-daemon
```

Scanning Files/Folders

To scan files/folders use the clamscan program.
clamscan runs as the user it is executed as so it needs read permissions to the files/folders it is scanning.
Using clamscan as root is dangerous because if a file is in fact a virus there is risk that it could use the root privileges.
To scan a file: clamscan /path/to/file.
To scan a directory: clamscan -r /path/to/folder.
You can use the -i switch to only print infected files.
Check clamscan’s man pages for other switches/options.

Rootkit Detection With Rkhunter (WIP)

Steps - How to detect Rootkits with Rkhunter

Install Rkhunter.

On Debian based systems:
```
 sudo apt install rkhunter
```

Make a backup of rkhunter’ defaults file:

 sudo cp -p /etc/default/rkhunter /etc/default/rkhunter-COPY-$(date +"%Y%m%d%H%M%S")

rkhunter’s configuration file is /etc/rkhunter.conf. Instead of making changes to it, create and use the file /etc/rkhunter.conf.local instead:
```
 sudo cp -p /etc/rkhunter.conf /etc/rkhunter.conf.local
```

Go through the configuration file /etc/rkhunter.conf.local and set to your requirements. My recommendations:

Setting	Note
`UPDATE_MIRRORS=1`
`MIRRORS_MODE=0`
`MAIL-ON-WARNING=root`
`COPY_LOG_ON_ERROR=1`	to save a copy of the log if there is an error
`PKGMGR=...`	set to the appropriate value per the documentation
`PHALANX2_DIRTEST=1`	read the documentation for why
`WEB_CMD=""`	this is to address an issue with the Debian package that disables the ability for rkhunter to self-update.
`USE_LOCKING=1`	to prevent issues with rkhunter running multiple times
`SHOW_SUMMARY_WARNINGS_NUMBER=1`	to see the actual number of warnings found

You want rkhunter to run every day and e-mail you the result. You can write your own script or check https://www.tecmint.com/install-rootkit-hunter-scan-for-rootkits-backdoors-in-linux/ for a sample cron script you can use.

On Debian based system, rkhunter comes with cron scripts. To enable them check /etc/default/rkhunter or use dpkg-reconfigure and say Yes to all of the questions:
```
 sudo dpkg-reconfigure rkhunter
```
After you’ve finished with all of the changes, make sure all the settings are valid:
```
 sudo rkhunter -C
```

Update rkhunter and its database:

 sudo rkhunter --versioncheck
 sudo rkhunter --update
 sudo rkhunter --propupd

If you want to do a manual scan and see the output:
```
 sudo rkhunter --check
```

Rootkit Detection With chrootkit (WIP)

Steps - How to detect Rootkits with chrootkit

Install chkrootkit.

On Debian based systems:
```
 sudo apt install chkrootkit
```

Do a manual scan:

 sudo chkrootkit

ROOTDIR is `/'
Checking `amd'...                                           not found
Checking `basename'...                                      not infected
Checking `biff'...                                          not found
Checking `chfn'...                                          not infected
Checking `chsh'...                                          not infected
...
Checking `scalper'...                                       not infected
Checking `slapper'...                                       not infected
Checking `z2'...                                            chklastlog: nothing deleted
Checking `chkutmp'...                                       chkutmp: nothing deleted
Checking `OSX_RSPLUG'...                                    not infected

Make a backup of chkrootkit’s configuration file /etc/chkrootkit.conf:

 sudo cp --archive /etc/chkrootkit.conf /etc/chkrootkit.conf-COPY-$(date +"%Y%m%d%H%M%S")

You want chkrootkit to run every day and e-mail you the result.

On Debian based system, chkrootkit comes with cron scripts. To enable them check /etc/chkrootkit.conf or use dpkg-reconfigure and say Yes to the first question:
```
 sudo dpkg-reconfigure chkrootkit
```

logwatch - system log analyzer and reporter

Your server will be generating a lot of logs that may contain important information. Unless you plan on checking your server everyday, you’ll want a way to get e-mail summary of your server’s logs. To accomplish this we’ll use logwatch.

How It Works

logwatch scans system log files and summarizes them. You can run it directly from the command line or schedule it to run on a recurring schedule. logwatch uses service files to know how to read/summarize a log file. You can see all of the stock service files in /usr/share/logwatch/scripts/services.

logwatch’s configuration file /usr/share/logwatch/default.conf/logwatch.conf specifies default options. You can override them via command line arguments.

Goals

Logwatch configured to send a daily e-mail summary of all of the server’s status and logs

Notes

Your server will need to be able to send e-mails for this to work
The below steps will result in logwatch running every day. If you want to change the schedule, modify the cronjob to your liking. You’ll also want to change the range option to cover your recurrence window. See https://www.badpenguin.org/configure-logwatch-for-weekly-email-and-html-output-format for an example.
If logwatch fails to deliver mail due to the e-mail having long lines please check https://blog.dhampir.no/content/exim4-line-length-in-debian-stretch-mail-delivery-failed-returning-message-to-sender as documented in issue #29. If you you followed Gmail and Exim4 As MTA With Implicit TLS then we already took care of this in step #7.

Steps

Install logwatch.

On Debian based systems:
```
 sudo apt install logwatch
```

To see a sample of what logwatch collects you can run it directly:

 sudo /usr/sbin/logwatch --output stdout --format text --range yesterday --service all

 ################### Logwatch 7.4.3 (12/07/16) ####################
        Processing Initiated: Mon Mar  4 00:05:50 2019
        Date Range Processed: yesterday
                              ( 2019-Mar-03 )
                              Period is day.
        Detail Level of Output: 5
        Type of Output/Format: stdout / text
        Logfiles for Host: host
 ##################################################################

 --------------------- Cron Begin ------------------------
...
...
 ---------------------- Disk Space End -------------------------


 ###################### Logwatch End #########################

Go through logwatch’s self-documented configuration file /usr/share/logwatch/default.conf/logwatch.conf before continuing. There is no need to change anything here but pay special attention to the Output, Format, MailTo, Range, and Service as those are the ones we’ll be using. For our purposes, instead of specifying our options in the configuration file, we will pass them as command line arguments in the daily cron job that executes logwatch. That way, if the configuration file is ever modified (e.g. during an update), our options will still be there.

Make a backup of logwatch’s daily cron file /etc/cron.daily/00logwatch and unset the execute bit:

 sudo cp --archive /etc/cron.daily/00logwatch /etc/cron.daily/00logwatch-COPY-$(date +"%Y%m%d%H%M%S")
 sudo chmod -x /etc/cron.daily/00logwatch-COPY*

By default, logwatch outputs to stdout. Since the goal is to get a daily e-mail, we need to change the output type that logwatch uses to send e-mail instead. We could do this through the configuration file above, but that would apply to every time it is run – even when we run it manually and want to see the output to the screen. Instead, we’ll change the cron job that executes logwatch to send e-mail. This way, when run manually, we’ll still get output to stdout and when run by cron, it’ll send an e-mail. We’ll also make sure it checks for all services, and change the output format to html so it’s easier to read regardless of what the configuration file says. In the file /etc/cron.daily/00logwatch find the execute line and change it to:

 /usr/sbin/logwatch --output mail --format html --mailto root --range yesterday --service all

#!/bin/bash

#Check if removed-but-not-purged
test -x /usr/share/logwatch/scripts/logwatch.pl || exit 0

#execute
/usr/sbin/logwatch --output mail --format html --mailto root --range yesterday --service all

#Note: It's possible to force the recipient in above command
#Just pass --mailto [email protected] instead of --output mail

 sudo sed -i -r -e "s,^($(sudo which logwatch).*?),# \1         # commented by $(whoami) on $(date +"%Y-%m-%d @ %H:%M:%S")\n$(sudo which logwatch) --output mail --format html --mailto root --range yesterday --service all         # added by $(whoami) on $(date +"%Y-%m-%d @ %H:%M:%S")," /etc/cron.daily/00logwatch

You can test the cron job by executing it:
```
 sudo /etc/cron.daily/00logwatch
```
Note: If logwatch fails to deliver mail due to the e-mail having long lines please check https://blog.dhampir.no/content/exim4-line-length-in-debian-stretch-mail-delivery-failed-returning-message-to-sender as documented in issue #29. If you you followed Gmail and Exim4 As MTA With Implicit TLS then we already took care of this in step #7.

ss - Seeing Ports Your Server Is Listening On

Ports are how applications, services, and processes communicate with each other – either locally within your server or with other devices on the network. When you have an application or service (like SSH or Apache) running on your server, they listen for requests on specific ports.

Obviously we don’t want your server listening on ports we don’t know about. We’ll use ss to see all the ports that services are listening on. This will help us track down and stop rogue, potentially dangerous, services.

Goals

find out non-localhost what ports are open and listening for connections

Steps

To see the all the ports listening for traffic:

 sudo ss -lntup

Netid  State      Recv-Q Send-Q     Local Address:Port     Peer Address:Port
udp    UNCONN     0      0                      *:68                  *:*        users:(("dhclient",pid=389,fd=6))
tcp    LISTEN     0      128                    *:22                  *:*        users:(("sshd",pid=4390,fd=3))
tcp    LISTEN     0      128                   :::22                 :::*        users:(("sshd",pid=4390,fd=4))

Switch Explanations:

l = display listening sockets
n = do now try to resolve service names
t = display TCP sockets
u = display UDP sockets
p = show process information

If you see anything suspicious, like a port you’re not aware of or a process you don’t know, investigate and remediate as necessary.

Lynis - Linux Security Auditing

From https://cisofy.com/lynis/:

Lynis is a battle-tested security tool for systems running Linux, macOS, or Unix-based operating system. It performs an extensive health scan of your systems to support system hardening and compliance testing.

Notes

CISOFY offers packages for many distributions. Check https://packages.cisofy.com/ for distribution specific installation instructions.

Steps

Install lynis. https://cisofy.com/lynis/#installation has detailed instructions on how to install it for your distribution.

On Debian based systems, using CISOFY’s community software repository:

 sudo apt install apt-transport-https ca-certificates host
 sudo wget -O - https://packages.cisofy.com/keys/cisofy-software-public.key | sudo apt-key add -
 sudo echo "deb https://packages.cisofy.com/community/lynis/deb/ stable main" | sudo tee /etc/apt/sources.list.d/cisofy-lynis.list
 sudo apt update
 sudo apt install lynis host

Update it:
```
 sudo lynis update info
```
Run a security audit:
```
 sudo lynis audit system
```
This will scan your server, report its audit findings, and at the end it will give you suggestions. Spend some time going through the output and address gaps as necessary.

OSSEC - Host Intrusion Detection

From https://github.com/ossec/ossec-hids

OSSEC is a full platform to monitor and control your systems. It mixes together all the aspects of HIDS (host-based intrusion detection), log monitoring and SIM/SIEM together in a simple, powerful and open source solution.

Steps

Install OSSEC-HIDS from sources

 sudo apt install -y libz-dev libssl-dev libpcre2-dev build-essential libsystemd-dev
 wget https://github.com/ossec/ossec-hids/archive/3.7.0.tar.gz
 tar xzf 3.7.0.tar.gz
 cd ossec-hids-3.7.0/
 sudo ./install.sh

Useful commands:

Agent information

    sudo /var/ossec/bin/agent_control -i <AGENT_ID>

AGENT_ID by default is 000, to be sure the command sudo /var/ossec/bin/agent_control -l can be used.

Run integrity/rootkit checking

OSSEC by default run rootkit check each 2 hours.

    sudo /var/ossec/bin/agent_control -u <AGENT_ID> -r 

Alerts

All:

  tail -f /var/ossec/logs/alerts/alerts.log

Integrity check:

  sudo cat /var/ossec/logs/alerts/alerts.log | grep -A4  -i integrity

Rootkit check:

   sudo cat /var/ossec/logs/alerts/alerts.log | grep -A4  "rootcheck,"

Licenses and Attributions

Speak Your Mind Cancel reply

-->

Related Books

Books / How To Secure A Linux Server / Chapter 5

Auditing Linux Security

In this chapter

File/Folder Integrity Monitoring With AIDE (WIP)

Steps

Updating The Database

Anti-Virus Scanning With ClamAV (WIP)

How It Works

Notes

Steps - Antivirus scan using ClamAV

Scanning Files/Folders

Rootkit Detection With Rkhunter (WIP)

Steps - How to detect Rootkits with Rkhunter

Rootkit Detection With chrootkit (WIP)

Steps - How to detect Rootkits with chrootkit

logwatch - system log analyzer and reporter

How It Works

Goals

Notes

Steps

ss - Seeing Ports Your Server Is Listening On

Goals

Steps

Lynis - Linux Security Auditing

Notes

Steps

OSSEC - Host Intrusion Detection

Steps

Licenses and Attributions

Original Content License

Speak Your Mind Cancel reply