Books / How To Secure A Linux Server / Chapter 5
Auditing Linux Security
In this chapter
- File/Folder Integrity Monitoring With AIDE (WIP)
- Anti-Virus Scanning With ClamAV (WIP)
- Rootkit Detection With Rkhunter (WIP)
- Rootkit Detection With chrootkit (WIP)
- logwatch - system log analyzer and reporter
- ss - Seeing Ports Your Server Is Listening On
- Lynis - Linux Security Auditing
- OSSEC - Host Intrusion Detection
File/Folder Integrity Monitoring With AIDE (WIP)
Steps
-
Install AIDE.
On Debian based systems:
sudo apt install aide aide-common
-
Make a backup of AIDE’s defaults file:
sudo cp -p /etc/default/aide /etc/default/aide-COPY-$(date +"%Y%m%d%H%M%S")
-
Go through
/etc/default/aide
and set AIDE’s defaults per your requirements. If you want AIDE to run daily and e-mail you, be sure to setCRON_DAILY_RUN
toyes
. -
Make a backup of AIDE’s configuration files:
sudo cp -pr /etc/aide /etc/aide-COPY-$(date +"%Y%m%d%H%M%S")
-
On Debian based systems:
- AIDE’s configuration files are in
/etc/aide/aide.conf.d/
. - You’ll want to go through AIDE’s documentation and the configuration files in to set them per your requirements.
- If you want new settings, to monitor a new folder for example, you’ll want to add them to
/etc/aide/aide.conf
or/etc/aide/aide.conf.d/
. - Take a backup of the stock configuration files:
sudo cp -pr /etc/aide /etc/aide-COPY-$(date +"%Y%m%d%H%M%S")
.
- AIDE’s configuration files are in
-
Create a new database, and install it.
On Debian based systems:
sudo aideinit
Running aide --init... Start timestamp: 2019-04-01 21:23:37 -0400 (AIDE 0.16) AIDE initialized database at /var/lib/aide/aide.db.new Verbose level: 6 Number of entries: 25973 --------------------------------------------------- The attributes of the (uncompressed) database(s): --------------------------------------------------- /var/lib/aide/aide.db.new RMD160 : moyQ1YskQQbidX+Lusv3g2wf1gQ= TIGER : 7WoOgCrXzSpDrlO6I3PyXPj1gRiaMSeo SHA256 : gVx8Fp7r3800WF2aeXl+/KHCzfGsNi7O g16VTPpIfYQ= SHA512 : GYfa0DJwWgMLl4Goo5VFVOhu4BphXCo3 rZnk49PYztwu50XjaAvsVuTjJY5uIYrG tV+jt3ELvwFzGefq4ZBNMg== CRC32 : /cusZw== HAVAL : E/i5ceF3YTjwenBfyxHEsy9Kzu35VTf7 CPGQSW4tl14= GOST : n5Ityzxey9/1jIs7LMc08SULF1sLBFUc aMv7Oby604A= End timestamp: 2019-04-01 21:24:45 -0400 (run time: 1m 8s)
-
Test everything works with no changes.
On Debian based systems:
sudo aide.wrapper --check
Start timestamp: 2019-04-01 21:24:45 -0400 (AIDE 0.16) AIDE found NO differences between database and filesystem. Looks okay!! Verbose level: 6 Number of entries: 25973 --------------------------------------------------- The attributes of the (uncompressed) database(s): --------------------------------------------------- /var/lib/aide/aide.db RMD160 : moyQ1YskQQbidX+Lusv3g2wf1gQ= TIGER : 7WoOgCrXzSpDrlO6I3PyXPj1gRiaMSeo SHA256 : gVx8Fp7r3800WF2aeXl+/KHCzfGsNi7O g16VTPpIfYQ= SHA512 : GYfa0DJwWgMLl4Goo5VFVOhu4BphXCo3 rZnk49PYztwu50XjaAvsVuTjJY5uIYrG tV+jt3ELvwFzGefq4ZBNMg== CRC32 : /cusZw== HAVAL : E/i5ceF3YTjwenBfyxHEsy9Kzu35VTf7 CPGQSW4tl14= GOST : n5Ityzxey9/1jIs7LMc08SULF1sLBFUc aMv7Oby604A= End timestamp: 2019-04-01 21:26:03 -0400 (run time: 1m 18s)
-
Test everything works after making some changes.
On Debian based systems:
sudo touch /etc/test.sh sudo touch /root/test.sh sudo aide.wrapper --check sudo rm /etc/test.sh sudo rm /root/test.sh sudo aideinit -y -f
Start timestamp: 2019-04-01 21:37:37 -0400 (AIDE 0.16) AIDE found differences between database and filesystem!! Verbose level: 6 Summary: Total number of entries: 25972 Added entries: 2 Removed entries: 0 Changed entries: 1 --------------------------------------------------- Added entries: --------------------------------------------------- f++++++++++++++++: /etc/test.sh f++++++++++++++++: /root/test.sh --------------------------------------------------- Changed entries: --------------------------------------------------- d =.... mc.. .. .: /root --------------------------------------------------- Detailed information about changes: --------------------------------------------------- Directory: /root Mtime : 2019-04-01 21:35:07 -0400 | 2019-04-01 21:37:36 -0400 Ctime : 2019-04-01 21:35:07 -0400 | 2019-04-01 21:37:36 -0400 --------------------------------------------------- The attributes of the (uncompressed) database(s): --------------------------------------------------- /var/lib/aide/aide.db RMD160 : qF9WmKaf2PptjKnhcr9z4ueCPTY= TIGER : zMo7MvvYJcq1hzvTQLPMW7ALeFiyEqv+ SHA256 : LSLLVjjV6r8vlSxlbAbbEsPcQUB48SgP pdVqEn6ZNbQ= SHA512 : Qc4U7+ZAWCcitapGhJ1IrXCLGCf1IKZl 02KYL1gaZ0Fm4dc7xLqjiquWDMSEbwzW oz49NCquqGz5jpMIUy7UxA== CRC32 : z8ChEA== HAVAL : YapzS+/cdDwLj3kHJEq8fufLp3DPKZDg U12KCSkrO7Y= GOST : 74sLV4HkTig+GJhokvxZQm7CJD/NR0mG 6jV7zdt5AXQ= End timestamp: 2019-04-01 21:38:50 -0400 (run time: 1m 13s)
-
That’s it. If you set
CRON_DAILY_RUN
toyes
in/etc/default/aide
then cron will execute/etc/cron.daily/aide
every day and e-mail you the output.
Updating The Database
Every time you make changes to files/folders that AIDE monitors, you will need to update the database to capture those changes. To do that on Debian based systems:
sudo aideinit -y -f
Anti-Virus Scanning With ClamAV (WIP)
How It Works
- ClamAV is a virus scanner
- ClamAV-Freshclam is a service that keeps the virus definitions updated
- ClamAV-Daemon keeps the
clamd
process running to make scanning faster
Notes
- These instructions do not tell you how to enable the ClamAV daemon service to ensure
clamd
is running all the time.clamd
is only if you’re running a mail server and does not provide real-time monitoring of files. Instead, you’d want to scan files manually or on a schedule.
Steps - Antivirus scan using ClamAV
-
Install ClamAV.
On Debian based systems:
sudo apt install clamav clamav-freshclam clamav-daemon
-
Make a backup of
clamav-freshclam
’s configuration file/etc/clamav/freshclam.conf
:sudo cp --archive /etc/clamav/freshclam.conf /etc/clamav/freshclam.conf-COPY-$(date +"%Y%m%d%H%M%S")
-
clamav-freshclam
’s default settings are probably good enough but if you want to change them, you can either edit the file/etc/clamav/freshclam.conf
or usedpkg-reconfigure
:sudo dpkg-reconfigure clamav-freshclam
Note: The default settings will update the definitions 24 times in a day. To change the interval, check the
Checks
setting in/etc/clamav/freshclam.conf
or usedpkg-reconfigure
. -
Start the
clamav-freshclam
service:sudo service clamav-freshclam start
-
You can make sure
clamav-freshclam
running:sudo service clamav-freshclam status
● clamav-freshclam.service - ClamAV virus database updater Loaded: loaded (/lib/systemd/system/clamav-freshclam.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2019-03-16 22:57:07 EDT; 2min 13s ago Docs: man:freshclam(1) man:freshclam.conf(5) https://www.clamav.net/documents Main PID: 1288 (freshclam) CGroup: /system.slice/clamav-freshclam.service └─1288 /usr/bin/freshclam -d --foreground=true Mar 16 22:57:08 host freshclam[1288]: Sat Mar 16 22:57:08 2019 -> ^Local version: 0.100.2 Recommended version: 0.101.1 Mar 16 22:57:08 host freshclam[1288]: Sat Mar 16 22:57:08 2019 -> DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav Mar 16 22:57:15 host freshclam[1288]: Sat Mar 16 22:57:15 2019 -> Downloading main.cvd [100%] Mar 16 22:57:38 host freshclam[1288]: Sat Mar 16 22:57:38 2019 -> main.cvd updated (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr) Mar 16 22:57:40 host freshclam[1288]: Sat Mar 16 22:57:40 2019 -> Downloading daily.cvd [100%] Mar 16 22:58:13 host freshclam[1288]: Sat Mar 16 22:58:13 2019 -> daily.cvd updated (version: 25390, sigs: 1520006, f-level: 63, builder: raynman) Mar 16 22:58:14 host freshclam[1288]: Sat Mar 16 22:58:14 2019 -> Downloading bytecode.cvd [100%] Mar 16 22:58:16 host freshclam[1288]: Sat Mar 16 22:58:16 2019 -> bytecode.cvd updated (version: 328, sigs: 94, f-level: 63, builder: neo) Mar 16 22:58:24 host freshclam[1288]: Sat Mar 16 22:58:24 2019 -> Database updated (6086349 signatures) from db.local.clamav.net (IP: 104.16.219.84) Mar 16 22:58:24 host freshclam[1288]: Sat Mar 16 22:58:24 2019 -> ^Clamd was NOT notified: Can't connect to clamd through /var/run/clamav/clamd.ctl: No such file or directory
Note: Don’t worry about that
Local version
line. Check https://serverfault.com/questions/741299/is-there-a-way-to-keep-clamav-updated-on-debian-8 for more details. -
Make a backup of
clamav-daemon
’s configuration file/etc/clamav/clamd.conf
:sudo cp --archive /etc/clamav/clamd.conf /etc/clamav/clamd.conf-COPY-$(date +"%Y%m%d%H%M%S")
-
You can change
clamav-daemon
’s settings by editing the file/etc/clamav/clamd.conf
or useingdpkg-reconfigure
:sudo dpkg-reconfigure clamav-daemon
Scanning Files/Folders
- To scan files/folders use the
clamscan
program. clamscan
runs as the user it is executed as so it needs read permissions to the files/folders it is scanning.- Using
clamscan
asroot
is dangerous because if a file is in fact a virus there is risk that it could use the root privileges. - To scan a file:
clamscan /path/to/file
. - To scan a directory:
clamscan -r /path/to/folder
. - You can use the
-i
switch to only print infected files. - Check
clamscan
’sman
pages for other switches/options.
Rootkit Detection With Rkhunter (WIP)
Steps - How to detect Rootkits with Rkhunter
-
Install Rkhunter.
On Debian based systems:
sudo apt install rkhunter
-
Make a backup of rkhunter’ defaults file:
sudo cp -p /etc/default/rkhunter /etc/default/rkhunter-COPY-$(date +"%Y%m%d%H%M%S")
-
rkhunter’s configuration file is
/etc/rkhunter.conf
. Instead of making changes to it, create and use the file/etc/rkhunter.conf.local
instead:sudo cp -p /etc/rkhunter.conf /etc/rkhunter.conf.local
-
Go through the configuration file
/etc/rkhunter.conf.local
and set to your requirements. My recommendations:Setting Note UPDATE_MIRRORS=1
MIRRORS_MODE=0
MAIL-ON-WARNING=root
COPY_LOG_ON_ERROR=1
to save a copy of the log if there is an error PKGMGR=...
set to the appropriate value per the documentation PHALANX2_DIRTEST=1
read the documentation for why WEB_CMD=""
this is to address an issue with the Debian package that disables the ability for rkhunter to self-update. USE_LOCKING=1
to prevent issues with rkhunter running multiple times SHOW_SUMMARY_WARNINGS_NUMBER=1
to see the actual number of warnings found -
You want rkhunter to run every day and e-mail you the result. You can write your own script or check https://www.tecmint.com/install-rootkit-hunter-scan-for-rootkits-backdoors-in-linux/ for a sample cron script you can use.
On Debian based system, rkhunter comes with cron scripts. To enable them check
/etc/default/rkhunter
or usedpkg-reconfigure
and sayYes
to all of the questions:sudo dpkg-reconfigure rkhunter
-
After you’ve finished with all of the changes, make sure all the settings are valid:
sudo rkhunter -C
-
Update rkhunter and its database:
sudo rkhunter --versioncheck sudo rkhunter --update sudo rkhunter --propupd
-
If you want to do a manual scan and see the output:
sudo rkhunter --check
Rootkit Detection With chrootkit (WIP)
Steps - How to detect Rootkits with chrootkit
-
Install chkrootkit.
On Debian based systems:
sudo apt install chkrootkit
-
Do a manual scan:
sudo chkrootkit
ROOTDIR is `/' Checking `amd'... not found Checking `basename'... not infected Checking `biff'... not found Checking `chfn'... not infected Checking `chsh'... not infected ... Checking `scalper'... not infected Checking `slapper'... not infected Checking `z2'... chklastlog: nothing deleted Checking `chkutmp'... chkutmp: nothing deleted Checking `OSX_RSPLUG'... not infected
-
Make a backup of chkrootkit’s configuration file
/etc/chkrootkit.conf
:sudo cp --archive /etc/chkrootkit.conf /etc/chkrootkit.conf-COPY-$(date +"%Y%m%d%H%M%S")
-
You want chkrootkit to run every day and e-mail you the result.
On Debian based system, chkrootkit comes with cron scripts. To enable them check
/etc/chkrootkit.conf
or usedpkg-reconfigure
and sayYes
to the first question:sudo dpkg-reconfigure chkrootkit
logwatch - system log analyzer and reporter
Your server will be generating a lot of logs that may contain important information. Unless you plan on checking your server everyday, you’ll want a way to get e-mail summary of your server’s logs. To accomplish this we’ll use logwatch.
How It Works
logwatch scans system log files and summarizes them. You can run it directly from the command line or schedule it to run on a recurring schedule. logwatch uses service files to know how to read/summarize a log file. You can see all of the stock service files in /usr/share/logwatch/scripts/services
.
logwatch’s configuration file /usr/share/logwatch/default.conf/logwatch.conf
specifies default options. You can override them via command line arguments.
Goals
- Logwatch configured to send a daily e-mail summary of all of the server’s status and logs
Notes
- Your server will need to be able to send e-mails for this to work
- The below steps will result in logwatch running every day. If you want to change the schedule, modify the cronjob to your liking. You’ll also want to change the
range
option to cover your recurrence window. See https://www.badpenguin.org/configure-logwatch-for-weekly-email-and-html-output-format for an example. - If logwatch fails to deliver mail due to the e-mail having long lines please check https://blog.dhampir.no/content/exim4-line-length-in-debian-stretch-mail-delivery-failed-returning-message-to-sender as documented in issue #29. If you you followed Gmail and Exim4 As MTA With Implicit TLS then we already took care of this in step #7.
Steps
-
Install logwatch.
On Debian based systems:
sudo apt install logwatch
-
To see a sample of what logwatch collects you can run it directly:
sudo /usr/sbin/logwatch --output stdout --format text --range yesterday --service all
################### Logwatch 7.4.3 (12/07/16) #################### Processing Initiated: Mon Mar 4 00:05:50 2019 Date Range Processed: yesterday ( 2019-Mar-03 ) Period is day. Detail Level of Output: 5 Type of Output/Format: stdout / text Logfiles for Host: host ################################################################## --------------------- Cron Begin ------------------------ ... ... ---------------------- Disk Space End ------------------------- ###################### Logwatch End #########################
-
Go through logwatch’s self-documented configuration file
/usr/share/logwatch/default.conf/logwatch.conf
before continuing. There is no need to change anything here but pay special attention to theOutput
,Format
,MailTo
,Range
, andService
as those are the ones we’ll be using. For our purposes, instead of specifying our options in the configuration file, we will pass them as command line arguments in the daily cron job that executes logwatch. That way, if the configuration file is ever modified (e.g. during an update), our options will still be there. -
Make a backup of logwatch’s daily cron file
/etc/cron.daily/00logwatch
and unset the execute bit:sudo cp --archive /etc/cron.daily/00logwatch /etc/cron.daily/00logwatch-COPY-$(date +"%Y%m%d%H%M%S") sudo chmod -x /etc/cron.daily/00logwatch-COPY*
-
By default, logwatch outputs to
stdout
. Since the goal is to get a daily e-mail, we need to change the output type that logwatch uses to send e-mail instead. We could do this through the configuration file above, but that would apply to every time it is run – even when we run it manually and want to see the output to the screen. Instead, we’ll change the cron job that executes logwatch to send e-mail. This way, when run manually, we’ll still get output tostdout
and when run by cron, it’ll send an e-mail. We’ll also make sure it checks for all services, and change the output format to html so it’s easier to read regardless of what the configuration file says. In the file/etc/cron.daily/00logwatch
find the execute line and change it to:/usr/sbin/logwatch --output mail --format html --mailto root --range yesterday --service all
#!/bin/bash #Check if removed-but-not-purged test -x /usr/share/logwatch/scripts/logwatch.pl || exit 0 #execute /usr/sbin/logwatch --output mail --format html --mailto root --range yesterday --service all #Note: It's possible to force the recipient in above command #Just pass --mailto [email protected] instead of --output mail
sudo sed -i -r -e "s,^($(sudo which logwatch).*?),# \1 # commented by $(whoami) on $(date +"%Y-%m-%d @ %H:%M:%S")\n$(sudo which logwatch) --output mail --format html --mailto root --range yesterday --service all # added by $(whoami) on $(date +"%Y-%m-%d @ %H:%M:%S")," /etc/cron.daily/00logwatch
-
You can test the cron job by executing it:
sudo /etc/cron.daily/00logwatch
Note: If logwatch fails to deliver mail due to the e-mail having long lines please check https://blog.dhampir.no/content/exim4-line-length-in-debian-stretch-mail-delivery-failed-returning-message-to-sender as documented in issue #29. If you you followed Gmail and Exim4 As MTA With Implicit TLS then we already took care of this in step #7.
ss - Seeing Ports Your Server Is Listening On
Ports are how applications, services, and processes communicate with each other – either locally within your server or with other devices on the network. When you have an application or service (like SSH or Apache) running on your server, they listen for requests on specific ports.
Obviously we don’t want your server listening on ports we don’t know about. We’ll use ss
to see all the ports that services are listening on. This will help us track down and stop rogue, potentially dangerous, services.
Goals
- find out non-localhost what ports are open and listening for connections
Steps
-
To see the all the ports listening for traffic:
sudo ss -lntup
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port udp UNCONN 0 0 *:68 *:* users:(("dhclient",pid=389,fd=6)) tcp LISTEN 0 128 *:22 *:* users:(("sshd",pid=4390,fd=3)) tcp LISTEN 0 128 :::22 :::* users:(("sshd",pid=4390,fd=4))
Switch Explanations:
l
= display listening socketsn
= do now try to resolve service namest
= display TCP socketsu
= display UDP socketsp
= show process information
-
If you see anything suspicious, like a port you’re not aware of or a process you don’t know, investigate and remediate as necessary.
Lynis - Linux Security Auditing
From https://cisofy.com/lynis/:
Lynis is a battle-tested security tool for systems running Linux, macOS, or Unix-based operating system. It performs an extensive health scan of your systems to support system hardening and compliance testing.
Notes
- CISOFY offers packages for many distributions. Check https://packages.cisofy.com/ for distribution specific installation instructions.
Steps
-
Install lynis. https://cisofy.com/lynis/#installation has detailed instructions on how to install it for your distribution.
On Debian based systems, using CISOFY’s community software repository:
sudo apt install apt-transport-https ca-certificates host sudo wget -O - https://packages.cisofy.com/keys/cisofy-software-public.key | sudo apt-key add - sudo echo "deb https://packages.cisofy.com/community/lynis/deb/ stable main" | sudo tee /etc/apt/sources.list.d/cisofy-lynis.list sudo apt update sudo apt install lynis host
-
Update it:
sudo lynis update info
-
Run a security audit:
sudo lynis audit system
This will scan your server, report its audit findings, and at the end it will give you suggestions. Spend some time going through the output and address gaps as necessary.
OSSEC - Host Intrusion Detection
From https://github.com/ossec/ossec-hids
OSSEC is a full platform to monitor and control your systems. It mixes together all the aspects of HIDS (host-based intrusion detection), log monitoring and SIM/SIEM together in a simple, powerful and open source solution.
Steps
- Install OSSEC-HIDS from sources
sudo apt install -y libz-dev libssl-dev libpcre2-dev build-essential libsystemd-dev wget https://github.com/ossec/ossec-hids/archive/3.7.0.tar.gz tar xzf 3.7.0.tar.gz cd ossec-hids-3.7.0/ sudo ./install.sh
- Useful commands:
Agent information
sudo /var/ossec/bin/agent_control -i <AGENT_ID>
AGENT_ID
by default is 000
, to be sure the command sudo /var/ossec/bin/agent_control -l
can be used.
Run integrity/rootkit checking
OSSEC by default run rootkit check each 2 hours.
sudo /var/ossec/bin/agent_control -u <AGENT_ID> -r
Alerts
- All:
tail -f /var/ossec/logs/alerts/alerts.log
- Integrity check:
sudo cat /var/ossec/logs/alerts/alerts.log | grep -A4 -i integrity
- Rootkit check:
sudo cat /var/ossec/logs/alerts/alerts.log | grep -A4 "rootcheck,"