Get best of the week

The book "Bash and cybersecurity: attack, defense and analysis from the Linux command line"

imageHello, habrozhiteli! The command line can be an ideal tool for cybersecurity. Incredible flexibility and absolute availability turn the standard command line interface (CLI) into a fundamental solution if you have the appropriate experience.

Authors Paul Tronkon and Karl Albing talk about tools and tricks of the command line that help to collect data with proactive protection, analyze logs and monitor network status. Pentesters will learn how to conduct attacks using colossal functionality built into almost any version of Linux.


Who is this book for?


The book "Bash and cybersecurity" is intended for those who wish to learn how to work with the command line in the context of computer security. Our goal is not to replace existing tools with command-line scripts, but rather to teach you how to effectively use the command line to improve existing functionality.

Throughout the book, we provide examples of security methods such as data collection, analysis and penetration testing. The purpose of these examples is to demonstrate command line capabilities and introduce you to some of the fundamental methods used in higher-level tools.

This book is not an introduction to programming, although some general concepts are covered in Part I.

Real-time log monitoring


The ability to analyze a magazine after an event has occurred is an important skill. But it is equally important to be able to extract information from the log file in real time in order to detect malicious or suspicious actions at the time they occur. In this chapter, we will consider methods for reading journal entries as they are created and formatted to display analytics and generate alerts based on known threat indicators for the operation of a system or network (indicators of compromise).

Text Log Monitoring


The easiest way to monitor the log in real time is to use the tail command with the –f parameter - it continuously reads the file and displays them in stdout as new lines are added. As in previous chapters, for examples we will use the Apache web server access log, but the described methods are relevant for any text log. To track the Apache access log using the tail command, enter the following:

tail -f /var/logs/apache2/access.log

The output from the tail command can be passed to the grep command, so only records matching certain criteria will be displayed. The following example tracks the Apache access log and displays entries corresponding to a specific IP address:

tail -f /var/logs/apache2/access.log | grep '10.0.0.152'

You can also use regular expressions. In this example, only records returning the HTTP 404 status code “Page not found” will be displayed; the -i option is added to ignore the case of characters:

tail -f /var/logs/apache2/access.log | egrep -i 'HTTP/.*" 404'

To clear from extraneous information, the output should be passed to the cut command. This example monitors the access log for queries leading to status code 404, and then uses the cut method to display only the date / time and the requested page: Next, to remove the square brackets and double quotes, you can direct the output to tr -d ' [] "'.

$ tail -f access.log | egrep --line-buffered 'HTTP/.*" 404' | cut -d' ' -f4-7
[29/Jul/2018:13:10:05 -0400] "GET /test
[29/Jul/2018:13:16:17 -0400] "GET /test.txt
[29/Jul/2018:13:17:37 -0400] "GET /favicon.ico



Note: the --line-buffering option of the egrep command is used here. This forces egrep to print to stdout every time a line break occurs. Without this parameter, buffering will occur and the output will not be sent to the cut command until the buffer is full. We do not want to wait so long. This option allows the egrep command to write each line as it is found.

COMMAND LINE BUFFERS

What happens during buffering? Imagine egrep finds a lot of strings matching the specified pattern. In this case, egrep will have a lot of output. But the output (in fact, any input or output) is much more expensive (takes more time) than data processing (text search). Thus, the fewer I / O calls, the more efficient the program will be.

grep , , . . grep . , grep 50 . , 50 , , . 50 !

egrep, , . egrep , , . , , , , . , .

, , tail -f ( ), . , « », . . .

, egrep , . , .


You can use the tail and egrep commands to monitor the log and display any entries that correspond to known patterns of suspicious or malicious activity, often called IOCs. You can create a simple intrusion detection system (IDS). First, create a file containing the regular expression patterns for the IOC, as shown in Example 8.1.

Example 8.1. ioc.txt (1) This template (../) is an indicator of a roundabout directory attack: an attacker tries to exit the current working directory and get to files that are not accessible to him. (2) Linux etc / passwd and etc / shadow files are used for system authentication and should never be accessed through a web server. (3)

\.\./ (1)
etc/passwd (2)
etc/shadow
cmd\.exe (3)
/bin/sh
/bin/bash





Serving the cmd.exe, / bin / sh, or / bin / bash files is an indication of the reverse connection returned by the web server. A reverse connection often indicates a successful operation attempt.

Note that IOCs must be in regex format, as they will be used later by the egrep command.

The ioc.txt file can be used with the egrep -f option. This parameter tells egrep to search the regular expression patterns from the specified file. This allows you to use the tail command to monitor the log file, and as each record is added, the read line will be compared with all the templates in the IOC file, displaying any corresponding record. Here is an example:

tail -f /var/logs/apache2/access.log | egrep -i -f ioc.txt

In addition, the tee command can be used to simultaneously display warnings on the screen and save them for later processing in its own file: Again, the --line-buffered option is needed to ensure that there are no problems caused by buffering the output of the command.

tail -f /var/logs/apache2/access.log | egrep --line-buffered -i -f ioc.txt |
tee -a interesting.txt



Windows Log Monitoring


As already mentioned, you must use the wevtutil command to access Windows events. Although this command is universal, it does not have such functionality as tail, which can be used to extract new incoming records. But there is a way out - use a simple bash script that can provide the same functionality (Example 8.2).

Example 8.2. wintail.sh

#!/bin/bash -
#
# Bash  
# wintail.sh
#
# :
#    tail   Windows
#
# : ./wintail.sh
#

WINLOG="Application" (1)

LASTLOG=$(wevtutil qe "$WINLOG" //c:1 //rd:true //f:text) (2)

while true
do
      CURRENTLOG=$(wevtutil qe "$WINLOG" //c:1 //rd:true //f:text) (3)
      if [[ "$CURRENTLOG" != "$LASTLOG" ]]
      then
            echo "$CURRENTLOG"
            echo "----------------------------------"
            LASTLOG="$CURRENTLOG"
      fi
done

(1) This variable defines the Windows log that you want to track. For a list of the logs currently available on the system, you can use the wevtutil el command.

(2) Here, wevtutil is executed to request the specified log file. The c: 1 parameter returns only one log entry. The rd: true parameter allows the command to read the most recent log entry. Finally, f: text returns the result in plain text, not in XML format, which makes it easy to read the result from the screen.

(3)The next few lines run the wevtutil command again and the newly received log entry is compared to the last one printed on the screen. If they differ from each other, this means that there have been changes in the log. In this case, a new entry is displayed. If the compared entries are the same, nothing happens and the wevtutil command goes back and starts searching and comparing again.

Real-time histogram creation


The tail -f command provides the current data stream. But what if you want to count the number of lines that were added to the file for a certain period of time? You can observe this data stream, start a timer and perform counting over a specified period of time; then the counting should be stopped and the results reported.

This work can be divided into two script processes: one script will read the lines, and the other will watch the time. The timer notifies the line counter using a standard POSIX interprocess communication mechanism called a signal. A signal is a software interrupt, and there are various kinds of signals. Some of them are fatal - they lead to the end of the process (for example, an exception in a floating point operation). Most of these signals can be either ignored or caught. Action is taken when a signal is caught. Many of these signals have a predefined purpose in the operating system. We will use one of two signals available to users. This is the signal SIGUSR1 (the other is SIGUSR2).

Shell scripts can catch interrupts using the built-in trap command. Using it, you can select a command that determines what action you want to perform when a signal is received, and a list of signals that trigger a call to this command. For example:

trap warnmsg SIGINT

This causes the warnmsg command (our own script or function) to be called whenever the shell script receives a SIGINT signal, for example, when you press Ctrl + C to interrupt a running process.

Example 8.3 shows a script that performs a count.

Example 8.3 looper.sh

#!/bin/bash -
#
# Bash  
# looper.sh
#
# :
#    
#
# : ./looper.sh [filename]
# filename —  ,   ,
#  : log.file
#

function interval ()                                           (1)
{
      echo $(date '+%y%m%d %H%M%S') $cnt                       (2)
      cnt=0
}

declare -i cnt=0
trap interval SIGUSR1                                          (3)

shopt -s lastpipe                                              (4)

tail -f --pid=$$ ${1:-log.file} | while read aline             (5)
do
     let cnt++
done

(1) The interval function will be called upon receipt of each signal. Of course, the interval must be defined before we can name it and use trap in our expression.

(2) The date command is invoked to provide a timestamp for the value of the cnt variable that we are printing. After the counter is displayed, we reset this value to 0 to begin the countdown of the next interval.

(3) Now that the interval is defined, we can indicate that the function is called whenever our process receives the signal SIGUSR1.

(4)This is a very important step. Usually, when there is a pipeline of commands (for example, ls-l | grep rwx | wc), parts of the pipeline (each command) are executed on subnets and each process ends with its own process identifier. This could be a problem for this scenario, because the while loop will be in a subshell with a different process identifier. Whatever process starts, the looper.sh script will not know the identifier of the while loop process to send a signal to it. In addition, changing the value of the variable cnt in the subshell does not change the value of cnt in the main process, so the signal for the main process will each time set the value to 0. To solve this problem, use the shopt command, which sets the -p parameter to lastpipe. It tells the shell not to create a subshell for the last command in the pipeline,and run this command in the same process as the script itself. In our case, this means that the tail command will be executed in a subshell (that is, in another process), and the while loop will become part of the main script process. Note: this shell option is available only in bash version 4.x and higher and only for non-interactive shells (i.e. scripts).

(5) This is the tail -f command with another --pid parameter. We indicate the identifier of the process, which, upon completion of this process, will terminate the tail command. We specify the process ID of the current shell script $$ to be viewed. This action allows you to clean processes and not leave the tail command executed in the background (if, say, this script runs in the background; Example 8.4).

The tailcount.sh script starts and stops the script with a stopwatch (timer) and counts the time intervals.

Example 8.4. tailcount.sh

#!/bin/bash -
#
# Bash  
# tailcount.sh
#
# :
#    n 
#
# : ./tailcount.sh [filename]
#     filename:  looper.sh
#

#  —    
function cleanup ()
{
      [[ -n $LOPID ]] && kill $LOPID          (1)
}

trap cleanup EXIT                             (2)
bash looper.sh $1 &                           (3)
LOPID=$!                                      (4)
#   
sleep 3

while true
do
      kill -SIGUSR1 $LOPID
      sleep 5
done >&2                                      (5)

(1) Since this script will run other scripts, it should clean up after work. If the process identifier was stored in LOPID, the variable will store the value, therefore the function will send a signal to this process using the kill command. If you do not specify a specific signal in the kill command, then the SIGTERM signal will be sent by default.

(2) The EXIT command is not a signal. This is a special case when the trap statement tells the shell to call this function (in this case cleanup) if the shell executing this script is about to shut down.

(3)Now the real work begins. The looper.sh script is launched, which will run in the background: for this script to work throughout the cycle (without waiting for the command to complete the work), it is disconnected from the keyboard.

(4) Here, the identifier of the script process that we just started in the background is stored.

(5) This redirect is simply a precaution. All output coming from a while loop or from kill / sleep statements (although we do not expect them) should not be mixed with any output from the looper.sh function, which, although it works in the background, sends them to stdout anyway. Therefore, we redirect data from stdout to stderr.

To summarize, we see that although the looper.sh function was placed in the background, its process identifier is stored in a shell variable. Every five seconds, the tailcount.sh script sends a signal SIGUSR1 to this process (which is executed in the looper.sh function), which, in turn, calls the looper.sh script to print the current number of lines fixed in it and restart the count. After exiting, the tailcount.sh script will be cleared by sending a SIGTERM signal to the looper.sh function to interrupt it.

With the help of two scripts - a script that performs line counting, and a script with a stopwatch (timer) that controls the first script - you can get the output (the number of lines for a certain period), based on which the next script will build a histogram. It is called like this:

bash tailcount.sh | bash livebar.sh

The livebar.sh script reads data from stdin and prints the output to stdout, one line for each input line (Example 8.5).

Example 8.5 livebar.sh

#!/bin/bash -
#
# Bash  
# livebar.sh
#
# :
#    «» 
#
# :
# <output from other script or program> | bash livebar.sh
#

function pr_bar ()                                         (1)
{
      local raw maxraw scaled
      raw=$1
      maxraw=$2
      ((scaled=(maxbar*raw)/maxraw))
      ((scaled == 0)) && scaled=1 #   
      for((i=0; i<scaled; i++)) ; do printf '#' ; done
      printf '\n'

} # pr_bar

maxbar=60     #         (2)
MAX=60
while read dayst timst qty
do
      if (( qty > MAX ))                                   (3)
      then
           let MAX=$qty+$qty/4    #   
           echo "                      **** rescaling: MAX=$MAX"
      fi
      printf '%6.6s %6.6s %4d:' $dayst $timst $qty         (4)
      pr_bar $qty $MAX
done

(1) The pr_bar function displays a string of hashtags that are scaled based on the provided parameters to the maximum size. This feature may seem familiar, as we previously used it in the histogram.sh script.

(2) This is the longest hashtag line size that we can allow (to do without line wrapping).

(3)How large will the values ​​to be displayed be? Without knowing this in advance (although this data can be provided to the script as an argument), the script will track the maximum. If this maximum is exceeded, the value will begin to “scale” and the lines that are now displayed, and future lines will also be scaled to a new maximum. The script adds 25% to the maximum value, so it does not have to scale the value if the next new value each time increases by only 1-2%.

(4)printf defines the minimum and maximum width of the first two fields to be printed. These are date and time stamps that will be truncated if the width values ​​are exceeded. To display the entire value, specify its width of four characters. In this case, despite the restrictions, all values ​​will be printed. If the number of characters in the values ​​is less than four, the missing ones will be supplemented with spaces.

Since this script is read from stdin, you can run it yourself to see how it behaves. Here is an example:

$ bash  livebar.sh
201010 1020 20
201010     1020 20:####################
201010 1020 70
                       **** rescaling: MAX=87
201010     1020 70:################################################
201010 1020 75
201010 1020 75:###################################################
^C

In this example, input is mixed with output. You can also put input in a file and redirect it to a script to see only the output:

$ bash livebar.sh < testdata.txt
bash livebar.sh < x.data
201010 1020 20:####################
                 **** rescaling: MAX=87
201010 1020 70:################################################
201010 1020 75:###################################################
$

»More information on the book can be found on the publisher’s website
» Contents
» Excerpt

For Khabrozhiteley 25% discount on the coupon - Bash

Upon payment of the paper version of the book, an electronic book is sent by e-mail.

More articles:


All Articles