Testing an SSL service with openssl

This example

• opens an SSL connection to a host
• sends a request
• waits for the response

echo "request body" | openssl s_client -connect <host>:<port> -quiet

For example:

ttreitlinger@IRE1ttreitliL1 /usr/bin
$ echo "-rfgetVersion -dv19901 -tr98 -x" | openssl s_client -connect 192.168.17.128:4433 -quiet
depth=1 C = US, ST = Virgina, L = Reston, O = Transaction Network Services Inc., OU = Development, CN = TNS Internal Test CA
verify error:num=19:self signed certificate in certificate chain
verify return:0
ACK
-rfgetVersion -tr98 -rc30  -ms"CU:02.01.73 AU:02.00.65 TU:02.01.76 "  -x

ttreitlinger@IRE1ttreitliL1 /usr/bin
$

Testing the log4j SyslogAppender with socat

Log4j includes a SyslogAppender which can be used to forward log messages directly to a Linux/Unix syslog daemon. This requires syslog to be configured for remote logging, by adding the -r switch to SYSLOGD_OPTIONS in /etc/sysconfig/syslog. Once configured this way, Syslog listens on UDP port 514 for log messages.

When you are developing on Windows, a local syslog daemon is not available. To provide an endpoint that simply prints messages sent from the log4j SyslogAppender to stdout, you can use socat. Start socat in UDP receive mode on port 514 like this:

$ socat stdio udp-recv:514
$    # or
$ socat - udp-recv:514

To send UDP datagrams to this emulator for testing purposes, socat can be used as well:

$ socat stdio udp:127.0.0.1:514
$    # or
$ socat - udp:127.0.0.1:514

This waits for messages on stdin. Under cygwin, sends the message, or sends the message with a terminating CRLF.

One of the differences between real syslog and this socat syslog simulator is, the real syslog interprets the number in brackets at the beginning of each message as facility.level.

Syslog facilities are defined in rfc3164 as follows:

Numerical             Facility
          Code

           0             kernel messages
           1             user-level messages
           2             mail system
           3             system daemons
           4             security/authorization messages (note 1)
           5             messages generated internally by syslogd
           6             line printer subsystem
           7             network news subsystem
           8             UUCP subsystem
           9             clock daemon (note 2)
          10             security/authorization messages (note 1)
          11             FTP daemon
          12             NTP subsystem
          13             log audit (note 1)
          14             log alert (note 1)
          15             clock daemon (note 2)
          16             local use 0  (local0)
          17             local use 1  (local1)
          18             local use 2  (local2)
          19             local use 3  (local3)
          20             local use 4  (local4)
          21             local use 5  (local5)
          22             local use 6  (local6)
          23             local use 7  (local7)

           Table 1.  syslog Message Facilities

The syslog levels are: (also in rfc3164, or see man syslog)

Numerical         Severity
          Code

           0       Emergency: system is unusable
           1       Alert: action must be taken immediately
           2       Critical: critical conditions
           3       Error: error conditions
           4       Warning: warning conditions
           5       Notice: normal but significant condition
           6       Informational: informational messages
           7       Debug: debug-level messages

           Table 2. syslog Message Severities

To convert facility.level to the integer required by syslog (quote from rfc3164):


   The Priority value is calculated by first multiplying the Facility
   number by 8 and then adding the numerical value of the Severity. For
   example, a kernel message (Facility=0) with a Severity of Emergency
   (Severity=0) would have a Priority value of 0.  Also, a "local use 4"
   message (Facility=20) with a Severity of Notice (Severity=5) would
   have a Priority value of 165.  In the PRI part of a syslog message,
   these values would be placed between the angle brackets as <0> and
   <165> respectively.




How does log4j translate from Java log level to syslog level?

This is set statically in class org.apache.log4.Level as follows:

final static public Level FATAL = new Level(FATAL_INT, "FATAL", 0);
final static public Level ERROR = new Level(ERROR_INT, "ERROR", 3);
final static public Level WARN  = new Level(WARN_INT, "WARN",  4);
final static public Level INFO  = new Level(INFO_INT, "INFO",  6);
final static public Level DEBUG = new Level(DEBUG_INT, "DEBUG", 7);
public static final Level TRACE = new Level(TRACE_INT, "TRACE", 7);

see http://www.docjar.com/html/api/org/apache/log4j/Level.java.html

Monitor Concurrent Load on a Server

Here is a one-liner that displays the number of currently active TCP connections on a given port, in intervals of 1 second.

 while true; do n=`netstat -n|grep 2707|wc -l`; echo $`date` --- $n ; sleep 1; done;

Useful for load testing a service.

Awk Samples

Count rows and compute sum for a column

grep "Summary" *.log|awk '{sum += $7} {count +=1} END {print count, sum}'

This counts each line containing the word "Summary" in all files ending in .log
Value in column 7 is summed.

Sample data:
                                        *               
Summary - Thread 145    20      16      4       16      4
Summary - Thread 148    20      15      5       15      5
Summary - Thread 147    20      15      5       15      5

* this is column 7

Conditional Sum

awk '$5 == 2 {{s += $8} {c += 1}} END {print 2, s/c }' timing.txt >> sum.txt

What this does:

* read file 'timing.txt'
* if 5th column has value 2:
- add value of column 8 to s
- add 1 to c (count occurrences)
* after last record read:
- print literal 2
- print average (sum/count)
* output: append to file 'sum.txt'

Disk Usage by Directory

Show Disk Usage by directory

du -s * | sort -nr

Show details of which files/directories consume how much space (Cygwin for C drive):

du -shc /cygdrive/c/*

Show the same info sorted by size, with largest directories last:

du -sc /cygdrive/c/* | sort -n

Count with sort and uniq

Question: Find all unique IP address that appear in a certain log message, and count how many times they appear.


Answer:

grep "log message" logfile.log | awk '{ print $18 }' | sort | uniq -c

Example:

[cygwin] $ grep "Missing post body" mpi.log |awk '{print $18}' |sort |uniq -c |sort -rn
      6 82.1.184.81
      6 212.121.212.149
      4 62.49.164.229
      2 62.140.196.160
 [cygwin] $

Note: the "sort -rn" at the end sorts results by number of occurrence

Log Monitoring - number of recent events exceeding treshold

This script checks a log file for a certain log event. If the number of these events in the last 3 minutes exceeds a treshold, an alert email is sent

#!/bin/bash
# Check timestamps up to 3 minutes ago
prev3=`date +"%y/%m/%d %H:%M" -d -3minute`
prev2=`date +"%y/%m/%d %H:%M" -d -2minute`
prev1=`date +"%y/%m/%d %H:%M" -d -1minute`
now=`date +"%y/%m/%d %H:%M"`

logevent="Exception"
treshold=2

# check the log file
failCount=` egrep "$prev3|$prev2|$prev1|$now" /var/log/app_logfile.log | grep $logevent | wc -l `


# 3 or more log events in previous 3-4 minutes
if [ $failCount -gt $treshold ]; then
    # notify someone
    echo "failCount ($failCount) exceeds treshold ($treshold)" | mail -s "Alert from ${HOSTNAME}" support@mycompany.com
fi

Compare two directory trees

diff -a -q -r dir1 dir2

Options used:

-a Treat all files as text and compare them line-by-line, even if they do not seem to be text.
-q Report only whether the files differ, not the details of the differences.
-r When comparing directories, recursively compare any subdirectories found.

Linux Runlevels

To quickly switch from GUI to shell-only runlevel, enter the following:

sudo /sbin/init 3    # switch to shell with networking 
sudo /sbin/init 5    # switch back to GUI

To set the default runlevel in CentOS, edit file /etc/inittab (set line id:3:initdefault: to the correct value).

Convert text file line delimiters

The simplest way to convert a file's line delimiters from DOS to Unix and back:

# from DOS/Windows to Unix
$ dos2unix filename 

# from Unix to DOS/Windows
$ unix2dos filename 

CentOS 5 Service Management

In CentOS5, services can be started and stopped from the graphical user interface. For example, if you're using Gnome, go to System...Administration...Server Settings...Services. This opens the "Service Configuration" window. Here you can start, stop and restart services, as well as configure them for runlevels 3 to 5.

I'm running CentOS in a VM, so the GUI takes up a lot of resources. Because of that, I usually login to the CentOS VM using putty. The functionality in the "Service Configuration" window is available on the command line as well.

# list all services and their configuration at all runlevels
[ccarddev@localhost]$ /sbin/chkconfig --list|less

# turn off a service at a certain runlevel...
[ccarddev@localhost]$ sudo /sbin/chkconfig --level 5 tomcat5 off
[sudo] password for ccarddev:
[ccarddev@localhost]$ /sbin/chkconfig --list|grep tomcat
tomcat5         0:off   1:off   2:off   3:off   4:off   5:off   6:off
[ccarddev@localhost]$

# ... and turn it back on
[ccarddev@localhost]$ sudo /sbin/chkconfig --level 5 tomcat5 on
[ccarddev@localhost]$ /sbin/chkconfig --list|grep tomcat
tomcat5         0:off   1:off   2:off   3:off   4:off   5:on    6:off
[ccarddev@localhost]$

# start/stop/restart a service
[ccarddev@localhost]$ sudo /etc/init.d/tomcat5 start
Starting tomcat5:                                          [  OK  ]
[ccarddev@localhost]$ sudo /etc/init.d/tomcat5 restart
Stopping tomcat5:                                          [  OK  ]
Starting tomcat5:                                          [  OK  ]
[ccarddev@localhost]$ sudo /etc/init.d/tomcat5 stop
Stopping tomcat5:                                          [  OK  ]
[ccarddev@localhost]$ 

Unix find

# find all files containing "string", starting in current directory, and print their path
find . -exec grep “string” {} /dev/null \;

Note: /dev/null shows name of the file before text that is found

For more details see http://en.wikipedia.org/wiki/Find and man find