clockwatching.net/~drew/

SiteMinder SSO with Websphere

Thinking about installing SiteMinder with Websphere for single sign on capabilities? Well, if you're like me and assigned such a task, you'll probably hit a few bumps I had to figure out along the way with no help from CA (Computer Associates) support department.

SiteMinder comes as a binary installer. According to CA support, it's packaged with it's own Java for the installer, which proved to be false.

In my current environment, we run Red Hat AS 4.6 which packages it's own java-compat-gcj-1.4.2 which sucks total ass and only usually breaks anything it touches or anything that attempts to use it.

Websphere has it's own Java packaged with it, which resides deep into the directory hierarchy. The instructions for the SiteMinder install claims to not have JAVA_HOME set and add the path to the jre bin directory for the JVM. I figured having to set this, SiteMinder does in fact use the Java on the OS and not within it's own package.

What I forgot though is the java-compat-gcj package, which will in fact have /usr/bin/java. Running:

which java

Will print out such info. Forgetting all about this, even setting up the /opt/IBM/Websphere/AppServer/java/jre/bin in your current PATH, the SiteMinder install will go with what's first in your PATH, which is /usr/bin/java.

So here's the trick around it to launch the ca-asa-6.0-was-unix.bin for SiteMinder. Before launching the binary installer, redo the PATH to exclude /usr/bin.

export PATH=/bin:/opt/IBM/Websphere/AppServer/java/jre/bin

I left /bin in the path cause well, I like the basic functional commands like ls, etc.

This should enable you to launch the installer and not get the strange errors associated with libgcj.so.5 or the like where SiteMinder is using /usr/bin/java instead of /opt/IBM/Websphere/AppServer/java/jre/bin/java to install.

Also I did find that running the installer from the console is more reliable as trying to run the GUI installer version might result in some libraries missing that might be required.

You can simply run:

./ca-asa-6.0-was-unix.bin -i console

Good luck with your SSO with your applications.

CFengine - Random Authentication and Challenge Response Errors

If you use cfengine for personal use or in a corporate environment, there's probably a good chance some of the machines it runs on might have multiple interfaces along with multiple A records pointing at them.

This might cause problems with cfengine and authentication at the intervals you have it set to run at. Basically cfengine will grab the IP of the machine when first run after you invoke the cfagent command and setup ppkeys for authentication, think of it as ssh keys so no password is required for authentication.

What happens with machines with multiple interfaces might try to use a different IP to authenticate with the main server you have configured. This can cause errors and generate an email that might look like this:


cfengine:somehost: Challenge response from server server.domain.com/10.15.1.87 was incorrect!
cfengine:somehost: Authentication dialogue with server.domain.com failed

After digging thru docs and groups for a solution and verifying and fixing any reverse DNS mismatches, we finally found that it was still occurring randomly cause cfengine would try to use another IP or interface on the client. With the main server having the ppkeys associated by the original IP, this caused the failure.

To fix this issue, simply add something like this to your cfserved.conf file on the main server:


    somehost::
      BindToInterface = ( xxx.xxx.xxx.xxx )

Basically this will force the client to always use the same interface when it runs cfagent looking for updates, etc.

Creating a Yum Repository

Need a local yum repository for your Red Hat or CentOS? Follow these steps to save yourself bandwidth usage to point your server farm to one local location. With this type of setup, you can drop your own RPM packages for easy installation as well.

Pick a server or setup a server with apache and web access. If you don't want to share this with the world, it's better to keep it on a private network. In this case, we've created a yum-repo directory on a /data partition with a reasonable amount of space to hold all of the base os RPM's, updates and our own RPM's.

Within the yum-repo directory, create a centos directory and within there, create subdirectories of the different versions you may have. In our case, we still used 3.8 and 4.4 CentOS installs.

In our case, we wanted the base, updates and another directory for our own RPM's, so within 3.8 and 4.4 we have such directories created. In the os directory, create the arch you use. In most cases you'll just have a i386. We also have some 64 bit servers so we created i386 and x86_64.

Grab the ISO's of each or one of them, whatever ones you have or use and extract the contents in the corresponding arch directory.

So in our case, we dropped the contents of the 3.8 ISO into /data/yum-repo/centos/3.8/os/i386/

Once extracted, run these two commands:


createrepo /data/yum-repo/centos/3.8/os/i386/
yum-arch /data/yum-repo/centos/3.8/os/i386/

This will create the xml files and headers. yum-arch is actually deprecated but we create these as most of the 3.8 servers still have an older version of yum that use header files instead of the new type that createrepo creates and reads from.

Now for 3.8 or older versions, edit the /etc/yum.conf file to point to your server instead of the mirrors.centos.org domain. You may also need to create a symlink to 3.8 as 3. We ran into issues where yum would look for 3 instead of 3.8 in the $releasever path variable in the yum.conf file.

4.0 or higher you'll need to edit your /etc/yum-repos.d/ files with your server info. You can keep some of the others handy for backups in case your server goes down but I just create a CentOS-Base.repo file that looks almost identical to your yum.conf file in Version 3 or older.

That's it, enjoy your local yum repository.

Port Scan Attack Detector - psad with iptables

Port Scan Attack Detector (psad) is a great utility for anyone that doesn't have hardware based firewalls and have servers that are accessible to the outside world. It was originally part of the Bastille project but evolved on it's own, with Michael Rash as the maintainer from cipherdyne.com who also creates and maintains a few other really network related programs.

From the features page for psad, here are a list of the main features it is capable of:

Detection for TCP SYN, FIN, NULL, and XMAS scans as well as UDP scans.
Detection of many signature rules from the Snort intrusion detection system.
Forensics mode Netfilter logfile analysis (useful as a forensics tool for extracting scan information from old Netfilter logfiles).
Passive operating system fingerprinting via TCP syn packets. Two different fingerprinting strategies are supported; a re-implementation of p0f that strictly uses Netfilter log messages (requires the --log-tcp-options command line switch), and a TOS-based strategy.
Email alerts that contain TCP/UDP/ICMP scan characteristics, reverse dns and whois information, snort rule matches, remote OS guess information, and more.
Content-based alerts for buffer overflow attacks, suspicious application commands, and other suspect traffic through the use of the Netfilter string match extension and fwsnort.
Icmp type and code header field validation.
Configurable scan thresholds and danger level assignments.
Iptables ruleset parsing to verify "default drop" policy stance.
IP/network danger level auto-assignment (can be used to ignore or automatically escalate danger levels for certain networks).
DShield alerts.
Auto-blocking of scanning IP addresses via Netfilter and/or tcpwrappers based on scan danger level. (This feature is NOT enabled by default.)
Parsing of Netfilter log messages and generation of CSV output that can be used as input to AfterGlow. This allows Netfilter logs to be visualized. Here are some example graphs created by parsing the Netfilter logs provided by the Honeynet Project: Scan30 and Scan34.
Status mode that displays a summary of current scan information with associated packet counts, Netfilter chains, and danger levels.

Basically before you install psad, you'll want to prepare your server with a few things, the main thing is to at least have iptables installed or ready, with some basic rules in place perhaps.

Here are a few basics I use on my own servers:


#!/bin/bash
 
# Reload and flush existing rules:
/usr/sbin/iptables -F
 
# Drop everything by default
/usr/sbin/iptables -P INPUT DROP
/usr/sbin/iptables -P OUTPUT DROP
 
# always allow the loopback
/usr/sbin/iptables -A INPUT -i lo -j ACCEPT
/usr/sbin/iptables -A OUTPUT --source 127.0.0.1 --destination 127.0.0.1 -j ACCEPT
 
# Allow return traffic
/usr/sbin/iptables -A INPUT -m state --state ESTABLISHED -j ACCEPT
/usr/sbin/iptables -A OUTPUT -m state --state ESTABLISHED -j ACCEPT
/usr/sbin/iptables -A INPUT -m state --state RELATED -j ACCEPT
 
# ICMP
/usr/sbin/iptables -A INPUT -p icmp -m icmp -j ACCEPT
/usr/sbin/iptables -A OUTPUT -p icmp -m icmp -j ACCEPT
 
# Log/Reject everything else (inbound)
/usr/sbin/iptables -A INPUT -j LOG --log-prefix "INPUT"
/usr/sbin/iptables -A INPUT -j REJECT
 
# Log/Reject everything else (outbound)
/usr/sbin/iptables -A OUTPUT -j LOG --log-prefix "OUTPUT"
/usr/sbin/iptables -A OUTPUT -j REJECT

Depending on what my server does, I'll then create rules to allow say HTTP for Apache, DNS if I'm running Bind, etc. Configure to your needs. The main portion though that psad needs is the LOG options. You can specify just LOG or give it a --log-prefix to narrow it down to have psad only monitor those with specific prefixes, it will ask you during the install.

Once you get iptables in place and how you like, download the latest psad package, unpack it and simply run the install script. It will take you thru the rest of the setup and will in most cases add the necessary kern.info messages for syslogd.

Now go read up on the rest to learn how to take advantage of this network detector to keep your system security tight and locked down from intrusion.

CentOS/Redhat Swtich MTA

So like most system administrators, most have a preference of which MTA or Mail Transfer Agent they use. The most popular are Sendmail, Postfix, Exim and Qmail.

I for one prefer Postfix. It's stable, easy to configure and secure.

In CentOS and Redhat, there is a simple utility to use to switch since by default Sendmail will be the MTA used.

You can start by installing Postfix by including the package during the install or using yum.

yum -y install postfix

To change the default MTA after installation, simply run the following command:

/usr/sbin/alternatives --set mta /usr/sbin/sendmail.postfix

Now you have Postfix setup as your default MTA and you can configure it to your liking.

USB Boot with CentOS

To simplify your life as a system administrator and deploying new servers with a base install, you can do this by eliminating the CDR with CentOS installed and easily create a USB boot disk to boot from to kick off an install with kickstart.

To create the bootdisk, download the bootdisk.img file of the version of CentOS you are using. This also works with Red Hat since CentOS is just a clone of the Enterprise version.

Create the boot disk with the dd command, disk dump for those not familiar with it.


dd if=bootdisk.img of=/dev/sdb

I used sdb as that is what my USB is recognized from. If your not sure, insert the USB disk and run dmesg to find out how it registers in the OS.

It's going to be rather small so you can easily use a 16MB USB disk for the boot info to boot from.

That's all, make sure the hardware your installing on allows USB boot or external storage devices to boot from and if you have kickstart in place like I do, it's as easy as this from the CentOS boot screen:


grub> linux ks=http://hostname/kickstart/ks.cfg ksdevice=eth0

If you don't have your kickstart auto do everything for you, You can throw in a vnc after linux to start a vnc server where you can remotely complete the install. Once all the boot info is loaded, you can simply remove the USB disk and start the next server. I had to install 10 servers and it took me about 15 minutes to have them all installed and ready for deployment with the base OS of course.

Handling MySQL Replication Errors

In an environment where you have a MySQL database and have replication slaves running, there are times when you might get an error and the replication slaves will stop processing the bin logs for replication.

Here are some quick steps to recover and get the slaves rolling again.

Login to the Master server and do:
mysql> show master status\G;
If the status looks okay, proceed to check the slaves by doing the following:
mysql> show slave status\G;
This should provide some details and look something like the following:


mysql> show slave status\G;
*************************** 1. row ***************************
             Slave_IO_State: Waiting for master to send event
                Master_Host: hostname.com
                Master_User: replicant
                Master_Port: 3306
              Connect_Retry: 60
            Master_Log_File: hostname-bin.000265
        Read_Master_Log_Pos: 10324
             Relay_Log_File: support-relay-bin.000159
              Relay_Log_Pos: 10483
      Relay_Master_Log_File: hostname-bin.000265
           Slave_IO_Running: Yes
          Slave_SQL_Running: Yes
            Replicate_Do_DB: 
        Replicate_Ignore_DB: 
         Replicate_Do_Table: 
     Replicate_Ignore_Table: 
    Replicate_Wild_Do_Table: 
Replicate_Wild_Ignore_Table: 
                 Last_Errno: 0
                 Last_Error: 
               Skip_Counter: 0
        Exec_Master_Log_Pos: 10324
            Relay_Log_Space: 10483
            Until_Condition: None
             Until_Log_File: 
              Until_Log_Pos: 0
         Master_SSL_Allowed: No
         Master_SSL_CA_File: 
         Master_SSL_CA_Path: 
            Master_SSL_Cert: 
          Master_SSL_Cipher: 
             Master_SSL_Key: 
      Seconds_Behind_Master: 0
1 row in set (0.00 sec)

If you have an error in replication most likely due to a bad query in the output above and the master and slave are out of sync, the quickest way to recover is to reset by telling MySQL to skip the problem query and continue.

On the replication slave do the following:


mysql> SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1;
mysql> SLAVE START SQL_THREAD;

At times, this can still cause the replication to skip or dump any data from the current bin long it's processing. The above works with 4.0.3 and above but will tell the slave to start replicating where the current master is at, so you may lose some transactions that have occured.

You can get around this at times by doing a simple stop and start:

mysql> SLAVE STOP;
mysql> SLAVE START;

IBM RSA II

The IBM RSA II cards in my opinion should just be standard in IBM servers is quite easy to configure and use. Probably the easiest way to update any firmware and BIOS on any type of computer or server I've worked with.

A quick overview of the RSA II which by the way stands for Remote Supervisor Adapter from the IBM site describes this device as:

Remote Supervisor Adapter II Enhancement now adds accelerated graphics and delivers advanced control and monitoring features to manage your IBM eServer xSeries server at virtually any time, from virtually any place.

The Remote Supervisor Adapter II Enhancement, an affordable, next-generation, systems-management adapter, makes everyday management tasks easier and more efficient throughout the life of your IBM eServer xSeries server. The Remote Supervisor Adapter II Enhancement simplifies remote management by providing around-the-clock remote access to selected xSeries servers.

So basically it's a built in KVM and remote administration console. You can power down a server, reboot, update firmware and just about anything else you can do all remotely. As long as power is plugged in, you can reach and power on the server while it's turned off.

Some of the new features in the latest generation include:

Scriptable command-line interface (CLI) and text-based serial console redirect accessible over Telnet or second serial port
Point-to-Point Protocol (PPP)
User authentication information stored on Lightweight Directory Access Protocol (LDAP) server
Enhanced user authority levels
Full Systems Network Management Protocol (SNMP)
Multiple remote drives and disk images
Remote disk image storage

Some of the existing features include:

Secure easy-to-use Web interface
Graphical console redirection with accelerated graphics
Remote drive support
Integration with IBM Director via Management Processor Assistant (MPA)
Improved keyboard and mouse support
Enhanced automated server restart (ASR)
Continuous health monitoring and control
Advanced Predictive Failure Analysis (PFA)
Automatic notification and alerts
Event log; time-stamped, non volatile attachment to e-mail alerts
LAN, serial, and Advanced System Management (ASM) interconnect remote access
Simple Network Management Protocol (SNMP) and e-mail alerts
Domain Name System (DNS) and Dynamic Host Configuration Protocol (DHCP) support
Remote power control
Remote firmware update and access to critical server settings

When you have limited access or the servers you administer are at a colocation that is across town or across the country, this makes life easy as a system administrator. Props to IBM for creating such an easy device but so powerful for performing those tasks that most need to do onsite physically.

Kickstart with LVM Partition Layout

So if your like me and work as a System Administrator in a Linux environment or workshop, you're most likely using Red Hat or some derivative of it like CentOS. You also most likely setup, configure and deploy new or old servers on a consistent basis. In such a environment, you're most likely using kickstart to automate installs.

There is an abundant amount of kickstart howto's but I found that many of them don't really go into details when having to setup LVM partitions that you want to span across two drives. They only really detail how to setup one Physical Volume on one given drive, which makes sense, since a majority of servers are going to be setup with RAID, hardware or software.

In my case though, we had five HP Proliant DL145 servers that had 2 SCSI drives but did not have a SCSI adapter capable of hardware RAID, also to mention that they were enclosed and not hot swappable. So even if they were setup with RAID, we'd have to shut these machines down, crack the case and swap out the drive.

These machines were production servers that were replaced so these were going to turn into QA and Staging type servers. Since these might end up with VMware Server, we wanted to just utilize all the disk space and create one large drive using LVM.

Now comes the part where we want to configure the partition scheme in the kickstart configuration file.

This is how I did it, since I couldn't find any documentation online where anyone was using two drives and wanted the LVM spanned across each.


clearpart --all
part /boot --fstype ext3 --size=100 --ondisk=sda --asprimary
part pv.1 --size=100 --grow --ondisk=sda --asprimary
part pv.2 --size=100 --grow --ondisk=sdb --asprimary
volgroup rootvg --pesize=16384 pv.1 pv.2
logvol /usr --fstype ext3 --name=usr --vgname=rootvg --size=4096
logvol /opt --fstype ext3 --name=opt --vgname=rootvg --size=4096
logvol /var --fstype ext3 --name=var --vgname=rootvg --size=4096
logvol /tmp --fstype ext3 --name=tmp --vgname=rootvg --size=1024
logvol /usr/local --fstype ext3 --name=usrlocal --vgname=rootvg --size=512
logvol / --fstype ext3 --name=root --vgname=rootvg --size=1024
logvol /home --fstype ext3 --name=home --vgname=rootvg --size=2048
logvol swap --fstype swap --name=swap --vgname=rootvg --size=4096

So basically for both disks, you'll have a part line for each disk. Both should look identical like in the above example except you'll specify the /dev name for each. In this case it's sda and sdb, since they're both SCSI drives.

Then the part to make them function under the same Volume Group should look like this like in the above example:


volgroup rootvg --pesize=16384 pv.1 pv.2

With this, you're basically telling which physical volumes to include in the volgroup you are creating where the partitions will reside.

Once you lay it all out how you like, after the initial installation, you can check the total volume size by running:


vgdisplay

The total VG size should amount to the size of both drives. If it doesn't, you did something wrong.