(uN)Tech blogs

scripting parted

2011-12-07T14:52:00.000+08:00

lots of SAN disks added today. So to create partitions
on them, I use the following:

#!/bin/sh
for i in $(multipath -ll | grep sd | awk '{print $3}')
do
# create a label first
parted /dev/$i --script -- mklabel msdos
# create partition take whole disk
parted /dev/$i --script -- mkpart primary 0 -1
# set type lvm
parted /dev/$i --script -- set 1 lvm on
parted /dev/$i --script print
done

WARNING: THIS CAN WIPE OUT YOUR LOCAL DISK PARTITIONS!
this is for a new system (all disks have no data) and the
local storage is on a H/W raid (/dev/c0dX).

hth.

Cloning an LPAR using an ISO via VIO Virtual Media Library

2011-07-04T17:47:00.000+08:00

Cloning an LPAR using VIO Virtual Media Library

This is a short howto on cloning an AIX LPAR using the
VIO Virtual Media Library. An ISO file will be created
from an AIX mksysb image of the source LPAR. This
iso file is then loaded thru the VIO server as a
virtual optical device to be used by the target LPAR.

For this howto, the mksysb and ISO files were created
on the same LPAR. This is not a requirement.Just make
sure ISO generation is on a system that is at least
the same level as the mksysb's source.

CREATE THE MKSYSB IMAGE

[root@HESTIA]/>mksysb -eim /export/mksysb/AIX53TL12SP1.img

Creating information file (/image.data) for rootvg.

Creating list of files to back up ..
Backing up 124218 files....................

124218 of 124218 files backed up (100%)
0512-038 mksysb: Backup Completed Successfully.

[root@HESTIA]/>du -ms /export/mksysb/AIX53TL12SP1.img
2923.00 /export/mksysb/AIX53TL12SP1.img

I have excluded directories to reduce size of mksysb image. Also,
/export is from another volumen group.

CREATE THE ISO IMAGE

[root@HESTIA]/> mkcd -L -S -I /export/iso/ -m /export/mksysb/AIX53TL12SP1.img

from mkcd man page:

-L creates final CD images that are DVD sized (up to 4.38 GB)
-S stops mkcd before writing to the CD-R, DVD-R or DVD-RAM 
 without removing the final CD images.
-I cd_image_dir specifies the directory or file system where
 the final CD images are stored before writing to 
 the CD-R, DVD-R or DVD-RAM device.
-m mksysb_image specifies a previously created mksysb image. 
 if not specified, mkcd will run the mksysb command.

[root@HESTIA]/> mkcd -L -S -I /export/iso/ -m /export/mksysb/AIX53TL12SP1.img

Initializing mkcd log: /var/adm/ras/mkcd.log...
Verifying command parameters...
Creating temporary file system: /mkcd/cd_fs...
Populating the CD or DVD file system...
Building chrp boot image...
Copying backup to the CD or DVD file system...
...
Creating Rock Ridge format image: /export/iso//cd_image_299194
Running mkisofs ...
...
mkrr_fs was successful.

Making the CD or DVD image bootable...


Removing temporary file system: /mkcd/cd_fs...

VIO SIDE
as padmin, create the Repository on the VIO server

$ lsrep
The DVD repository has not been created yet.

$ lssp
Pool              Size(mb)   Free(mb)  Alloc Size(mb)    BDs Type
rootvg              279552     222464             256      0 LVPOOL

$ mkrep -sp rootvg -size 9G
Virtual Media Repository Created
Repository created within "VMLibrary" logical volume

NOTE: a new FS /var/vio/VMLibrary will be created and mounted.

then as root, copy the CD ISO image:

# pwd
/var/vio/VMLibrary
# scp ZEUS:/Apps/iso/cd_image_545020 AIX53TL12.iso
root@ZEUS's password:
cd_image_545020                                                             100% 3264MB  35.1MB/s   01:33
# ls -ltr
total 15809960
-r--r--r--    1 root     staff    3129395200 Jun 25 09:00 AIX53TL12SP1.iso
-r--r--r--    1 root     staff    3422056448 Jun 25 09:00 AIX53TL12.iso
-r--r--r--    1 root     staff    1543245824 Jul 04 09:00 AIX61TL4SP7.iso

List the pool:

$ lsrep
Size(mb) Free(mb) Parent Pool         Parent Size      Parent Free
    9179     1458 rootvg                   279552           222464

Name                                    File Size Optical         Access
AIX53TL12.iso                                3264 None            ro
AIX53TL12SP1.iso                             2985 None            ro
AIX61TL4SP7.iso                              1472 None            ro

create the adapter

$ mkvdev -fbo -vadapter vhost1
vtopt0 Available

from mkvdev man page:

The -fbo flag is used to create a virtual target device which maps 
the VirtualServerAdapter to a file backed virtual optical device. 
Virtual optical devices cannot be used until virtual media is 
loaded into the device.


$ lsmap -vadapter vhost1
SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
vhost1          U8233.E8B.0633B8P-V1-C4                      0x00000003

VTD                   vtopt0
Status                Available
LUN                   0x8100000000000000
Backing device
Physloc
Mirrored              N/A

load the image:

$ mkvdev -fbo -vadapter vhost1
vtopt0 Available

$ loadopt -f -vtd vtopt0 -disk AIX53TL12SP1.iso

$ lsrep
Size(mb) Free(mb) Parent Pool         Parent Size      Parent Free
    9179     1458 rootvg                   279552           222464

Name                                    File Size Optical         Access
AIX53TL12.iso                                3264 None            ro
AIX53TL12SP1.iso                             2985 vtopt0          ro
AIX61TL4SP7.iso                              1472 None            ro

$ lsmap -vadapter vhost1
SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
vhost1          U8233.E8B.0633B8P-V1-C4                      0x00000003

VTD                   HESTIAsys00
Status                Available
LUN                   0x8200000000000000
Backing device        hdisk5
Physloc               U5796.001.0620413-P1-C1-T1-W50001FE1501C6C69-L2000000000000
Mirrored              false

VTD                   vtopt0
Status                Available
LUN                   0x8100000000000000
Backing device        /var/vio/VMLibrary/AIX53TL12SP1.iso
Physloc
Mirrored              N/A

then boot the LPAR from the scsi device detected

 SMS 1.7 (c) Copyright IBM Corp. 2000,2008 All rights reserved.                 
------------------------------------------------------------------------------- 
 Select Task                                                                    
                                                                                
SCSI CD-ROM                                                                     
    ( loc=U8233.E8B.0633B8P-V3-C2-T1-L8100000000000000 )                        
                                                                                
 1.   Information                                                               
 2.   Normal Mode Boot                                                          
 3.   Service Mode Boot                                                         
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
 -------------------------------------------------------------------------------
 Navigation keys:                                                               
 M = return to Main Menu                                                        
 ESC key = return to previous screen         X = eXit System Management Services
 -------------------------------------------------------------------------------
 Type menu item number and press Enter or select Navigation key:2

 Version AL710_097                                                              
 SMS 1.7 (c) Copyright IBM Corp. 2000,2008 All rights reserved.                 
------------------------------------------------------------------------------- 
 Device Information                                                             
    /vdevice/v-scsi@30000002/disk@8100000000000000                              
                   :  (Bootable)                                                
DEVICE             : SCSI CD-ROM                                                
    ( loc=U8233.E8B.0633B8P-V3-C2-T1-L8100000000000000 )                        
NAME               : disk                                                       
DEVICE-TYPE        : block                                                      
                                                                                
 Parent Information                                                             
IBM,FW-ADAPTER-NAME: Virtual SCSI                                               
NAME               : v-scsi                                                     
DEVICE-TYPE        : vscsi                                                      
                                                                                
                                                                                
                                                                                
                                                                                
 -------------------------------------------------------------------------------
 Navigation keys:                                                               
 M = return to Main Menu                                                        
 ESC key = return to previous screen         X = eXit System Management Services
 -------------------------------------------------------------------------------
 Type menu item number and press Enter or select Navigation key:


IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM     
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM     
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM     
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM     
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM     
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM     
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM     
|                                                                               
Elapsed time since release of system processors: 352759 mins 32 secs            
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
------------------------------------------------------------------------------- 
                                Welcome to AIX.                                 
                       boot image timestamp: 07:07 07/04                        
                 The current time and date: 08:35:48 07/04/2011                 
               number of processors: 2    size of memory: 4096MB                
boot device: /vdevice/v-scsi@30000002/disk@8100000000000000:\ppc\chrp\bootfile.e
                     kernel size: 12544763; 32 bit kernel                       
-------------------------------------------------------------------------------

Once done with the image, unload it:

$ unloadopt -release -vtd vtopt0

$ lsmap -vadapter vhost1
SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
vhost1          U8233.E8B.0633B8P-V1-C4                      0x00000003

VTD                   HESTIAsys00
Status                Available
LUN                   0x8200000000000000
Backing device        hdisk5
Physloc               U5796.001.0620413-P1-C1-T1-W50001FE1501C6C69-L2000000000000
Mirrored              false

VTD                   vtopt0
Status                Available
LUN                   0x8100000000000000
Backing device
Physloc
Mirrored              N/A

and if the Virtual adapter is no longer needed:

$ rmdev -dev vtopt0
vtopt0 deleted
$ lsmap -vadapter vhost1

check_printq - AIX QUEUE MONITOR NRPE PLUGIN FOR NAGIOS

2011-06-08T17:33:00.007+08:00

There are instances when some of our AIX printer queues encounter
problems on their corresponding physical devices (most commonly paper
supply) causing jobs to build up. Since I couldn't find one that will
show the status the way I wanted to, I created a simple script called
"check_printq".
Its a crude one but it works. You can find the script below.

tested on:
AIX 5.3/6.1 with Version 2.5.x
Nagios® Core™ 3.2.3
Icinga 1.3.1
NRPE Version: 2.12

WHAT THE SCRIPT DOES
it will monitor specific queues on the AIX machine. Since our NRPE
setup was compiled without command argument processing, each monitored
queue needs to be defined in the corresponding nrpe configuration.
although the script can be used with arguments if your NRPE daemon
supports it.

DEFINING IT IN NRPE (AIX SIDE):
edit:
/opt/nagios/etc/nrpe_check_commands.cfg

add:

command[check_PTISAP11]=/opt/nagios/libexec/check_printq  -Q PTISAP11 -w 3 -c 10
command[check_PTISAP05]=/opt/nagios/libexec/check_printq  -Q PTISAP05 -w 4 -c 5

reload nrpe daemon:

kill -1 $PID
or restart:
/opt/nagios/sbin/nrpe -n -c /opt/nagios/etc/nrpe.cfg -d

NAGIOS SERVER SIDE DEFINITION

define service{
        hostgroup_name          AIX-TEST
        service_description     PTISAP11
        check_command           check_nrpe!check_PTISAP11
        use                     template-service-test
        }
define service{
        hostgroup_name          AIX-TEST
        service_description     PTISAP05
        check_command           check_nrpe!check_PTISAP05
        use                     template-service-test
        }

save and do a syntax check;
sudo nagios -v ../nagios.cfg

and then reload nagios.

SAMPLE RESULTS

qchk -WA
Queue                Dev            Status       Job Files              User         PP   %  Blks  Cp Rnk
-------------------- -------------- --------- ------ ------------------ ---------- ---- --- ----- --- ---
PTISAP11             hp@PTISAP11    DOWN
                                    QUEUED      1193 STDIN.426216       root                    1   1   1
PTISAP08             hp@PTISAP08    READY
PTISAP0801           hp@PTISAP08    READY
PTISAP05             hp@PTISAP05    READY
PTISAP0501           hp@PTISAP05    READY

nagios image:

THE SCRIPT

#! /bin/sh
#####
#  check_printq - plugin to check AIX printer queues
#  will show queue status and queue job number
#  v1.2 - updated some queue checking codes
####
ECHO="/usr/bin/echo"
QCHK="/usr/bin/qchk"
GREP="/usr/bin/egrep"
PROGNAME=`/usr/bin/basename $0`
PROGPATH=`echo $0 | sed -e 's,[\\/][^\\/][^\\/]*$,,'`
REVISION=`echo '$Revision: 1.2 $' | sed -e 's/[^0-9.]//g'`

. $PROGPATH/utils.sh

ShowUsage() {
    echo "Usage: $PROGNAME -Q QueueName -w # -c #"
    echo "Usage: $PROGNAME --help"
    echo "Usage: $PROGNAME --version"
}

ShowHelp() {
    print_revision $PROGNAME $REVISION
    echo ""
    ShowUsage
    echo ""
    echo "AIX Qdaemon monitor for Nagios"
    echo ""
    support
}

# Make sure the correct number of command line
# arguments have been supplied

if [ $# -lt 1 ]; then
    ShowUsage
    exit $STATE_UNKNOWN
fi
# Grab the command line arguments
exitstatus=$STATE_WARNING #default
while test -n "$1"; do
    case "$1" in
        --help)
            ShowHelp
            exit $STATE_OK
            ;;
        -h)
            ShowHelp
            exit $STATE_OK
            ;;
        --version)
            print_revision $PROGNAME $VERSION
            exit $STATE_OK
            ;;
        -V)
            print_revision $PROGNAME $VERSION
            exit $STATE_OK
            ;;
        -Q)
            qNAME=$2
            shift
            ;;
        -w)
            qWarnCount=$2
           shift
                ;;
        -c)
            qCritCount=$2
           shift
                ;;
        *)
            echo "Unknown argument: $1"
            ShowUsage
            exit $STATE_UNKNOWN
        ;;
esac
shift
done
# check if valid printer queue
qCheck=`${QCHK} -WP${qNAME} > /dev/null 2>&1 ; echo $?`
#invalid queue will spit code 64
if [ ${qCheck} -ne 0 ]
then
#if not valid, exit
        exit  $STATE_UNKNOWN
fi

# get jobcount and compare with threshold
qJobCount=`${QCHK} -P${qNAME} | ${GREP} -c QUEUED`
nSTATUS=$STATE_OK
if [ ${qJobCount} -gt  ${qWarnCount} -a  ${qJobCount}  -lt ${qCritCount}  ]
then
        nSTATUS=$STATE_WARNING
else
  if [ ${qJobCount} -gt  ${qWarnCount} -o  ${qJobCount}  -gt ${qCritCount}  ]
    then
        nSTATUS=$STATE_CRITICAL
  fi
fi

# check status if queue is DOWN, READY or RUNNING
qSTAT=`${QCHK} -WP${qNAME} | ${GREP} ${qNAME} | awk '{print $3}'`
case "${qSTAT}" in
READY)
    $ECHO "$qNAME is ${qSTAT} with [ ${qJobCount} ] queued Jobs"
    exit $nSTATUS
        ;;
RUNNING)
    $ECHO "$qNAME is ${qSTAT} with [ ${qJobCount} ] queued Jobs"
    exit $nSTATUS
        ;;
DOWN)
    $ECHO "$qNAME is ${qSTAT} with [ ${qJobCount} ] queued Jobs"
    exit $STATE_CRITICAL
        ;;
*)
        # any other state = unknown
    ${ECHO} "[ ${qSTAT} ] queued Jobs with Unknown STATUS: ${qSTAT}"
        nSTATUS=$STATE_UNKNOWN
        exit $STATE_UNKNOWN
        ;;
esac
    exit $nSTATUS
## EOF

CHECKING THE SCRIPT
checking from AIX side (local to the script):
run it using:
sh -x /opt/nagios/libexec/check_printq -Q PTISAP11 -w 2 -c 10
this will run the script in debug mode and will show you
what exactly is happening.

from nagios server side:
[user@NAGIOS objects]$ /usr/lib64/nagios/plugins/check_nrpe -H UX0017 -n -p 5666 -c check_PTISAP11; echo $?
PTISAP11 is DOWN with [ 0 ] queued Jobs
2

[user@NAGIOS objects]$ /usr/lib64/nagios/plugins/check_nrpe -H UX0017 -n -p 5666 -c check_PTISAP05; echo $?
PTISAP05 is READY with [ 0 ] queued Jobs
0

this will verify the return codes needed for the monitoring based on the defined NAGIOS PLUGIN RETURN CODES.

Further reading:
http://nagiosplug.sourceforge.net/developer-guidelines.html#PLUGOUTPUT

SOLARIS 10 x86 Jumpstart Server on GNU/Linux

2011-03-31T10:36:00.002+08:00

okay i know this maybe a bit late but i'll post this anyway.
i got too busy the past few months (translation: i got lazy :D).

This is how I've set my Centos box for both kickstart and jumpstart.

Centos machine is a kickstart Server for RHEL/CENTOS releases.
i'll be using the same dhcp and tftp server to serve the Solaris
installation.

Jumpstart server:
CENTOS 5.5

SOLARIS 10 u8 target machines
LX0050 - ProLiant BL460c G1 with Smart Array E200i controller
LX0130 - HP ProLiant BL45p G1 Smart Array 6i Controller

components of my JumpStart server:

DHCP server - dhcpd config

## start of /etc/dhcpd.conf
authoritative;
ddns-update-style none;
allow booting;
allow bootp;
option grubmenu code 150 = text;

subnet 10.129.6.0 netmask 255.255.255.0 {
        option routers                  10.129.6.1;
        option subnet-mask              255.255.255.0;
        option domain-name              "mydomain.org";
        option domain-name-servers      10.129.8.11;
        option ntp-servers              10.129.8.19;
        range dynamic-bootp 10.129.6.101  10.129.6.101;
        default-lease-time 21600;
        max-lease-time 43200;
        option tftp-server-name "10.129.6.125";
}

host LX0130.mydomain.org{
    next-server 10.129.6.125;
    hardware ethernet 00:16:35:C4:89:00;
    fixed-address 10.129.6.74;
    filename "/solaris/pxegrub.sol10u8";
}

host LX0050.mydomain.org{
    next-server 10.129.6.125;
    hardware ethernet 00:18:FE:30:C1:F2;
    fixed-address 10.129.6.201;
    filename "/solaris/pxegrub.sol10u8";
}

## end of /etc/dhcpd.conf

TFTP server tree

/tftpboot/
|-- solaris
|   |-- pxegrub.sol10u6
|   |-- pxegrub.sol10u8
|   |-- sol10u6.miniroot
|   |-- sol10u6.multiboot
|   |-- sol10u8.miniroot
|   `-- sol10u8.multiboot
|-- menu.lst.01001635C48900
`-- menu.lst.010018FE30C1F2

contents of /tftpboot/menu.lst.<MAC>

## menu.lst.01001635C48900
default=0
timeout=4
title Solaris_10 Jumpstart
kernel /solaris/sol10u8.multiboot kernel/unix -v -m verbose  install w nfs://10.129.6.125/tools/solaris/sol10u8/config/LX0050.tar  -B install_media=10.129.6.125:/tools/solaris/sol10u8/image
module /solaris/sol10u8.miniroot
## end of  menu.lst.01001635C48900

## menu.lst.010018FE30C1F2
default=0
timeout=4
title Solaris_10 Jumpstart
kernel /solaris/sol10u8.multiboot kernel/unix -v -m verbose  install w nfs://10.129.6.125/tools/solaris/sol10u8/config/LX0050.tar  -B install_media=10.129.6.125:/tools/solaris/sol10u8/image
module /solaris/sol10u8.miniroot
## end of menu.lst.010018FE30C1F2

- 01001635C48900 and 010018FE30C1F2 are MAC addresses of the first
network interface for the machines i'm install with solaris:

- the *.miniroot and *.miniboot are from the image/boot of the Solaris 10 image.

NOTE: installation for the HP proliant blades I use require a modified miniroot files.
I included the CPQary3 modules for the HP SMART Arrays. more details. pls follow these links:

Solaris installation guide for HP proliant servers

for the smart array driver i use

NFS (hosting Solaris installation image and installation scripts)
here's an overview of the nfs tree i use for solaris:

/tools/solaris/sol10u8/
|-- config
|   |-- LX0050.tar
|   |-- LX0130.tar
|   `-- LXISAP0140.tar
|-- image
|   |-- Copyright
|   |-- JDS-THIRDPARTYLICENSEREADME
|   |-- License
|   |-- Solaris_10
|   |-- boot
|   `-- installer
`-- share
    |-- CPQary3
    `-- CPQary3.pkg

contents for /tools/solaris/sol10u8/image

creating the solaris installation tree:

create a "/bin/bar" symlink to "/bin/tar"

ln -s /bin/tar /bin/bar
ln -s /bin/sed /usr/bin/sed
ln -s /usr/bin/gdb /usr/bin/adb

and for the "mach" script:

echo "#!/bin/bash" > /bin/mach
echo "uname -p" >> /bin/mach
chmod +x /bin/mach

from the downloaded solaris x86 ISO:

sudo mount -o loop sol-10-u8-ga-x86-dvd.iso /mnt/iso
ls -ltr /mnt/iso
total 490
-r--r--r-- 1 root root 487593 Aug 22  2009 JDS-THIRDPARTYLICENSEREADME
-r--r--r-- 1 root root   6582 Aug 22  2009 Copyright
-r-xr-xr-x 1 root root    257 Sep 17  2009 installer
dr-xr-xr-x 2 root root   2048 Sep 17  2009 License
dr-xr-xr-x 7 root root   2048 Sep 17  2009 Solaris_10
dr-xr-xr-x 5 root root   2048 Sep 17  2009 boot


cd /mnt/iso/Solaris_10/Tools

sudo ./setup_install_server /tools/solaris/sol10u8/image
Verifying target directory...
Calculating the required disk space for the Solaris_10 product
|\b\c
/\b\c
-\b\c
\ \b\b\c
...
...

and wait for it to finish.

my installation source tree looks like this:

/tools/solaris/sol10u8/image
|-- Copyright
|-- JDS-THIRDPARTYLICENSEREADME
|-- License
|   `-- 819-0764-10_SLA_Multi.pdf
|-- Solaris_10
|   |-- Docs
|   |-- Misc
|   |-- Patches
|   |-- Product
|   `-- Tools
|-- boot
|   |-- amd64
|   |-- grub
|   |-- multiboot
|   |-- platform
|   `-- x86.miniroot
`-- installer

contents for /tools/solaris/sol10u8/config/
this serves the tar files of the automation files needed for the
jumpstart installation:

LX0050.tar
LX0130.tar
LXISAP0140.tar

LX0050.tar has the following contents:

tar tf LX0050.tar

LX0050
LX0050_post.sh
rules
rules.ok
sysidcfg

ff are the contents of the files.

## LX0050
install_type    initial_install
system_type     standalone
partitioning    explicit
filesys c1t0d0s0 8192 /
filesys c1t0d0s1 4096 /tmp
filesys c1t0d0s4 8192 swap
filesys c1t0d0s5 3000 /var
filesys c1t0d0s6 5000 /usr
filesys c1t0d0s7 4096 /u01
# the packages here include those needed
# for an oracle instllation
#cluster SUNWCall
cluster SUNWCrnet
cluster SUNWCssh add
package SUNWbash add
package SUNWgss  add
package SUNWarc add
package SUNWbtool add
package SUNWhea add
package SUNWlibm add
package SUNWlibms  add
package UNWsprot add
package SUNWtoo add
package SUNWi1of add
package SUNWi1cs add
package SUNWi15cs add
package SUNWxwfnt add
package SUNWman add
package SUNWdoc add
package SUNWnfscr add
package SUNWnfscu add
package SUNWnfsckr add
package SUNWdtrp add
package SUNWdtrc add
# tcp/ip tools
package SUNWbip add
# needed by Veritas SF
package SUNWscpr add
package SUNWscpu add
package SUNWrcmdc add
package SUNWtftp add
package SUNWtnetc add
package SUNWpool add
package SUNWntpu add
## end of LX0050

## LX0050_post.sh
#!/bin/sh
#

# Add packages
BASE=/a
MNT=/a/mnt
ADMIN_FILE=/a/tmp/admin

mkdir ${MNT}
mount -f nfs 10.129.y.d:/tools/solaris/sol10u8/share/ ${MNT}
cat >${ADMIN_FILE} <<DONT_ASK
mail=root
instance=overwrite
partial=nocheck
runlevel=nocheck
idepend=nocheck
rdepend=nocheck
space=ask
setuid=nocheck
conflict=nocheck
action=nocheck
basedir=default
DONT_ASK

/usr/sbin/pkgadd -a ${ADMIN_FILE} -d ${MNT} -R ${BASE} CPQary3
umount ${MNT}
rmdir ${MNT}

# Customize root's environment
#
echo "***adding customizations in /.cshrc"
test -f /a/.cshrc || {
cat >> /a/.cshrc <<EOF
set history=100 savehist=200 filec ignoreeof prompt="\$user@`uname -n`> "
alias cp cp -i
alias mv mv -i
alias rm rm -i
alias ls ls -FC
alias h history
alias c clear
unset autologout
EOF
}

echo "***adding customizations in /etc/profile"
test -f /a/.cshrc || {
cat >> /a/.cshrc <<EOF
set history=100 savehist=200 filec ignoreeof prompt="\$user@`uname -n`> "
alias cp cp -i
alias mv mv -i
alias rm rm -i
alias ls ls -FC
alias h history
alias c clear
unset autologout
EOF
}


### NET related
echo "***adding network customizations..."
# resolv.conf
cat > /a/etc/resolv.conf << EOF
nameserver 10.129.x.a
nameserver 10.129.x.b
nameserver 10.129.x.3
search mydomain.org
EOF

# motd and issue/.net
cat > /a/etc/motd << EOF
----------------------------------
Access to this system is strictly for authorized personnel only.
If you are not authorized, please disconnect now!
Any unauthorized access attempts shall be prosecuted.
-----------------------------------

EOF
cat > /a/etc/issue << EOF
----------------------------------
Access to this system is strictly for authorized personnel only.
If you are not authorized, please disconnect now!
Any unauthorized access attempts shall be prosecuted.
-----------------------------------

EOF

# ssh
/usr/bin/ed <<EOT /a/etc/ssh/sshd_config
/^PermitRootLogin
s/no/yes
.
w
q
EOT

#this sets the second interface
cat > /a/etc/hostname.bge1 << EOF
192.168.100.130
EOF

cat >> /a/etc/hosts << EOF
192.168.100.50  node01
192.168.100.130 node02
EOF

cat >> /a/etc/system << EOF
set lwp_default_stksize=0x6000
set rpcmod:svc_default_stksize=0x6000
EOF
### end of NET related

## end of LX0050_post.sh

## rules
hostname        LX0050 -       LX0050 LX0050_post.sh
any              -               -      any_machine LX0050_post.sh
# version=2 checksum=num
## end of rules

# rules.ok = rules - this basically make sure
# rules file is correct
## rules.ok
hostname        LX0050 -       LX0050 LX0050_post.sh
any              -               -      any_machine LX0050_post.sh
# version=2 checksum=num
## end of rules.ok


# this automates most the basic stuff

## sysidcfg
system_locale=en_US
keyboard=US-English
terminal=xterms
network_interface=PRIMARY
        {hostname=LX0050
        ip_address=10.129.x.201
        default_route=10.129.x.1
        netmask=255.255.255.0
        protocol_ipv6=no}
security_policy=none
name_service=DNS{ domain_name=mydomain.org
        name_server=10.129.8.11
        search=mydomain.org}
nfs4_domain=dynamic
timezone=Singapore
timeserver=localhost
root_password=$1$8op9W$5kF3KWEmEJ8Du0fL/MIo31
#keyboard=us101 {kbdfile=us101.kbd}
## end of sysid cfg

once done with the automation files, put them in the tar filenamed LX0050.tar.add the ff to /etc/sysconfig/nfs:

RPCNFSDARGS="--no-tcp --no-nfs-version 4"

and to /etc/exports:

/tools/solaris LX0130(ro,no_root_squash) LX0050(ro,no_root_squash)

start the services:

sudo /sbin/service dhcpd restart
sudo /sbin/chkconfig tftp on
sudo /sbin/service nfs restart

installation images:

PXE Booting Damn Small Linux

2010-12-22T11:27:00.001+08:00

Damn Small Linux is very compact Linux distribution. This howto shows how to boot a machine via PXE boot using DSL

download the initrd ISO here:

http://ftp.heanet.ie/mirrors/damnsmalllinux.org/current/
http://ftp.heanet.ie/mirrors/damnsmalllinux.org/current/dsl-4.4.10-initrd.iso

(Thanks to heanet.ie for the very fast mirror.)

loopback mount the iso:

sudo mount -o loop dsl-4.4.10-initrd.iso /mnt/iso

sudo cp -pR /mnt/iso /tftpboot/linux-install/DSL

network settings
dhcp/pxeboot server - ip: 192.168.1.1
pxeclient - ip: 192.168.1.23
mac address: 00:18:fe:30:d1:f4

in /etc/dhcpd.conf:

#DSL boot
host LX0050 {
        next-server 192.168.1.1;
        hardware ethernet  00:18:fe:30:d1:f4;
        fixed-address 192.168.1.23;
        filename "linux-install/pxelinux.0";
}

TFTP layout:

/tftpboot/linux-install/DSL/
|-- boot.cat
|-- boot.msg
|-- f2
|-- f3
|-- german.kbd
|-- isolinux.bin
|-- linux24
|-- logo.16
`-- minirt24.gz

in /tftpboot/linux-install/pxelinux.cfg/, i have a file named dsl.cfg
with ff contents:

PROMPT 1
DISPLAY DSL/boot.msg
F1 DSL/boot.msg
F2 DSL/f2
F3 DSL/f3
LABEL dsl
KERNEL DSL/linux24
APPEND ramdisk_size=100000 init=/etc/init lang=us apm=power-off vga=791 initrd=DSL/minirt24.gz nomce noapic quiet BOOT_IMAGE=knoppix
LABEL memtest
KERNEL memtest
APPEND initrd=
LABEL dsl-txt
KERNEL DSL/linux24
APPEND ramdisk_size=100000 init=/etc/init lang=us apm=power-off vga=normal initrd=DSL/minirt24.gz nomce noapic quiet BOOT_IMAGE=knoppix
LABEL expert
KERNEL DSL/linux24
APPEND ramdisk_size=100000 init=/etc/init lang=us apm=power-off vga=791 initrd=DSL/minirt24.gz nomce BOOT_IMAGE=expert
LABEL fb1280x1024
KERNEL DSL/linux24
APPEND ramdisk_size=100000 init=/etc/init lang=us apm=power-off vga=794 xmodule=fbdev initrd=DSL/minirt24.gz nomce noapic quiet BOOT_IMAGE=knoppix
LABEL fb1024x768
KERNEL DSL/linux24
APPEND ramdisk_size=100000 init=/etc/init lang=us apm=power-off vga=791 xmodule=fbdev initrd=DSL/minirt24.gz nomce noapic quiet BOOT_IMAGE=knoppix
LABEL fb800x600
KERNEL DSL/linux24
APPEND ramdisk_size=100000 init=/etc/init lang=us apm=power-off vga=788 xmodule=fbdev initrd=DSL/minirt24.gz nomce noapic quiet BOOT_IMAGE=knoppix
LABEL lowram
KERNEL DSL/linux24
APPEND ramdisk_size=100000 init=/etc/init lang=us apm=power-off vga=normal initrd=DSL/minirt24.gz noscsi noideraid nosound nousb nofirewire noicons minimal nomce noapic noapm quiet BOOT_IMAGE=knoppix
LABEL install
KERNEL DSL/linux24
APPEND ramdisk_size=100000 init=/etc/init lang=us apm=power-off vga=normal initrd=DSL/minirt24.gz noscsi noideraid nosound nofirewire legacy base norestore _install_ nomce noapic noapm quiet BOOT_IMAGE=knoppix
LABEL failsafe
KERNEL DSL/linux24
APPEND ramdisk_size=100000 init=/etc/init lang=us vga=normal atapicd nosound noscsi nousb nopcmcia nofirewire noagp nomce nodhcp xmodule=vesa initrd=DSL/minirt24.gz BOOT_IMAGE=knoppix
LABEL userdef
KERNEL DSL/linux24
APPEND ###############################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################

these contents are from the isolinux.cfg file from the DSL ISO image
with the linux24 and minirt24.gz path modified (notice the DSL/ entries
in the lines above). of course, these paths may vary with your setup, so
keep it in mind.
this will show the menu as seen when booting DSL from a CD/DVD or ISO image.

to prep the setup, the command sequence:

as root:


cd /tftpboot/linux-install/pxelinux.cfg

./iptohex.pl 192.168.1.23

IP address 192.168.1.23, converts to Hex: C0A80117

ln -fs dsl.cfg C0A80117

service dhcpd start

chkconfig tftp on  (or service xinetd restart)

and booting up the machine:

Credits:
IPtoHex is a perl script from Yngve Berthelsen
and downloaded from BigAdmin.

RHEL4 kickstart for oracle 9i install

2010-11-29T11:28:00.002+08:00

i missed posting part of my ks file for an oracle 9iR2 installation.
this one is for a RH Enterprise Linux 4 (update 8 to be exact).
also tested on Centos 4.x.

you need to change:
- NFS source entries
- disk entry in the partition layout
- network entries
- shared memory entries (sysctl.conf)

CAUTION: This will erase all data on the target disk!

### START of ks file

install
nfs --server=10.129.a.xyz --dir=/export/isapsys/redhat/RHEL4U8/x86_64
lang en_US.UTF-8
langsupport --default=en_US.UTF-8 en_US.UTF-8
keyboard us
skipx
network --device eth0 --bootproto static --ip 10.129.a.bc --netmask 255.255.255.0 --gateway 10.129.x.1 --nameserver 10.129.x.yy --hostname LX0130
# password is rootroot 
rootpw --iscrypted $1$2jq4s/$u/xCKr9Gpk.ynPy.V/LbW.
firewall --disabled
selinux --disabled
authconfig --enableshadow --passalgo=md5
timezone Asia/Shanghai
reboot
device scsi cciss
clearpart --all --initlabel --drives=cciss/c0d0
bootloader --location=mbr --driveorder=cciss/c0d0
part /boot --fstype ext3 --size=250 --ondisk=cciss/c0d0 --asprimary
part pv.17 --size=100 --grow --ondisk=cciss/c0d0 --asprimary
volgroup vg00 --pesize=32768 pv.17
logvol / --fstype ext3 --name=lvsys00 --vgname=vg00 --size=8192
logvol swap --fstype swap --name=lvsys01 --vgname=vg00 --size=16384
logvol /var --fstype ext3 --name=lvsys02 --vgname=vg00 --size=4096
logvol /opt --fstype ext3 --name=lvsys03 --vgname=vg00 --size=1024
logvol /tmp --fstype ext3 --name=lvsys04 --vgname=vg00 --size=2048
logvol /home --fstype ext3 --name=lvsys05 --vgname=vg00 --size=1024
logvol /apps --fstype ext3 --name=lvapps00 --vgname=vg00 --size=8192

%packages
e2fsprogs
e2fsprogs.i386
kernel
kernel-devel
kernel-largesmp
kernel-largesmp-devel
openssh-clients
openssh-server
postfix
vnc-server
device-mapper
device-mapper-multipath
net-snmp
net-snmp-utils
lm_sensors

#oracle
ntp
strace
libaio
libaio-devel
ncompress
sg3_utils
ncurses
openssl
openssl-devel
openmotif
openmotif21
xorg-x11-deprecated-libs-devel
xorg-x11-deprecated-libs
#gnome-libs-devel
sysstat
compat-gcc-32
compat-libstdc++-33
compat-gcc-32-c++
compat-libgcc-296
compat-glibc-headers
compat-glibc.x86_64
libgcc
libgcc.i386
compat-libstdc++-296
compat-db
gcc
gcc-c++
glibc-devel
glibc-devel.i386
ncurses.i386
compat-glibc.i386
glibc.i686

#java-1.4.2-gcj-compat
# VNCserver requirements
#xorg-x11- does not exist in RHEL5
xorg-x11-twm
xorg-x11-tools
xterm
screen
sharutils
pdksh
krb5-libs
krb5-libs.i386

#nagios nrpe
perl-Digest-HMAC
perl-Digest-SHA1

# remove
-openldap
-bluez-hcidump
-bluez-bluefw
-desktop-file-utils
-irda-utils
-nano
-isdn4k-utils
-NetworkManager
-pcmcia-cs
-switchdesk
-kudzu
-bluez-libs
-rsh
-redhat-menus
-fontconfig
-minicom
-wpa_supplicant
-bluez-utils
-rp-pppoe
-ppp
-wvdial
-gcc-java
-libgcj-devel
-java-1.4.2-gcj-compat
-libgcj
-gtk2
-pango


%post
# Modify DNS Search
/bin/cat <<EOT > /etc/resolv.conf
nameserver 10.129.x.y
options attempts:5
options timeout:15
EOT

# KERNEL PARAMS
cat > /etc/sysctl.conf << EOT
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 1800
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 0
kernel.shmall = 3617282
kernel.shmmax = 14816389632
kernel.shmmni = 4096
kernel.sem = 1010 129280 1010 128
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=4194304
net.core.wmem_max=1048576
vm.swappiness=40
EOT

# oracle user and groups
/usr/sbin/groupadd -g 1001 dba
/bin/mkdir /home/DBA
/bin/chgrp -R dba /home/DBA
# oracle pass - oracl3pass
/usr/sbin/useradd -u 1001 -g dba -d '/home/DBA/oracle' -p '$1$AIlZX$tJxG2xWT8SfSOAPGjnhuu.' oracle


# Add hangcheck-timer module for RAC
#echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >> /etc/modprobe.conf
#echo "modprobe hangcheck-timer" >> /etc/rc.local

cat >> /etc/bashrc << EOF
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
set -o vi
EOF

# .bash_profile
cat > /home/DBA/oracle/.bash_profile << EOF
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
      . ~/.bashrc
fi

# User specific environment and startup programs
export ORACLE_BASE=/apps/oracle
export ORACLE_HOME=\${ORACLE_BASE}/product/9i
export ORA_CRS_HOME=/apps/crs
export ORACLE_PATH=\${ORACLE_BASE}/common/oracle/sql:.:\${ORACLE_HOME}/rdbms/admin
export CV_JDKHOME=/usr/local/java
export LD_ASSUME_KERNEL=2.4.19

# Each RAC node must have a unique ORACLE_SID. (i.e. orcl1, orcl2,...)
export ORACLE_SID=ISITEST

export PATH=.:\${JAVA_HOME}/bin:${PATH}:$HOME/bin:\${ORACLE_HOME}/bin
export PATH=\${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export PATH=\${PATH}:\${ORACLE_BASE}/common/oracle/bin
export ORACLE_TERM=xterm
export TNS_ADMIN=\${ORACLE_HOME}/network/admin
export ORA_NLS10=\${ORACLE_HOME}/nls/data
export LD_LIBRARY_PATH=\${ORACLE_HOME}/lib
export LD_LIBRARY_PATH=\${LD_LIBRARY_PATH}:\${ORACLE_HOME}/oracm/lib
export LD_LIBRARY_PATH=\${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export CLASSPATH=\${ORACLE_HOME}/JRE
export CLASSPATH=\${CLASSPATH}:\${ORACLE_HOME}/jlib
export CLASSPATH=\${CLASSPATH}:\${ORACLE_HOME}/rdbms/jlib
export CLASSPATH=\${CLASSPATH}:\${ORACLE_HOME}/network/jlib
export THREADS_FLAG=native
export TEMP=/tmp
export TMPDIR=/tmp
alias ls="ls -FA"
set -o vi

EOF

# limits
cat >> /etc/security/limits.conf <<EOF
oracle       soft    nofile          65536
oracle       hard    nofile          101062
oracle       soft    nproc           16384
oracle       hard    nproc           101062
oracle       soft    memlock            unlimited
oracle       hard    memlock            unlimited
EOF

# vncserver stuff
cat << EOT > /opt/vnc_xstartup
#!/bin/sh

# run vncserver and copy to your $HOME/.vnc/xstartup file
# Uncomment the following two lines for normal desktop:
# unset SESSION_MANAGER
# exec /etc/X11/xinit/xinitrc

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
xterm -geometry 130x30+12+12 -ls -bg black -fg green -title "$VNCDESKTOP Desktop" &
mwm &
EOT


### END of ks file

The Awesome Kinect

2010-11-18T16:03:00.000+08:00

i'm taking a timeout just to express some excitement that I had
yesterday (17 November Singapore) - i got KINECTed (if thats a term. :D)).

there is a large crowd lining up to purchase the Kinect controller and the xbox 360 S package with kinect controller. got the controller for my old Xbox 360 from the pre-launch held
at Plaza Singapura.

the included game Kinect Adventures is pretty good.
my son, athrun karl, is still learning to use it though.
we've tried kinect joy ride two player mode - very nice (no pics/vids to post here
though).

I'm so awed by this gadget. And I believe the Kinect controller is a major breakthrough and the very one piece of technology that will re-shape how we interact with machines.
I won't be surprised if Kinect usage becomes widespread outside the gaming world.
could be very useful to help people with disability and also with robotics.

maybe Sony (PS3) and Nintendo (Wii) should try to intergrate Kinect with their gaming consoles. frankly speaking - i've lost interest with Wii when i used the Kinect yesterday. good thing i never bought a wii. :)

RHEL6 64-bit and Oracle 11g R2

2010-11-18T14:07:00.022+08:00

ok so RH Enterprise Linux 6 has been released. I'm installing it to test an oracle 11g installation. included below is the kickstart script used.

BIG CAUTION: ORACLE 11g (nor any other oracle versions afaik) is not certified (yet) to run on RHEL6. SO DON'T PUT THIS ON PRODUCTION.

first issue encountered is the installation over NFS. my NFS RHEL6 sources are on anAIX 5L machine. and anaconda won't mount the export.

i resorted to using an RHEL5U5 system to host the ISOs.

here's the directory structure:

/apps/erik/RHEL6/

|-- i386
|   |-- images
|   |   |-- README
|   |   |-- TRANS.TBL
|   |   |-- install.img
|   |   |-- product.img
|   |   `-- pxeboot
|   |-- rhel-server-6.0-i386-dvd.iso
|   |-- rhel-server-supplementary-6.0-i386-dvd.iso
|   `-- sha256sum.txt
`-- x86_64
    |-- images
    |   |-- README
    |   |-- TRANS.TBL
    |   |-- efiboot.img
    |   |-- efidisk.img
    |   |-- install.img
    |   |-- product.img
    |   `-- pxeboot
    |-- rhel-server-6.0-x86_64-dvd.iso
    |-- rhel-server-supplementary-6.0-x86_64-dvd.iso
    `-- sha256sum.txt

the "images" directory is copied from the server iso image. the nfs based installationwon't proceed without that directory. appears to be a mandatory requirement now for the NFS install.

kickstart file used:

#START of ks file
# Kickstart file automatically generated by anaconda.

#version=RHEL6
install
nfs --server=LXISAP0050 --dir=/apps/erik/RHEL6/x86_64
lang en_US.UTF-8
keyboard us
network --device eth0 --bootproto static --ip 10.a.b.c --netmask 255.255.255.0 --gateway 10.a.6.1 --nameserver 10.a.8.11 --hostname LX0130.local
#r00tpassw0rd
rootpw  --iscrypted $1$ib6Rr/$wuTR7.6MF4DalT4WdirT90
# System services
services --disabled="sendmail,xfs,bluetooth,kudzu,firstboot,cups,ip6tables"
# Reboot after installation
reboot
firewall --disabled
authconfig --enableshadow --passalgo=sha512 --enablefingerprint
selinux --enforcing
timezone --utc Asia/Singapore
bootloader --location=mbr --driveorder=cciss/c0d0 --append="crashkernel=auto crashkernel=auto rhgb quiet" 
clearpart --linux --initlabel --drives=cciss/c0d0
part /boot --fstype=ext4 --asprimary --size=200
part pv.None --grow --size=200
volgroup vg00 --pesize=16384 pv.None
logvol / --fstype=ext4 --name=lvsys00 --vgname=vg00 --size=1024
logvol swap --name=lvsys01 --vgname=vg00 --size=8192
logvol /usr --fstype=ext4 --name=lvsys02 --vgname=vg00 --size=2048
logvol /var --fstype=ext4 --name=lvsys03 --vgname=vg00 --size=1024
logvol /opt --fstype=ext4 --name=lvsys04 --vgname=vg00 --size=1024
#this will produce error during oracle install
logvol /tmp --fstype ext4 --name=lvsys05 --vgname=vg00 --size=2048 --fsoptions=noexec,nodev
logvol /home --fstype ext4 --name=lvsys06 --vgname=vg00 --size=1024
logvol /apps --fstype ext4 --name=lvora00 --vgname=vg00 --size=2048 --grow


%packages
@Base
@Core
@additional-devel
@base
@compat-libraries
@console-internet
@core
@hardware-monitoring
@legacy-software-support
binutils
compat-db gcc
compat-gcc-34-c++
compat-glibc
compat-glibc-headers
compat-glibc.i386
compat-libgcc-296
compat-libstdc++-296
compat-libstdc++-33
dejavu-lgc-fonts
device-mapper
device-mapper-event
device-mapper-multipath
e2fsprogs
elfutils-libelf
elfutils-libelf-devel
gcc
gcc-c++
glibc
glibc-common
glibc-devel
glibc-headers
kernel
libX11-devel
libXext
libXp
libXt
libXtst
libXxf86dga
libXxf86misc
libaio
libaio-devel
libdmx
libgcc
libgcc.i386
libstdc++
libstdc++-devel
make
ncompress
ncurses
ncurses.i386
ntp
openmotif
openssh-clients
openssh-server
pdksh
postfix
pyOpenSSL
sg3_utils
strace
sysstat
unixODBC
unixODBC-devel
vnc-server
xorg-x11-deprecated-libs
xorg-x11-deprecated-libs-devel
xorg-x11-utils
xterm
yum
-bluez-gnome
-bluez-utils
-firstboot-tui
-ftp
-irda-utils
-nano
-notification-daemon
-openswan
-pango
-ppp
-rdist
-rp-pppoe
-rsh
-setserial
-talk
-usbutils
-wireless-tools
-wpa_supplicant
%end

%post --log=/root/ks-post.log
{
#~-> Remove GRUB's splash image
/bin/ed < User limits
ed /etc/security/limits.conf << EOF
$
.a
oracle       soft    nofile          65536
oracle       hard    nofile          101062
oracle       soft    nproc           16384
oracle       hard    nproc           101062
.
w
EOF

#kernel params
/bin/cat < /etc/sysctl.conf
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.ip_local_port_range = 9000 65500
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 1800
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 0
fs.file-max = 6815744
kernel.shmall = 10523004
kernel.shmmax = 6465333657
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=4194304
net.core.wmem_max=1048576
fs.aio-max-nr = 1048576
EOF_sysctl

/bin/chown root:root /etc/sysctl.conf
/bin/chmod 0600 /etc/sysctl.conf
/sbin/sysctl -p

# create group
/usr/sbin/groupadd -g 306 dba
/bin/mkdir --mode=0750 /home/dba
chgrp dba /home/dba
# temp password - oracl3pass
/usr/sbin/useradd -c 'oracle user' -d /home/dba/oracle -u 306 -p '$1$diNEr/$gu0FHftsYmIm/T6PT9yIN/' -g dba -G dba   oracle


# oracle user .bash_profile
cat > /home/dba/oracle/.bash_profile << EOF
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
      . ~/.bashrc
fi

# User specific environment and startup programs
export ORACLE_BASE=/apps/oracle
export ORACLE_HOME=\${ORACLE_BASE}/product/11g/
export ORA_CRS_HOME=/apps/crs
export ORACLE_PATH=\${ORACLE_BASE}/common/oracle/sql:.:\${ORACLE_HOME}/rdbms/admin
export CV_JDKHOME=/usr/local/java
# Each RAC node must have a unique ORACLE_SID. (i.e. orcl1, orcl2,...)
export ORACLE_SID=ISITEST
export PATH=.:\${JAVA_HOME}/bin:${PATH}:$HOME/bin:\${ORACLE_HOME}/bin
export PATH=\${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export PATH=\${PATH}:\${ORACLE_BASE}/common/oracle/bin
export ORACLE_TERM=xterm
export TNS_ADMIN=\${ORACLE_HOME}/network/admin
export ORA_NLS10=\${ORACLE_HOME}/nls/data
export LD_LIBRARY_PATH=\${ORACLE_HOME}/lib
export LD_LIBRARY_PATH=\${LD_LIBRARY_PATH}:\${ORACLE_HOME}/oracm/lib
export LD_LIBRARY_PATH=\${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export CLASSPATH=\${ORACLE_HOME}/JRE
export CLASSPATH=\${CLASSPATH}:\${ORACLE_HOME}/jlib
export CLASSPATH=\${CLASSPATH}:\${ORACLE_HOME}/rdbms/jlib
export CLASSPATH=\${CLASSPATH}:\${ORACLE_HOME}/network/jlib
export THREADS_FLAG=native
export TEMP=/tmp
export TMPDIR=/tmp
alias ls="ls -FA"
set -o vi
EOF
# end oracle profile
chown -R oracle.dba /apps
%end
} 1>/root/ks-post.log 2>&1
##END of kickstart file

AFTER RHEL6 Install

[root@LX0130 /]# uname -a
Linux LX0130.isap.asia.cib 2.6.32-71.el6.x86_64 #1 SMP Wed Sep 1 01:33:01 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

[root@LX0130 /]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.0 (Santiago)

[root@LX0130 /]# df -Ph | column -t
Filesystem                Size   Used  Avail  Use%  Mounted   on
/dev/mapper/vg00-lvsys00  1008M  203M  754M   22%   /
tmpfs                     1.9G   0     1.9G   0%    /dev/shm
/dev/cciss/c0d0p1         194M   25M   159M   14%   /boot
/dev/mapper/vg00-lvsys06  1008M  34M   924M   4%    /home
/dev/mapper/vg00-lvsys04  1008M  34M   924M   4%    /opt
/dev/mapper/vg00-lvsys05  2.0G   68M   1.9G   4%    /tmp
/dev/mapper/vg00-lvsys02  2.0G   1.5G  471M   76%   /usr
/dev/mapper/vg00-lvsys03  1008M  92M   866M   10%   /var
/dev/mapper/vg00-lvora00  51G    180M  49G    1%    /apps

the kickstart install sets an oracle user and needed environment for
an 11g installation. my oracle11g sources are hosted on an AIX 5L machine
(UX0027). installation will be done via vnc session.

the usual nfs mounting procedure won't work. it spits out an error:

mount.nfs   UX0027:/export/isapsys/oracle11g /mnt/nfs   -v
mount.nfs: timeout set for Tue Nov 16 14:36:19 2010
mount.nfs: trying text-based options 'vers=4,addr=UX0027,clientaddr=LX013'
mount.nfs: mount(2): Remote I/O error
mount.nfs: Remote I/O error

the fix is to use v3 or v2 option:

[root@LX0130 ~]# mount.nfs   UX0027:/export/isapsys/oracle11g /mnt/nfs -o v3
[root@LX0130 ~]# ls /mnt/nfs
AIX  AIXclient  Linux64  Linuxclient  linux.x64_11gR1_database_1013.zip
[root@LX0130 ~]#

Installation of Oracle 11g R2

in a vnc session, running runInstaller encountered a permission denied
error due to the mount optins of /tmp:

/dev/mapper/vg00-lvsys05 on /tmp type ext4 (rw,noexec,nodev)

to fix (just for the install)

[root@LX0130 ~]# mount -o exec,remount /tmp
[root@LX0130 ~]# mount | grep /tmp
/dev/mapper/vg00-lvsys05 on /tmp type ext4 (rw,nodev)

install then started but encountered another error during a pre-req
check - there are "missing" packages:

libaio-0.3.105
compat-libstdc++-33-3.2.3
libaio-devel-0.3.105
libgcc-3.4.6
libstdc++-3.4.6
unixODBC-2.2.11
unixODBC-devel-2.2.11
pdksh-5.2.14

these packages are old versions. RHEL6 ships with the following:

libaio-0.3.107-10.el6.x86_64
libaio-0.3.107-10.el6.i686
libaio-devel-0.3.107-10.el6.x86_64
libaio-devel-0.3.107-10.el6.i686
compat-libstdc++-33-3.2.3-69.el6.x86_64
compat-libstdc++-33-3.2.3-69.el6.i686
libgcc-4.4.4-13.el6.x86_64
libgcc-4.4.4-13.el6.i686
unixODBC-2.2.14-11.el6.x86_64
unixODBC-devel-2.2.14-11.el6.x86_64

(these packages will be installed by the kickstart file above).

pdksh is no longer part of rhel6. i did install the following:

ksh-20100621-2.el6.x86_64
mksh-39-4.el6.x86_64

So i chose "Ignore All" and continued the installation. it finished with
no additional complaints.

[oracle@LX0130 ~]$ env | grep ORA
ORA_CRS_HOME=/apps/crs
ORACLE_PATH=/apps/oracle/common/oracle/sql:.:/apps/oracle/product/11g//rdbms/admin
ORA_NLS10=/apps/oracle/product/11g//nls/data
ORACLE_SID=ISITEST
ORACLE_BASE=/apps/oracle
ORACLE_TERM=xterm
ORACLE_HOME=/apps/oracle/product/11g/
[oracle@LX0130 ~]$ which sqlplus
/apps/oracle/product/11g/bin/sqlplus
[oracle@LX0130 ~]$ sqlplus -v

SQL*Plus: Release 11.2.0.1.0 Production

[oracle@LX0130 ~]$

hth.

update (in reference of comments below from Bob):

i did a re-install. this time with a database created. it proceeded with no problems. haven't tried doing it manually AFTER oracle SW only installation.

vncserver not accessible/won't start in AIX 6.1

2010-11-10T10:14:00.000+08:00

after installing vncserver under AIX 6.1 (TL3), vncserver won't run properly to accept incoming vncviewer connections.

NOTE: vncserver version here is rather old - so be warned.

from the logfile in user's $HOME/.vnc/ directory:

29/10/10 14:22:47 Xvnc version 3.3.3r2
29/10/10 14:22:47 Copyright (C) AT&T Laboratories Cambridge.
29/10/10 14:22:47 All Rights Reserved.
29/10/10 14:22:47 See http://www.uk.research.att.com/vnc for information on VNC
29/10/10 14:22:47 Desktop name 'X' (DEMETER:1)
29/10/10 14:22:47 Protocol version supported 3.3
29/10/10 14:22:47 Listening for VNC connections on TCP port 5901
29/10/10 14:22:47 Listening for HTTP connections on TCP port 5801
29/10/10 14:22:47   URL http://DEMETER:5801
Font directory '/usr/lib/X11/fonts/Speedo/' not found - ignoring

Fatal server error:
could not open default font 'fixed'
xrdb: A remote host refused an attempted connect operation.
1356-605 xrdb: Can't open display 'DEMETER:1'
1356-265 xsetroot:  Unable to open display:  DEMETER:1.
xterm Xt error: Can't open display: DEMETER:1
twm:  unable to open display "DEMETER:1"

solution was to edit the script /usr/bin/X11/vncserver

and add this line (in my system its in line 151:

$cmd .= " -fp /usr/lib/X11/fonts/,/usr/lib/X11/fonts/misc/";

after saving the changes and starting vncserver:

29/10/10 14:44:03 Xvnc version 3.3.3r2
29/10/10 14:44:03 Copyright (C) AT&T Laboratories Cambridge.
29/10/10 14:44:03 All Rights Reserved.
29/10/10 14:44:03 See http://www.uk.research.att.com/vnc for information on VNC
29/10/10 14:44:03 Desktop name 'X' (DEMETER:1)
29/10/10 14:44:03 Protocol version supported 3.3
29/10/10 14:44:03 Listening for VNC connections on TCP port 5901
29/10/10 14:44:03 Listening for HTTP connections on TCP port 5801
29/10/10 14:44:03   URL http://DEMETER:5801

i can now access the vnc session.

ps -ef | grep vnc

user04 4644886 6135868   1 14:47:24  pts/0  0:00 grep vnc
  user04 6004822       1   0 14:44:02  pts/0  0:04 Xvnc :1 -desktop X -httpd /usr/local/vnc/classes -auth /home/admin/user04/.Xauthority -geometry 1024x768 -depth 8 -rfbwait 120000 -rfbauth /home/admin/user04/.vnc/passwd -rfbport 5901 -fp /usr/lib/X11/fonts/,/usr/lib/X11/fonts/misc/

installing vncserver in AIX 6.1

2010-11-10T10:07:00.001+08:00

installing vncserver rpm (this is an outdated version) complaints of some dependency problems:

root@DEMETER]# rpm -ivh vnc-3.3.3r2-3.aix5.1.ppc.rpm
error: failed dependencies:
        libX11.a(shr4.o) is needed by vnc-3.3.3r2-3
        libXext.a(shr.o) is needed by vnc-3.3.3r2-3
        libXt.a(shr4.o) is needed by vnc-3.3.3r2-3

even if the libraries are already there.

to fix:

run

/usr/sbin/updtvpkg

[root@DEMETER]# /usr/sbin/updtvpkg

Please wait...


[root@DEMETER]# rpm -ivh vnc-3.3.3r2-3.aix5.1.ppc.rpm
vnc                         #################################################################

$ vncserver

1356-364 xauth:  creating new authority file /home/DBA/oracle/.Xauthority

New 'X' desktop is DEMETER:1

Creating default startup script /home/DBA/oracle/.vnc/xstartup
Starting applications specified in /home/DBA/oracle/.vnc/xstartup
Log file is /home/DBA/oracle/.vnc/DEMETER:1.log

AIX 5L IPSec (packet filtering)

2010-08-24T09:54:00.001+08:00

This short howto sets an ipsec Filter that allows all connections by default
and then how to create rule to block certain hosts and ports.

This is tested on two TLs:

5300-12-01-1016
5300-08-07-0920

Default installation of AIX 5L has IPSec disabled:

lsfilt -a -v4 -O
Can not open device /dev/ipsec4_filt.

to Start IPSEc, use smitty:

smitty ipsec4

Select Start/Stop IP SEcurity > Start IPsecurity
Deny All Non_Secure IP Packets [no]

then ENTER, this will set default to PERMIT.
the filter will then look like the ff by default:

lsfilt -a -v4 -O

1|*** Dynamic filter placement rule for IKE tunnels ***|no
2|permit|0.0.0.0|0.0.0.0|0.0.0.0|0.0.0.0|yes|all|any|0|any|0|both|both|no|all packets|0|all|0|||

now to edit the ruleset, export default rule to a file:

(NOTE: i only ran the export command so i can have the param names)

[root@UXI0017]/root>expfilt -v4 -f /tmp/ip_sec/ -l all -r
Filter rule(s) have been exported to /tmp/ip_sec/ipsec_fltr_rule.exp successfully.

the file /tmp/ip_sec/ipsec_fltr_rule.exp will then have the following:

#IPver:flt_id:act:src_addr:src_mask:dst_addr:dst_mask:src_rte:proto:src_p_op:src_p:dst_p_op:dst_p:if:scope:dir:log:frag:tun_id:auto_gen:exptime:pattern_type:pattern

which is actully just the name of the parameters used by a filter.

for an example ruleset, this will block (deny) ftp (20/21) from one host and
then block ssh from another. other machines are allowed access:

#IPver:flt_id:act:src_addr:src_mask:dst_addr:dst_mask:src_rte:proto:src_p_op:src_p:dst_p_op:dst_p:if:scope:dir:log:frag:tun_id:auto_gen:exptime:pattern_type:pattern
4 3 deny 10.129.6.122 255.255.255.255 0.0.0.0 0.0.0.0 y tcp any 0 eq  20 all local inbound yes yes 0 no 0 patt_none
4 3 deny 10.129.6.122 255.255.255.255 0.0.0.0 0.0.0.0 y tcp any 0 eq  21 all local inbound yes yes 0 no 0 patt_none
4 4 deny 10.129.12.116 255.255.255.255 0.0.0.0 0.0.0.0 y tcp any 0 eq  22 all local inbound yes yes 0 no 0 patt_none

#import the rule

[root@UX0017]/root>impfilt -v4 -f /tmp/ip_sec/

Filter rule 3 for IPv4 imported as rule 3.
Filter rule 3 for IPv4 imported as rule 4.
Filter rule 4 for IPv4 imported as rule 5.
Filter rule(s) have been imported successfully.

# update/activate the new ruleset
[root@UX0017]/root>mkfilt -v4 -u

#list/check the running v4 ruleset
[root@UX0017]/root>lsfilt -a -v4 -O

1|*** Dynamic filter placement rule for IKE tunnels ***|no
2|deny|10.129.6.122|255.255.255.255|0.0.0.0|0.0.0.0|yes|tcp|any|0|eq|20|local|inbound|yes|all packets|0|all|0|||
3|deny|10.129.6.122|255.255.255.255|0.0.0.0|0.0.0.0|yes|tcp|any|0|eq|21|local|inbound|yes|all packets|0|all|0|||
4|deny|10.129.12.116|255.255.255.255|0.0.0.0|0.0.0.0|yes|tcp|any|0|eq|22|local|inbound|yes|all packets|0|all|0|||
5|permit|0.0.0.0|0.0.0.0|0.0.0.0|0.0.0.0|yes|all|any|0|any|0|both|both|no|all packets|0|all|0|||

hth.

Kickstart file for oracle 11g R2 on CEntOS5 / RHEL5 x86_64

2010-07-01T12:05:00.021+08:00

what my kickstart file does
- installs the OS via cmdline mode (no gui/tui)
- creates the oracle user and filesystem
- sets kernel parameters needed for 11g install
- install required packages for 11g install
- creates an oracle group and user
- sets an oracle environemt var declaration (via .bash_profile)
this is tested with CENTOS5u5/RHEL5u5 x86_64

NOTE:
you need to change the following for your environment
- installation source (here i'm using NFS)
- network settings (ip address, network interface, etc)
- disk drive (i'm using an HP blade with Smart Array 6i controllers
hence the cciss/c0d0 and ccissc0d1 disks

here's the kickstart contents (some company
specific entries here were removed):

##START of kickstart file
install
reboot
skipx
cmdline
nfs --server=<your_nfs_server> --dir=/export/centos/CENTOS5U5/x86_64
lang en_US.UTF-8
keyboard us
#uncomment for RHEL5 install
#key --skip
#network --device eth0 --onboot no --bootproto dhcp --hostname LX0040
#network --device eth1 --onboot no --bootproto dhcp --hostname LX0040
network --device eth2 --bootproto static --ip 10.129.a.b --netmask 255.255.255.0 --gateway 10.129.a.1 --nameserver 10.129.c.d --hostname LX0040
#network --device eth3 --onboot no --bootproto dhcp --hostname LX0040
# root pass is r00tm3pass
rootpw --iscrypted $1$4Y6Je/$CF8Cts/dWgo4d7DpasKSu0
firewall --disabled
authconfig --enableshadow --enablemd5
selinux --permissive
timezone Asia/Singapore
device scsi cciss
bootloader --location=mbr --driveorder=cciss/c0d0,cciss/c0d1 --append="rhgb quiet"
##############################
##this is for a two SMart Array volume setup  
## CHANGE THIS FOR YOUR MACHINE!!!
clearpart --linux --initlabel --drives=cciss/c0d0,cciss/c0d1
part /boot --fstype ext3 --size=100 --ondisk=cciss/c0d0 --asprimary
part pv.6 --size=100 --grow --ondisk=cciss/c0d0 --asprimary
part pv.7 --size=100 --grow --ondisk=cciss/c0d1 --asprimary
volgroup vg00 --pesize=32768 pv.6
volgroup vgdata00 --pesize=32768 pv.7
logvol / --fstype ext3 --name=lvsys00 --vgname=vg00 --size=512
logvol swap --fstype swap --name=lvsys01 --vgname=vg00 --size=8192
logvol /usr --fstype ext3 --name=lvsys02 --vgname=vg00 --size=2048
logvol /var --fstype ext3 --name=lvsys03 --vgname=vg00 --size=1024
logvol /opt --fstype ext3 --name=lvsys04 --vgname=vg00 --size=256
#will produce error during oracle install
#logvol /tmp --fstype ext3 --name=lvsys05 --vgname=vg00 --size=2048 --fsoptions=noexec,nodev
logvol /tmp --fstype ext3 --name=lvsys05 --vgname=vg00 --size=2048
logvol /home --fstype ext3 --name=lvsys06 --vgname=vg00 --size=256
logvol /tools --fstype ext3 --name=lvtools00 --vgname=vg00 --size=5000
logvol /u01 --fstype ext3 --name=lvora00 --vgname=vgdata00 --size=10000

services --disabled sendmail,xfs,bluetooth,kudzu,firstboot,cups,ip6tables,iptables

%packages
@legacy-software-support
e2fsprogs
kernel
openssh-clients
openssh-server
postfix
vnc-server
yum
device-mapper
device-mapper-multipath
device-mapper-event

#oracle
ntp
strace
ncompress
sg3_utils
ncurses
xorg-x11-deprecated-libs-devel
xorg-x11-deprecated-libs
xorg-x11-utils
libX11-devel
libXext
libdmx
libXxf86misc
libXxf86dga
sysstat
compat-gcc-34-c++
compat-libgcc-296
compat-glibc-headers
compat-glibc
libgcc
compat-db
gcc
gcc-c++
ncurses.i386
compat-glibc.i386
glibc-devel
dejavu-lgc-fonts
xterm
#system-config-cluster
binutils
compat-libstdc++-296
compat-libstdc++-33
elfutils-libelf
gcc
gcc-c++
glibc
glibc-common
glibc-devel
glibc-headers
libaio
libaio-devel
libgcc.i386
libstdc++
libstdc++-devel
libXp
libXt
libXtst
make
openmotif
sysstat
unixODBC
unixODBC-devel
pyOpenSSL
#11g
elfutils-libelf-devel
pdksh

#unwanted
-ftp
-talk
-setserial
-wireless-tools
-rp-pppoe
-ppp
-wpa_supplicant
-bluez-utils
-bluez-gnome
-rsh
-rdist
-pango
-notification-daemon
-usbutils
-irda-utils
-firstboot-tui

%post --log=/root/ks-post.log

{
#~-> Remove GRUB's splash image
/bin/ed <<EOT /etc/grub.conf
1,$ s/^splashimage/#splashimage/
.
wq
EOT

#~-> User limits
ed /etc/security/limits.conf << EOF
$
.a
oracle       soft    nofile          65536
oracle       hard    nofile          101062
oracle       soft    nproc           16384
oracle       hard    nproc           101062
.
w
EOF

#kernel params
/bin/cat <<EOF_sysctl > /etc/sysctl.conf
# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1

# increase the local port range
net.ipv4.ip_local_port_range = 9000 65500

# enable tcp syn cookie protection
net.ipv4.tcp_syncookies = 1

# disable icmp redirect acceptance
net.ipv4.conf.all.accept_redirects = 0

# enable ip spoofing protection
net.ipv4.conf.all.rp_filter = 1

# log spoofed/source routed/redirected packets
#net.ipv4.conf.all.log_martians = 1

# want to become a smurf amplifier?
net.ipv4.icmp_echo_ignore_broadcasts = 1

# enable bad error message protection
net.ipv4.icmp_ignore_bogus_error_responses = 1

# decrease time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 30

# decrease time default value for tcp_keepalive_time connection
net.ipv4.tcp_keepalive_time = 1800

# turn off tcp_sack
net.ipv4.tcp_sack = 0

# turn off tcp_timestamps
net.ipv4.tcp_timestamps = 0

# turn off tcp_window_scaling
net.ipv4.tcp_window_scaling = 0

fs.file-max = 6815744

kernel.shmall = 10523004
kernel.shmmax = 6465333657
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128

# Default setting in bytes of the socket receive buffer
net.core.rmem_default=262144
# Default setting in bytes of the socket send buffer
net.core.wmem_default=262144
# Maximum socket receive buffer size which may be set by using the SO_RCVBUF socket option
net.core.rmem_max=4194304
# Maximum socket send buffer size which may be set by using the SO_SNDBUF socket option
net.core.wmem_max=1048576
fs.aio-max-nr = 1048576
EOF_sysctl


/bin/chown root:root /etc/sysctl.conf
/bin/chmod 0600 /etc/sysctl.conf
/sbin/sysctl -p


# Modify DNS Search
/bin/cat <<EOT > /etc/resolv.conf
nameserver 10.129.c.20
nameserver 10.129.c.21
domain example.com
options attempts:5
options timeout:15
EOT

# create group
/usr/sbin/groupadd -g 306 dba
/bin/mkdir --mode=0750 /home/dba

chgrp dba /home/dba
#oracle password is oracl3pass
/usr/sbin/useradd -c 'oracle user' -d /home/dba/oracle -u 306 -p '$1$F.iIe/$vfXBG1CqjXB5xOu.LLEdJ.' -g dba -G dba   oracle

cat <<EOT >> /etc/auto.isapsys
oracle11g -ro,soft,intr,rsize=8192,wsize=8192 uxisap0027:/export/isapsys/oracle11g
EOT

# vncserver stuff
cat << EOT > /opt/vnc_xstartup
#!/bin/sh

# run vncserver and copy to your $HOME/.vnc/xstartup file
# Uncomment the following two lines for normal desktop:
# unset SESSION_MANAGER
# exec /etc/X11/xinit/xinitrc

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
#xterm -geometry 130x30+12+12 -ls -bg black -fg green -title "$VNCDESKTOP Desktop" &
xterm -geometry 130x30+12+12 -ls -title "$VNCDESKTOP Desktop" &
mwm &
EOT

# oracle user .bash_profile
cat > /home/dba/oracle/.bash_profile << EOF
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
      . ~/.bashrc
fi

# User specific environment and startup programs
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=\${ORACLE_BASE}/product/11g/db_1
export ORA_CRS_HOME=/u01/app/crs
export ORACLE_PATH=\${ORACLE_BASE}/common/oracle/sql:.:\${ORACLE_HOME}/rdbms/admin
export CV_JDKHOME=/usr/local/java
# Each RAC node must have a unique ORACLE_SID. (i.e. orcl1, orcl2,...)
export ORACLE_SID=orcl1
export PATH=.:\${JAVA_HOME}/bin:${PATH}:$HOME/bin:\${ORACLE_HOME}/bin
export PATH=\${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export PATH=\${PATH}:\${ORACLE_BASE}/common/oracle/bin
export ORACLE_TERM=xterm
export TNS_ADMIN=\${ORACLE_HOME}/network/admin
export ORA_NLS10=\${ORACLE_HOME}/nls/data
export LD_LIBRARY_PATH=\${ORACLE_HOME}/lib
export LD_LIBRARY_PATH=\${LD_LIBRARY_PATH}:\${ORACLE_HOME}/oracm/lib
export LD_LIBRARY_PATH=\${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export CLASSPATH=\${ORACLE_HOME}/JRE
export CLASSPATH=\${CLASSPATH}:\${ORACLE_HOME}/jlib
export CLASSPATH=\${CLASSPATH}:\${ORACLE_HOME}/rdbms/jlib
export CLASSPATH=\${CLASSPATH}:\${ORACLE_HOME}/network/jlib
export THREADS_FLAG=native
export TEMP=/tmp
export TMPDIR=/tmp
alias ls="ls -FA"
set -o vi
EOF
# end oracle profile
chown -R oracle.dba /u01
mount | column -t
} 1>/root/ks-post.log 2>&1
##END of kickstart file

ks install in progress:

space usage after the install:

Filesystem                    Size  Used   Avail  Use%  Mounted   on
/dev/mapper/vg00-lvsys00      496M  238M   234M   51%   /
/dev/mapper/vg00-lvsys03      992M  56M    885M   6%    /var
/dev/mapper/vg00-lvsys02      2.0G  1012M  870M   54%   /usr
/dev/mapper/vg00-lvsys06      248M  11M    226M   5%    /home
/dev/mapper/vg00-lvsys04      248M  11M    226M   5%    /opt
/dev/mapper/vg00-lvsys05      2.0G  70M    1.8G   4%    /tmp
/dev/cciss/c0d0p1             99M   13M    82M    14%   /boot
tmpfs                         3.9G  0      3.9G   0%    /dev/shm
/dev/mapper/vg00-lvtools00    4.8G  138M   4.4G   4%    /tools
/dev/mapper/vgdata00-lvora00  9.5G  4.1G   4.9G   46%   /u01

After the OS install, system will reboot. from here on, Oracle 11g R2 installation
is a breeze. I just login as oracle user, start vncserver and connect via
vncviewer from my workstation to start the install.

these are the last part of the oracle installation via
vnc-viewer session:

Wacom Bamboo Pen in Debian Linux

2010-04-27T14:58:00.013+08:00

This is how I configured a Wacom Bamboo Pen (model CTL-460) inside
a Debion (5.0.3) based HP Thinclient workstation for use in a Citrix
session.

1. downloaded the linuxwacom tar ball from sourceforge.net:

http://linuxwacom.sourceforge.net/

then extract:

tar jxvf linuxwacom-0.8.6.tar.bz

go into the new directory

cd /home/user/linuxwacom-0.8.6

then run this
./configure --enable-wacom --prefix=/usr

output:

##############################################
----------------------------------------
BUILD ENVIRONMENT:
architecture - i486-linux-gnu
linux kernel - yes 2.6.26
module versioning - no 
kernel source - yes /lib/modules/2.6.26-2-686/build
XFree86 source - no 
Xorg SDK - yes /usr/include/xorg
XSERVER64 - no
dlloader - yes
XLib - yes /usr/lib
xf86config - no
TCL - yes /usr/include/tcl8.5
TK - no 
ncurses - no

BUILD OPTIONS:
wacom.o - yes
wacdump - no 
xidump - yes (no ncurses)
libwacomcfg - yes
libwacomxi - no
xsetwacom - yes
wacomxrrd - no
hid.o - no 
wacom_drv.so - yes /usr/lib/xorg/modules/input 
wacom_drv.o - no
wacom*_drv quirks - libc-wrapper IsXExtensionPointer key-events dixScreenOrigins Uninit-called
----------------------------------------
##############################################

- Check kernel version

uname -r

##############################################
mine is:
2.6.26-2-686
##############################################

if the configure command above is OK, cd into

/home/user/linuxwacom-0.8.6/src/2.6.26

contents:

##############################################
-rwxr-xr-x 1 user user 23584 2010-04-10 07:02 wacom_sys.c
-rwxr-xr-x 1 user user  6807 2010-04-10 07:02 wacom.h
-rwxr-xr-x 1 user user  1491 2010-04-10 07:02 Makefile.in
-rwxr-xr-x 1 user user  2746 2010-04-26 11:11 wacom_wac.h
-rwxr-xr-x 1 user user 50431 2010-04-26 11:11 wacom_wac.c
-rw-r--r-- 1 user user     0 2010-04-26 11:11 Module.symvers
-rw-r--r-- 1 user user    64 2010-04-26 11:11 modules.order
-rw-r--r-- 1 user user  1491 2010-04-26 11:26 Makefile
##############################################

now run make:

##############################################
user@TCISAP8610:~/linuxwacom-0.8.6/src/2.6.26$ make
cp -f ../2.6.27/wacom_wac.c .
cp -f ../2.6.27/wacom_wac.h .
Building linuxwacom drivers for 2.6 kernel.
***Note: Drivers not enabled as modules in your kernel config but requested through configure are NOT built
make -C /lib/modules/2.6.26-2-686/build M=/writable/home/user/linuxwacom-0.8.6/src/2.6.26
make[1]: Entering directory `/usr/src/linux-headers-2.6.26-2-686'
LD      /writable/home/user/linuxwacom-0.8.6/src/2.6.26/built-in.o
CC [M]  /writable/home/user/linuxwacom-0.8.6/src/2.6.26/wacom_wac.o
CC [M]  /writable/home/user/linuxwacom-0.8.6/src/2.6.26/wacom_sys.o
LD [M]  /writable/home/user/linuxwacom-0.8.6/src/2.6.26/wacom.o
Building modules, stage 2.
MODPOST 1 modules
CC      /writable/home/user/linuxwacom-0.8.6/src/2.6.26/wacom.mod.o
LD [M]  /writable/home/user/linuxwacom-0.8.6/src/2.6.26/wacom.ko
make[1]: Leaving directory `/usr/src/linux-headers-2.6.26-2-686'
##############################################

New files will be produced (kernel modules):

##############################################
-rwxr-xr-x 1 user user 23584 2010-04-10 07:02 wacom_sys.c
-rwxr-xr-x 1 user user  6807 2010-04-10 07:02 wacom.h
-rwxr-xr-x 1 user user  1491 2010-04-10 07:02 Makefile.in
-rw-r--r-- 1 user user     0 2010-04-26 11:11 Module.symvers
-rw-r--r-- 1 user user  1491 2010-04-26 11:26 Makefile
-rwxr-xr-x 1 user user  2746 2010-04-26 11:28 wacom_wac.h
-rwxr-xr-x 1 user user 50431 2010-04-26 11:28 wacom_wac.c
-rw-r--r-- 1 user user     8 2010-04-26 11:28 built-in.o
-rw-r--r-- 1 user user 24328 2010-04-26 11:28 wacom_wac.o
-rw-r--r-- 1 user user  9504 2010-04-26 11:28 wacom_sys.o
-rw-r--r-- 1 user user 31965 2010-04-26 11:28 wacom.o
-rw-r--r-- 1 user user  5387 2010-04-26 11:28 wacom.mod.c
-rw-r--r-- 1 user user    64 2010-04-26 11:28 modules.order
-rw-r--r-- 1 user user 10376 2010-04-26 11:28 wacom.mod.o
-rw-r--r-- 1 user user 41728 2010-04-26 11:28 wacom.ko
##############################################

2. Install the module

now, as root, copy the new module to kernel module directory:

cd /home/user/linuxwacom-0.8.6/src/2.6.26

cp wacom.ko /lib/modules/$(uname -r)/kernel/drivers/input/tablet/

in my system, the command will be

cp wacom.ko /lib/modules/2.6.26-2-686/kernel/drivers/input/tablet/

##############################################
user@TCISAP8610:~/linuxwacom-0.8.6$ ls -ltr /lib/modules/2.6.26-2-686/kernel/drivers/input/tablet
total 88
-rw-r--r-- 1 root root  7437 Aug 19  2009 kbtab.ko
-rw-r--r-- 1 root root 10395 Aug 19  2009 gtco.ko
-rw-r--r-- 1 root root 20841 Aug 19  2009 aiptek.ko
-rw-r--r-- 1 root root  7501 Aug 19  2009 acecad.ko
-rw-r--r-- 1 root root 41728 Apr 26 11:12 wacom.ko
##############################################

Then run the following (as root)

(as root)
depmod -a

modprobe -r wacom

modprobe wacom

3. Test it

Plug in the wacom device to a test hp thinclient workstation.

from dmesg

[20188.832647] usb-storage: device scan complete
[20188.833742] scsi 0:0:0:0: Direct-Access     RIM      BlackBerry       1002 PQ: 0 ANSI: 4 CCS
[20188.852275] usb-storage: device scan complete
[20188.855377] scsi 1:0:0:0: Direct-Access     RIM      BlackBerry SD    0002 PQ: 0 ANSI: 0 CCS
[20189.077568] Driver 'sd' needs updating - please use bus_type methods
[20189.082293] sd 0:0:0:0: [sda] Attached SCSI removable disk
[20189.085329] sd 1:0:0:0: [sdb] Attached SCSI removable disk
[20652.800029] usb 3-2: new full speed USB device using ohci_hcd and address 2
[20652.964009] usb 3-2: configuration #1 chosen from 1 choice
[20652.981247] input: Wacom Bamboo 4x5 Pen as /class/input/input6
[20653.026234] input: Wacom Bamboo 4x5 Finger as /class/input/input7
[20653.063422] usb 3-2: New USB device found, idVendor=056a, idProduct=00d4
[20653.063434] usb 3-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[20653.063439] usb 3-2: Product: CTL-460
[20653.063443] usb 3-2: Manufacturer: Wacom Co.,Ltd.

via lsmod | grep wacom

wacom                  28768  0
usbcore               118192  6 usb_storage,wacom,usbhid,ehci_hcd,ohci_hcd

via lsusb

Bus 006 Device 003: ID 0fca:8004 Research In Motion, Ltd.
Bus 006 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 003: ID 046d:c018 Logitech, Inc. Optical Wheel Mouse
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 003 Device 002: ID 056a:00d4 Wacom Co., Ltd
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

The bamboo should be responding by now.

to make it persistent across reboots, edit the file (as root):

/etc/modules

and append wacom (as show below):

##############################################
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.

# Load the ALSA OSS emulation layer
# The only known application which require OSS still
# is ICA hopefully they will switch to modern APIs someday
# padsp should also be checked out in the future so everything
# goes through PulseAudio
snd-pcm-oss
lp
wacom
# end of /etc/modules
##############################################

Then reboot. Inside a citrix session, i'm able to use the bamboo pen
device inside MS paint. the device is positioned horizontally on my desk
- where the cableend is at my lefthand side.

Below is the xorg.conf file i'm using
(resolution is 1440x900 - with a samsung SyncMaster 933 LCD monitor)

##############################################
# start xorg.conf
Section "Files"
FontPath "/usr/share/fonts/X11/misc"
FontPath "/usr/X11R6/lib/X11/fonts/misc"
FontPath "/usr/share/fonts/X11/cyrillic"
FontPath "/usr/X11R6/lib/X11/fonts/cyrillic"
FontPath "/usr/share/fonts/X11/100dpi/:unscaled"
FontPath "/usr/X11R6/lib/X11/fonts/100dpi/:unscaled"
FontPath "/usr/share/fonts/X11/75dpi/:unscaled"
FontPath "/usr/X11R6/lib/X11/fonts/75dpi/:unscaled"
FontPath "/usr/share/fonts/X11/Type1"
FontPath "/usr/X11R6/lib/X11/fonts/Type1"
FontPath "/usr/share/fonts/X11/100dpi"
FontPath "/usr/X11R6/lib/X11/fonts/100dpi"
FontPath "/usr/share/fonts/X11/75dpi"
FontPath "/usr/X11R6/lib/X11/fonts/75dpi"
# path to defoma fonts
FontPath "/var/lib/defoma/x-ttcidfont-conf.d/dirs/TrueType"
EndSection 
Section "Module"
Load "bitmap"
Load "ddc"
Load "dri"
Load "extmod"
Load "freetype"
Load "glx"
Load "int10"
Load "vbe"
EndSection

Section "InputDevice"
Identifier "Generic Keyboard"
Driver  "kbd"
Option          "XLeds" "1 2 3"
Option  "CoreKeyboard"
Option  "XkbRules" "xorg"
Option "XkbModel" "pc104"
Option "XkbLayout" "us"
Option "XkbVariant" ""
EndSection
Section "Monitor"
Identifier "Primary Monitor"
Option  "DPMS"  "true"
Option  "TargetRate"  "60"
Option  "TargetRefresh"  "60"
Option  "PreferredMode" "1440x900"
EndSection
Section "Monitor"
Identifier "Secondary Monitor"
Option  "DPMS" "true"
Option  "TargetRate"  "60"
Option  "TargetRefresh"  "60"
Option  "PreferredMode" "1440x900"
EndSection
Section "Device"
Identifier "Driver0"
Screen  0
Driver  "fglrx"  
BusID  "PCI:01:05:0" 
Option  "Monitor-CRT1" "Primary Monitor" 
EndSection
Section "Device"
Identifier "Driver1"
Screen  1
Driver  "fglrx"  
BusID  "PCI:01:05:0" 
Option  "Monitor-DFP2" "Secondary Monitor" 
EndSection
Section "Screen"
Device  "Driver0"
Monitor  "Primary Monitor"
Identifier "Primary Screen"
DefaultDepth 24
SubSection "Display"
Viewport 0 0
Modes "1440x900"
EndSubSection
EndSection
Section "Screen"
Device  "Driver1"
Monitor  "Secondary Monitor"
Identifier "Secondary Screen"
DefaultDepth 24
SubSection "Display"
Viewport 0 0
Modes "1440x900"
EndSubSection
EndSection
Section "ServerLayout"
Identifier "Default Layout"
InputDevice "Generic Keyboard"
Screen  0 "Primary Screen" 0 0
Screen  1 "Secondary Screen" RightOf "Primary Screen" 
Option  "xinerama" "on" 
EndSection
Section "DRI"
Mode 0666
EndSection
Section "ServerFlags"
Option "DontVTSwitch" "true"
Option "DontZap" "true"
Option "DontZoom" "true" 
EndSection
# end xorg.conf
##############################################

Removing multipath disks

2010-04-22T11:06:00.005+08:00

in one of our RHEL4 blade servers, i need to remove
1x500G HP XP1000 LUN:

PV         VG             Fmt  Attr PSize   PFree
  /dev/dm-17 lx0080apps00vg lvm2 a-   200.00G  11.50G
  /dev/dm-18 lx0080apps00vg lvm2 a-   200.00G  11.50G
  /dev/dm-19 lx0080apps02vg lvm2 a-   500.00G 100.00G
  /dev/dm-20 lx0080apps01vg lvm2 a-   250.00G  50.00G
  /dev/dm-21 lx0080apps01vg lvm2 a-   100.00G  50.00G
  /dev/dm-22 lx0080apps01vg lvm2 a-   200.00G  10.00G
  /dev/dm-23 lx0080apps01vg lvm2 a-   200.00G  10.00G
  /dev/dm-24 lx0080apps01vg lvm2 a-   200.00G  10.00G
  /dev/sda2  vg00           lvm2 a-    67.88G   4.47G

[root@lx0080 ~]# pvdisplay /dev/dm-19

--- Physical volume ---
  PV Name               /dev/dm-19
  VG Name               lx0080apps02vg
  PV Size               500.00 GB / not usable 1.25 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              128000
  Free PE               25600
  Allocated PE          102400
  PV UUID               O9b3hw-oSUO-pUpe-oGJw-zEBp-tKXo-3ba2g9

so here's how to do it without causing disruptions on the machine:
btw, this machine hosts a test environment.

CAUTION: DON'T DO THIS IN PRODUCTION ENVIRONMENT ON A REGULAR
BUSINESS DAY (Even if you are sure it won't cause any problems)

make sure no i/o activity on the disk
- i've removed it from nfs exports
- unmounted the FS
remove from VG group
remove from multipath definitions
delete from system

below here are the actual steps.

1. remove from VG group

[root@lx0080 ~]# vgremove lx0080apps02vg

Found duplicate PV O9b3hwoSUOpUpeoGJwzEBptKXo3ba2g9: using /dev/sdb1 not /dev/sdq1
Do you really want to remove volume group "lx0080apps02vg" containing 1 logical volumes? [y/n]: y
Do you really want to remove active logical volume "lx0080apps02LV"? [y/n]: y
  Logical volume "lx0080apps02LV" successfully removed
  Volume group "lx0080apps02vg" successfully removed

pvs

PV         VG             Fmt  Attr PSize   PFree
  /dev/dm-17 lx0080apps00vg lvm2 a-   200.00G  11.50G
  /dev/dm-18 lx0080apps00vg lvm2 a-   200.00G  11.50G
  /dev/dm-19                lvm2 --   500.00G 500.00G

/dev/dm-20 lx0080apps01vg lvm2 a-   250.00G  50.00G
  /dev/dm-21 lx0080apps01vg lvm2 a-   100.00G  50.00G
  /dev/dm-22 lx0080apps01vg lvm2 a-   200.00G  10.00G
  /dev/dm-23 lx0080apps01vg lvm2 a-   200.00G  10.00G
  /dev/dm-24 lx0080apps01vg lvm2 a-   200.00G  10.00G
  /dev/sda2  vg00           lvm2 a-    67.88G   4.47G

as you can see - /dev/dm-19 is now an orphan. since its the only 500G
LUN in the server, its easy to identify.

2. remove from multipath

from multipath -ll (other entries removed):

mpath8 (360060e8004a51a000000a51a00000200)
[size=500 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 3:0:0:0 sdb 8:16   [active][ready]
 \_ 4:0:0:0 sdq 65:0   [active][ready]

it has two active paths.

multipath -f mpath8

(-f flush a multipath device map specified as parameter, if unused)

[root@lx0080 ~]# multipath -f mpath8
[root@lx0080 ~]# multipath -ll | grep 360060e8004a51a000000a51a00000200

360060e8004a51a000000a51a00000200 is the UUID of the disk from
scsi_id -gus /block/sdb (or /block/sdq)

now the multipath map is removed. but the disk is still in the system.

[root@lx0080 ~]# fdisk -l /dev/sdq

Disk /dev/sdq: 536.8 GB, 536877465600 bytes
255 heads, 63 sectors/track, 65271 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdq1               1       65271   524289276   8e  Linux LVM

[root@lx0080 ~]# fdisk -l /dev/sdb

Disk /dev/sdb: 536.8 GB, 536877465600 bytes
255 heads, 63 sectors/track, 65271 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       65271   524289276   8e  Linux LVM

[root@lx0080 ~]#

[root@lx0080 ~]# pvs

Found duplicate PV O9b3hwoSUOpUpeoGJwzEBptKXo3ba2g9: using /dev/sdb1 not /dev/sdq1
  PV         VG             Fmt  Attr PSize   PFree
  /dev/dm-17 lx0080apps00vg lvm2 a-   200.00G  11.50G
  /dev/dm-18 lx0080apps00vg lvm2 a-   200.00G  11.50G
  /dev/dm-20 lx0080apps01vg lvm2 a-   250.00G  50.00G
  /dev/dm-21 lx0080apps01vg lvm2 a-   100.00G  50.00G
  /dev/dm-22 lx0080apps01vg lvm2 a-   200.00G  10.00G
  /dev/dm-23 lx0080apps01vg lvm2 a-   200.00G  10.00G
  /dev/dm-24 lx0080apps01vg lvm2 a-   200.00G  10.00G
  /dev/sda2  vg00           lvm2 a-    67.88G   4.47G
  /dev/sdb1                 lvm2 --   500.00G 500.00G

3. delete from the system

[root@lx0080 ~]# echo 1 > /sys/block/sdb/device/delete
[root@lx0080 ~]# echo 1 > /sys/block/sdq/device/delete
[root@lx0080 ~]# dmesg

Synchronizing SCSI cache for disk sdb:
Synchronizing SCSI cache for disk sdq:

[root@lx0080 ~]# fdisk -l /dev/sdb
[root@lx0080 ~]# fdisk -l /dev/sdq

here are some entries generated thru syslog:

- when multipath -f was run:

Apr 22 09:17:20 lx0080 multipathd: dm map mpath8 removed
Apr 22 09:17:20 lx0080 udevd[2444]: udev done!

- when deleting sdb

Apr 22 09:22:52 lx0080 kernel: Synchronizing SCSI cache for disk sdb:
Apr 22 09:22:52 lx0080 multipathd: remove sdb path checker
Apr 22 09:22:52 lx0080 udevd[2444]: udev done!

- when deleting sdq

Apr 22 09:22:57 lx0080 kernel: Synchronizing SCSI cache for disk sdq:
Apr 22 09:22:57 lx0080 multipathd: remove sdq path checker
Apr 22 09:22:57 lx0080 udevd[2444]: udev done!
Apr 22 09:23:02 lx0080 udevd[2444]: udev done!

our storage admin can now remove the disk definition from storage
server.

Adding multipath disks without rebooting

2010-04-13T13:45:00.001+08:00

this is done on a CEntOS 5.3 x86_64 Box with HP EVA 8000 LUNs.
HBA is Emulex.

after disks have been presented on the OS, they can be "discovered"
by scanning the hba ports.

doing a scsi bus rescan or reset causes connectivity problems
and should be avoided if you can't stop I/O activities on the
server.

i'm doing this on a CEntOs 5.3 machine with several existing
HP EVA 8000 LUNs with multipath as shown below:

mpath2 (3600508b40005459d0000c00001120000) dm-10 HP,HSV210
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=110][active]                      
 \_ 1:0:0:2 sdc 8:32  [active][ready]                    
 \_ 2:0:0:2 sdh 8:112 [active][ready]                    
 \_ 2:0:1:2 sdm 8:192 [active][ready]                    
mpath1 (3600508b40005459d0000c000010f0000) dm-9 HP,HSV210
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=110][active]                      
 \_ 1:0:0:1 sdb 8:16  [active][ready]                    
 \_ 2:0:0:1 sdg 8:96  [active][ready]                    
 \_ 2:0:1:1 sdl 8:176 [active][ready]                    
mpath5 (3600508b40005459d0000e00000450000) dm-13 HP,HSV210
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw] 
\_ round-robin 0 [prio=70][active]                        
 \_ 1:0:0:5 sdf 8:80  [active][ready]                     
 \_ 2:0:0:5 sdk 8:160 [active][ready]                     
 \_ 2:0:1:5 sdp 8:240 [active][ready]                     
mpath4 (3600508b40005459d0000c000011a0000) dm-12 HP,HSV210
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw] 
\_ round-robin 0 [prio=110][active]                       
 \_ 1:0:0:4 sde 8:64  [active][ready]                     
 \_ 2:0:0:4 sdj 8:144 [active][ready]                     
 \_ 2:0:1:4 sdo 8:224 [active][ready]                     
mpath3 (3600508b40005459d0000c00001150000) dm-11 HP,HSV210
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw] 
\_ round-robin 0 [prio=110][active]                       
 \_ 1:0:0:3 sdd 8:48  [active][ready]                     
 \_ 2:0:0:3 sdi 8:128 [active][ready]                     
 \_ 2:0:1:3 sdn 8:208 [active][ready]

what's needed:
1. current LUN IDs of the server (via multipath -ll or lsscsi)
2. WWN of the storage server
3. the number and channel of HBA

current LUN IDs are shown in the multipath -ll output above.
or, via lsscsi, its in the leftmost column, the 4th number:

[2:0:1:1]    disk    HP       HSV210           5000  /dev/sdl
[2:0:1:2]    disk    HP       HSV210           5000  /dev/sdm
[2:0:1:3]    disk    HP       HSV210           5000  /dev/sdn
[2:0:1:4]    disk    HP       HSV210           5000  /dev/sdo
[2:0:1:5]    disk    HP       HSV210           5000  /dev/sdp

the current LUN ids are 1,2,3,4 and 5.

0x50001fe1500a7340 is the WWN of the storage server
(from /sys/class/fc_transport/target1\:0\:0/node_name)

to list all existing hba number/channel, run:

grep 0x50001fe1500a7340 /sys/class/fc_transport/*/node_name

/sys/class/fc_transport/target1:0:0/node_name:0x50001fe1500a7340  -> 1st path
/sys/class/fc_transport/target2:0:0/node_name:0x50001fe1500a7340  -> 2nd path
/sys/class/fc_transport/target2:0:1/node_name:0x50001fe1500a7340  -> 3rd path

now let the OS know of the new storage device(s):

echo "c t l" > /sys/class/scsi_host/hosth/scan

h is the HBA number
c is the channel on the HBA
t is the SCSI target ID,   
l is the LUN.

in my case, i need to use the following commands:

echo "0 0 06" > /sys/class/scsi_host/host1/scan
echo "0 0 06" > /sys/class/scsi_host/host2/scan
echo "0 1 06" > /sys/class/scsi_host/host2/scan

from dmesg, the following is for the first echo command:

and for the next two echo commands:

now we have 3 new disk: sdq, sdr and sds (these are not persistent
and may change during reboots).

to verify, run scs_id on each new disks to see
if they have the same UID:

now i run fdisk on /dev/sdq to create one partition wity type 8e
(Linux LVM):

and then to re-enable multipath for the new drive, run:

[root@LXI*****60 ~]#multipath

mpath6 is then created:

[root@LXI*****60 ~]# multipath -ll

mpath6 (3600508b40005459d0000e00001370000) dm-22 HP,HSV210
[size=750G][features=1 queue_if_no_path][hwhandler=0][rw] 
\_ round-robin 0 [prio=70][active]                        
 \_ 1:0:0:6 sdq 65:0  [active][ready]                     
 \_ 2:0:0:6 sdr 65:16 [active][ready]                     
 \_ 2:0:1:6 sds 65:32 [active][ready]

Adding to existing Volume Group

run kpartx on the new mpath to list available mapping

[root@LXI*****60 ~]#kpartx -l /dev/mapper/mpath6

mpath6p1 : 0 1572859827 /dev/mapper/mpath6 63

now create the partition mappings:

[root@LXI*****60 ~]# kpartx -a /dev/mapper/mpath6

[root@LXI*****60 ~]# ls /dev/mapper/mpath6*

/dev/mapper/mpath6  /dev/mapper/mpath6p1

[root@LXISAP0060 ~]# ls -ltr /dev/mapper/mpath6*

brw-rw---- 1 root disk 253, 22 Apr  7 12:00 /dev/mapper/mpath6
brw-rw---- 1 root disk 253, 23 Apr  7 13:43 /dev/mapper/mpath6p1

initiliaze the device for LVM use:

[root@LXI*****60 ~]# pvcreate /dev/mapper/mpath6p1

returns:

Physical volume "/dev/mapper/mpath6p1" successfully created

listing via pvs:

/dev/dm-23 is the one to be used:

[root@LXI*****60 ~]# vgextend lx0060apps00vg /dev/dm-23

  Volume group "lx0060apps00vg" successfully extended

Resizing logical volume and the FS

[root@LXI*****60 ~]# lvresize -i1 -L +200G /dev/mapper/lx0060apps00vg-lx0060data00LV

  Extending logical volume lx0060data00LV to 930.00 GB
  Logical volume lx0060data00LV successfully resized

and then for the fs:

[root@LXI*****60 ~]# resize2fs /dev/mapper/lx0060apps00vg-lx0060data00LV

till next time. :)

HMC V7 "Connection to the Command Server failed"

2010-03-30T10:50:00.003+08:00

searching via google about the error and found this:

http://www-01.ibm.com/support/docview.wss?uid=isg3T1011162

(steps to fix are also mentioned there).

our HMCs have all the symptoms mentioned there.

here are two snapshots:

i decided to do a clean install of a new version. so from IBM's

fix central, i need to get the following:

HMC V7R3.5.0M0 Released 30 Oct 2009 MH01194

(which i can use to a fresh install - i don't have any

special settings with our HMC so reconfiguration is easy).

HMC V7R3.5.0M1 MH01212

(This is the current recommended HMC Service Pack for MH01194

and was made available on 15 Mar 2010).

info for above updates are available from the following links:

http://www-933.ibm.com/support/fixcentral/firmware/readme?fixid=MH01194

http://www-933.ibm.com/support/fixcentral/firmware/readme?fixid=MH01212

installation is straight forward (make sure you have notes of the HMC and

managed systems' network settings and logins where applicable.

(psss...blogger is getting harder and harder to use everytime i make a new post. :()..

postfix canonical maps

2010-03-10T15:15:00.005+08:00

mail admins complain that there are bounces messages
due to invalid non-routable email addresses in our environments.
since our *nix servers send out regular system and application
messages with the sender local to those systems, there are a lot
of instances were replies to these bounce back or sent to
mail admin catch-all folders due to the invalid
addresses.

example:

hostA.localdomain.cib sends out messages (from root) to
an application user.with email user@validdomain.com,
user is on vacation and has set out an out-of-office
reply.
ooo reply sends out email addressed to root@hostA.localdomain.cib
with the message bouncing back to user@validdomain.com or
gets sent to a mail admin inbox folder.

so as not to touch any application settings, canonical
mappings in postfix have been used.

in /etc/postfix/canonical

@localdomain.cib    admin.user@validdomain.com

then run (cwd is /etc/postfix):
postmap canonical

(the file /etc/postfix/canonical.db gets created or updated)

in /etc/postfix/main.cf

canonical_maps = hash:/etc/postfix/canonical

then run:
/sbin/service postfix reload

(if additional changes are needed in the canonical file, no need
to reload postfix).

from here on, messages will appear to have come from the email address:

admin.user@validdomain.com

NOTE: above example affects all messages coming from that machine.
it can be configured on a per user or per address setup. in this case, it
is "domain wide" from the machine's point of view.
please read the /etc/postfix/canonical file - it
contains documentation on how it is used.

pam_tally2 - lock account after failed logins

2009-12-10T17:17:00.009+08:00

this post shows how to temporarily block an account after reaching a certain number of failled logins.
system is a CentOS 5.4 x86_64.

in the file

/etc/pam.d/system-auth

i've added this line at the beginning of the auth section:

auth        required      pam_tally2.so  file=/var/log/tallylog deny=5 quiet unlock_time=3600

then this at the account section:

account     required      pam_tally2.so

the auth line above will lockout and account after 5 failed logins. user will then have to wait for an hour to have the account available.
account can be re-enabled immediately by root (or one with root privileges) by running this:

pam_tally2 -r -u account_name_to_unlock

[root@node02 ~]# tail -20 /var/log/secure

Dec 10 17:10:44 node02 sshd[10623]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=node01-priv  user=oracleDec 10 17:10:46 node02 sshd[10623]: Failed password for oracle from 192.168.100.10 port 51233 ssh2
Dec 10 17:10:52 node02 last message repeated 2 times
Dec 10 17:10:52 node02 sshd[10624]: Connection closed by 192.168.100.10
Dec 10 17:10:52 node02 sshd[10623]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=node01-priv  user=oracle
Dec 10 17:10:56 node02 sshd[10626]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=node01-priv  user=erik
Dec 10 17:10:58 node02 sshd[10626]: Failed password for erik from 192.168.100.10 port 51234 ssh2
Dec 10 17:11:07 node02 last message repeated 2 times
Dec 10 17:11:07 node02 sshd[10627]: Connection closed by 192.168.100.10
Dec 10 17:11:07 node02 sshd[10626]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=node01-priv  user=erik
Dec 10 17:11:11 node02 sshd[10629]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=node01-priv  user=oracle
Dec 10 17:11:13 node02 sshd[10629]: Failed password for oracle from 192.168.100.10 port 51235 ssh2
Dec 10 17:11:17 node02 sshd[10629]: Failed password for oracle from 192.168.100.10 port 51235 ssh2
Dec 10 17:11:21 node02 sshd[10629]: pam_tally2(sshd:auth): user oracle (1000) tally 6, deny 5
Dec 10 17:11:23 node02 sshd[10629]: Failed password for oracle from 192.168.100.10 port 51235 ssh2
Dec 10 17:11:23 node02 sshd[10630]: Connection closed by 192.168.100.10
Dec 10 17:11:23 node02 sshd[10629]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=node01-priv  user=oracle
Dec 10 17:11:26 node02 sshd[10631]: pam_tally2(sshd:auth): user oracle (1000) tally 7, deny 5
Dec 10 17:11:27 node02 sshd[10631]: Failed password for oracle from 192.168.100.10 port 51236 ssh2
Dec 10 17:11:28 node02 sshd[10632]: Connection closed by 192.168.100.10

check the counter

[root@node02 ~]# pam_tally2

Login           Failures Latest failure     From
oracle              7    12/10/09 17:11:26  node01-priv
erik                3    12/10/09 17:11:05  node01-priv

unlock oracle:

[root@node02 ~]# pam_tally2 -r -u oracle

Login           Failures Latest failure     From

oracle              7    12/10/09 17:11:26  node01-priv

re-check, oracle has been cleared and should be able to login:

[root@node02 ~]# pam_tally2

Login           Failures Latest failure     From
erik                3    12/10/09 17:11:05  node01-priv

Reference: /usr/share/doc/pam-0.99.6.2/txts/README.pam_tally2

WDTV Live and uShare Media Server

2009-12-03T14:40:00.021+08:00

i recently acquired a WDTV Live media player as i wanted to play bluray format movies

especially kiddie movies for my 2 yo son Athrun Karl. i got it from Challenger in SG and it comes with a free HDMI cable. i used it for a couple of days and then i brought to the Philippines for my wife and son to use.

i've been waiting for WDTV Live as i've wanted builtin network support compared to the old WDTV (although you can load a custom firmware and plug a USB Lan device - checkout http://b-rad.cc/ so i can access files from my old desktop cum server.

the WDTV Live is pretty neat and plays almost all media format. and it has support for DTS

(users of old WDTV still needs to convert to AC-3/Dolby Digital format to enjoy media with DTS).

more about DTS here

and youtube videos - wdtv live can access youtube. :D

anyway, for playing media over the network, i use uShare on a heavily battered CentOs 5 x86 machine. Before, i only use it with my xbox 360 to view images and play Divx media.

when i got the wdtv live, i just modified my ushare.conf file, restarted the service
and the share can then be accessed from the Main menu > Videos > Network servers and my ushare "share" named uMedia is accessible. from here, i can play DivX movies - with both my xbox 360 and wdtv live.

NOTE: i only tried playing bluray movies via USB HDD plugged directly to the
wdtv live player due to space constraints with my Centos Box (these bluray movies
do eat lots of disk space).

here's my ushare configuration (xbox 360 and wdtv live compatible):

OS: CentOS 5.2 x86
eth1 is my internal LAN interface

ushare package downloaded from DAG's repo http://dag.wieers.com/.

network layout - might go out of whack in your browser:

+------------+          +-----------------+
internet<- eth0 -> + centos_box +<- eth1 ->+ wlan/lan router +--> wdtv live
                   +------------+          +--------+--------+
                                                    |
                                                    +-------> xbox 360

$ rpm -qa | grep ushare ushare-1.1a-2.el5.rf
$ rpm -qi ushare

Name        : ushare                       Relocations: (not relocatable)
Version     : 1.1a                              Vendor: Dag Apt Repository, http://dag.wieers.com/apt/
Release     : 2.el5.rf                      Build Date: Tue 18 Dec 2007 02:10:55 AM SGT
Install Date: Sat 27 Sep 2008 10:12:10 AM SGT      Build Host: lisse.leuven.wieers.com
Group       : Applications/Multimedia       Source RPM: ushare-1.1a-2.el5.rf.src.rpm
Size        : 126519                           License: GPL
Signature   : DSA/SHA1, Tue 18 Dec 2007 10:42:08 AM SGT, Key ID a20e52146b8d79e6
Packager    : Dag Wieers
URL         : http://ushare.geexbox.org/
Summary     : Universal Plug'nPlay (uPNP) Media Server
Description :

uShare is a UPnP (TM) A/V Media Server. It implements the server component that provides UPnP media devices with information on available multimedia files. uShare uses the built-in http server of libupnp to stream the files to clients.

main configuration file /etc/ushare.conf:

#START of config file:

# /etc/ushare.conf
# Configuration file for uShare

# uShare UPnP Friendly Name (default is 'uShare').
USHARE_NAME=uMedia

# Interface to listen to (default is eth0).
# Ex : USHARE_IFACE=eth1
USHARE_IFACE=eth1

# Port to listen to (default is random from IANA Dynamic Ports range)
# Ex : USHARE_PORT=49200
USHARE_PORT=49200

# Port to listen for Telnet connections
# Ex : USHARE_TELNET_PORT=1337
USHARE_TELNET_PORT=

# Directories to be shared (space or CSV list).
# Ex: USHARE_DIR=/dir1,/dir2

USHARE_DIR=/home/media

# Use to override what happens when iconv fails to parse a file name.
# The default uShare behaviour is to not add the entry in the media list
# This option overrides that behaviour and adds the non-iconv'ed string into
# the media list, with the assumption that the renderer will be able to
# handle it. Devices like Noxon 2 have no problem with strings being passed
# as is. (Umlauts for all!)
#
# Options are TRUE/YES/1 for override and anything else for default behaviour
USHARE_OVERRIDE_ICONV_ERR=

# Enable Web interface (yes/no)
ENABLE_WEB=

# Enable Telnet control interface (yes/no)
ENABLE_TELNET=no

# Use XboX 360 compatibility mode (yes/no)
ENABLE_XBOX=yes

# Use DLNA profile (yes/no)
# This is needed for PlayStation3 to work (among other devices)
ENABLE_DLNA=yes

#END of config file

then run:
$ sudo /sbin/service ushare restart

to check

$ sudo netstat -patun | grep ushare
Password:

tcp        0      0 0.0.0.0:49200               0.0.0.0:*                   LISTEN      15777/ushare
tcp        0      0 0.0.0.0:1337                0.0.0.0:*                   LISTEN      15777/ushare
udp        0      0 127.0.0.1:55830             0.0.0.0:*                               15777/ushare
udp        0      0 0.0.0.0:1900                0.0.0.0:*                               15777/ushare

Things to watch out:
1. iptables ruleset
2. your cabling - if you need to direcly connect wdtv live to your PC (no router or switch)
use a crossover cable
3. SELinux policies

hth.

Building rsyslog 4.4.1 RPMs

2009-10-09T16:35:00.003+08:00

Today i finished packaging rsyslog-4.4.1. most of the steps involved (some not mentioned) in this post were from
http://telinit0.blogspot.com/2009/04/packaging-rsyslog-and-adding-it-to-yum.html

latest stable release as of this post:

rsyslog 4.4.1 (v4-stable)

tarball can be downloaded here:
http://www.rsyslog.com/Downloads-req-viewsdownload-sid-1.phtml

Build environment is a CEntOs 5.3 x86_64 system with gcc version 3.4.6 20060404.

i initially thought that the old rsyslog.spec file i used before will work. so i just edited some entries mainly just to change the rsyslog version and then did an rpmbuild -ba immediately.

i encountered this error (just a snippet as the error lines are quite long):

libtool: link: x86_64-redhat-linux-gcc -O2 -g -m64 -mtune=generic -W -Wall -Wformat-security -Wshadow -Wcast-align -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o rsyslogd-omfwd.o rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o rsyslogd-pidfile.o -Wl,--export-dynamic  -lz -lpthread ../runtime/.libs/librsyslog.a -ldl -lrt
../runtime/.libs/librsyslog.a(librsyslog_la-parser.o): In function `sanitizeMessage':
/apps/rpmbuild/BUILD/rsyslog-4.4.1/runtime/parser.c:213: undefined reference to `rpl_realloc'
../runtime/.libs/librsyslog.a(librsyslog_la-stringbuf.o): In function `rsCStrExtendBuf':
/apps/rpmbuild/BUILD/rsyslog-4.4.1/runtime/stringbuf.c:178: undefined reference to `rpl_realloc'
collect2: ld returned 1 exit status
make[2]: *** [rsyslogd] Error 1
make[2]: Leaving directory `/apps/rpmbuild/BUILD/rsyslog-4.4.1/tools'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/apps/rpmbuild/BUILD/rsyslog-4.4.1'
make: *** [all] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.39758 (%build)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.39758 (%build)

from the ever reliable google, i need to run the configure command like this:

ac_cv_func_malloc_0_nonnull=yes ./configure --enable-mysql --host=x86_64-redhat-linux

to get rid of the rpl_realloc errors. from the command line, the make command following the above custom configure command works.

but i need to have that done within the rpm spec file. and since i can't find anything on the web how to use cache variables from the rpm spec file, i did the following:

1. extract the tar ball
2. ran configure with the following:

# this is a one-liner:
ac_cv_func_malloc_0_nonnull=yes ./configure --enable-mysql --host=x86_64-redhat-linux --prefix=/usr --libdir=/usr/lib64 --sbindir=/sbin/

3. re-created tar ball after ./configure
4. remove %configure part of rpm spec file so rpmbuild will jump to the "make" portion

from the CLI configure process:

config.status: creating plugins/omoracle/Makefile
config.status: creating plugins/cust1/Makefile
config.status: creating tests/Makefile
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands
config.status: executing libtool commands
****************************************************
rsyslog will be compiled with the following settings:

    Multithreading support enabled:           yes
    Large file support enabled:               yes
    Networking support enabled:               yes
    Regular expressions support enabled:      yes
    Zlib compression support enabled:         yes
    rsyslog runtime will be built:            yes
    rsyslogd will be built:                   yes
    custom module 1 will be built:            no

---{ input plugins }---
    Klog functionality enabled:               yes (linux)
    imdiag enabled:                           no
    file input module enabled:                no
    input template module will be compiled:   no

---{ output plugins }---
    Mail support enabled:                     no
    omprog module will be compiled:           no
    omstdout module will be compiled:         no
    output template module will be compiled:  no

---{ database support }---
    MySql support enabled:                    yes
    libdbi support enabled:                   no
    PostgreSQL support enabled:               no
    Oracle (OCI) support enabled:             no

---{ protocol support }---
    GnuTLS network stream driver enabled:     no
    GSSAPI Kerberos 5 support enabled:        no
    RELP support enabled:                     no
    SNMP support enabled:                     no

---{ debugging support }---
    Testbench enabled:                        no
    Debug mode enabled:                       no
    Runtime Instrumentation enabled:          no
    Diagnostic tools enabled:                 no
    Valgrind support settings enabled:        no

looks OK. i then recreate the tarball:

cd /apps/rpmbuild/SOURCES
tar zcvf rsyslog-4.4.1.tar.gz rsyslog-4.4.1

and then re-ran rpmbuild -ba rsyslog.spec

but there's still some errors:

Checking for unpackaged file(s): /usr/lib/rpm/check-files /var/tmp/rsyslog-4.4.1-0-root-admin
error: Installed (but unpackaged) file(s) found:
   /usr/lib64/rsyslog/lmstrmsrv.so
   /usr/sbin/rsyslogd


RPM build errors:
    File not found: /var/tmp/rsyslog-4.4.1-0-root-admin/sbin/rsyslogd
    Installed (but unpackaged) file(s) found:
   /usr/lib64/rsyslog/lmstrmsrv.so
   /usr/sbin/rsyslogd

the FIX: modify the specfile.
the sbindir in the specfile must be the same as what is defined in Makefile.
/usr/lib64/rsyslog/lmstrmsrv.so is missing and thus added in specfile

after that, rpmbuild was successful:

Executing(%doc): /bin/sh -e /var/tmp/rpm-tmp.58716
+ umask 022
+ cd /apps/rpmbuild/BUILD
+ cd rsyslog-4.4.1
+ DOCDIR=/var/tmp/rsyslog-4.4.1-0-root-admin/usr/share/doc/rsyslog-mysql-4.4.1
+ export DOCDIR
+ rm -rf /var/tmp/rsyslog-4.4.1-0-root-admin/usr/share/doc/rsyslog-mysql-4.4.1
+ /bin/mkdir -p /var/tmp/rsyslog-4.4.1-0-root-admin/usr/share/doc/rsyslog-mysql-4.4.1
+ cp -pr plugins/ommysql/createDB.sql /var/tmp/rsyslog-4.4.1-0-root-admin/usr/share/doc/rsyslog-mysql-4.4.1
+ exit 0
Provides: ommysql.so()(64bit)
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Requires: libc.so.6()(64bit) libc.so.6(GLIBC_2.2.5)(64bit) libc.so.6(GLIBC_2.3.4)(64bit) libc.so.6(GLIBC_2.4)(64bit) libcrypt.so.1()(64bit) libcrypto.so.6()(64bit) libm.so.6()(64bit) libmysqlclient.so.15()(64bit) libmysqlclient.so.15(libmysqlclient_15)(64bit) libnsl.so.1()(64bit) libssl.so.6()(64bit) libz.so.1()(64bit) rsyslog = 4.4.1-0 rtld(GNU_HASH)
Checking for unpackaged file(s): /usr/lib/rpm/check-files /var/tmp/rsyslog-4.4.1-0-root-admin
Wrote: /apps/rpmbuild/SRPMS/rsyslog-4.4.1-0.src.rpm
Wrote: /apps/rpmbuild/RPMS/x86_64/rsyslog-4.4.1-0.x86_64.rpm
Wrote: /apps/rpmbuild/RPMS/x86_64/rsyslog-mysql-4.4.1-0.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.68227
+ umask 022
+ cd /apps/rpmbuild/BUILD
+ cd rsyslog-4.4.1
+ rm -rf /var/tmp/rsyslog-4.4.1-0-root-admin
+ exit 0

some changes: for rsyslog-sysconfig (this is the file copied into /etc/sysconfig/rsyslog: from SYSLOGD_OPTIONS="-c3" to SYSLOGD_OPTIONS="-c4" for the spec file (rsyslog.spec)

2d1
<
5c4
< Version: 3.20.5
---
> Version: 4.4.1
48d46
< %configure --sbindir=%{sbindir} --enable-mysql  # removed in the new spec file
112a111
> %{_libdir}/rsyslog/lmstrmsrv.so   # added
119a119,121
> * Tue Oct 07 2009 Roderick Tapang   4.4.1-0
> - use 4.4.1 stable release
>

INSTALLING THE RPMs

after adding to my custom yum repo:

sudo yum install rsyslog-mysql.x86_64

Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Setting up Install Process
Parsing package install arguments
Resolving Dependencies
--> Running transaction check
---> Package rsyslog-mysql.x86_64 0:4.4.1-0 set to be updated
--> Processing Dependency: rsyslog = 4.4.1-0 for package: rsyslog-mysql
--> Running transaction check
---> Package rsyslog.x86_64 0:4.4.1-0 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================
 Package               Arch           Version          Repository      Size
============================================================================
Installing:
 rsyslog-mysql         x86_64         4.4.1-0          custom          25 k
Installing for dependencies:
 rsyslog               x86_64         4.4.1-0          custom         1.3 M

Transaction Summary
============================================================================
Install      2 Package(s)
Update       0 Package(s)
Remove       0 Package(s)

Total size: 1.3 M
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : rsyslog                                           [1/2]
  Installing     : rsyslog-mysql                                     [2/2]

Installed: rsyslog-mysql.x86_64 0:4.4.1-0
Dependency Installed: rsyslog.x86_64 0:4.4.1-0
Complete!

i'll be using this for a new machine that will host our syslog server.
ciao!

yet another oracle shared memory tip

2009-10-08T10:40:00.030+08:00

there have been a few instances of shared memory related DB startup failures (whew!) in our environment the past few days (should be permanently fixed with upgrade to newer versions of oracle). some of the possible actions we've taken to
fix it are the following:

1. kill all the running processes related to that instance
2. shutdown all the DBs until the problematic DB is the only
one left (is a No No in most cases)
3. follow the procedure in a previous post
4. doing some ipcs comparisons which i''ll be posting today.

problem: we can't restart the DBTWND1 instance. shuttting it down has this error:

ERROR at line 1: ORA-01034: ORACLE not available SQL> 
ORA-01034: ORACLE not available 
ORA-27101: shared memory realm does not exist 
Linux-x86_64 Error: 2: No such file or directory SQL> 
Disconnected Database "DBTWND1" not shut down.

even a shutdown abort is useless:

[oracle@lx0090 ~]$ export ORACLE_SID=DBTWND1

[oracle@lx0090 ~]$ sqlplus /nolog

SQL*Plus: Release 9.2.0.8.0 - Production on Thu Oct 8 09:32:37 2009 Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved. 
SQL> conn / as sysdba; Connected to an idle instance. 
SQL> shutdown abort; ORACLE instance shut down. 
SQL> exit Disconnected 

[oracle@lx0090 ~]$ db_open DBTWND1
Database is already Running : DBTWND1

checking the pmon listing:

[oracle@lx0090 ~]$ ps -ef | grep pmon | grep -v grep

oracle    1499     1  0 Sep22 ?        00:00:01 ora_pmon_DBCHND1
oracle 1529 1 0 Sep22 ? 00:00:00 ora_pmon_DBSGPD2
oracle 1584 1 0 Sep22 ? 00:00:01 ora_pmon_DBMYLD2
oracle 1603 1 0 Sep22 ? 00:00:20 ora_pmon_DBASIT1
oracle 1625 1 0 Sep22 ? 00:00:01 ora_pmon_DBJPSD2
oracle 1652 1 0 Sep22 ? 00:00:01 ora_pmon_DBTWND1 - can't shutdown
oracle 1711 1 0 Sep22 ? 00:00:00 ora_pmon_DBAUSD2
oracle 1731 1 0 Sep22 ? 00:00:01 ora_pmon_DBSGPD1
oracle 1791 1 0 Sep22 ? 00:00:01 ora_pmon_DBTHAD3
oracle 1878 1 0 Sep22 ? 00:00:00 ora_pmon_DBBOAUD1
oracle 3842 1 0 Sep22 ? 00:00:01 ora_pmon_DBCHND3
oracle 6228 1 0 Sep20 ? 00:00:01 ora_pmon_DBJPND2
oracle 6320 1 0 Sep20 ? 00:00:00 ora_pmon_DBBORT1
oracle 16540 1 0 Sep20 ? 00:00:01 ora_pmon_DBJPND1
oracle 22482 1 0 Oct06 ? 00:00:00 ora_pmon_DBGLFD1
oracle 27242 1 0 Sep25 ? 00:00:01 ora_pmon_DBMYLD
oracle 27715 1 0 Sep30 ? 00:00:00 ora_pmon_DBJPSD1

process listing for the DB:

oracle 1652 1 0 Sep22 ? 00:00:01 ora_pmon_DBTWND1
oracle 1654 1 0 Sep22 ? 00:00:42 ora_dbw0_DBTWND1
oracle 1656 1 0 Sep22 ? 00:00:33 ora_lgwr_DBTWND1
oracle 1658 1 0 Sep22 ? 00:00:35 ora_ckpt_DBTWND1
oracle 1660 1 0 Sep22 ? 00:00:14 ora_smon_DBTWND1
oracle 1662 1 0 Sep22 ? 00:00:00 ora_reco_DBTWND1
oracle 1664 1 0 Sep22 ? 00:00:02 ora_cjq0_DBTWND1
oracle 1666 1 0 Sep22 ? 00:00:00 ora_arc0_DBTWND1
oracle 1668 1 0 Sep22 ? 00:00:00 ora_arc1_DBTWND1

since we can't shutdown all the DBs and steps 2 and 3 are a failure, here's what i did:

get the shmid listing with the process id and look for the entry close to the process ids for the DB in error:

[oracle@lx0090 ~]$ ipcs -pm

------ Shared Memory Creator/Last-op --------
shmid owner cpid lpid
0 oracle 6225 19957
32769 oracle 6318 7119
65538 oracle 16538 19646
98307 oracle 1497 19950
131076 oracle 1517 16772
622597 oracle 22480 19616
196614 oracle 1576 8796
229383 oracle 1601 19966
262152 oracle 1623 19894
294921 oracle 1644 19891 -> this is the closest
327690 oracle 1703 15403
360459 oracle 1729 19743
557068 oracle 27240 19927
425997 oracle 1783 19942
589838 oracle 27426 19861
491535 oracle 1867 9035
524304 oracle 3840 19713

checking the time stamps:

[oracle@lx0090 ~]$ ipcs -mt

------ Shared Memory Attach/Detach/Change Times --------
shmid owner attached detached changed
0 oracle Oct 8 09:22:46 Oct 8 09:22:46 Sep 20 10:57:10
32769 oracle Oct 7 21:40:11 Oct 7 21:41:32 Sep 20 10:59:19
65538 oracle Oct 8 09:21:41 Oct 8 09:22:45 Sep 20 15:31:17
98307 oracle Oct 8 09:22:52 Oct 8 09:22:52 Sep 22 08:42:28
131076 oracle Sep 24 10:27:35 Sep 24 10:33:20 Sep 22 08:42:38
622597 oracle Oct 8 09:20:57 Oct 8 09:22:23 Oct 6 09:59:46
196614 oracle Sep 30 16:47:22 Sep 30 17:39:40 Sep 22 08:43:01
229383 oracle Oct 8 09:23:21 Oct 8 09:23:21 Sep 22 08:43:12
262152 oracle Oct 8 09:19:06 Oct 8 09:20:07 Sep 22 08:43:19
294921 oracle Oct 8 09:19:04 Oct 8 09:20:05 Sep 22 08:43:30 -> a logical choice*
327690 oracle Sep 30 14:25:29 Sep 30 14:39:35 Sep 22 08:43:40
360459 oracle Oct 8 09:22:59 Oct 8 09:22:59 Sep 22 08:43:49
557068 oracle Oct 8 09:19:39 Oct 8 09:20:40 Sep 25 11:56:43
425997 oracle Oct 8 09:19:44 Oct 8 09:20:44 Sep 22 08:44:11
589838 oracle Oct 8 09:18:52 Oct 8 09:20:46 Sep 30 15:28:17
491535 oracle Oct 6 14:26:27 Oct 6 14:26:35 Sep 22 08:44:30
524304 oracle Oct 8 09:22:29 Oct 8 09:23:29 Sep 22 09:14:16

* - since the other entries from the pmon listing with the same date have pids too far from the cpid=1644

i then removed the shmid:

ipcrm -m 294921

and was able to restart the DB:

[oracle@lx0090 ~]$ db_open DBTWND1

SQL*Plus: Release 9.2.0.8.0 - Production on Thu Oct 8 09:48:03 2009
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.
SQL> Connected to an idle instance.
SQL> ORACLE instance started.

Total System Global Area 286260616 bytes

Fixed Size 740744 bytes
Variable Size 218103808 bytes
Database Buffers 67108864 bytes
Redo Buffers 307200 bytes
Database mounted.
Database opened.
SQL> Disconnected from Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.8.0 - Production

Database "DBTWND1" started.

there might be other ways to handle this problem by i'll stick with above procedure for now.

hth.

Oracle 9i R2 on CEntOS 5 / RHEL 5 x86_64

2009-09-30T11:21:00.010+08:00

i've read a lot of queries online about installing oracle 9i R2 on rhel 5.x or centos 5.x 64 bit systems. Some answers aren't clear and doesn't offer any solution. And most say that it won't work.
Technically, installing Oracle 9i R2 directly to a Centos 5.x or RHEL 5.x system will just result into failure (at least that's what happened to all my attempts).

BUT you can run it.

yes. you can't install but you can run it under RHEL5/CENTOS5.

here's what i did. i have a couple of systems running on RHEL4.8 x86_64 (1 an HP blade. the other an IBM). i've installed oracle 9i R2 on them (oracle 9.2.4 first, then the patches to make them 9.2.8.0). that's very easy and a lot of guides are posted on the net so i won't include it here.

my installation is in an fs called /apps (our env has SAN so the fs housing oracle is on some SAN disks with LVM2/ext3 on them).

after setting up Oracle 9iR2 on rhel4.8, there are a few ways to move the installation run on centos 5.x or rhel 5.x (here i'm using CEntOS 5.3 x86_64):

1. install centos 5.3 on another machine and move the LUNs (hosting Oracle) from the rhel4 installation.

NOTE: copying the Oracle installation will also work. i've tried it - making a tar file of the whole oracle installation (i've installed oracle in /apps/oracle - this is not standard so your path will vary), scp'ing over to the new machine and untarring it. if you do this, make sure you have enough disk space and that the oracle user accounts for both machines have the same attributes (ie uid and gid).

2. install over the RHEL4 system and use Centos 5.3 on the same system keeping the /apps filesystem intact (for KS installs, use the option --useexisting to preserve an FS).

though i've tried both, i settled for option1 since the existing rhel4 machine will be decommissioned.

Making it work in Centos 5.3

i've tried this on Centos 5.3 x86_64. it shold also work with RHEL 5.x installs.

for the oracle environment variables, i've copied all defined from the rhel4 system.

here it is:

#  export LD_ASSUME_KERNEL=2.4.19
# Oracle Environment
export ORACLE_BASE=/apps/oracle
export ORACLE_HOME=$ORACLE_BASE/9270
export ORACLE_TERM=xterm
export NLS_LANG=AMERICAN;
export ORA_NLS33=$ORACLE_HOME/ocommon/nls/admin/data
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH

# Set shell search paths
export PATH=$PATH:$ORACLE_HOME/bin:/apps/scripts

notice that the LD_ASSUME is commented out. that's the only difference between centos 5.3 x86_64 and an rhel4 x86_64 oracle environment declarations.
if that line is enabled and used by the oracle account, i get errors like the ff:

[oracle@UX0107 ~]$ clear
clear: error while loading shared libraries: libdl.so.2: wrong ELF class: ELFCLASS32
[oracle@UX107 ~]$ sqlplus
sqlplus: error while loading shared libraries: libdl.so.2: wrong ELF class: ELFCLASS32

and that's basically it:

here are some of the packages from my centos 5.3 x86_64 system.
these are part of my kickstart scripts. not all are needed if you are going to move or copy and oracle install as there's no linking involved there.

compat-gcc-34-3.4.6.4 (x86_64)
compat-gcc-34-c++-3.4.6.4 (x86_64)
compat-glibc-2.3.4.2.26 (i386)
compat-glibc-2.3.4.2.26 (x86_64)
compat-glibc-headers-2.3.4.2.26 (x86_64)
compat-libgcc-296-2.96.138 (i386)
gcc-4.1.2.44.el5 (x86_64)
gcc-c++-4.1.2.44.el5 (x86_64)
glibc-2.5.34 (i686)
glibc-2.5.34 (x86_64)
glibc-common-2.5.34 (x86_64)
glibc-devel-2.5.34 (i386)
glibc-devel-2.5.34 (x86_64)
glibc-headers-2.5.34 (x86_64)
libaio-0.3.106.3.2 (i386)
libaio-0.3.106.3.2 (x86_64)
libaio-devel-0.3.106.3.2 (i386)
libaio-devel-0.3.106.3.2 (x86_64)
libcap-1.10.26 (x86_64)
libgcc-4.1.2.44.el5 (i386)
libgcc-4.1.2.44.el5 (x86_64)

hth.

CAUTION: Oracle 9i R2 on RHEL5 or CENTOS 5.x is not a supported combination. so don't try running this in production and then call Oracle if you have problems. :)

UPDATE:
i've just tested this with RHEL5 update 4 x86_64 - works perfectly fine.

Clearing Shared Memory of crashed Oracle instances

2009-09-23T15:16:00.010+08:00

in some cases, maybe twice or thrice a month, our helpdesk encounters some problems when restarting an oracle 9iR2 Database - with an error like this:

SQL*Plus: Release 9.2.0.8.0 - Production on Wed Sep 23 00:21:59 2009
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

SQL> Connected to an idle instance.
SQL> ORA-27100: shared memory realm already exists
Linux-x86_64 Error: 17: File exists

what they are encountering here is that the database instance has not released its shared memory. since there are several instances running, they can't just restart the machine. and running

ipcsm -mt

or

ipcs -t

and comparing it with the time stamp of running instances to eliminate erring shmid will take some time for them to verify, here's what i've advised them to do.

Our Oracle DBs are hosted on CEntOS 4.x (for test/dev) and RHEL4 (in prod).

you need two sessions here for the oracle user . in session 1, open a local sql session:

export ORACLE_SID=env_name (ie export ORACLE_SID=ISIMYLD)
sqlplus /nolog

here's an example:

in session 1:

in session 2:

get the PID of the sqlsession, in this example its 29723. to do this, in session 2:

ps -ef | grep ISIMYLD | grep LOCAL

go back to session 1. and from the sql session, run this:

SQL> oradebug setospid 29723

(29723) is PID from session 2.

still in session 1, run this:

SQL> oradebug ipc

then, in the system command line, cd into the udump directory of the instance,
in this example its:

cd /apps/isis/oradb/ISIMYLD/oracle/udump

then look for the latest .trc file (use ls -ltr *.trc)

scroll down a bit and look for the line with Shmid (if you are using vi, use
/Shmid

view isimyld_ora_29723.trc

take note of the value under it. here it is 393228

close the file. and from the command line, remove that shmid:

ipcrm -m 393228

the DB was then able to start.

NOTE: always make sure you are in the correct instance before doing the above process. removing the wrong shmid will make the instances crash.

Most of the stuff on this post is from http://www.idevelopment.info/data/Oracle/DBA_tips/Unix/UNIX_2.shtml

Apache Active Directory Login for Nagios Access

2009-09-07T15:40:00.006+08:00

in a previous post, i've configured nagios 3.2.0 on a centos 5.3 x86_64 server. I now have set up M$ Active Directory based authentication for nagios users.

I initially asked for a new AD group with the nagios users login names as members but it turned out to be unusable. so what i did is to just browse for the
IT department level, filter out the login names and require the specific users that need to access the nagios site.

I did a trial and error for the required fields. The ldapsearch utility (part of openldap-clients) packages came in very handy.

one command i used to test the filtering is this:

ldapsearch -b 'OU=Departments,OU=Users,OU=domain,DC=asia,DC=org' -D 'CN=srv_nagios,OU=Service Accounts,OU=Operations,DC=asia,DC=org' -h ldapsearch -x -W sAMAccountName

(it will ask for the srv_nagios password)

srv_nagios is an unprivileged AD account used to bind the process to the Active Directory (since the AD doesn't allow anonymous browsing). without the account, one error that i had is:

auth_ldap authenticate: user user01 authentication failed; URI /nagios/ [ldap_search_ext_s() for user failed][Operations error]

also, from ldapsearch, as i still don't have the correct parameters to use, i encountered mostly this message:

# search result
search: 2
result: 1 Operations error
text: 00000000: LdapErr: DSID-0C090627, comment: In order to perform this ope
ration a successful bind must be completed on the connection., data 0, vece

in the nagios web config file:

/etc/httpd/conf.d/nagios.conf

the following is the one that worked for me (your requirement will
vary so try to work with your AD admin of the correct fields to use).

ScriptAlias /nagios/cgi-bin "/usr/lib64/nagios/cgi"
<Directory "/usr/lib64/nagios/cgi">
Options ExecCGI
AllowOverride None
Order allow,deny
Allow from all
AuthBasicProvider ldap
AuthzLDAPAuthoritative  off
AuthLDAPURL "ldap://ldap_server:389/OU=Departments,OU=Users,OU=domain,DC=asia,DC=org?sAMAccountName?sub?(objectClass=*)"
AuthLDAPBindDN "CN=srv_nagios,OU=Service Accounts,OU=Operations,DC=asia,DC=org"
AuthLDAPBindPassword "secretpassword"
AuthLDAPGroupAttribute  memberOf
AuthLDAPGroupAttributeIsDN off
AuthName "Nagios Access"
AuthType Basic
Require ldap-user user1 user2 user3
Require ldap-user user4 user5
</Directory>

Alias /nagios "/usr/share/nagios"

<Directory "/usr/share/nagios">
Options None
AllowOverride None
Order allow,deny
Allow from all
AuthBasicProvider ldap
AuthzLDAPAuthoritative  off
AuthLDAPURL "ldap://ldap_server:389/OU=Departments,OU=Users,OU=domain,DC=asia,DC=org?sAMAccountName?sub?(objectClass=*)"
AuthLDAPBindDN "CN=srv_nagios,OU=Service Accounts,OU=Operations,DC=asia,DC=org"
AuthLDAPBindPassword "secretpassword"
AuthLDAPGroupAttribute  memberOf
AuthLDAPGroupAttributeIsDN off
AuthName "Nagios Access"
AuthType Basic
Require ldap-user user1 user2 user3
Require ldap-user user4 user5
</Directory>

Watch for errors in /var/log/httpd/error_log - if there's a problem, log entries are a very big help.

NOTE: this works with 2.2.x. i was trying with a 2.0.x version but the mod_authnz_ldap module is not builtin to it.

Once it is working, i've re-defined the contacts (for each nagios user) and then assigned them to groups. these groups are then used in templates and used by service definitions.

Nagios Setup on CentOS 5.3

2009-08-31T17:12:00.023+08:00

this is a long one. :) i'm trying out the latest nagios release (v 3.2.0)
on centos 5 update 3 x86_64.
Since the nagios server and plugin tarballs come with their own spec
files, building the RPMs is easy.

i needed to change some directory names so i had to untar
the tarball and modify the file "nagios.spec" the "./configure" is in line
189. here, the target directories are defined and can be modified.

BUILDING THE RPMs

how to build the rpms are in a previous post:

http://telinit0.blogspot.com/2009/06/building-nagios-rpms.html

for nrpe:

if openssl-devel is not installed, this error appears:

checking for type of socket size... size_t
checking for SSL headers... configure: error: Cannot find ssl headers
error: Bad exit status from /var/tmp/rpm-tmp.55397 (%build)

RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.55397 (%build)

running

yum install openssl-devel.x86_64

will do the following:

Downloading Packages:
(1/6): keyutils-libs-devel-1.2-1.el5.x86_64.rpm | 27 kB 00:00
(2/6): libselinux-devel-1.33.4-5.1.el5.x86_64.rpm | 149 kB 00:00
(3/6): libsepol-devel-1.15.2-1.el5.x86_64.rpm | 194 kB 00:00
(4/6): e2fsprogs-devel-1.39-20.el5.x86_64.rpm | 636 kB 00:00
(5/6): openssl-devel-0.9.8e-7.el5.x86_64.rpm | 1.8 MB 00:00
(6/6): krb5-devel-1.6.1-31.el5.x86_64.rpm | 1.9 MB 00:00

after that, rpmbuild for nrpe went without a hitch. (the ssl feature is retained
just to be prepared in case SSL is implemented with our setup).

after building the rpms, i have this in the RPMS directory:

nagios-3.2.0-1.x86_64.rpm
nagios-devel-3.2.0-1.x86_64.rpm
nagios-plugins-1.4.13-1.x86_64.rpm
nagios-www-3.2.0-1.x86_64.rpm
nrpe-2.12-1.x86_64.rpm
nrpe-plugin-2.12-1.x86_64.rpm

you can just manually run the rpm -ivh command to install the packages.
in my case, i added them to a yum repo - easier if they are needed to be installed
on other servers.
for other rpm requirements - i got them from dag.wieers.com (yey Dag!)
like for this one:

rpm -ivh nagios-plugins-1.4.13-1.x86_64.rpm
error: Failed dependencies:
perl(Net::SNMP) is needed by nagios-plugins-1.4.13-1.x86_64

i've added the packages to my yum repo for easy installation on target hosts.

CONFIGURATION

The following are default configuration files that came from the package:

just for the fun of it, i ran a check on the configuration using

#tip: do this everytime you modify any config file for nagios
nagios -v nagios.cfg

initial run reports a missing directory:

/var/log/nagios/spool/checkresults

after creating it and assigning the permissions (must have the same selinux context and ownership as the rest of the directories there):

Checking for circular paths between hosts...
Checking for circular host and service dependencies...
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check
....

but it is still useless. :D

configuring nagios is a tricky one. which ever part you want to start (ie hosts
or templates or check commands), you can always use the verification
check to see if everything is ok. you do a problem elimination.

for this post, i'll be using several files as follows:

cfg_file=/etc/nagios/objects/commands.cfg
cfg_file=/etc/nagios/objects/contacts.cfg
cfg_file=/etc/nagios/objects/timeperiods.cfg
cfg_file=/etc/nagios/objects/templates.cfg
cfg_file=/etc/nagios/objects/contactgroups.cfg
cfg_file=/etc/nagios/objects/hosts.cfg
cfg_file=/etc/nagios/objects/hostgroups.cfg
cfg_file=/etc/nagios/objects/services.cfg

TIP: having object definitions grouped in their own descriptive configuration
files will make editing/troubleshooting easier.

these are defined in /etc/nagios/nagios.cfg. config files doesn't have any standard names - as long as you define it in nagios.cfg. entries after ";" are comments.

#contacts.cfg

define contact{
contact_name                    nagiosadmin             ; Short name of user
use                             generic-contact         ; Inherit default values from generic-contact template (defined above)
alias                           Nagios Admin            ; Full name of user
email                           your_emailadress@yourcompany.com
}

#end - contacts.cfg

#contactgroups.cfg

define contactgroup{
contactgroup_name       admins
alias                   Nagios Administrators
members                 nagiosadmin
}

#end - ontactgroups.cfg

#hosts.cfg

define host{
use                     linux-server
host_name               node01
alias                   node01
address                 192.168.122.10
}

define host{
use                     linux-server
host_name               node02
alias                   node02
address                 192.168.122.20
}

#end - hosts.cfg

#hostgroups.cfg

define hostgroup{
hostgroup_name  linux-cluster ; The name of the hostgroup
alias           Linux Cluster ; Long name of the group
members         node01,node02 ; Comma separated list of hosts that belong to this group
}

#end - hostgroups.cfg

#templates.cfg

define contact{
name                            generic-contact         ; The name of this contact template
service_notification_period     24x7                    ; service notifications can be sent anytime
host_notification_period        24x7                    ; host notifications can be sent anytime
service_notification_options    w,u,c,r,f,s             ; send notifications for all service states, flapping events, and scheduled downtime events
host_notification_options       d,u,r,f,s               ; send notifications for all host states, flapping events, and scheduled downtime events
service_notification_commands   notify-service-by-email ; send service notifications via email
host_notification_commands      notify-host-by-email    ; send host notifications via email
register                        0                       ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL CONTACT, JUST A TEMPLATE!
}
define host{
name                            generic-host    ; The name of this host template
notifications_enabled           1               ; Host notifications are enabled
event_handler_enabled           1               ; Host event handler is enabled
flap_detection_enabled          1               ; Flap detection is enabled
failure_prediction_enabled      1               ; Failure prediction is enabled
process_perf_data               1               ; Process performance data
retain_status_information       1               ; Retain status information across program restarts
retain_nonstatus_information    1               ; Retain non-status information across program restarts
notification_period             24x7            ; Send host notifications at any time
register                        0               ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL HOST, JUST A TEMPLATE!
}

# Linux host definition template - This is NOT a real host, just a template!
define host{
name                            linux-server    ; The name of this host template
use                             generic-host    ; This template inherits other values from the generic-host template
check_period                    24x7            ; By default, Linux hosts are checked round the clock
check_interval                  5               ; Actively check the host every 5 minutes
retry_interval                  1               ; Schedule host check retries at 1 minute intervals
max_check_attempts              10              ; Check each Linux host 10 times (max)
check_command                   check-host-alive ; Default command to check Linux hosts
notification_period             workhours       ; Linux admins hate to be woken up, so we only notify during the day
; Note that the notification_period variable is being overridden from
; the value that is inherited from the generic-host template!
notification_interval           120             ; Resend notifications every 2 hours
notification_options            d,u,r           ; Only send notifications for specific host states
contact_groups                  admins          ; Notifications get sent to the admins by default
register                        0               ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL HOST, JUST A TEMPLATE!
}

define service{
name                            generic-service         ; The 'name' of this service template
active_checks_enabled           1                       ; Active service checks are enabled
passive_checks_enabled          1                       ; Passive service checks are enabled/accepted
parallelize_check               1                       ; Active service checks should be parallelized (disabling this can lead to major performance problems)
obsess_over_service             1                       ; We should obsess over this service (if necessary)
check_freshness                 0                       ; Default is to NOT check service 'freshness'
notifications_enabled           1                       ; Service notifications are enabled
event_handler_enabled           1                       ; Service event handler is enabled
flap_detection_enabled          1                       ; Flap detection is enabled
failure_prediction_enabled      1                       ; Failure prediction is enabled
process_perf_data               1                       ; Process performance data
retain_status_information       1                       ; Retain status information across program restarts
retain_nonstatus_information    1                       ; Retain non-status information across program restarts
is_volatile                     0                       ; The service is not volatile
check_period                    24x7                    ; The service can be checked at any time of the day
max_check_attempts              3                       ; Re-check the service up to 3 times in order to determine its final (hard) state
normal_check_interval           10                      ; Check the service every 10 minutes under normal conditions
retry_check_interval            2                       ; Re-check the service every two minutes until a hard state can be determined
contact_groups                  admins                  ; Notifications get sent out to everyone in the 'admins' group
notification_options            w,u,c,r                 ; Send notifications about warning, unknown, critical, and recovery events
notification_interval           60                      ; Re-notify about service problems every hour
notification_period             24x7                    ; Notifications can be sent out at any time
register                        0                      ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL SERVICE, JUST A TEMPLATE!
}
# Local service definition template - This is NOT a real service, just a template!
define service{
name                            linux-service           ; The name of this service template
use                             generic-service         ; Inherit default values from the generic-service definition
max_check_attempts              4                       ; Re-check the service up to 4 times in order to determine its final (hard) state
normal_check_interval           5                       ; Check the service every 5 minutes under normal conditions
retry_check_interval            1                       ; Re-check the service every minute until a hard state can be determined
register                        0                       ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL SERVICE, JUST A TEMPLATE!
}

#end - templates.cfg

NOTE about templates: these basically shortens out how objects are defined. contents of a template used repeatedly for objects that has the same characteristics.

then, to verify (do this everytime you make changes to configuration files:

nagios -v /etc/nagios/nagios.cfg

ideal output (last few lines) is:

Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check

although it is ok if you have some warnings. just make sure there are no errors.

than start it:

/sbin/service nagios start

in my initial startup, i had an error (from /var/log/nagios/nagios.log):

[1251702022] Error: Could not create external command file '/var/log/nagios/rw/nagios.cmd'
as named pipe: (2) -> No such file or directory.  If this file already exists and you are
sure that another copy of Nagios is not running, you should delete this file.

i then created /var/log/nagios/rw/ and change owner/group to nagios/nagios.

second startup attempt is ok.

WEB Access

here's my apache config for the nagios wui:

/etc/httpd/conf.d/nagios.conf

ScriptAlias /nagios/cgi-bin "/usr/lib64/nagios/cgi"
<Directory "/usr/lib64/nagios/cgi">
Options ExecCGI
AllowOverride None
Order allow,deny
Allow from all
AuthName "Nagios Access"
AuthType Basic
AuthUserFile /etc/nagios/htpasswd.users
Require valid-user
</Directory>

Alias /nagios "/usr/share/nagios"

<Directory "/usr/share/nagios">
Options None
AllowOverride None
Order allow,deny
Allow from all
AuthName "Nagios Access"
AuthType Basic
AuthUserFile /etc/nagios/htpasswd.users
Require valid-user
</Directory>

Authentication SETUP

this is how i defined the nagios web user account:

htpasswd -c /etc/nagios/htpasswd.users sysadmin
New password:
Re-type new password:
Adding password for user sysadmin

when i tied to access my nagios web site, the left frame is out of whack. i turns out
i am missing the php package. :D

i did a yum install php:

=====================================================================================================
Package                 Arch                Version                    Repository              Size
=====================================================================================================
Installing:
php                     x86_64              5.1.6-23.el5               centos5u3              1.2 M
Installing for dependencies:
gmp                     x86_64              4.1.4-10.el5               centos5u3              201 k
php-cli                 x86_64              5.1.6-23.el5               centos5u3              2.2 M
php-common              x86_64              5.1.6-23.el5               centos5u3              154 k

and then restarted httpd:

service httpd start

and:

there's nothing important on that yet as systems to be monitored
are still unconfigured.

SETUP HOSTS TO BE MONITORED

this is NRPE based. i have two test hosts to be monitored by nagios:

node01 and node02.

since i've added the RPMs on a yum repo, for each host, i can just do:

yum install nrpe

Dependencies Resolved

=====================================================================================================
Package                     Arch              Version               Repository                 Size
=====================================================================================================
Installing:
nrpe                        x86_64            2.12-1                custom-RPMS                34 k
Installing for dependencies:
nagios-plugins              x86_64            1.4.13-1              custom-RPMS               837 k
perl-Crypt-DES              x86_64            2.05-3.rf             custom-RPMS                38 k
perl-Digest-HMAC            noarch            1.01-15               centos5-Server             12 k
perl-Digest-SHA1            x86_64            2.11-1.2.1            centos5-Server             49 k
perl-Net-SNMP               noarch            5.2.0-1.rf            custom-RPMS                95 k
perl-Socket6                x86_64            0.19-3.fc6            centos5-Server             21 k

Transaction Summary
=====================================================================================================
Install      7 Package(s)
Update       0 Package(s)
Remove       0 Package(s)

Total download size: 1.1 M

and:

after installing the packages, some changes are needed in the config file:

[root@node01 nagios]# pwd; ls -ltr
/etc/nagios
total 32
-rw-rw-r-- 1 root root 15920 Aug 27 10:13 command.cfg
-rw-r--r-- 1 root root 7215 Aug 31 15:56 nrpe.cfg
[root@node01 nagios]#

i only changed the following:

server_port=5666
server_address=192.168.122.10 # node01's ip
allowed_hosts=192.168.122.1 # ip of the nagios server

(i'm running NRPE in standalone mode. it can also be configured to run
via xinetd).

after saving the file:

/sbin/service nrpe start

make sure that if you have an iptables ruleset or tcpwrappers, you allow access from
your nagios server to hosts' port 5666.

do this on machines you need to monitor.

as for the check commands, there are some examples defined
in the configuration file, like the following:

command[check_users]=/usr/lib64/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib64/nagios/plugins/check_load -w 15,10,5 -c 30,25,20
command[check_hda1]=/usr/lib64/nagios/plugins/check_disk -w 20% -c 10% -p /dev/hda1
command[check_zombie_procs]=/usr/lib64/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib64/nagios/plugins/check_procs -w 150 -c 200

from the nagios server, to check, we can run the following:

/usr/lib64/nagios/plugins/check_nrpe  -H node01 -p 5666 -c check_hda1
DISK OK - free space: /boot 81 MB (87% inode=99%);| /boot=12MB;78;88;0;98

/usr/lib64/nagios/plugins/check_nrpe  -H node01 -p 5666 -c check_total_procs
PROCS OK: 121 processes

once you've configured your other hosts for nrpe, you can do similar checks.

as i've configured node02:

/usr/lib64/nagios/plugins/check_nrpe  -H node02 -p 5666 -c check_hda1
DISK OK - free space: /boot 81 MB (87% inode=99%);| /boot=12MB;78;88;0;98

/usr/lib64/nagios/plugins/check_nrpe  -H node02 -p 5666 -c check_total_procs
PROCS OK: 123 processes

/usr/lib64/nagios/plugins/check_nrpe  -H node02 -p 5666 -c check_users
USERS OK - 1 users currently logged in |users=1;5;10;0

and as an example, i created some custom check commands:

# custom check commands

command[check_ssh]= /usr/lib64/nagios/plugins/check_ssh  -p 22 localhost
command[check_http]=  /usr/lib64/nagios/plugins/check_http localhost
command[check_disk_root]=/usr/lib64/nagios/plugins/check_disk -w 25 -c 10 -p /

DEFINING SERVICES TO BE MONITORED IN NAGIOS

NOTE: for monitoring MS windows machines, checkout a previous post:

http://telinit0.blogspot.com/2009/06/nagios-and-nsclient-for-m-windoze.html

since i'm using check_nrpe, i needed to define a generic check_nrpe command that
i can use with different arguments. these arguments are the check_commands defined
for each host. i placed the generic check_nrpe command in

/etc/nagios/objects/commands.cfg

define command{
command_name    check_nrpe
command_line    /usr/lib64/nagios/plugins/check_nrpe -H $HOSTADDRESS$ -p 5666 -c $ARG1$
}

and then, under the file services.cfg:

#start services.cfg

define service{
use                             linux-service         ; Name of service template to use
hostgroup_name                  linux-cluster
service_description             PING
check_command                   check_ping!100.0,20%!500.0,60%
}

define service{
use                             linux-service         ; Name of service template to use
hostgroup_name                       linux-cluster
service_description             Total Processes
check_command                   check_nrpe!check_total_procs
}

define service{
use                             linux-service         ; Name of service template to use
hostgroup_name                       linux-cluster
service_description             SSH
check_command                   check_nrpe!check_ssh
notifications_enabled           0
}
# Define a service to check HTTP on the local machine.
# Disable notifications for this service by default, as not all users may have HTTP enabled.
define service{
use                             linux-service; Name of service template to use
hostgroup_name                  linux-cluster
service_description             HTTP
check_command                   check_nrpe!check_http
notifications_enabled           0
}

# end - services.cfg

so restarting nagios daemon. the interface now shows:

trying other pages, i get some authorization error:

i need to add the "syadmin" user in /etc/nagios/cgi.cfg. the following
entries should fix authorization issues for sysadmin:

grep sysadmin cgi.cfg

authorized_for_system_information=sysadmin
authorized_for_configuration_information=sysadmin
authorized_for_system_commands=sysadmin
authorized_for_all_services=sysadmin
authorized_for_all_hosts=sysadmin
authorized_for_all_service_commands=sysadmin
authorized_for_all_host_commands=sysadmin

then restart nagios. the following images show the test setup:

tactical overview
map
host
services
hostgroup details
service problems
service state details
as you can see, notification is still disabled for some services (spefically
for HTTP and SSH monitoring). the line:

notifications_enabled 0

is in the check_http definitions in services.cfg. removing it and
restart/reload nagios will enable notifications (provided that you have
configured your mail server to send out alerts.

here's two sample alerts generated

If you are doing some tests, make sure alerts are limited to your email address.

And that's it. customizations and additional hosts/services can then be added.
just remember to run the verification command for any configuration changes you make. this release of Nagios (3.2.0) is quite selinux friendly (compared to the version that is live in our environment - 2.5 :D). I might put in production to replace our ageing setup in coming weeks.
'till next time.

standards.h problem after upgrading to 5300-08-07-0920

2009-08-03T16:36:00.004+08:00

after applying SP 7 to our test/dev systems (bring AIX to 5300-08-07-0920),
our application support team encountered two problems:

1. the linker error described here
2. problem while doing another compile for an application patch that
involves some oracle libraries (see below):

proc include=/Apps/oracle/product/9.2.0.7/rdbms/demo include=/Apps/oracle/product/9.2.0.7/sqllib/public include=/Apps/isis/ISIVNMD2/COMMON/COMMON_C/src/c/incl iname=ora_cnx.pc

Pro*C/C++: Release 9.2.0.8.0 - Production on Mon Aug 3 15:15:21 2009

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

System default option values taken from: /Apps/oracle/product/9.2.0.7/precomp/admin/pcscfg.cfg

Syntax error at line 135, column 2, file /usr/include/standards.h:
Error at line 135, column 2 in file /usr/include/standards.h
#warning The -qdfp option is required to process DFP code in headers.
.1
PCC-S-02014, Encountered the symbol "warning" when expecting one of the followin
g:

a numeric constant, newline, define, elif, else, endif,
error, if, ifdef, ifndef, include, line, pragma, undef,
an immediate preprocessor command, a C token,
The symbol "newline," was substituted for "warning" to continue.

Syntax error at line 69, column 56, file /Apps/oracle/product/9.2.0.7/rdbms/demo/ociapr.h:
Error at line 69, column 56 in file /Apps/oracle/product/9.2.0.7/rdbms/demo/ocia
pr.h
sword odessp(struct cda_def *cursor, OraText *objnam, size_t onlen,
.......................................................1
PCC-S-02201, Encountered the symbol "size_t" when expecting one of the following
:

... auto, char, const, double, enum, float, int, long,
ulong_varchar, OCIBFileLocator OCIBlobLocator,
OCIClobLocator, OCIDateTime, OCIExtProcContext, OCIInterval,
OCIRowid, OCIDate, OCINumber, OCIRaw, OCIString, register,
short, signed, sql_context, sql_cursor, static, struct,
union, unsigned, utext, uvarchar, varchar, void, volatile,
a typedef name, exec oracle, exec oracle begin, exec,
exec sql, exec sql begin, exec sql type, exec sql var,
The symbol "enum," was substituted for "size_t" to continue.

### and more blah blah blah blah errors ####

previous cases for this in our environment was due to the updated

/usr/include/standards.h

owned by the fileset bos.adt.include:

lslpp -w /usr/include/standards.h

File Fileset Type
----------------------------------------------------------------------------
/usr/include/standards.h bos.adt.include File

lslpp -L bos.adt.include
Fileset Level State Type Description (Uninstaller)
----------------------------------------------------------------------------
bos.adt.include 5.3.8.5 C F Base Application Development
Include Files

copying an older version fixes the problem.

here's a diff output between the old version and the new one:

< #ifdef __STDC_DEC_FP__< #ifndef __STDC_WANT_DEC_FP__< #define __STDC_WANT_DEC_FP__129,130d111< #endif< #endif132,141d112< #if (defined(__IBMC__) || defined(__IBMCPP__))< #if ((defined(__STDC_WANT_DEC_FP__)) && !(defined(__IBM_DFP__)))< #if defined(__IBM_PP_WARNING)< #warning The -qdfp option is required to process DFP code in headers.< #else< #error The -qdfp option is required to process DFP code in headers.< #endif< #endif< #endif<190,203d160<< /*< * Determine what type should be used for the boolean type.< * In C++ the type bool should be used. If C99 is supported< * then use _Bool. Otherwise use unsigned char.< */< #ifdef __cplusplus< #define __bool__ bool< #elif (__STDC_VERSION__ >= 199901L)
< #define __bool__ _Bool
< #else
< #define __bool__ unsigned char
< #endif
<

hth.

some HMC screen caps

2009-08-03T15:48:00.008+08:00

some of you might not be familiar with the HMC web interface so here are some screen captures.
this is from hmc v 7.3.X.

after shutting down LPARs - take note of the reference codes. the managed system is still running (the physical machine is still ON).

with only the LPAR status listing - they are all Not Available

This is a console window of an LPAR after shutdown - notice it says "Halt Completed".

with the LPARs running (one is not activated - with reference code 0000000).

only the lpar window:

And the following is to startup an LPAR. the old HMC version (6.x) has a different layout (but still familiar menu entries).

usually, there are more than 1 profile for each LPAR - some used for installation. here, my Normal profile is the one usually used. Marking the option to open a terminal window will show the startup sequence.

and the startup sequence begins (this is from a VIO LPAr):

and then:

this will then be followed by a login prompt.

iptables and some dumb-o restrictions

2009-07-17T14:06:00.005+08:00

yesterday i got a request to block access from certain development boxes that appears to be able
to connect to a production database.

initially, our DBAs thought that they can restrict access by not defining the oracle SIDs in the tnsnames.ora file, but i've pointed them out before that:

sqlplus 'SCHEMA_MYL/schePAss0@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=lxAADB.prod)(PORT=1549)))(CONNECT_DATA=(SERVICE_NAME=DBMYLP)))'

will connect to the DB. i proposed to use iptables and allow only access to valid systems and i didn't get any response.

anyway, after quite some time, they asked to block connection to certain machines (of which
developers groups are working on). I've been asking to give the list of valid machines that will be allowed so deny can be implicitly defined. but i got only the list of which machines to block.

here's a few lines of iptables commands that i used:

# be very careful with iptables -F - if you have the default policy to be
# drop, running iptables -F will lock you out of your current ssh session!
# so make sure you have local login access to the machine before
# playing with iptables

iptables -F
iptables -A INPUT -s 127.0.0.1 -j ACCEPT
iptables -A OUTPUT -s 127.0.0.1 -j ACCEPT
iptables -A INPUT -p tcp -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -j ACCEPT
iptables -A INPUT -s 0/0 -p icmp -m icmp --icmp-type any -j ACCEPT
iptables -A OUTPUT -p tcp -s 0/0 --dport 22 -j ACCEPT
iptables -A INPUT -s a.b.c.d -j DROP
iptables -A INPUT -s a.b.x.y -j DROP

since i'm too lazy to create more rules, i've dropped everything from the requested machines (he he he its easier).
with the ruleset above, the production machine can connect to the development boxes but not
the other way around. Only ping is allowed from the dev boxes.
do remember this is kind of firewall setup is not the recommended way of doing port filtering.
it should have been deny everything and then accept what is allowed.

after checking that is ok, i saved the rule:

sudo /sbin/service iptables save

and made sure iptables will run during startup (we previously disabled it):

sudo /sbin/chkconfig iptables on

to list the running ruleset:

sudo /sbin/iptables -L

Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- localhost.localdomain anywhere
ACCEPT tcp -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT icmp -- anywhere anywhere icmp any
DROP all -- LX0xy0.dev.asia anywhere
DROP all -- LX0ab0.dev.asia anywhere
DROP all -- LX00ed.dev.asia anywhere
DROP all -- UX00mn.dev.asia anywhere
DROP all -- LX0e3b.dev.asia anywhere
DROP all -- LX0d20.dev.asia anywhere
DROP all -- LX0mnl.dev.asia anywhere
DROP all -- UX0uio.dev.asia anywhere

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- localhost.localdomain anywhere
ACCEPT all -- anywhere anywhere
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh

IBM HS21 Blade internal disk error

2009-07-17T13:43:00.007+08:00

a few days ago, one of our Blade HS21 servers

Jul 9 13:18:31 lx0080 kernel: mptscsi: ioc0: attempting target reset! (sc=000001021a2dc080)
Jul 9 13:18:31 lx0080 kernel: scsi0 : destination target 0, lun 0
Jul 9 13:18:31 lx0080 kernel: command = Write (10) 00 00 95 73 61 00 00 02 00
Jul 9 13:18:31 lx0080 kernel: mptbase: ioc0: RAID STATUS CHANGE for PhysDisk 0
Jul 9 13:18:53 lx0080 kernel: mptbase: ioc0: PhysDisk is now failed
Jul 9 13:18:53 lx0080 kernel: mptbase: ioc0: RAID STATUS CHANGE for PhysDisk 0
Jul 9 13:18:53 lx0080 kernel: mptbase: ioc0: PhysDisk is now failed, out of sync
Jul 9 13:18:53 lx0080 kernel: mptbase: ioc0: RAID STATUS CHANGE for VolumeID 0
Jul 9 13:18:53 lx0080 kernel: mptbase: ioc0: volume is now degraded, enabled
Jul 9 13:18:53 lx0080 kernel: mptscsi: ioc0: target reset: SUCCESS (sc=000001021a2dc080)
Jul 9 13:18:53 lx0080 kernel: mptscsi: ioc0: attempting target reset! (sc=000001021a2dc080)
Jul 9 13:18:53 lx0080 kernel: scsi0 : destination target 0, lun 0
Jul 9 13:18:53 lx0080 kernel: command = Test Unit Ready 00 00 00 00 00
Jul 9 13:18:53 lx0080 kernel: mptscsi: ioc0: target reset: SUCCESS (sc=000001021a2dc080)
Jul 9 13:18:53 lx0080 kernel: mptscsi: ioc0: attempting bus reset! (sc=000001021a2dc080)
Jul 9 13:18:53 lx0080 kernel: scsi0 : destination target 0, lun 0
Jul 9 13:18:53 lx0080 kernel: command = Write (10) 00 00 95 73 61 00 00 02 00
Jul 9 13:18:53 lx0080 kernel: mptscsi: ioc0: bus reset: SUCCESS (sc=000001021a2dc080)

from the blade center management module:

since the internal disk drives of the blade are mirrored (via internal raid controller - in RAID1)
and this is just hosting test/dev applications, the replacement was done after a few days (he heh).

so after shutting down the machine, pulling out the module, the faulty one (which is disk #2) was removed and replaced with a good one (the replacement disk arrived after a few hours).
after booting in, disk resycn was started automatically and we left it running in the background (yes, rhel4 booted without a hitch - just a little slow).

after about 30+ minutes:

Jul 15 10:14:38 lx0080 kernel: mptbase: ioc0: RAID STATUS CHANGE for PhysDisk 1
Jul 15 10:14:38 lx0080 kernel: mptbase: ioc0: PhysDisk is now online
Jul 15 10:14:38 lx0080 kernel: mptbase: ioc0: RAID STATUS CHANGE for VolumeID 0
Jul 15 10:14:38 lx0080 kernel: mptbase: ioc0: volume is now optimal, enabled, resync in progress
Jul 15 10:14:38 lx0080 kernel: mptbase: ioc0: RAID STATUS CHANGE for VolumeID 0
Jul 15 10:14:38 lx0080 kernel: mptbase: ioc0: volume is now optimal, enabled

the blade group is about 2.5 years old. so i think this will be the start of a lot of h/w problems.
disk replacement in our HP blades are easier to handle - just pull out the failed drive and plug in a new one. then go home. :D

Command line DLPAR for HMC, VIO and AIX

2009-07-08T16:10:00.008+08:00

since accessing the HMC remotely takes a bit of a while (well, its java based so
its a bit slow) I usually do DLPAR tasks via command line from an SSH session.
Today's post illustrates how I do it.

NOTE: In our environment, I have allowed SSH access to the HMC and have the
services/communcations established between HMC and the LPARs to initiate
dynamic lpar configuration.
You can visit the DLPAR checklist site if you need more info.

First, confirm which resource to move and where is the resource currently attached.
Once you know and you have located it, the DLPAR process will be very easy.
If you haven't done any IO profiling for your machine, better start doing it now.
You'll be wasting a lot of time if you only do the IO mapping just when you need it.
Its a good idea to create a spreadsheet for your machine where you have the IO
definitions. its a tedious process but is worth the effor when you need to move
resources around frequently.

Today, i'm going to move a tape drive to another AIX LPAR from a VIO server.
I'm going to move device rmt0 from VIO1 to ux0004

since i know the physical location code of the adapter i want to move,
i login to HMC as hscroot and check which LPAR is holding it using the command:

lshwres -r io --rsubtype slot -m Server-9113-550-SN65XXXXF | grep C1

(from my spreadsheet, C1 is the location of the pysical adapter i'm looking)

and:
it is in VIO1. from VIO1:

lsdev -type tape
then trace the parent of the device:

to check if there are anyother devices attached to the
slot, we do the reverse:
here, cd0 and rmt0 are connected on the same slot (pci6)

so removing pci6 WILL remove cd0 and rmt0 from the system.

to check it agains the drc_name from the HMC:

lsdev -slots
since cd0 is not in use, i then remove pci6 recursively
using the command:

rmdev -dev pci6 -recursive

from the HMC, i will then move the resource to ux0004. what i need to use
are the following:
- name of the lpar where resource is currently attached
- name of the target lpar where resource is to ve moved
- the drc_index value of the resource to move

the lpar names can be obtained using the following command:

lssyscfg -r lpar -m Server-9113-550-SNXXXXXXXX -F name

in my machine, i have the following:

now, to move, i used this command:

chhwres -r io -m Server-9113-550-SN6XXXXXF -o m -p ux02_VIO1 -t ux0004_DB -l 21030003

from the chhwres man page:

-r The type of hardware resources to change. Valid values are io for physical I/O,
virtualio for virtual I/O, mem for memory, proc for processing resources,
procpool for shared processor pool, and hea for Host Ethernet Adapter (HEA)
resources.

-m The name of the managed system for which the hardware resource configuration is
to be changed.

-o The operation to perform. Valid values are a to add hardware resources to a
partition, r to remove hardware resources from a partition, m to move
hardware resources from one partition to another, and s to set hardware
resource related attributes for a partition, a shared processor pool, an HEA, or
the managed-system.

-p The name of the partition for which the operation is to be performed. For a
move operation, this is the source partition (the partition the resources
will be moved from) for the operation. To perform an add, remove, or move
operation, the partition must be in the running state.

-t The name of the target partition for a move operation. The partition must be in
the running state.

-l When adding, removing, or moving a physical I/O slot, use this option to specify
the DRC index of the slot.

now to configure the resource to ux0004 (yes, the system doesn't know yet
that it has the tape drive). running the command:

sudo cfgmgr

will do the h/w detection. to check if the tape (and cdrom) drive was added:

lsdev -Cc tape; lsdev -Cc cdrom

and

lscfg | grep rmt0

take note that ux0004 is an AIX LPAR runnig AIX 5.3 TL8 SP5
and VIO1 is a VIO LPAR with ioslevel 1.5.2.1-FP-11.1. the dev
commands for AIX and ios are different. for instance, in
the process above, to trace the parent slot of rmt0, what can be
used is

odmget -q name=rmt0 CuDv | grep parent

and to trace from slot name to the child:

lsdev -p pci4

and so on. but since we know the slot location, we can just use
the command:

lsdev -c slot
and from there, remove pci4 recursively from ux0004 (again, difference
in command):

rmdev -dRl pci4

the AIX rmdev command:

-d Removes the device definition from the Customized Devices object class. This flag cannot
be used with the -S flag.

-l Name
Specifies the logical device, indicated by the Name parameter, in the Customized Devices
object class.

-R
Unconfigures the device and its children. When used with the -d or -S flags, the
children are undefined or stopped, respectively.

and then use the chhwres to move the resource to another LPAR If you need to.

playing with KVM and physical HDDs

2009-06-26T15:39:00.005+08:00

I tried installing KVM (Kernel-based Virtual Machine) just to play around with my notebook
hard drive and old DOS games (yeah, i know i can do it with http://www.dosbox.com/).

since i can't boot from my cdrom drive and dd'ing dos images directly to the hdd won't work (or i
couldn't make it work :)), i then resorted to using virtualization.

I initially want to use VirtualBox but it appears it can't do installation on a physical drive directly.

so i then tried kvm. here's what i did on my notebook with kubuntu 9.04 running on it.

first, install kvm:

sudo apt-get install kvm

# boot the floppy image to install "DOS" on the HDD
sudo kvm -hda /dev/sda -fda /tmp/win98c/WIN98C.IMG -boot a -m 256

(NOTE: WIN98C.IMG is from www.bootdisk.com (i downloaded the win98 zip file))

this is the virtual machine's startup sequence:

the virtual machine with an A: prompt:

run fdisk (WARNING: be sure you don't have important data in the disks you are playing with):

after creating a DOS partition, i then formatted my new drive:

format c: /s

/s is to make "c:" bootable (copying the sytem files from floopy to c:

and copy other commands from a: to c: - doing "cp" and wondering why command is not found - :D

now, after shutting down and removing by bootable usb drive, i can boot
on my notebooks hard drive and have DOS running.

Now boot the hard drive
sudo kvm -hda /dev/sda -boot c -m 256

how/where else can you use it? maybe if you want to install M$ Windoze, like XP Pro
on your HDD, but you can't boot from a CDROM/or you don't have a CDROM but you have
the installation files copied on disk, you can start the installation from DOS:

http://support.microsoft.com/kb/307848

ciao!

AIX 5L NIM mksysb restore

2009-06-24T15:13:00.017+08:00

In replicating AIX systems, I use mksysb based restorations so as to have the exact filesets especially if this is for a test/dev environment requirement. I use NIM based restores. And below is how I usually do it.

what i have

NIM server: ux0027
NIM Client: ux0026 (target for mksysb restoration)

resources i used:
name type
MyMksysbFile mksysb
SP5TL8 spot
ux26bosinst bosinst_data # for non-interactive restore
ux26image image_data
postInstall script # optional - only for customization
ux26resolv resolv_conf

i'll be using smitty since it's easier to use (and pretty much self explanatory
and easier to follow in making documentations).

so what i'll be doing is to restore a previously created image of another
AIX LPAR and restore it to another LPAR hosting ux0026.

the source mksysb image was created using this command:

mksysb -e -i MyMksysb.img

-e
Excludes files listed in the /etc/exclude.rootvg
-i
Calls the mkszfile command, which generates the /image.data file.

RTFM for mksysb for more details.

SP5TL8 or the spot resource used here, is created from the AIX installation
media (@TL5) that came with one of our p550 machines and then applied with
TL8 SP5 (hence the name).

prepare the NIM server definitions

i've already defined ux0026 before, so i'll be just adding the resources needed for the restoration.

1. define the mksysb resource - for this setup, i'm only defining the mksysb resource since the rest have already been defined (being used to often).

sudo smitty nim_mkres

then scroll down and select mksysb. for the values:

here, MyMksysbFile is just the name to call the resource. Server of Resource, is where it is located, in this case, it is master (my NIM server name). Location of Resource is then
local to the NIM server, /export/mksysb/MyMksysb.img. this img file
was created from another AIX LPAR. then hit ENTER.
if there are no conflicts, the definition should be OK.

2. allocating resources and starting the mksysb restore

sudo smitty nim_bosinst

i select the target, which is ux0026, then just hit ENTER

the type of instllation, select mksysb, then hit ENTER

select the mksysb resource

the SPOT resource, then hit ENTER

and then the rest of the resources I used:

the following are previously defined resources:

BOSINST_DATA - ux26bosinst (points to /export/mksysb/ux0026/ux26bosinst.data)
IMAGE_DATA - ux26image (points to /export/mksysb/ux0026/ux26image.data)
RESOLV_CONF - resolv.conf (points to /export/custom/resolv.conf)
Customization SCRIPT - postInstall (points to /export/custom/post_install.ksh)

BOSINST_DATA and IMAGE_DATA are generated from the mksysb command used to
produce the source mksysb image. these files will be in the / of the
system where image is created.

BOSINST_DATA is needed for the *almost* unattended installation. i said almost
as what i'm doing still needs manual boot device selection.

IMAGE_DATA - has the mapping for the partitions

RESOLV_CONF - my /etc/resolv.conf

Customization SCRIPT - script that will be executed after restore. it defines
the automount setup (automap definition), login banners, sudo entries
and configures the backup client software.

mksysb restores are actually replication of another AIX system. but for this case
since the target machine is for a different requirement, the custom script
is needed.

these resources have been allocated to ux0026 as shown above.

after defining the resources, select ENTER and the target LPAR (here it is ux0026, will reboot
by itself). in this example, the LPAR will go through the SMS menu. so i'm opening a
console window for the LPAR from the HMC.

status of the LPAR as shown by the HMC:

settings for the network adapter used for the nim install.

SERVER IP ADDRESS is the ipaddress of ux0027 - the nIM server.

since this is configured, i select X and then 1 to exit from SMS
and bootup the LPAR.

and the installation will then start:

installation in process

from the HMC, status of the LPAR (ux0026)

and the rest:

after this, LPAR will restart and go to the SMS prompt again. this time, i've specified the boot device to be the hard drive. after exiting from SMS:

ux0026 booting up:

and then logging onto the system:

checking the FS sizes and oslevel:

and that's it... ux0026 is now available. the whole process (excluding resource creation) took around 35 minutes.

some commands to remember (must have root privileges):

nim fast paths:

# manage machines
sudo smitty nim_mac

# define a machine for NIM operations
sudo smitty nim_mkmac

# perform operation on a machine
sudo smitty nim_mac_op

# create a NIM resource
sudo smitty nim_mkres

# manage NIM resources
sudo smitty nim_res

# actually, all you need to remember is this one
smitty nim

if you need more info, specifically in doing things via CLI, you can check out this guide from IBM.

building Nagios RPMs

2009-06-05T14:11:00.004+08:00

we've been running nagios v2.5 for quite some time now. and the latest stable release is 3.0.6.
i downloaded the tarballs from this site:

http://www.nagios.org/download/download.php

i got the following files:

nagios-3.0.6.tar.gz
nagios-plugins-1.4.13.tar.gz
nrpe-2.12.tar.gz

since the tarballs have the rpm spec files built-in, you can easily run rpmbuild on the tarballs to
build the binary rpms and source rpms.

from the rpmbuild man page:

The argument used is -b if a spec file is being used to build the package and -t if
rpmbuild should look inside of a (possibly compressed) tar file for the spec file
to use. After the first argument, the next character (STAGE) specifies the stages
of building and packaging to be done and is one of:

-ba Build binary and source packages (after doing the %prep, %build, and
%install stages).

so, to build the binary rpm files for nagios, i ran the following (tarballs are in my cwd):

rpmbuild -ta nagios-3.0.6.tar.gz

produced the following:

Wrote: /apps/rpmbuild/SRPMS/nagios-3.0.6-1.src.rpm
Wrote: /apps/rpmbuild/RPMS/x86_64/nagios-3.0.6-1.x86_64.rpm
Wrote: /apps/rpmbuild/RPMS/x86_64/nagios-www-3.0.6-1.x86_64.rpm
Wrote: /apps/rpmbuild/RPMS/x86_64/nagios-devel-3.0.6-1.x86_64.rpm
Wrote: /apps/rpmbuild/RPMS/x86_64/nagios-debuginfo-3.0.6-1.x86_64.rpm

rpmbuild -ta nrpe-2.12.tar.gz

produced the following:

Wrote: /apps/rpmbuild/SRPMS/nrpe-2.12-1.src.rpm
Wrote: /apps/rpmbuild/RPMS/x86_64/nrpe-2.12-1.x86_64.rpm
Wrote: /apps/rpmbuild/RPMS/x86_64/nrpe-plugin-2.12-1.x86_64.rpm
Wrote: /apps/rpmbuild/RPMS/x86_64/nrpe-debuginfo-2.12-1.x86_64.rpm

rpmbuild -ta nagios-plugins-1.4.13.tar.gz

produced the following:

Wrote: /apps/rpmbuild/SRPMS/nagios-plugins-1.4.13-1.src.rpm
Wrote: /apps/rpmbuild/RPMS/x86_64/nagios-plugins-1.4.13-1.x86_64.rpm
Wrote: /apps/rpmbuild/RPMS/x86_64/nagios-plugins-debuginfo-1.4.13-1.x86_64.rpm

your SRPMS and RPMS location will be different. mine is from the rpmmacros file definition. if you are not using rpmmacros, the rpm files by default will go to /usr/src/redhat/.

that's basically it.

and if you want to modify the default options of building the binaries (the rpmbuild commands above are doing a compile with default configuration directives), you can untar/unzip the tarball and modify the spec files, and look for the "./configure" line. you can then change the options, save the file and recreate the tarball. i did this to change the lib directory to lib64 from lib as my current nagios installation uses /usr/lib64.

Nagios and NSCLient++ for M$ Windoze monitoring

2009-06-01T11:00:00.016+08:00

'been a while since my last post as i was pretty much occupied at work. anyway, i'm posting the steps i did to use NSClient++ to monitor MS Windows systems from Nagios. We've been using Nagios for monitoring our servers for quite sometime now - we're still stuck with Version 2.5. :)

we also have a few Business Objects systems on Windows and our BO Guys asked to monitor BO related processes and inform them in case any of them goes down.

download the nsclient package (choose from the list):

http://nsclient.org/nscp/downloads

i used the zip file and extracted it in the BO machine on this location:

C:\Program Files\NSClient++-Win32-0.3.5

Configuration
edit the file NSC.ini, i only set the following:

#start of NSC.ini
[modules]
FileLogger.dll
CheckSystem.dll
CheckDisk.dll
NSClientListener.dll
NRPEListener.dll
CheckEventLog.dll
[Settings]
allowed_hosts=my_nagios_server_hostname
use_file=1
[log]
[NRPE]
port=5666
allow_arguments=1
allowed_hosts=my_nagios_server_hostname
#end of nsc.ini

now install it (here, i'm inside C:\Program Files\NSClient++-Win32-0.3.5)

NSClient++ /install

by default, you can control the nsclient using the following commands:

NSClient++ /start
NSClient++ /stop

but the elegant way to do it is to define it as a service and have it run
during startup:

sc config NSClientpp obj= "NT AUTHORITY\LocalService"
if not exist nsclient.log type NUL: > nsclient.log
cacls nsclient.log /e /g "NT AUTHORITY\LocalService":c

after that, you can now control it using the ff:

net stop NSClientpp
net start NSClientpp

Nagios Server side
in the nagios server, i created several config files for Windows machines, namely:

wincheckcommands.cfg
winhostgroups.cfg
winhosts.cfg
winservices.cfg

and all defined in the /etc/nagios.cfg config file. i won't go into the contents of those files - they are just the same config files in nagios created only for our windows machines. but for the wincheckcommands.cfg, i defined the BO process monitoring like this:

# 'check_BOprocs' command definition
define command{
command_name check_BOProcs
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -p 5666 -c checkServiceState -a CheckAll exclude=SysmonLog exclude=BOBJCrystalReportApplicationServer
}

so that is basically a generic entry, saying monitor all except the following:

SysmonLog
BOBJCrystalReportApplicationServer

where did thse come from? in windows, there's a command call sc - and to list the services, you run it from the command prompt:

sc query | more

and it will list the services and their current status.

you need the entries for SERVICE_NAME: if you want to be specific. for instance, the NSClient service i created earlier, i can query like this:

then, in winservices.cfg, i have this:

define service{
hostgroup_name BO_Servers
service_description BO Services
check_command check_BOProcs
use template-bo-services
}

so what it does is monitor the BO_Servers group using the check_BOProcs check command. In the web interface of nagios, for one of the BO servers:

you can also monitor other aspects of the servers using different command modules, like CheckDriveSize:

/usr/lib64/nagios/plugins/check_nrpe -H server_hostname -p 5666 -c CheckDriveSize -a ShowAll MinWarnFree=10% MinCritFree=5% Drive=C:
OK: C:: 36.4G|'C:'=47%;10;5;

please refer to the nsclient site for other CheckCommands.

TIP: when making changes to a running a nagios setup, always make backup copies of the config files. and after making changes, always run:

nagios -v nagios.cfg

to check for errors before doing a reload or restart of the nagios service.
nagios is a very nice template-based configuration but very confusing at times.

F981 3694 PRINTER ERROR

2009-05-07T14:50:00.008+08:00

one of our branches have been having some problems with an HP LJ P4515 printer for the past
few days. From the printer web interface Event Log, there are entries with F981 3694 PRINTER ERROR. everytime this error appears, jobs destined for the printer are queued in an AIX system hosting the queue:

From a quick google search with F981 3694 PRINTER ERROR, brought me to an HP Support forum. The error is due a problem with the firmware v04.047.2 and our user's printer has the same version installed:

the current work around according to the thread is to backdate the firmware to v04.042.0 while a new firmware update is being developed.
Right now, we are doing a spooler restart. I gave a (crude :)) script for our Helpdesk to run. It has no error checking (feel free to add to it) but does the job while waiting for a permanent fix.

here's the script:

# start
#!/usr/bin/ksh

# restart_spooler.ksh
# check if there are queued jobs
qJobS=`enq -W -A | grep QUEUED | wc -l`
if [ ${qJobS} != 0 ]
then
qNamE="";
echo "there are queued jobs. stopping active ones..."
for qNamE in $(ps -ef | grep qdaemon | grep bktprint | awk '{print $10}')
do
echo "disabling ${qNamE}..."
sudo /usr/bin/enq -K -P${qNamE}
done

# stop the spooler
nohup sudo stopsrc -s qdaemon &

# this will check if the qdaemon actually stopped
# before doing a startsrc
qCount=`ps -ef | grep qdaemon | grep -v grep | wc -l`

while [ ${qCount} != 0 ]
do
echo "qdaemon is still running - waiting..."
sleep 1
qCount=`ps -ef | grep qdaemon | grep -v grep | wc -l`
done

# start the spooler
sudo startsrc -s qdaemon

#enable queues with status DOWN
for qNamE in $(enq -W -s -A| grep DOWN | awk '{print $1}')
do
echo "enabling ${qNamE}...."
sudo /usr/bin/enq -U -P${qNamE}
done
fi
sudo /usr/bin/enq -W -A
exit
# end

the script disables the active queue first as qdaemon won't stop while a queue has a running job. queus with status DOWN will then be re-enabled after qdaemon restarts. The account used by our Helpdesk here has sudo privileges (with NOPASSWD option) for enq, stopsrc and startsrc.

NOTE: disabling/re-enabling a queue will "re-submit" the job to the printer. so if a print job is already halfway through and you disable/re-enable the queue, the job will be re-submitted and re-printed. i don't know of any fix for this right now so let me know if you have any. :)

c ya.

controlling crontab users

2009-04-21T11:56:00.005+08:00

we needed to limit the use of cron jobs but we don't want them defined at
the user level. it has to be done solely by root but still runs under
the user's environment settings.

to limit usage of crontab, the following files in /etc/
can be used:

cron.allow
cron.deny

from the man page:

If the cron.allow file exists, then you must be listed therein in
order to be allowed to use this command. If the cron.allow file
does not exist but the cron.deny file does exist, then you must
not be listed in the cron.deny file in order to use this command. If
neither of these files exists, only the super user will be allowed to
use this command.

all is needed is to remove /etc/cron.deny and /etc/cron.allow (in
RHEL4/5 systems, an empty /etc/cron.deny exists). By default, any user
can use crontab to define schduled jobs.

so removing /etc/cron.deny and/or /etc/cron.allow:

[user00@lx0050 ~]$ crontab -l
You (user00) are not allowed to use this program (crontab)
See crontab(1) for more information
[user00@lx0050 ~]$ crontab -e
You (user00) are not allowed to use this program (crontab)
See crontab(1) for more information

But we need to schedule user defined jobs in our system!

define the job in /etc/cron.d instead. jobs here will
be defined in a file. the format for each job entry
is as folows:

MM HH DD MM DayOfWeek user_to_run_job command

it is similar to a crontab entry except that before the command, the user
who needs to run the command must be provided.

sample entry:

*/2 * * * * user00 echo "test" >> $HOME/x.log

*/2 * * * * - run every two minutes all the time
user00 - the user who needs to run the command
echo "test" >> $HOME/x.log - the scheduled command

Note that in this definition, the command will run as if user00 defined
the job using crontab.

this line is in the file /etc/cron.d/admin_scripts. permissions
for the files in /etc/cron.d are the following (take note of the
selinux settings):

for RHEL4
-rw-r--r-- 1 system_u:object_r:etc_t root root 192 May 6 2008 admin_scripts

for RHEL5
-rw-r--r-- 1 system_u:object_r:system_cron_spool_t root root 192 Sep 16 2008 admin_scripts

hth.

Oracle Buys Sun

2009-04-21T11:14:00.007+08:00

Sun is now under Oracle. Are Solaris admins jumping for joy right now?
Oh, and MySQL is owned by Sun.
what now?

rsyslog and phplogcon

2009-04-16T16:40:00.008+08:00

so my php-syslog-ng evaluation encountered an early booting. now i'm
switching to another free tool - phplogcon

installation is pretty easy:

download (latest stable) - phplogcon-2.6.2.tar.gz

http://www.phplogcon.org/Downloads-req-getit-lid-54.phtml

and follow the steps here

http config
my new web config file for phplog site was changed to this:

Alias /isaplog /apps/isaplog

<Directory "/apps/isaplog">
Options -Indexes FollowSymLinks MultiViews
AllowOverride All
Order allow,deny
Allow from all
</Directory>

for my setup, i've untarred the downloaded file (phplogcon-2.6.2.tar.gz) and moved the src folder to /apps/isaplog

NOTE: my previous php-syslog-ng was renamed so i can use the same site
for phplogcon).

since i'm using selinux, i needed to make the config.php
writable by the web server:

chcon -t public_content_rw_t config.php

NOTE: don't forget to change config.php back to its original context after installation.

DB setup:

i've create a new DB (since i plan to import
old tables used by php-syslog-ng to phplogcon).

the new DB is phpISAPLog

so:

mysql -u root -p
mysql> create DATABASE phpISAPLog;
mysql> grant all privileges on phpISAPLog.* to isaplog@"localhost" identified by 'admin_pass';

then, to create the tables, i've used the createDB.sql script
that comes with the rsyslog source that i packaged

it is in /usr/share/doc/rsyslog-mysql-3.20.5

-rw-r--r-- 1 root root 1050 Apr 16 11:16 createDB.sql

i've modified the file to reflect the DB name i use
and commented out the CREATE line:

#CREATE DATABASE Syslog;
USE phpISAPLog
CREATE TABLE SystemEvents
(
ID int unsigned not null auto_increment primary key,
CustomerID bigint,
ReceivedAt datetime NULL,
DeviceReportedTime datetime NULL,
Facility smallint NULL,
<entries removed>

to create the table:

mysql -u isaplog -p -D phpISAPLog < createDB.sql

then in my browser (which is firefox ;)):

http://lx0050/isaplog/

the prompts are self-explanatory. for the source, here's
what i provided:

for my rsyslog.conf, i had to make some changes since i'm using
a new DB:

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$EscapeControlCharactersOnReceive off
$ModLoad ommysql.so
$ModLoad imklog
$ModLoad immark
$ModLoad imuxsock
$ModLoad imudp
# receive messages from network
$UDPServerAddress 10.x.y.z
$UDPServerRun 514

$template isapDBLog,"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', '%syslogfacility-text%', '%FROMHOST%', '%syslogpriority%', '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%programname%')",sql

if $msg contains 'START: nrpe' then ~
*.* :ommysql:127.0.0.1,phpISAPLog,isaplog,admin_pass;isapDBLog

and after a restart of rsyslog, phplogcon is now showing some log entries:

to mimic the php-syslog-ng view (that is easier to read by my target users),
i created a custom view:

anyone has any tips on how to import the php-syslog-ng tables to phplogcon?

Packaging rsyslog and adding it to a yum repo

2009-04-14T17:37:00.011+08:00

the rsyslog version that comes with RHEL5 u3 is an old
one - v 2.0.6. since me thinks this version has some problems
with the discard option in my config file, i've decided
to try the latest stable release and build an RPM file
from it and then include the packages in my yum repository.

here are the steps i did:

setup rpm macros
my .rpmmacros has the following:

$ cat .rpmmacros
%packager Roderick Tapang
%distribution Centos RPMS

%_topdir /apps/rpmbuild
{_builddir} %{_topdir}/BUILD
{_rpmdir} %{_topdir}/RPMS
{_sourcedir} %{_topdir}/SOURCES
{_specdir} %{_topdir}/SPECS
{_srcrpmdir} %{_topdir}/SRPMS

/apps/rpmbuild has the following structure

# generated using tree command

|-- BUILD
|-- RPMS
| |-- noarch
| `-- x86_64
|-- SOURCES
|-- SPECS
|-- SRPMS
`-- tmp

now, i've downloaded the tar ball from here:

http://download.rsyslog.com/rsyslog/rsyslog-3.20.5.tar.gz

and used Peter Vrabec's spec file from:

http://people.redhat.com/pvrabec/rpms/rsyslog/rsyslog.spec

the tar ball is placed in SOURCES along with other files that
will be included in the package:

../SOURCES/rsyslog-3.20.5.tar.gz -> tar ball is copied as is.
../SOURCES/rsyslog.conf -> sample configuration
../SOURCES/rsyslog.init -> startup script (without the rklogd entries)
../SOURCES/rsyslog.sysconfig -> will be copied to /etc/sysconfig/rsyslog

the spec file i used:

#START of SPEC file
%define sbindir /sbin

Summary: Enhanced system logging and kernel message trapping daemons
Name: rsyslog
Version: 3.20.5
Release: 0%{?dist}
License: GPLv2+
Group: System Environment/Daemons
URL: http://www.rsyslog.com/
Source0: http://download.rsyslog.com/rsyslog/%{name}-%{version}.tar.gz
#Source0: %{name}-%{version}.tar.gz
Source1: rsyslog.init
Source2: rsyslog.sysconfig
Source3: syslog.log
Source4: rsyslog.conf
BuildRequires: zlib-devel
BuildRequires: autoconf automake
Requires: logrotate >= 3.5.2
Requires: bash >= 2.0
Requires(post): /sbin/chkconfig coreutils
Requires(preun): /sbin/chkconfig /sbin/service
Requires(postun): /sbin/service
Provides: syslog
BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)

%package mysql
Summary: MySQL support for rsyslog
Group: System Environment/Daemons
Requires: %name = %version-%release
BuildRequires: mysql-devel >= 4.0

%description
Rsyslog is an enhanced multi-threaded syslogd supporting, among others, MySQL,
syslog/tcp, RFC 3195, permitted sender lists, filtering on any message part,
and fine grain output format control. It is quite compatible to stock sysklogd
and can be used as a drop-in replacement. Its advanced features make it
suitable for enterprise-class, encryption protected syslog relay chains while
at the same time being very easy to setup for the novice user.

%description mysql
The rsyslog-mysql package contains a dynamic shared object that will add
MySQL database support to rsyslog.

%prep
%setup -q

%build
%configure --sbindir=%{sbindir} --enable-mysql
make %{?_smp_mflags}

%install
rm -rf $RPM_BUILD_ROOT

make install DESTDIR=$RPM_BUILD_ROOT

install -d -m 755 $RPM_BUILD_ROOT%{_initrddir}
install -d -m 755 $RPM_BUILD_ROOT%{_sysconfdir}/sysconfig
install -d -m 755 $RPM_BUILD_ROOT%{_sysconfdir}/logrotate.d
install -p -m 755 %{SOURCE1} $RPM_BUILD_ROOT%{_initrddir}/rsyslog
install -p -m 644 %{SOURCE2} $RPM_BUILD_ROOT/%{_sysconfdir}/sysconfig/rsyslog
install -p -m 644 %{SOURCE3} $RPM_BUILD_ROOT%{_sysconfdir}/logrotate.d/syslog
install -p -m 644 %{SOURCE4} $RPM_BUILD_ROOT%{_sysconfdir}/rsyslog.conf

#get rid of *.la
rm $RPM_BUILD_ROOT/%{_libdir}/rsyslog/*.la
#get rid of useless files in doc
rm -rf doc/Makefile*

%clean
rm -rf $RPM_BUILD_ROOT

%post
/sbin/chkconfig --add rsyslog
for n in /var/log/{messages,secure,maillog,spooler}
do
[ -f $n ] && continue
umask 066 && touch $n
done

%preun
if [ $1 = 0 ]; then
service rsyslog stop >/dev/null 2>&1 ||:
/sbin/chkconfig --del rsyslog
fi

%postun
if [ "$1" -ge "1" ]; then
service rsyslog condrestart > /dev/null 2>&1 ||:
fi

%files
%defattr(-,root,root,-)
%doc AUTHORS COPYING NEWS README doc
%dir %{_libdir}/rsyslog
%config(noreplace) %{_sysconfdir}/rsyslog.conf
%config(noreplace) %{_sysconfdir}/sysconfig/rsyslog
%config(noreplace) %{_sysconfdir}/logrotate.d/syslog
%{_initrddir}/rsyslog
%{sbindir}/rsyslogd
%{_mandir}/*/*
%{_libdir}/rsyslog/imklog.so
%{_libdir}/rsyslog/immark.so
%{_libdir}/rsyslog/imtcp.so
%{_libdir}/rsyslog/imudp.so
%{_libdir}/rsyslog/imuxsock.so
%{_libdir}/rsyslog/lmnet.so
%{_libdir}/rsyslog/lmnetstrms.so
%{_libdir}/rsyslog/lmnsd_ptcp.so
%{_libdir}/rsyslog/lmregexp.so
%{_libdir}/rsyslog/lmtcpclt.so
%{_libdir}/rsyslog/lmtcpsrv.so
%{_libdir}/rsyslog/omtesting.so

%files mysql
%defattr(-,root,root)
%doc plugins/ommysql/createDB.sql
%{_libdir}/rsyslog/ommysql.so

%changelog

# END of SPEC file

NOTE: if anyone needs a tar file of the spec file and other files a used
to do the build, let me know so i can send it to you.

to build the src and binary rpms, i ran the following:

rpmbuild -bb rsyslog.spec

and produced the following (relevant to /apps/rpmbuild/SPECS):

../RPMS/x86_64/rsyslog-3.20.5-0.x86_64.rpm
../RPMS/x86_64/rsyslog-debuginfo-3.20.5-0.x86_64.rpm
../RPMS/x86_64/rsyslog-mysql-3.20.5-0.x86_64.rpm
../SRPMS/rsyslog-3.20.5-0.src.rpm

this works in RHEL5 u3 i386 and x86_64 releases and variants (like CEntOS). i then created a new entry in my existing yum repo using "createrepo", for my i386 repository:

/apps/repo/rhel5u3/i386/Custom
# ls -ltr
total 748
-rw-r--r-- 1 root root 723690 Apr 14 17:23 rsyslog-3.20.5-0.i386.rpm
-rw-r--r-- 1 root root 25337 Apr 14 17:23 rsyslog-mysql-3.20.5-0.i386.rpm
[root@lxisap0050 Custom]# createrepo .
2/2 - rsyslog-mysql-3.20.5-0.i386.rpm
Saving Primary metadata
Saving file lists metadata
Saving other metadata
[root@lxisap0050 Custom]# ls -ltr
total 756
-rw-r--r-- 1 root root 723690 Apr 14 17:23 rsyslog-3.20.5-0.i386.rpm
-rw-r--r-- 1 root root 25337 Apr 14 17:23 rsyslog-mysql-3.20.5-0.i386.rpm
drwxr-xr-x 2 root root 4096 Apr 14 17:27 repodata

and then, from another rhel5 system, running:

yum clean all
yum list rsyslog

installing it:

yum install rsyslog

and then:

[root@localhost ~]# chkconfig syslog off
[root@localhost ~]# chkconfig rsyslog on
[root@localhost ~]# service rsyslog start
Starting system logger: [ OK ]

rsyslog and php-syslog-ng for gnu/linux and AIX logging

2009-04-09T10:06:00.020+08:00

php-syslog-ng is a free tool to display reports from
a centralized log server. i'm evaluating it now to see
how it fits our rhel and aix systems.

currently, a windows based application is being used
by our infra team to handle centralized log reporting.

install rsyslog and mysql

yum install rsyslog rsyslog-mysql php-mysql mysql-server mysql

NOTE: a web server is also needed. httpd and php have been previously
installed on my machine.

download php-syslog-ng from:

http://code.google.com/p/php-syslog-ng/

for the directory location, i used the ff:

php-syslog-ng:
/apps/isaplog

mysql database:
/database/

mysql server

by default, RHEL5 mysql creates DBs in /var/lib/mysql. i
stopped the mysql daemons first and then moved the /var/lib/mysql
files to /database/mysql.
take note of the selinux settings here.

# /etc/my.cnf
[mysqld]
datadir=/database/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1

[mysql.server]
user=mysql
basedir=/database

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

# set password
/usr/bin/mysqladmin -u root password 'root_db_pass'
/usr/bin/mysqladmin -u root -h lx0050.domain password 'root_db_pass'

# create database
mysql -u root -p
create DATABASE ISAPLog;
grant all privileges on ISAPLog.* to isaplog@"localhost" identified by 'admin_pass';
grant select on ISAPLog.* to sysloguser@"%" identified by 'ro_passwd';
Query OK, 0 rows affected (0.00 sec)

php-syslog-ng

installation of php-syslog-ng is straight forward. you can
follow the steps in the guide .

just watch out for the following:

1. ownership of the files
- i set apache to own the contents of /apps/isaplog
2. permissions
- html/jpcache and html/config needs to be writable
during installation
chcon -t public_content_rw_t html/jpcache
chcon -R -t public_content_rw_t html/config
(a blank config.php is present, apache needs to write
to it during configuration).

after installation, make sure to remove the write permissions
in config/ and its contents. install directory also needs to be
removed.

for info:
man 8 httpd_selinux

my webserver configuration for my php-syslog-ng site is as follows:

#/etc/httpd/conf.d/phpsyslog.conf
Alias /isaplog /apps/isaplog/html

<Directory "/apps/isaplog/html">
Options -Indexes FollowSymLinks MultiViews
AllowOverride All
Order allow,deny
Allow from all
</Directory>
# added after configuration is done
<Directory "/apps/isaplog/scripts">
Deny from all
</Directory>
<Directory "/apps/isaplog/includes">
Deny from all
</Directory>
<Directory "/apps/isaplog/config">
Deny from all
</Directory>
# end

my site, is then accessible as:

http://lx0050/isaplog/

after installation, /apps/isaplog looks like this:

drwxr-xr-x apache root root:object_r:public_content_t html
drwxr-xr-x apache root root:object_r:public_content_t scripts
drwxr-xr-x apache root root:object_r:public_content_t upgrades

html/config:

drwxr-xr-x apache root root:object_r:public_content_t config

my initial rsyslog.conf:
$ModLoad ommysql.so
$template isapDBLog,"insert into logs(host, facility, priority, level, tag, datetime, program, msg) values ('%HOSTNAME%', '%syslogfacility-text%', '%syslogseverity-text%', '%syslogpriority-text%', '%syslogtag%', '%timereported:::date-mysql%', '%programname%', '%msg%')",sql
*.* :ommysql:127.0.0.1,ISAPLog,isaplog,admin_pass;isapDBLog

isapDBLog - is just the template name
ISAPLog - is the DB created to hold logs
isaplog - the DB user with admin privileges on ISAPLog DB
admin_pass - isaplog password

for the AIX Audit logs, entries in AIX are like this:

UXI0025 AIXAudit 2 14398 Wed Apr 08 16:21:42 2009 PROC_Execute grep iriXXXXX user00 979110 6758
74 OK euid: 6206 egid: 6200 epriv: 0:0 name grep 14024

UX0025 AIXAudit 2 14399 Wed Apr 08 16:21:42 2009 PROC_Execute grep iriXXXXX user00 1364214 6758
74 OK euid: 6206 egid: 6200 epriv: 0:0 name grep PRINCIPAL

UX0025 AIXAudit 2 14400 Wed Apr 08 16:21:42 2009 PROC_Execute awk iriXXXXX user00 675874 4671
68 OK euid: 6206 egid: 6200 epriv: 0:0 name awk -F | { print $5}

UX0025 AIXAudit 2 14401 Wed Apr 08 16:21:42 2009 PROC_Execute cat iriXXXXX user00 1085528 6758
76 OK euid: 6206 egid: 6200 epriv: 0:0 name cat /xMount/adsfasf/ISIXXXXX/IRIS/IRIS_C/extraction/out/Variables.dat

UX0025 AIXAudit 2 14402 Wed Apr 08 16:21:42 2009 PROC_Execute grep iriXXXXX user00 1364216 6758
76 OK euid: 6206 egid: 6200 epriv: 0:0 name grep 14024

UX0025 AIXAudit 2 14403 Wed Apr 08 16:21:42 2009 PROC_Execute grep iriXXXXX user00 1085530 6758
76 OK euid: 6206 egid: 6200 epriv: 0:0 name grep PRINCIPAL

However, using the current rsyslog configuration above to log to the mysql database,
the AIX Audit log entries are badly messed up. what i did, for a logging that is just *OK* with me:

- process the AIX Audit logs separetely (the AIX boxes still logs using traditional syslog message
- log them to DB with the raw,messed up AIX audit messages intact.

i then have the ff rsyslog.conf entries:

# start of rsyslog.conf
$ModLoad ommysql.so
# for other systems using syslog
$template isapDBLog,"insert into logs(host, facility, priority, level, tag, datetime, program, msg) values ('%HOSTNAME%', '%syslogfacility-text%', '%syslogseverity-text%', '%syslogpriority-text%', '%syslogtag%', '%timereported:::date-mysql%', '%programname%', '%msg%')",sql

# for AIX Audit
$template AIXDBLog,"insert into logs(host, facility, priority, level, tag, datetime, program, msg) values ('%FROMHOST%', 'syslog', '%syslogseverity-text%', '%syslogpriority-text%', '%syslogtag%', '%timereported:::date-mysql%', 'AIXAudit', '%msg:::drop-last-lf%')",sql

# for specific AIX Audit messages - log to sql
:msg,contains,"AIXAudit" :ommysql:127.0.0.1,ISAPLog,isaplog,admin_pass;AIXDBLog
& ~

#:msg, startswith, "START: nrpe " ~
if $msg contains 'START: nrpe' then ~
# log the rest to sql
*.* :ommysql:127.0.0.1,ISAPLog,isaplog,admin_pass;isapDBLog
# end of rsyslog.conf

from php-syslog-ng, an AIX audit entry appears like this:

one problem i'm still trying to fix - discarding unwanted log entries (like the
status msgs from nagios).

the line in rsyslog.conf:

if $msg contains 'START: nrpe' then ~

was supposed to fix it. and i think i'm doing something wrong. :D

more readings:
php-syslog-ng
rsyslog

update: i've scheduled some log rotation and cache reloading (scripts are part of php-syslog-ng):

#a file in /etc/cron.d has this:
*/5 * * * * root php /apps/isaplog/scripts/reloadcache.php >> /var/log/reloadcache.log

and had the following problem (msg was sent to my email):

PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 19731248 bytes) in /apps/isaplog/html/includes/common_funcs.php on line 203

i've set before the memory allocation to 128M then to 256M - and it's been working for a few days. now it is complaining again. from a google issue post, i've set the following:

in line 201 of html/includes/common_funcs.php, i added this line:

if (!in_array($row['program'],$cacheProgramValues))

and that fixed the problem.

update:

i've upgraded the rsyslog package using the rpm file i created and discarding of nrpe entries is now working. but the AIX Audit entries now has some different entries:

under Program column, instead of the "AIXAudit" entry, it now has the delimiter (?) included (the #011 etc).
AND from this very helpful post, i added $EscapeControlCharactersOnReceive off in my rsyslog.conf.

my new rsyslog.conf now has the following:
# start rsyslog.conf
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$EscapeControlCharactersOnReceive off
$ModLoad ommysql.so
$ModLoad imklog
$ModLoad immark
$ModLoad imuxsock
$ModLoad imudp
# receive messages from network
$UDPServerAddress 10.x.y.z
$UDPServerRun 514

$template isapDBLog,"insert into logs(host, facility, priority, level, tag, datetime, program, msg) values ('%HOSTNAME%', '%syslogfacility-text%', '%syslogseverity-text%', '%syslogpriority-text%', '%syslogtag%', '%timereported:::date-mysql%', '%programname%', '%msg%')",sql

if $msg contains 'START: nrpe' then ~
*.* :ommysql:127.0.0.1,ISAPLog,isaplog,admin_pass;isapDBLog
# end rsyslog.conf

notice that the AIX template is no longer used. isapDBLog now handles everything (for RHEL and AIX messages). and entries now look neat and clean:

will update how it goes.

building a YUM repository

2009-03-25T15:11:00.013+08:00

its always nice to have a local yum repository for your own environment. since we have systems with different packages installed, we sometimes need to add more packages for certain setup requirements.

here's how i setup my own yum repo (this one is using rhel5u3):

loopback mount iso file

mount -o loop rhel-server-5.3-x86_64-dvd.iso /mnt/temp

cp the contents to /apps/repo/rhel5u3

cp -pR /mnt/temp /apps/repo/rhel5u3

# ls -ldtr /apps/repo/rhel5u3/*/
dr-xr-xr-x 4 root root 4096 Jan 7 06:36 /apps/repo/rhel5u3/images/
dr-xr-xr-x 2 root root 4096 Jan 7 06:36 /apps/repo/rhel5u3/isolinux/
dr-xr-xr-x 3 root root 204800 Jan 7 08:40 /apps/repo/rhel5u3/Server/
dr-xr-xr-x 3 root root 4096 Jan 7 08:40 /apps/repo/rhel5u3/VT/
dr-xr-xr-x 3 root root 4096 Jan 7 08:40 /apps/repo/rhel5u3/ClusterStorage/
dr-xr-xr-x 3 root root 4096 Jan 7 08:40 /apps/repo/rhel5u3/Cluster/

set up the site:

/etc/httpd/conf.d/rhel5u3.conf
# start
Alias /rhel5u3 /apps/repo/rhel5u3/

Options Indexes
AllowOverride None
Order allow,deny
Allow from all

# end

NOTE: i'm using selinux. i had to set correct context so yum
clients can access the repo:

chcon -R root:object_r:public_content_t /apps/repo
chcon root:object_r:public_content_t /apps

man 8 httpd_selinux for more details.

restart or reload httpd:

/sbin/service httpd reload
or
/sbin/service httpd restart

then, on the client machine that needs to use the repo, i create
/etc/yum.repos.d/rhel5.repo

with the following content:

[rhel5-Server]
name=Red Hat Enterprise Linux Server
baseurl=http://lxisap0050.isap.asia.cib/rhel5u3/Server
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

[rhel5-VT]
name=Red Hat Enterprise Linux VT
baseurl=http://lxisap0050.isap.asia.cib/rhel5u3/VT
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

[rhel5-ClusterStorage]
name=Red Hat Enterprise Linux ClusterStorage
baseurl=http://lxisap0050.isap.asia.cib/rhel5u3/ClusterStorage
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

[rhel5-Cluster]
name=Red Hat Enterprise Linux Cluster
baseurl=http://lxisap0050.isap.asia.cib/rhel5u3/Cluster
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

then:

# yum clean all
Loaded plugins: rhnplugin, security
Cleaning up Everything

# yum repolist
Loaded plugins: rhnplugin, security
This system is not registered with RHN.
RHN support will be disabled.
rhel5-Server | 1.3 kB 00:00
primary.xml.gz | 825 kB 00:00
rhel5-Server 2985/2985
rhel5-Cluster | 1.3 kB 00:00
primary.xml.gz | 6.0 kB 00:00
rhel5-Cluster 32/32
rhel5-VT | 1.3 kB 00:00
primary.xml.gz | 8.8 kB 00:00
rhel5-VT 40/40
rhel5-ClusterStorage | 1.3 kB 00:00
primary.xml.gz | 7.2 kB 00:00
rhel5-ClusterStorage 36/36
repo id repo name status
rhel5-Cluster Red Hat Enterprise Linux Cluster enabled : 32
rhel5-ClusterStorage Red Hat Enterprise Linux ClusterStorage enabled : 36
rhel5-Server Red Hat Enterprise Linux Server enabled : 2,985
rhel5-VT Red Hat Enterprise Linux VT enabled : 40
repolist: 3,093

# yum grouplist
Loaded plugins: rhnplugin, security
This system is not registered with RHN.
RHN support will be disabled.
Setting up Group Process
comps-rhel5-server-core.xml | 987 kB 00:00
comps-rhel5-cluster.xml | 99 kB 00:00
comps-rhel5-vt.xml | 101 kB 00:00
comps-rhel5-cluster-st.xml | 104 kB 00:00
Installed Groups:
Administration Tools
Development Libraries
Editors
Graphics
Legacy Network Server
Legacy Software Development
Legacy Software Support
Mail Server
MySQL Database
Network Servers
PostgreSQL Database
Server Configuration Tools
System Tools
Text-based Internet
Virtualization
Web Server
Windows File Server
Available Groups:
Authoring and Publishing
Cluster Storage
Clustering
DNS Name Server
Development Tools
Engineering and Scientific
FTP Server
GNOME Desktop Environment
GNOME Software Development
Games and Entertainment
Graphical Internet
Java Development
KDE (K Desktop Environment)
KDE Software Development
News Server
Office/Productivity
OpenFabrics Enterprise Distribution
Printing Support
Sound and Video
X Software Development
X Window System
Done

my very own yum repo.

installing a package:
# yum install mysql-server
Loaded plugins: rhnplugin, security
This system is not registered with RHN.
RHN support will be disabled.
Setting up Install Process
Parsing package install arguments
Resolving Dependencies
--> Running transaction check
---> Package mysql-server.x86_64 0:5.0.45-7.el5 set to be updated
--> Processing Dependency: mysql = 5.0.45-7.el5 for package: mysql-server
--> Processing Dependency: libmysqlclient.so.15(libmysqlclient_15)(64bit) for package: mysql-server
--> Processing Dependency: perl-DBD-MySQL for package: mysql-server
--> Processing Dependency: libmysqlclient_r.so.15(libmysqlclient_15)(64bit) for package: mysql-server
--> Processing Dependency: libmysqlclient_r.so.15()(64bit) for package: mysql-server
--> Processing Dependency: libmysqlclient.so.15()(64bit) for package: mysql-server
--> Running transaction check
---> Package mysql.x86_64 0:5.0.45-7.el5 set to be updated
---> Package perl-DBD-MySQL.x86_64 0:3.0007-2.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================
Package Arch Version Repository Size
============================================================================================================================================
Installing:
mysql-server x86_64 5.0.45-7.el5 rhel5-Server 9.7 M
Installing for dependencies:
mysql x86_64 5.0.45-7.el5 rhel5-Server 4.2 M
perl-DBD-MySQL x86_64 3.0007-2.el5 rhel5-Server 147 k

Transaction Summary
============================================================================================================================================
Install 3 Package(s)
Update 0 Package(s)
Remove 0 Package(s)

Total download size: 14 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): perl-DBD-MySQL-3.0007-2.el5.x86_64.rpm | 147 kB 00:00
(2/3): mysql-5.0.45-7.el5.x86_64.rpm | 4.2 MB 00:00
(3/3): mysql-server-5.0.45-7.el5.x86_64.rpm | 9.7 MB 00:00
--------------------------------------------------------------------------------------------------------------------------------------------
Total 15 MB/s | 14 MB 00:00
warning: rpmts_HdrFromFdno: Header V3 DSA signature: NOKEY, key ID 37017186
Importing GPG key 0x37017186 "Red Hat, Inc. (release key) " from /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : mysql [1/3]
Installing : perl-DBD-MySQL [2/3]
Installing : mysql-server [3/3]

Installed: mysql-server.x86_64 0:5.0.45-7.el5
Dependency Installed: mysql.x86_64 0:5.0.45-7.el5 perl-DBD-MySQL.x86_64 0:3.0007-2.el5
Complete!

the GPG key check prompt is just for the initial yum install. next installations won't have this prompt (if it is the same GPG key used):

# yum install php-mysql
Loaded plugins: rhnplugin, security
This system is not registered with RHN.
RHN support will be disabled.
Setting up Install Process
Parsing package install arguments
Resolving Dependencies
--> Running transaction check
---> Package php-mysql.x86_64 0:5.1.6-23.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================
Package Arch Version Repository Size
============================================================================================================================================
Installing:
php-mysql x86_64 5.1.6-23.el5 rhel5-Server 87 k

Transaction Summary
============================================================================================================================================
Install 1 Package(s)
Update 0 Package(s)
Remove 0 Package(s)

Total download size: 87 k
Is this ok [y/N]: y
Downloading Packages:
php-mysql-5.1.6-23.el5.x86_64.rpm | 87 kB 00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : php-mysql [1/1]

Installed: php-mysql.x86_64 0:5.1.6-23.el5
Complete!

HINT: createrepo-0.4.11-3.el5 provides a way to setup a custom yum repository for your
RPMS. this way, you can host 3rd party rpms (and rpm updates) local to your network.
i use this with rhel4/5 and centos4/5 repositories. this is very useful for machines that doesn't have internet connection.
and no need for that satellite or rhn proxy purchase (at least for me :)).

syslogd and splunk

2009-03-25T10:59:00.005+08:00

i'm evaluating splunk for centralized log management. and syslog messages from AIX systems have the additional "Message forwarded from ux0026"

Mar 10 15:24:21 UX0026.xxxx.yyyy Mar 10 15:24:21 Message forwarded from ux0026: su: BAD SU from user1 to root at /dev/pts/0

from the man page of syslogd:

-n
Supresses the "Message forwarded from : " string added to the beginning
of the syslog message that is forwarded to a remote log host.

currently:

#ps -ef | grep syslog
root 233616 123026 0 12:47:53 - 0:00 /usr/sbin/syslogd

to pass the -n parameter:

#stopsrc -s syslogd
0513-044 The syslogd Subsystem was requested to stop.

#startsrc -s syslogd -a -n
0513-059 The syslogd Subsystem has been started. Subsystem PID is 233620.

#ps -ef | grep syslog
root 233620 123026 0 12:48:44 - 0:00 /usr/sbin/syslogd -n

now, this is not persistent. you need to edit /etc/rc.tcpip and change the syslog entry to this:

start /usr/sbin/syslogd "$src_running" -n

so the "-n" is passed to syslogd on system restart.

any idea what is the command to change it other than editing the file directly?

i think chrctcp can do it since enabling/disabling can be done this way:

# disable syslogd
chrctcp -d syslogd
# enable syslogd
chrctcp -a syslogd

these commands modify /etc/rc.tcpip directly. but documentation for chrctcp is scarce.

native MPIO on VIO hosted AIX LPARs

2009-03-23T15:30:00.008+08:00

we have AIX 5L LPARs hosted on a dual VIO server setup. Each Vio servers (lets call them vio1 and vio2), have 1 fiber connection each.
for this setup, MPIO is needed to be done at the client LPARs.

our IBM p570 VIO hosted Lpars setup is similar to the one illustrated here. in whole, our system has the following:
2xVIO servers (1 fiber port each)
1 dedicated LPAR (2 fiber ports)
3 AIX 5L LPARs (all hosted via VIO).

from a client LPAR named ux0018, a disk defined in both VIO servers appear like this:

lspath | grep hdisk5
Enabled hdisk5 vscsi0
Enabled hdisk5 vscsi1

vscsi0 - is through vio1
vscsi1 - is through vio2

how do we know this?

digging more in the path configuration for hdisk5 in ux0018:

#get the pvid:
lspv | grep hdisk5
hdisk5 00c14ebbea6747a8 ux0018apps03 active

this pvid is the same in both vio servers.

lspath -F'status name path_id parent connection' | grep -w hdisk5
Enabled hdisk5 0 vscsi0 860000000000
Enabled hdisk5 1 vscsi1 880000000000

this shows the LUN id that you can use to check in each vio server.

from vio1:
$ lspv | grep 00c14ebbea6747a8
hdisk22 00c14ebbea6747a8 None

from vio2:
$ lspv | grep 00c14ebbea6747a8
hdisk20 00c14ebbea6747a8 None

the same "disk" has different disk names but the same PVID in both vio servers. now check the LUN ID. here, vhost5 is the adapter assigned for ux0018:

vio1:
lsmap -vadapter vhost5 | grep -E "LUN|Backing device"

LUN 0x8600000000000000
Backing device hdisk22

vio2:
lsmap -vadapter vhost5 | grep -E "LUN|Backing device"
LUN 0x8800000000000000
Backing device hdisk20

checking the attributes for hdisk5:
lsattr -El hdisk5
PCM PCM/friend/vscsi Path Control Module False
algorithm fail_over Algorithm True
hcheck_cmd test_unit_rdy Health Check Command True
hcheck_interval 0 Health Check Interval True
hcheck_mode nonactive Health Check Mode True
max_transfer 0x40000 Maximum TRANSFER Size True
pvid 00c14ebbea6747a80000000000000000 Physical volume identifier False
queue_depth 3 Queue DEPTH True
reserve_policy no_reserve Reserve Policy True

for load balancing to work, algorithm must be set to round_robin. but:

sudo chdev -l hdisk5 -a algorithm=round_robin
Method error (/etc/methods/chgdisk):
0514-018 The values specified for the following attributes
are not valid:
algorithm

since native mpio only works in failover mode with this kind of setup,we have to do the "load balancing" manually. this means we redirect io to the unused path while maintaining the failover capability of the configuration. this would be a static LUN based load balancing setup.

to do this, we set the priority for hdisk5 path via vscsi0 to a lower value. meaning the path via vscsi1 is the primary path. since most of the disks are using vscsi0, hdisk5 will then not contend to the bandwidth on vscsi0. so we set the path to vscsi0 to priority=2:

chpath -l hdisk5 -p vscsi0 -w 860000000000 -a priority=2
path Changed

checking the two paths:

lspath -l hdisk5 -p vscsi0 -E
priority 2 Priority True

lspath -l hdisk5 -p vscsi1 -E
priority 1 Priority True

any I/O access now to/from hdisk5 will pass through vio2's fiber connection while still maintaining connection to vio1 in failover mode.

further readings:
configuring MPIO for AIX client logical partitions
Multiple path I/O

multipathing with RHEL5 and HP EVA 8K storage

2009-03-23T14:15:00.011+08:00

one of our RHEL5 test servers was given a second fiber connection (for multipathing) to improve SAN i/o. it currently has 2x100G of HP EVA 8k LUNs:

[root@lx0050 /]# pvs
Found duplicate PV BlHTUbYi2WYRSFFQjyxKcUyKPjtd4Aq3: using /dev/sdc1 not /dev/sda1
Found duplicate PV 11qdvncWddtqdWRV8nqQ1VT2QaGgRzdC: using /dev/sdd1 not /dev/sdb1
Found duplicate PV BlHTUbYi2WYRSFFQjyxKcUyKPjtd4Aq3: using /dev/sde1 not /dev/sdc1
Found duplicate PV 11qdvncWddtqdWRV8nqQ1VT2QaGgRzdC: using /dev/sdf1 not /dev/sdd1
PV VG Fmt Attr PSize PFree
/dev/cciss/c0d0p2 vg00 lvm2 a- 68.22G 50.66G
/dev/sde1 lx0050vg lvm2 a- 100.00G 50.00G #-> EVA
/dev/sdf1 lx0050vg lvm2 a- 100.00G 50.00G #-> EVA

the machine is running on RHEL5 update 3 with device-mapper-multipath-0.4.7-23.el5.

currently:

# multipath -ll
# dmsetup ls --target multipath

confirms that multipathing is not working.

checking the current disks on the system (for simplicity, i've just grep'd for entries
with 100.00 in size), we can see that there are disks with duplicate PV IDs. these
are due to the second path being active (but not yet utilized).

# lvmdiskscan | grep 100.00
Found duplicate PV BlHTUbYi2WYRSFFQjyxKcUyKPjtd4Aq3: using /dev/sdc1 not /dev/sda1
Found duplicate PV 11qdvncWddtqdWRV8nqQ1VT2QaGgRzdC: using /dev/sdd1 not /dev/sdb1
Found duplicate PV BlHTUbYi2WYRSFFQjyxKcUyKPjtd4Aq3: using /dev/sde1 not /dev/sdc1
Found duplicate PV 11qdvncWddtqdWRV8nqQ1VT2QaGgRzdC: using /dev/sdf1 not /dev/sdd1
/dev/sda1 [ 100.00 GB] LVM physical volume
/dev/sdb1 [ 100.00 GB] LVM physical volume
/dev/sdc1 [ 100.00 GB] LVM physical volume
/dev/sdd1 [ 100.00 GB] LVM physical volume
/dev/sde1 [ 100.00 GB] LVM physical volume
/dev/sdf1 [ 100.00 GB] LVM physical volume

to verify, the disk uuids (that can also be seen from the storage server side) are the following:

# for the first LUN
# scsi_id -g -u -s /block/sdc
3600508b4000704db0000600000070000
# scsi_id -g -u -s /block/sde
3600508b4000704db0000600000070000
# scsi_id -g -u -s /block/sda
3600508b4000704db0000600000070000

# for the second LUN
# scsi_id -g -u -s /block/sdd
3600508b4000704db00006000000a0000
# scsi_id -g -u -s /block/sdf
3600508b4000704db00006000000a0000
# scsi_id -g -u -s /block/sdb
3600508b4000704db00006000000a0000

notice that there are 3 disks with the same UUID.

to configure multipath for the EVA LUNs, i used the multipath device mapper for linux from HP. installation instructions are also on that page.

i'm not sure where the error is, but the installation fails with this:

** HPDMmultipath-4.2.0 kit Installation. Date : Fri Mar 20 09:21:49 SGT 2009 **

Checking for previous installation. Please wait...

This version of Enterprise Linux is not supported!!!.

the installation script is perl. and it looks like it's reading /etc/issue* in the system to check whether its a compatible release or not. but i have custom /etc/issue* files in our boxes.
since i only need the configuration file, i just copied HPDMmultipath-4.2.0/conf/multipath.conf.HPTemplate.RHEL5 to /etc/multipath.conf

then:

# deactivate the volume group
# vgchange -an lx0050vg
Found duplicate PV BlHTUbYi2WYRSFFQjyxKcUyKPjtd4Aq3: using /dev/sdc1 not /dev/sda1
Found duplicate PV 11qdvncWddtqdWRV8nqQ1VT2QaGgRzdC: using /dev/sdd1 not /dev/sdb1
Found duplicate PV BlHTUbYi2WYRSFFQjyxKcUyKPjtd4Aq3: using /dev/sde1 not /dev/sdc1
Found duplicate PV 11qdvncWddtqdWRV8nqQ1VT2QaGgRzdC: using /dev/sdf1 not /dev/sdd1
0 logical volume(s) in volume group "lx0050vg" now active

# multipath -v2 -d
create: mpath0 (3600508b4000704db0000600000070000) HP,HSV210
[size=100G][features=0][hwhandler=0][n/a]
\_ round-robin 0 [prio=100][undef]
\_ 0:0:1:1 sdc 8:32 [undef][ready]
\_ 1:0:0:1 sde 8:64 [undef][ready]
\_ round-robin 0 [prio=10][undef]
\_ 0:0:0:1 sda 8:0 [undef][ready]
create: mpath1 (3600508b4000704db00006000000a0000) HP,HSV210
[size=100G][features=0][hwhandler=0][n/a]
\_ round-robin 0 [prio=100][undef]
\_ 0:0:1:2 sdd 8:48 [undef][ready]
\_ 1:0:0:2 sdf 8:80 [undef][ready]
\_ round-robin 0 [prio=10][undef]
\_ 0:0:0:2 sdb 8:16 [undef][ready]

# service multipathd restart
Stopping multipathd daemon: [ OK ]
Starting multipathd daemon: [ OK ]
#

# list the active paths
# dmsetup ls --target multipath
mpath1 (253, 8)
mpath0 (253, 7)

# multipath -ll

mpath1 (3600508b4000704db00006000000a0000) dm-8 HP,HSV210
[size=100G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
\_ 0:0:1:2 sdd 8:48 [active][ready]
\_ 1:0:0:2 sdf 8:80 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 0:0:0:2 sdb 8:16 [active][ready]
mpath0 (3600508b4000704db0000600000070000) dm-7 HP,HSV210
[size=100G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
\_ 0:0:1:1 sdc 8:32 [active][ready]
\_ 1:0:0:1 sde 8:64 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 0:0:0:1 sda 8:0 [active][ready]

# pvs
PV VG Fmt Attr PSize PFree
/dev/cciss/c0d0p2 vg00 lvm2 a- 68.22G 50.66G
/dev/dm-10 lx0050vg lvm2 a- 100.00G 50.00G
/dev/dm-9 lx0050vg lvm2 a- 100.00G 50.00G

# then reactivate the volume group:
# vgchange -ay lx0050vg
3 logical volume(s) in volume group "lx0050vg" now active

now, the entries in /etc/fstab must be updated to use the correct device entries for the FS mount points. more details about this in Red Hat Knowledgebase. in my system, the logical volume in lx0050vg (now with multipath) is changed to use the device entry in /dev/mapper:

/dev/mapper/lx0050vg-lvapps /apps ext3 defaults,noatime 1 2

/apps can then be mounted. it now has multipath access to the storage for failover and load balancing.

and then to set multipathd to start during bootup:

chkconfig multipathd on

NOTE: the 2nd path we used here are on a different switch.

more readings can be searched at RH kbase.

linker error after moving to AIX TL8 SP5

2009-03-11T16:21:00.006+08:00

our developers encountered a problem yesterday while applying patches for an in-house developed application. initially, their compile scripts were calling an older version of gcc and needed gcc 4.0.0.
'fixed that by changing their environment declarations to point to the correct gcc version.
second, during linking, they got this error:

ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information.
ld: 0711-317 ERROR: Undefined symbol: .decimal128ToNumber
ld: 0711-317 ERROR: Undefined symbol: .decContextDefault
ld: 0711-317 ERROR: Undefined symbol: .decNumberTestClass
ld: 0711-317 ERROR: Undefined symbol: .decNumberMultiply
ld: 0711-317 ERROR: Undefined symbol: .decimal128FromNumber
ld: 0711-317 ERROR: Undefined symbol: .decNumberCompare
ld: 0711-317 ERROR: Undefined symbol: .decNumberExtractBiasedExponent
ld: 0711-317 ERROR: Undefined symbol: .decNumberInsertBiasedExponent
ld: 0711-317 ERROR: Undefined symbol: .decNumberTestSignificance
ld: 0711-317 ERROR: Undefined symbol: .decNumberQuantize
ld: 0711-317 ERROR: Undefined symbol: .decNumberToIntegralValue
ld: 0711-317 ERROR: Undefined symbol: .decimal32ToNumber
ld: 0711-317 ERROR: Undefined symbol: .decNumberZero
ld: 0711-317 ERROR: Undefined symbol: .decNumberAdd
ld: 0711-317 ERROR: Undefined symbol: .decimal64FromNumber
ld: 0711-317 ERROR: Undefined symbol: .decimal32FromNumber
ld: 0711-317 ERROR: Undefined symbol: .decimal64ToNumber
ld: 0711-317 ERROR: Undefined symbol: .decNumberFromLongLong
ld: 0711-317 ERROR: Undefined symbol: .decNumberToLongLong
ld: 0711-317 ERROR: Undefined symbol: .decNumberTestExponent
ld: 0711-317 ERROR: Undefined symbol: .decNumberFromInt
ld: 0711-317 ERROR: Undefined symbol: .decNumberSubtract
make: The error code from the last command is 8.

added the -bnoquiet in the ld line in the make script but the output doesn't make sense to me (heh, i'm not a programmer :P) but checking this line:

ld -bnoquiet -r -o $/lib/cmrtslib.o $lib/bun.a $/lib/zlib.a $/lib/clbase.a /usr/lib/libm.a

i thought of /usr/lib/libm.a being the culprit since that has been affected by the upgrade.

found out it was just a soft link:

$ ls -ltr /usr/lib/libm.a
lrwxrwxrwx 1 bin bin 19 Sep 23 2006 /usr/lib/libm.a -> /usr/ccs/lib/libm.a

and:

$ lslpp -w /usr/ccs/lib/libm.a

File Fileset Type
----------------------------------------------------------------------------
/usr/ccs/lib/libm.a bos.adt.libm File

/usr/ccs/lib/libm.a is owned by the bos.adt.libm fileset which was upgraded:

$ lslpp -L bos.adt.libm

Fileset Level State Type Description (Uninstaller)
----------------------------------------------------------------------------
bos.adt.libm 5.3.8.1 A F Base Application Development
Math Library

since i can't downgrade the whole fileset, i decided to restore just the file /usr/ccs/lib/libm.a from a mksysb file created before the upgrade:

$restore -xvqf /mnt//hostname.img ./usr/ccs/lib/libm.a

and this will restore the file, ./usr/ccs/lib/libm.a, (with full directory path) in the current directory.
i then moved the upgraded /usr/ccs/lib/libm.a to ./usr/ccs/lib/libm.a.tl8sp5 and copied the restored file as /usr/ccs/lib/libm.a.
the linking then went ok.

another way to use the old libm.a file w/out disturbing the upgraded one is to just put the old one somewhere and modify the make/link scripts to it. but then:

"would you expect us to modify all the linker scripts for each environment?!!"
- youknowwho

commands used:
# chech which fileset owns the file /usr/ccs/lib/libm.a
lslpp -w /usr/ccs/lib/libm.a

# check the version/desc of the fileset bos.adt.libm
lslpp -L bos.adt.libm

# extract a file from a mksysb image
restore -xvqf mksysb_file.img ./usr/ccs/lib/libm.a

copying printer queue definitions between LPARs

2009-03-09T15:17:00.002+08:00

i just don't want to lose this process so i'm posting this (in case web archives lose it. heh)

in a previous work, i needed to "copy" the printer queue definitions and their respective settings to another AIX 5.2 LPAR. here's what i came up with:

install printer drivers to target
append printer host entries to /etc/hosts of target
stopsrc -s lpd | stopsrc -s qdaemon
cp /etc/qconfig to target
cp /var/spool/lpd dir of source to target
chown -R root.printq /var/spool/lpd
chmod -R g+rw /var/spool/lpd
startsrc -s lpd | startsrc -s qdaemon

might not be the correct way but it worked.

IBM p550 and p570 HMC, Firmware and OS upgrade

2009-03-09T11:49:00.004+08:00

we recently had a maintenance window where we can accommodate upgrading firmware and patching AIX 5.3. These are the steps I did after a couple of weeks planning (with the help of IBM's Fix Level Recommendation Tool (FLRT).

old version:

AIX OS/TL Level	Firmware	vio	HMC version
5300-06-04-0748	SF240_332	1.5.1.1-FP-10.1	V6.1.2 build 20070201.1

0. BACKUP
1. upgrade HMC first (done 1 week ahead of yesterday's maint sked).
2. shutdown all LPARs
3. upgrade firmware (can be done via ftp - needs the rpm and xml file
5. apply VIO updates (commit installed patches first)
5. update NIM server -> ( if you use NIM to apply AIX patches)
6. apply AIX patches (do a commit first before applying new patches)

the HMC update DVDs were ordered from IBM website including the SP1 disc (MH01084). same thing with the VIO update discs. The VIO updates were copied locally to the VIO servers. we have 2 VIOs on each machine.

HMC upgrade
we two HMCs (1 for each machine). upgrade was done during business hours since the lpars can still run without it. Upgrade was started as follows:
1. Save Upgrade Data - HMC configuration will then be automatically loaded once the update is done.
2. boot from the 1st DVD update.
3. once done with disc 1, it will prompt you for the second disk.
4. once done, the new HMC web based interface will show up (very differenct, and kind of slow when accessed remotely).
5. to apply SP1, insert the SP1 Disc (MH01084) , login as hscroot. look for "Updates" on the left fram of the HMC interface.
6. click on Update HMC and click on OK. the default is to look for the update in the DVD/CD drive..

and just reply to the prompts. if there are no problmes, it will take about 15-20 minutes for the service pack. HMC upgrade, in total, will take about an hour to finish.

detailed info can be found here.

Firmware Update
firwmare updates for the two machines (we actually have 3 - p550, p550q and p570) were all the same. so need only to burn 1 ISO file. But the HMC won't read the discs. so downloaded the RPM and xml files and setup an FTP server (good thing i have proftpd installed in a GNU/Linux machine).
after starting up proftpd and making sure the files are accessible, i then went back to the HMC session. to access the firmware update menu, you need to:
1. under Systems management, go to "Servers", select the machine name
2. at the bottom from, go to "Updates"
4. either use "Change Licensed Internal Code for the current release" or "Upgrade Licensed internal Code to a new release".
I chose the latter.
5. HMC will do some system readiness check. No errors, so i hit OK.
6. select FTP site and hit OK. i then provided the ftp server, username, password
and directory location of the rpm and xml files.
the rest of the process is straight forward so i won't include it here.
once done, the firmware should be upgraded and should

Updates > View System Information

select "None - Display current values" and hit OK.

more info here

from the command line, once the an LPAR is online, the firmware version can be viewed using lsmcode:

#lsmcode -c
The current permanent system firmware image is SF240_332
The current temporary system firmware image is SF240_358
The system is currently booted from the temporary firmware image.

(running lsmcode alone will bring you to a smitty based session).

NOTE: the LPARs here are all powered-off.

VIO update commands:

#commit all uncommitted (really?) updates
updateios -commit

# apply the patches from the directory VIO accepting all license agreements by default
updateios -dev VIO -install -accept

after applying, VIO servers need to be restarted (NOTE: AIX Lpars are still "powered off" here.

# check VIO level:
$ ioslevel
1.5.2.1-FP-11.1
$

NOTE: client lpars here are still powered down.

AIX update

the VIO updates took a while to install. i waited to finish both VIO updates before i started updating the AIX LPARs.

AIX 5.3 TL8 SP5 update commands
#commit
sudo installp -c all

# update (assuming patches are in the current directory).
sudo update_all -d . -i

# update and accept licencse agreements
sudo update_all -d . -Y

for the TL8 sp5 update, i had to re-run the update_all command twice. the first run misses this fileset:

ifor_ls.html.en_US.base.cli

and from this post

suggests to a 2nd run of the update which worked for me.

additional commands i needed:

# i needed XL C/C++ Runtime messages (wasn't picked up the first update_all run
installp -d . -gX xlC.msg.en_US 9.0.0.0

# get installed ML or TLs
instfix -i | grep ML

# check missing fileset needed to satisfy 53300-08_AIX_ML
instfix -ciqk "5300-08_AIX_ML" | grep ":-:"

once update is done, AIX LPARs have been rebooted:

#oslevel -s
5300-08-05-0846

TIP: the AIX updates are NFS mounted on the AIX LPARs. patches were supposed to be applied via NIM servers (the update resources have already been defined) but during a test run, the application of patches were a bit slow.

once everything is done, i now have the following:

updated versions
AIX OS/TL Level	Firmware	VIO	HMC version
TL8sp5	SF240_358	1.5.2.1-FP-11.1	7.3.2.0 sp1

now i can tell internal auditors our systems are on the same patch level. :)