What is this blog about?
This blog is about DBA’s who run databases on linux. Linux sysadmins are target audience too; this blog is about the stop/start system on linux, and to be more precise: about the specific implementation of the stop/start system on redhat and it’s derivatives (Oracle’s enterprise linux for example, but also centos)

What about it?
The redhat stop/start system looks and feels like the traditional unix system V stop/start system found on most unixes. Correct usage of the redhat stop/start system is in fact quite different from that.

The traditional unix system V stop/start uses an executable script in the init.d directory (often /etc/init.d or /etc/rc.d/init.d) and uses a symbolic link in the runlevel directories (rc1.d, rc2.d, rc3.d, etc.) from the script in init.d, but prefixing the name it with ‘S’ (for start) or ‘K’ (for kill, thus stop) and a two digit number to indicate the order for starting or stopping.

The redhat stop/start system works with an executable script in the init.d directory (/etc/rc.d/init.d). The correct way to enable stop/start is using the chkconfig command:
chkconfig --add yourscript
This command check if the script yourscript is present in /etc/rc.d/init.d, then reads a line looking like this in the stop/start script:
#chkconfig: 2345 80 05
This tells chkconfig that the script should run in runlevels 2,3,4 and 5 with order number 80 and stopped (killed) with order number 05.
To check if chkconfig did understand your script, use:
chkconfig --list
Further reading about chkconfig:
man chkconfig

Alright, but what about crashing the database?
A lesser known fact is that the redhat stop/start system uses a file with the same name as the stop/start script in /var/lock/subsys to determine if the script is already started (so it doesn’t get started if it’s already started in a previous runlevel) and needs to be created in the start part, but more importantly it determines if the script needs to execute the stop procedure by checking the existence of this file in /var/lock/subsys. As part of the stop procedure the file needs to be removed, so the stop procedure isn’t run again in another runlevel.

In many custom made scripts this file doesn’t get created in /var/lock/subsys in the start procedure, so the stop procedure actually is never executed, because the script is already stopped from the perspective of the stop/start system. This typically means that the oracle stop/start script does a perfect job starting the database, but actually never stops the database, which means that during a shutdown or a reboot, the database actually crashes.

Sadly, I see very much scripts forgetting this (even ones from respectable sources, like redhat; see: Redhat summit beginner’s guide to running oracle on RHEL, for example, there are much more). Check your stop/start script today!

For this recognition I must thank Doug Burns, but also all other friends who helped me solving problems, help with presentations, or helping me in other aspects. Thank you! Last but not least my employer, VX Company..

If you are interested in enhancing security on linux/unix based systems and making DBA usage audit-able, fetch the paper. Of course, any comments are appreciated!

If someone is using Redhat Enterprise Linux, or one of the clones (Oracle Enterprise Linux, Centos), and are using yum (and I can see only very limited reasons for not using it; if you have to administer multiple machines, setting up a yum server will be a real timesaver, if the number is very small, use the internet repositories directly), add the EPEL repository: EPEL.

EPEL contains packages not present in the package collection of RHEL. One of these packages is rlwrap, which can give SQL*Plus commandline history. Installing rlwrap with EPEL is as simple as:

yum install rlwrap

Please notice this gives the SQL*Plus commandline some of the properties you have in the bash shell (when sqlplus is invoked with rlwrap of course: rlwrap sqlplus username/password@database). This means you can also search the history with the key combination CTRL + R.

Another property of the bash shell is autocompletion with the TAB key (something I use extensively). It would be extremely handy to have TAB autocompletion for Oracle keywords and a list of choices (if it can’t autocomplete because several options exist) with TAB - TAB (TAB two times).

Johannes Gritsch has written a little script to overcome some rlwrap difficulties and lists with autocompletion for keywords and data dictionary: http://www.linuxification.at/rlwrap_ext.html.en

Please note the tar needs to be extracted in ‘/usr/share/rlwrap’ when using the rlwrap version of EPEL, instead of the path listed in the readme. This path also needs to be modified in the ’sql+’ script.

I’ve just copied my plain linux operating system install and installed oracle 10 and oracle 11. Of course, I’ve renamed the machine. That is what this blogpost is about.

To honest, I am quite lazy. So lazy that I forgot to link the system name set in /etc/sysconfig/network to the ip-address given (done in /etc/hosts).

This does not affect the software installation, but does affect the creating of the database. It is reported as an internal error (ORA-600):

ORA-00600: internal error code, arguments: [keltnfy-ldmInit], [46], [1], [], [], [], [], []

Meaning: the hostname can not be translated to an ip-address.
Resolution: lookup hostname using the hostname command, lookup the ip-address using the /sbin/ifconfig -a command, and link both in /etc/hosts

In most situations where you do configuration, tuning or other settings of web environments (apache, oracle http server, OC4J, weblogic, websphere, etc.) you have access from ‘the inside’, most of the time access via a separate LAN (often called ‘administration LAN’) which let’s you get access in another way than the regular web traffic, which comes in from ‘the outside’ which is firewalled different (or is the only connection being firewalled), thus is handled different.

There are situations where you want to see what traffic from ‘the outside’ does instead of your ‘inside connection’, for example:
- ssl/certificates/wallets
- connecting webservers with the webcache
- altering hostnames
- review firewalling and webtraffic routing
- review NIDS (network intrusion detection)

This is where tor comes in.

What does Tor do? The best explanation is on the tor website. What it does is route your traffic encrypted through an anonymous, distributed network. Tor accomplishes much security related things, for this subject it’s important it setups an connection randomly on the internet.

Tor is not available on the RHEL/OEL/CentOS CD/DVD’s and network repositories. It’s also not present in EPEL (extra packages for Enterprise Linux), so it must be downloaded from the tor download page (choose ’stable’).

In order to fully use it, you need privoxy, which is present in the repository (thus can be installed using yum).

After the installation of both, there’s a little configuration necessary, but then you’ve got a proxy setup which let’s you browse the internet (anonymously) from a random place on the internet!

For people who have security considerations, and/or doubts about anonymity of their traffic, anything which can use a proxy can use it.

This means nikto can be used entirely anonymous, whilst things like nmap, nessus can’t with this setup.

The man-page says ‘Generate debugging information for this system’. This is getting even more interesting…

Lets run it!

$ /usr/sbin/sosreport

sosreport requires root permissions to run.

The utility requires root permissions. Not very surprising, root got all permissions to get internal system information.

Again, now as root:

# sosreport

Please send this file to your support representative.


Let’s see what it provides, and see if it got any benefits above Oracle’s RDA…
It produces two files in /tmp:

-rw-r--r-- 1 root root 33 May 1 14:34 sosreport-fritshoogland.9999-725705-3400ad.tar.bz2.md5
-rw------- 1 root root 631098 May 1 14:34 sosreport-fritshoogland.9999-725705-3400ad.tar.bz2


The list of files when extracting sosreport-fritshoogland.9999-725705-3400ad.tar.bz2 simply is too big to put in the blogpost.
It contains all linux configuration files (yes, that’s a bold statement), all startup/shutdown scripts, all kind of current system information (dmidecode, lsmod, lspci, netstat, iptables, rpm, network routes, LVM, device mapper, EMC multipathing, etc.), could go on and on!

This command is able to let anyone who can get access to the system make a file with all relevant settings for troubleshooting. That’s handy! It’s very hard to get information about network configuration, firewall (iptables), etc. out of inexperienced administrators. This command lets you get all information so you can do the troubleshooting out of the archive instead of a question and answer game with the administrator!

All other information (location, price, etc.) are on that page too. If you are an Oracle database professional and want to learn something and be able to discuss various things with likeminded professionals, be there!

First of all, there’s Direct IO (DIO) and Concurrent IO (DIO). The advantage of DIO and CIO is the data for both reading and writing is not buffered at the operating system level. This means ‘double buffering’ is omitted, which saves CPU cycles. If the database uses the operating system buffering (for example if there’s plenty of memory on the system), performance can degrade when DIO or CIO is used, because then all IO’s are done physically from the operating system’s perspective (it could be buffered at the lower layer, which could be a SAN or NAS).

The difference between DIO and CIO is the way write concurrency is handled (this is a bit simplified). DIO serialises IO’s to prevent writes from corrupting each other, whilst CIO does not serialise, because it assumes the writer has done proper serialisation. With the Oracle database, this is the case (remember, Oracle can work without a filesystem too).

The IBM JFS filesystem has DIO implemented, the IBM JFS2 filesystem has CIO implemented.

In order to implement CIO, there are two ways:
-Use a database parameter: FILESYSTEMIO_OPTIONS; set it to ’setall’ (which also implements Asynchronous IO), or ‘directIO’ (which opens files in DIO mode on JFS, and in CIO mode on JFS2)
-Use a mount option (’cio’) to make all IO’s to the filesystem behave as if CIO was done

A note on Oracle’s metalink (’Direct I/O or Concurrent I/O on AIX 5L’; ML 272520.1) describes this behavior. This note also states that BOTH should be enabled. I don’t find this the best way to make a database do CIO; merely setting FILESYSTEMIO_OPTIONS properly (’setall’,'directIO’) will do, mounting the filesystem with the ‘cio’ option will not add anything, it will make regular IO’s do CIO, of which some do not benefit. For example, if the database software and the dump destinations are on this filesystem too.

The default page of the website (http://public-yum.oracle.com/) contains information about how to setup OEL4, OEL5 and Oracle VM 2 to use it, both the i386 and the x86_64 version.

As I’ve guessed in my previous post, the yum server does not contain an update channel, it just contains the contents of the installation media.