Frits Hoogland Weblog

What is orion? Well it used to be the name of the oracle java server, which eventually became ‘OC4J’, or ‘Oracle Containers for Java’. It’s a product oracle uses or bought (don’t know). It’s still available from the original vendor.

But oracle now has a new orion: ORacle IO Numbers. This is a tool to test IO with! Other than other tools, it uses the exact libraries oracle 10g uses, and issues ‘database like’ requests. In other words, this tool enables you to make a IO prediction of a database fairly easy, or enables you to calculate optimal numbers of IO slaves (for oracle’s parallel query).

Orion is available on otn for the following platforms:
-Linux AMD x86-64
-Linux i386
-Windows

If you are interested: follow this link. It also contains a document about usage, output, examples and troubleshooting.

What does it do, and how does it work?

First, you have to have raw devices, or files (both are used and written during the test, and destroy any current data in them. So generate them for this test!!!)

Next, make a file (in the same directory as where the executable out of the zip/msi appeared, that’s the most easy way), named anything you wish, with the extention “.lun”; for example: testrun.lun. In that file, put the whole path to these files or the raw devices:

/oracle/orion/file1
/oracle/orion/file2


For a simple testrun, execute the following:
./orion -run simple -testname testrun -num_disks 2
This runs using the ’simple’ mode of orion (meaning ’small’ and ‘large’ IO’s are tested in isolation (not concurrent)). ‘-num_disks’ is used for determining the number of concurrent IO’s (to my understanding; I’ve gotten 1-5 concurrent IO’s when set to ‘1′, and 1-10 concurrent IO’s when set to ‘2′)

Currently, I’ve used this tool to measure the number of IO’s possible for a performance problem.
This problem is with a 9.2 database (but I think using orion the measurement still is viable), in the nightbatch. Despite most batches, most of the waiting in on ‘db_file_sequential_read’, which means (among others) index unique scans, index range scans and table fetches by index rowid.

To simulate this type of IO, I’ve used the following commandline:
orion -run advanced -testname test -num_disks 2 -type rand -write 5 -matrix row -num_large 1
This commandline means:
-num_disks 2: I want to know throughput up to 10 concurrent processes
-type rand: most of my IO’s are not sequentially, but random (from an o/s perspective, we walk through some branch blocks, and through the leafs (if range-scanned)
-write 5: ultimate goal of the batch is to process data. So we do writing
-matrix row: the calculation is based upon small IO’s, we set the number of large IO’s using ‘num_large’
-num_large 1: we do some ‘large’ IO’s (multiblock reads; full table scans, fast full index scans)

I’ve gotten up to about 500 IOPS (IO’s per second), from windows 2003, iSCSI, 1Gb ethernet and netapp filer. The filer is massively overloaded (cpu 100% almost entire day), and results of two tests on the same files where not consistent, which is in line with an overloaded filer, of which the throughput is depended upon other IO’s being done at the same time.

There (still) is much speak about OFA, Oracle Flexible Architecture. The reason for OFA is obvious (OFA is designed to organize large amounts of complicated software and data on disk, simplify administration tasks, maximise performance and assist switching between Oracle databases.)

I work in a team which administers +200 databases and we thought out a OFA strategy, for all the reasons stated above. (this document applies to unix/linux filesystems, but can be adjusted to windows too)

First of all: use a single place for all “oracle stuff”. This means everything we use for the oracle database. We use “/oracle”. This way, all the oracle database stuff is concentrated on one place on our system, and it’s easy for the linux/unix system administrator to see if something is belonging to oracle.
Before you ask: please remember it is not mandatory to mount a filesystem directly under root.

Next, there are more oracle products than the database. So the second stage should be “db” (/oracle/db)

At this level, we have a place where all the stuff for an oracle database should be. At this place we can make a distinction of what we could encounter at this point. That can be any of the two following things:
-the database software of a certain version
-a database

So, to install the database software, it should be in “/oracle/db/” (eg, /oracle/db/10.2.0.1)
There are a few more things which are very important for the database software.
-We use a seperate mountpoint for each version (meaning /oracle/db/10.2.0.1 is a mountpoint). This way the space can be given back when we stop using the software of a certain version.
-No instance specific files are allowed in the software filesystem. Some files (like network files) can be moved using a environment variable (TNS_ADMIN), some are needed at a certain point, so we use symbolic links to ‘link them back’ to an instance specific place (like password file and pfile/spfile). This helps the instance directory (eg, /oracle/db/myinstance) in being perfectly movable to another machine, because it’s self contained)

And, for all the files concerning a database instance, it should be in “/oracle/db/” (eg, /oracle/db/test). Underneath this directory, we have a large number of instance specific things:
-udump, bdump, adump, cdump (the dump dirs)
-pfile (for the pfile/spfile and password file)
-utlfile (if UTL_FILE_DIR is used)
-exp (exports directory/mountpoint)
-arch (archive directory/mountpoint)
-scripts (instance specific scripts, creation scripts)
-data (for database files and redologfiles)
this is the directory where the important stuff is going. There is a little structure here:
-rd01
-rd02
->we need at least two online redolog directories. These need to be mountpoints (to spread IO), and can be even more, depending on your needs. Minimum is two (which should at least contain two redolog groups each, so redo-IO is switched back and forth between the two mountpoints).
-fs01
->this is the mountpoint for the datafiles. Of course, ‘fs’ means filesystem, and we could add as many of this mountpoints as we need. Also, it’s easy to monitor IO’s with the Oracle disk manager to see of the IO load is spread well enough for any given number of mountpoints and databases.

There are a few more things which we need to specify:
-oracle net
-oracle scripts and config files

Oracle net is global to the databases. Even if we got more databases, we use one listener most of the time (so the listener configuration should not be at the database level). The networking files (tnsnames.ora, sqlnet.ora, listener.ora, ldap.ora, etc.) should be in /oracle/net/admin. Update the .profile of the oracle database user to set TNS_ADMIN to this directory.

The oracle scripts are the ones which root.sh puts in /usr/local/bin by default. Because we want to have access to these scripts as dba’s, and we do not have root rights most of the time, we move these files to /oracle/admin/bin (after root.sh has put them there, we have root rights at that time), and put a symlink in /usr/local/bin. This way we have full access to these scripts, without the need for root.

There are two files (if emtab is needed, it follows oratab) left which we want inside /oracle, but all the oracle software expects them at another place:
-oratab
-oraInst.loc
these files are needed for some scripts and tools, so it’s wise to put them in the /oracle structure as well, and put a symbolic link at the place oracle expects them to be. Put the files in /oracle/admin/etc.

This way we achieved the following:
-all oracle files are in /oracle
-all files concerning an instance are grouped in /oracle/db/
-if a new version or a new patchlevel of the software is installed, it’s easy to move an instance to this software
-to move a database to a new machine, it’s already grouped and simple to move
-the distribution of filesystems is easy to see
-it’s easy to add filesystems to a database for both database files and online redologfiles

First question here is: why would I want to install the Oracle database on debian? Well, there is very much to say about that, but, in essence: I very much like debian. And I could tell you, of course, about the brilliant packaging system, debian being true free software, debian being standards based, easy upgrading and security updates, the merry help from the community, but I don’t. I just happen to like debian.

Second question is: why in a xen user domain? Well, that’s very obvious for me:
-the userdomain I describe here is build in a logical volume. the logical volume can very easy and very fast being copied to a second one, and voila: I have a second, seperate machine.
-the userdomain can be snapshoted at any time, so we can have a ‘frozen’ image of a domain. comes in very handy when testing or when trailing errors.
-I can use a single piece of metal for running various versions of linux (yes, also the bsd’s and solaris, but I do not run that now), running various versions of oracle (each running different patchlevels and CPU’s (the infamous Cummulative Patch Update)). Very handy for investigating quality and stability.
-When software alters files or something, and needs to be recovered (in testing or investigation), it’s very convenient to use the snapshot possiblities of LVM.

This is how I create a debian/sarge domU in a logical volume. There are more versions of this description (in fact, that is how got to know the trick), but they have little omissions which do not give me what I want (like being unable to use swap because the block special device is missing in /dev)

nb. dom0 is called ’server’, the domU to be created is called ’sarge’.
nb. this version is simplified. I install the oracle software in a seperate logical volume, so I can use a single installation with different o/s installs.

1. create logical volume for the root:
frits@server:~# sudo lvcreate -n sarge-root -L 5G /dev/vg00
2. create logical volume for swap:
frits@server:~$ sudo lvcreate -n sarge-swap -L 1G /dev/vg00
3. create ext3 filesystem on the root logical volume:
frits@server:~$ sudo /sbin/mkfs.ext3 /dev/vg00/sarge-root
4. create swap:
frits@server:~$ sudo /sbin/mkswap /dev/vg00/sarge-swap
5. mount root logical volume:
frits@server:~$ sudo mount /dev/vg00/sarge-root /mnt/install
6. install sarge using debootstrap:
frits@server:~$ sudo debootstrap sarge /mnt/install http://ftp.nl.debian.org/debian
7. alter the feshly installed server:
frits@server:~$ cd /mnt/install/dev
frits@server:~$ sudo ./MAKEDEV tty1 tty2 tty3 tty4 tty5 tty6
frits@server:~$ sudo ./MAKEDEV sda
frits@server:~$ cd ../etc
frits@server:~$ sudo vi fstab
here we put the following rows, so we mount the minimal three things:

proc /proc proc defaults 0 0
/dev/sda1 / ext3 defaults,errors=remount-ro 0 1
/dev/sda2 none swap sw 0 0

frits@server:~$ cd network
frits@server:~$ sudo vi interfaces

here we put the following rows, so we activate localhost and our public ip address:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

next we configure the last few things before we boot our virtual host ’sarge’:

frits@server:~$ cd ..
frits@server:~$ sudo echo "sarge" > hostname
frits@server:~$ sudo printf "127.0.0.1\tlocalhost\n192.168.0.18\tsarge\n" > hosts

As we see on the last line, 192.168.0.18 is the address I have gotten from my dhcp server.

8. The filesystem is now configured for booting as root-filesystem. Next we have to define a host for xen, which looks like this (name of the file needs to be the name of machine, in order to start it by name with xen):


kernel = "/boot/vmlinuz-2.6.12.6-xenU"
builder='linux'
memory = 512
name = "sarge"
vif = [ 'mac=aa:00:00:00:00:15, bridge=xenbr0' ]
disk = [ 'phy:vg00/sarge-root,sda1,w','phy:vg00/sarge-swap,sda2,w' ]
dhcp="on"
hostname= "sarge"
root = "/dev/sda1 ro"
extra = "4"


9. Next, we start our new host ’sarge’:
frits@server:~$ sudo xm create sarge -c

The experienced xen user will notice I have forgotten to move the /lib/tls directory. Well, I have done that on purpose! Xen will complain about the “normal” tls version and warn us.

After the complaint, our machine will boot nicely and we get a prompt. We log in as root, and set a password. (root needs to have a password, especially because we get public access shortly!) After that, we finish the configuration using ‘base-config’.

10. Next we have to modify the ‘/etc/apt/sources.list’ file. Apparently, base-config puts the following line:
deb http://ftp.nl.debian.org/debian/ testing main
Which results in downloading packages of etch, instead of sarge. So we alter the line to:
deb http://ftp.nl.debian.org/debian/ stable main non-free
(we add ‘non-free’ to it too, because we need some packages out of non-free in a short while)
Refresh the apt cache and renew our new system:
sarge:~# apt-get update
sarge:~# apt-get upgrade

11. Next we alter the sources list once more, in order to get a xen-friendly libc. Add the following line to /etc/apt/sources.list:
deb http://www.hodek.net/debian sarge roman
After that, refresh the apt cache, and get the xen-friendly libc:
sarge:~# apt-get update
sarge:~# apt-get install libc6-i686

At this point, we got a working version of debian/sarge, with the bare minimum of packages. This is a good point to save a copy of the root logical volume, in order to be able to start with an installed and configured debian/sarge installation.

Because we want to install the oracle database server on our debian host ’sarge’, we get on configuring it.

12. Because the ssh daemon is not a part of the bare minumum packages of debian, we have to install it:
sarge:~# apt-get install ssh
After this installation, review the ssh_config and sshd_config in /etc/ssh, in order to be able to use X-tunneling. This way we are able to get a ssh session to sarge, and can conveniently tunnel our X connection through it. No messing with xhost for me

13. We need to have a X environment, because of the graphical installation of the oracle software. So we install it:
sarge:~# apt-get install x-window-system

14. There are some reports of errors during the link phase of the installations due to incorrect version of libaio. The most simple (and elegant) way to get this, is from oracle itself. We add the oracle repository to /etc/apt/sources.list by inserting the following line:
deb http://oss.oracle.com/debian unstable main non-free
Next we refresh our apt cache and install libaio:
apt-get install libaio

15. Because we got a minimized version of the install, we have installed less than a default debian/sarge installation. One of the missing things (which gives a nice little error message during linktime:)
Error in invoking target 'ntcontab.o' of makefile '/oracle/db/10201/network/lib/ins_net_client.mk'
is libc6-dev. So we install it:
apt-get install libc6-dev

16. Next a whole bunch of other needed software:
apt-get install gcc make binutils libmotif3 lesstif2 rpm

At this point, we can do the oracle installation. There are no error messages or other faults during the install. I have used a nfs share exported from dom0 with the software.

17. Follow this link to go to Graham Williams description of installing oracle on debian.

Goodluck!