Steve Harville's Blog

Migrating databases to new storage

steveharville — Mon, 09 Jan 2012 11:58:01 +0000

This was a last minute request to move five databases on two servers from an old failing storage system to a new one. There was a total of 5.6 terabytes to move and the databases were in heavy use by the development team. The DBA who usually supported these systems was unavailable so I started working with the storage team to get this done.

I was able to get the storage team to create 67 new disks with the same sizes as the original disks and assign them to the correct servers. I researched how to get the disks to show up for ASM on Windows. There were 37 diskgroups and I assigned the new disks to each disk group according to the size of the original disks. So each diskgroup now had double the storage.

The next step shows the power of this technique: I dropped the old disks from each diskgroup and ASM moved all the data from the old disks to the new disks, then released the old disks. I started these late in the day and they finished the next morning. No downtime, no impact to the development team.

Lessons learned: use standard disk sizes, minimize the number of diskgroups

RAC on AIX

steveharville — Thu, 27 Oct 2011 23:06:48 +0000

I just finished the class: X0161 Oracle RAC on AIX Systems Workshop taught by Andrei Socoliuc of IBM Romania. Andrei knew a lot about Oracle RAC and AIX so it was a good class. I thought the best part was the hands on labs, especially the pre-installation preparation of the operating system. There was also a lot of good information on hardware and LPAR configuration. This is my first RAC class since 2003 when it had just been released, so the overview of RAC was a good way for me to get a refresher on RAC internals. My only complaint is that we spent a lot of time on IBM’s shared disk solution (GPFS) and very little on Oracle’s ASM .

How do I know if the cardinality estimates in a plan are accurate?

steveharville — Thu, 04 Aug 2011 16:54:06 +0000

Maria Colgan has another great blog post today about using the GATHER_PLAN_STATISTICS hint to see the actual rows returned as well as the optimizer’s estimate of number of rows returned. This is very good information and will make it obvious if your statistics are wrong.

Opening a standby READ_ONLY and continuing to apply logs

steveharville — Tue, 05 Apr 2011 22:40:28 +0000

After reading Guenadi’s post I decided to try it myself. Creating Oracle 11g active standby database from physical standby database.

10:34:21 PROD-DB1SYS>select * from v$instance;
INSTANCE_NUMBER INSTANCE_NAME    HOST_NAME                                                        VERSION
--------------- ---------------- ---------------------------------------------------------------- -----------------
              1 PROD-DB             dev-box                                                     11.1.0.7.0
1 row selected.
Elapsed: 00:00:00.01
10:37:23 PROD-DB1SYS>select open_mode from v$database;
OPEN_MODE
----------
MOUNTED
1 row selected.
Elapsed: 00:00:00.03
10:37:48 PROD-DB1SYS>alter database recover managed standby database cancel;
Database altered.
Elapsed: 00:00:02.06
10:38:52 PROD-DB1SYS>alter database open read only;
Database altered.
Elapsed: 00:00:23.96
10:39:45 PROD-DB1SYS>alter database recover managed standby database using current logfile disconnect;
Database altered.
Elapsed: 00:00:09.08
10:40:40 PROD-DB1SYS>select open_mode from v$database;
OPEN_MODE
----------
READ ONLY
1 row selected.
Elapsed: 00:00:00.02
……………………………………………………………………………….
SQL> select HOST_NAME from v$instance;

HOST_NAME
----------------------------------------------------------------

prod-box

SQL> create user sharville identified by secret account unlock;
User created.
SQL> grant unlimited tablespace to sharville;
Grant succeeded.
SQL> create table sharville.sh (col_id number(10));
Table created.
……………………………………………………………………………..
SQL> select HOST_NAME from v$instance;

HOST_NAME
----------------------------------------------------------------
  dev-box  
11.1.0.7.0        04-APR-11 OPEN         NO           1 STARTED
ALLOWED    NO  ACTIVE            PRIMARY_INSTANCE   NORMAL    NO
SQL> select object_name from all_objects where owner = 'SHARVILLE';
OBJECT_NAME
------------------------------
SH

WAIT_FOR_GAP status on standby

steveharville — Mon, 04 Apr 2011 22:36:39 +0000

This morning I noticed one of the managed standby databases was in a pending shutdown state. I restarted it and verified the logs were getting applied from production. However, I noticed the WAIT_FOR_GAP status on one of the old logs. Some of the logs had not been applied and were no longer on the production system. So I found out which logs were in the gap and used RMAN to recover them in production. Oracle automatically shipped them to te standby and applied them.

Discover a gap:

1* SELECT PROCESS, CLIENT_PROCESS, SEQUENCE#, STATUS FROM V$MANAGED_STANDBY

PROCESS   CLIENT_P  SEQUENCE# STATUS
--------- -------- ---------- ------------
ARCH      ARCH         136626 CLOSING
ARCH      ARCH              0 CONNECTED
ARCH      ARCH         136627 CLOSING
ARCH      ARCH         136628 CLOSING
RFS       LGWR         136629 IDLE
RFS       UNKNOWN           0 IDLE
RFS       UNKNOWN           0 IDLE
MRP0      N/A          130107 WAIT_FOR_GAP
RFS       UNKNOWN           0 IDLE


Find out the extent of the gap:

  1  SELECT high.thread#, "LowGap#", "HighGap#"
  2  FROM
  3       (
  4       SELECT thread#, MIN(sequence#)-1 "HighGap#"
  5       FROM
  6       (
  7           SELECT a.thread#, a.sequence#
  8           FROM
  9           (
 10               SELECT *
 11               FROM v$archived_log
 12           ) a,
 13           (
 14               SELECT thread#, MAX(next_change#)gap1
 15               FROM v$log_history
 16               GROUP BY thread#
 17           ) b
 18           WHERE a.thread# = b.thread#
 19           AND a.next_change# > gap1
 20       )
 21       GROUP BY thread#
 22   ) high,
 23   (
 24       SELECT thread#, MIN(sequence#) "LowGap#"
 25       FROM
 26       (
 27           SELECT thread#, sequence#
 28           FROM v$log_history, v$datafile
 29           WHERE checkpoint_change# <= next_change#
 30           AND checkpoint_change# >= first_change#
 31       )
 32       GROUP BY thread#
 33   ) low
 34*  WHERE low.thread# = high.thread#
11:13:08 BIPROD1SYS>/
   THREAD#    LowGap#   HighGap#
---------- ---------- ----------
         1     130106     130153

Recover the logs in Production:

RMAN>  restore archivelog from sequence 130106 until sequence 130153;

I had to repeat this sequence of commands several times because there were multiple gaps.

Hotsos Symposium

steveharville — Sat, 12 Mar 2011 00:48:30 +0000

This was the best conference I have ever attended. Very smart speakers and attendees. It was fun and I learned a lot about Oracle performance. Some of the themes were : statistics, Exadata and fine grained measurement. I would definitely go again! Hotsos Symposium 2011

CRS-2800: Cannot start resource ‘ora.asm’ as it is already in the INTERMEDIATE state on server

steveharville — Fri, 11 Feb 2011 17:35:15 +0000

I got this error when I was installing Oracle RAC 11.2 on Red Hat Enterprise Linux 5.6. I was installing Clusterware using ASM on VMware shared disks. When I created the independent persistent virtual disks, I left the “allocate all disk space now” option unselected. Oracleasm was happy on both RAC nodes. The Oracle installer was happy when it created +ASM1 on the first RAC node. But when the ASM instance started on node 2 it did not like the “virtually provisioned” disk. The +ADSM2 instance was not open and was complaining about one of the shared disks being corrupt at a certain byte. When I checked the virtual disk files, I saw that Oracle was trying to read past the end of the file. I started all over with new fully allocated shared disk and that fixed the problem. Everything is up and running now.

ERROR 1045 (28000): Access denied for user – Setting up MySQL Enterprise Dashboard Agent account

steveharville — Wed, 15 Dec 2010 16:37:21 +0000

Adding MySQL servers to the dashboard forced me to really understand the way MySQL handles security. I have two servers running on the same physical host and I wanted the same agent to monitor both. Simple, right? Except I couldn’t get the new agent account to work. I dumped the user table from the first server and used the SQL to insert the account info into the other server’s mysql.user table. But the agent couldn’t log on. I finally figured out that I had to “flush privileges;” after inserting into the user table. Now it’s working and I have a script to add the agent to other servers.

Oracle error 19505 returned by remote Oracle server

steveharville — Tue, 16 Nov 2010 22:14:57 +0000

I was re-running an rman duplicate from active database today and got this error. I had assumed that the duplicate command would overwrite the files from the previous run. Instead it created new filenames and caused the ASM disk group to fill up.

Here’s the rman message on the target system:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 11/16/2010 10:08:27
RMAN-03015: error occurred in stored script Memory Script
RMAN-03009: failure of backup command on ORA_DISK_6 channel at 11/16/2010 10:00:25
ORA-17628: Oracle error 19505 returned by remote Oracle server

Here’s the alert log entry on the auxiliary system:

ORA-19505: failed to identify file "+data"
ORA-17502: ksfdcre:4 Failed to create file +data
ORA-15041: diskgroup "DATA" space exhausted

The target (source) is on ASM using OMF (Oracle Managed Files). The auxiliary (destination) is also using ASM and OMF. I looked through the rman log and found a lot of set newname commands like this:

executing command: SET NEWNAME

No filename is specified so it makes up new filenames, which are different from the previous run. So that’s why I ran out of space.

I used asmcmd to remove all of the files and reclaim the space.

Easy fix, but initially confusing.

Installing Webutil for Oracle Forms

steveharville — Thu, 08 Jul 2010 14:20:20 +0000

Thanks to Zeeshan Baig for the great article about installing Webutil for Oracle Forms. It is much better than the Oracle documentation.