How to fix problem with ASM not recognizing multipathed disks after a reboot

After installing ASM 11g on Red Hat 5.5 using the Linux multipath software the disk permissions are not set properly by ASMLib when the system is rebooted. This results in the following error in /var/log/oracleasm:

oracleasm-read-label: Unable to open device "/dev/dm-0": No such file or directory

ASM will not mount the disks and of course the database will not start.

I tried the following:
- I checked that all the correct versions of the ASM RPMs were installed.
- I performed the oracleasm configure, which is supposed to properly set the permissions on reboot.
- I tried setting the ORACLEASM_SCANORDER per the documentation about setting up ASM on Linux Multipath disks [Document Id 602952.1].

None of these things worked.

Finally I added these lines to the top of the script /etc/init.d/oracleasm and the problem is solved:

chown grid:oinstall /dev/dm*
chmod 775 /dev/dm*

Deinstall ASM on standalone

First deconfigure clusterware
Run as root

cd <GRID_HOME>/crs/install
perl rootcrs.pl -deconfig -force

Now Deinstall
Run as ASM owner

cd <GRID_HOME>/deinstall

./deinstall

If you want to reuse the ASM volumes under a different user name you can wipe the headers like this.

Determine run mode of database instance

SQL> select status from v$instance;


STATUS
------------------------------------
OPEN

May be OPEN, UPGRADE, OPEN MIGRATE, etc.

Determine pool sizes

SQL> select pool, sum(bytes)/1024/1024 Mbytes from v$sgastat group by pool;

POOL             MBYTES
------------ ----------
java pool             4
large pool            4
shared pool         148

Find location of online redo logs

SQL> select member from v$logfile;

MEMBER
--------------------------------------------------------------------------------
/te11/u02/ORACLE/te11/te11.redo.g01.m01.rdo
/te11/u03/ORACLE/te11/te11.redo.g01.m02.rdo
/te11/u02/ORACLE/te11/te11.redo.g02.m01.rdo
/te11/u03/ORACLE/te11/te11.redo.g02.m02.rdo
/te11/u02/ORACLE/te11/te11.redo.g03.m01.rdo
/te11/u03/ORACLE/te11/te11.redo.g03.m02.rdo
/te11/u02/ORACLE/te11/te11.redo.g04.m01.rdo
/te11/u03/ORACLE/te11/te11.redo.g04.m02.rdo
/te11/u02/ORACLE/te11/te11.redo.g05.m01.rdo
/te11/u03/ORACLE/te11/te11.redo.g05.m02.rdo

10 rows selected.

Determine what database components are installed

set pagesize    40

column componentName    format a40  heading 'Component Name'
column version          format a15  heading 'Version'
column status           format a8   heading 'Status'

select  COMP_NAME   componentName, 
        VERSION     version, 
        STATUS      status
from    dba_registry;

Example:

Component Name                           Version         Status
---------------------------------------- --------------- --------
OWB                                      11.1.0.6.0      VALID
Oracle Application Express               3.0.1.00.08     VALID
Oracle Enterprise Manager                11.1.0.6.0      VALID
Oracle Ultra Search                      11.1.0.6.0      VALID
OLAP Catalog                             11.1.0.6.0      VALID
Spatial                                  11.1.0.6.0      VALID
Oracle Multimedia                        11.1.0.6.0      VALID
Oracle XML Database                      11.1.0.6.0      VALID
Oracle Text                              11.1.0.6.0      VALID
Oracle Expression Filter                 11.1.0.6.0      VALID
Oracle Rules Manager                     11.1.0.6.0      VALID
Oracle Workspace Manager                 11.1.0.6.0      VALID
Oracle Database Catalog Views            11.1.0.6.0      VALID
Oracle Database Packages and Types       11.1.0.6.0      VALID
JServer JAVA Virtual Machine             11.1.0.6.0      VALID
Oracle XDK                               11.1.0.6.0      VALID
Oracle Database Java Packages            11.1.0.6.0      VALID
OLAP Analytic Workspace                  11.1.0.6.0      VALID
Oracle OLAP API                          11.1.0.6.0      VALID

List all ORACLE_SIDs and their home directories on a server (UNIX)

Look in oratab, typically located in /var/opt/oracle or the /etc directories.

$ more /var/opt/oracle/oratab
#
ora9:/oracle/product/10.2.0/db_1:N
EIC:/oracle/product/10.2.0/db_1:N
cmstdev:/oracle/product/10.2.0/db_1:N

List Tablespace Usage

set pagesize 9999
set linesize 160 

column tablespaceName       heading 'Tablespace'
column kBytesUsed           heading 'Used (KB)'
column kBytesFree           heading 'Free (KB)'
column largest              heading 'Largest'
column percentUsed          heading '% Used'

select      a.TABLESPACE_NAME           tablespaceName,
            a.BYTES  / 1024             kBytesUsed,
            b.BYTES / 1024              kBytesFree,
            b.largest / 1024            largest,
            round(((a.BYTES - b.BYTES)
                    / a.BYTES)
                    *100
                    ,2) percentUsed
from        (
            select     TABLESPACE_NAME,
                       sum(BYTES) BYTES
            from       dba_data_files
            group by   TABLESPACE_NAME
            ) a,
            (
            select      TABLESPACE_NAME,
                        sum(BYTES) BYTES ,
                        max(BYTES) largest
            from        dba_free_space
            group by    TABLESPACE_NAME
            ) b
where       a.TABLESPACE_NAME = b.TABLESPACE_NAME
order by    ((a.BYTES-b.BYTES)/a.BYTES) desc
;

Example:

Tablespace                      Used (KB)  Free (KB)    Largest     % Used
------------------------------ ---------- ---------- ---------- ----------
SYSTEM                             716800       7808       7104      98.91
SYSAUX                             601024      29120      29120      95.15
EXAMPLE                            102400      22848      20416      77.69
USERS                                5120       1920       1920       62.5
UNDOTBS1                            81920      60416      45888      26.25

5 rows selected.

Find location of data files

SQL> select name from v$datafile;

NAME
--------------------------------------------------------------------------------
/oradata/emprddb/disk1/system01.dbf
/oradata/emprddb/disk1/undotbs01.dbf
/oradata/emprddb/disk1/sysaux01.dbf
/oradata/emprddb/disk1/users01.dbf
/oradata/emprddb/disk1/mgmt.dbf
/oradata/emprddb/disk1/mgmt_ecm_depot1.dbf
/oradata/emprddb/disk1/rman_01.dbf

Locate the Oracle Inventory Directory

Solaris

$ more /var/opt/oracle/oraInst.loc
inventory_loc=/oracle/oraInventory
inst_group=dba

On Linux the file will be under /etc

Move data files to new location

1-Shutdown the DB
2-copy all .DBF .LOG to the new location. Do not copy the .CTL files at this step.
3-startup mount
3-alter data~base rename file 'old location' to 'new location' ; ... for all .DBF and .LOG files
Oracle finds them in the new location
4-alter system set control_file=[list of CTL files in new location] scope=spfile
5-alter system set core_dump_dest=[new empty location for cdump] scope=spfile
6-alter system set user_dump_dest=[new empty location for udump] scope=spfile
7-alter system set background_dump_dest=[new empty location for bdump] scope=spfile
(CTL are still in old location but contain information about the new location)
8-shutdown
9-copy all .CTL files to the new location
10-startup

Determine background dump dest (bdump.) Location of alert Log

select value from v$parameter where name = 'background_dump_dest';

--OR--

show parameter dump;