Create single instance standby database for RAC Database

Here I created a physical standby database drdb for RAC database racdb instance names racdb1 and racdb2, server names: - NODE1 and NODE2.

PRIMARY RAC DATABASE INSTANCE RACDB1 IN HOST MACHINE NODE1 PRIMARY RAC DATABASE INSTANCE RACDB2 IN HOST MACHINE NODE2 CREATING A STANDBY DATABASE IN SERVER NODE2 WITH NAME DRDB

ASSUMPTION: Primary:
1. The RAC Primary Database is in fully operating condition in Archivelog mode using ASM storage (shared location) as archivelog destination. 2. The Primary Database is using ASM as Storage and OMFs. 3. Listener LISTENER has already been created.

Steps in primary database Servers:1. Database must in Archivelog mode. 2. Enable Force Logging. 3. Create the Standby Redo logs. 4. Create a password file for all instances (Password must same in all nodes). 5. Update listener.ora file to include SID information on each node in cluster. 6. Verify cluster Service is available. 7. Verify TNSNAMES.ORA file. 8. Modify init.ora Parameters For DataGuard Configuration. 9. Create temporary directory to hold the RMAN backup of this database. 10. Backup the Primary Database for Standby. 11. Copy the Primary Database backup to the standby database server to create standby database.

Steps in Standby database server:1. Create required folders and directories. 2. Modify the init.ora parameter on standby database. 3. Create a password file (Password must same as primary database instances) 4. Update listener.ora file to include SID information 5. Add required TNS entries for standby database on all servers. 6. Verify TNSNAMES.ORA file. 7. Create standby database.

[oracle@node1 dbs]$ . oraenv ORACLE_SID = [black] ? racdb1 The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1 is /u01/app/oracle [oracle@node1 dbs]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.1.0 Production on Sun Jan 22 03:41:53 2012 Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options SQL> archive log list; Database log mode Automatic archival Archive destination Oldest online log sequence Next log sequence to archive Current log sequence Archive Mode Enabled USE_DB_RECOVERY_FILE_DEST 16 17 17

SQL> alter database force logging; Database altered. SQL> select instance_name,status from gv$instance; INSTANCE_NAME STATUS ---------------- -----------racdb1 OPEN racdb2 OPEN SQL> select name,db_unique_name,database_role from gv$database; NAME DB_UNIQUE_NAME DATABASE_ROLE --------- ------------------------------ ---------------RACDB racdb PRIMARY RACDB racdb PRIMARY

IN PRIMARY DATABASE (any one of the node) ======================================

SQL> alter system set log_file_name_convert='+DATA/drdb','+DATA/racdb' scope=spfile sid='*'; SQL> alter system set db_file_name_convert='+DATA/drdb','+DATA/racdb' scope=spfile sid='*'; SQL> alter system set fal_client='racdb1' sid='racdb1';

SQL> alter system set fal_client='racdb2' sid='racdb2'; SQL> alter system set fal_server='drdb' sid='*'; SQL> alter system set log_archive_config='dg_config=(racdb,drdb)' sid='*'; SQL> alter system set log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ONLINE_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=racdb' sid='*'; SQL> alter system set log_archive_dest_state_2='ENABLE' sid='*'; SQL> alter system set log_archive_dest_2='SERVICE=drdb LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=drdb' sid='*'; SQL> alter system set log_archive_dest_3='LOCATION=+DATA/racdb/STANDBYLOG VALID_FOR=(STANDBY_LOGFILES,STANDBY_ROLE) DB_UNIQUE_NAME=racdb' sid='*'; SQL> alter system set log_archive_dest_state_1='enable' sid='*'; SQL> alter system set log_archive_dest_state_3='enable' sid='*'; SQL> alter system set log_archive_max_processes=7 sid='*'; SQL> alter system set log_archive_min_succeed_dest=2 sid='*'; SQL> alter system set remote_login_passwordfile='EXCLUSIVE' sid='*'; SQL> alter system set standby_file_management='auto' sid='*'; racdb2.__db_cache_size=520093696 racdb1.__db_cache_size=520093696 racdb2.__java_pool_size=16777216 racdb1.__java_pool_size=16777216 racdb2.__large_pool_size=16777216 racdb1.__large_pool_size=16777216 racdb2.__pga_aggregate_target=587202560 racdb1.__pga_aggregate_target=587202560 racdb2.__sga_target=872415232 racdb1.__sga_target=872415232 racdb2.__shared_io_pool_size=0 racdb1.__shared_io_pool_size=0 racdb2.__shared_pool_size=301989888 racdb1.__shared_pool_size=301989888 racdb2.__streams_pool_size=0 racdb1.__streams_pool_size=0 *.audit_file_dest='/u01/app/oracle/admin/racdb/adump' *.audit_trail='db' *.cluster_database=true *.compatible='11.2.0.0.0' *.control_files='+DATA/racdb/controlfile/current.466.773206191','+DATA/ racdb/controlfile/current.465.773206191' *.db_block_size=8192 *.db_create_file_dest='+DATA'

*.db_domain='' *.db_file_name_convert='+DATA/drdb','+DATA/racdb' *.db_name='racdb' *.db_recovery_file_dest='+DATA' *.db_recovery_file_dest_size=4039114752 *.diagnostic_dest='/u01/app/oracle' *.dispatchers='(PROTOCOL=TCP) (SERVICE=racdbXDB)' racdb1.fal_client='racdb1' racdb2.fal_client='racdb2' *.fal_server='drdb' racdb2.instance_number=2 racdb1.instance_number=1 *.log_archive_config='dg_config=(racdb,drdb)' *.log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ONLINE_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=racdb' *.log_archive_dest_2='SERVICE=drdb LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=drdb' *.log_archive_dest_3='LOCATION=+DATA/racdb/STANDBYLOG VALID_FOR=(STANDBY_LOGFILES,STANDBY_ROLE) DB_UNIQUE_NAME=racdb' *.log_archive_dest_state_1='enable' *.log_archive_dest_state_2='ENABLE' *.log_archive_dest_state_3='enable' *.log_archive_format='%t_%s_%r.dbf' *.log_archive_max_processes=7 *.log_archive_min_succeed_dest=2 *.memory_target=1449132032 *.open_cursors=300 *.processes=150 *.remote_listener='rac-scan:1521' *.remote_login_passwordfile='exclusive' *.sec_case_sensitive_logon=FALSE *.standby_archive_dest='+DATA/racdb/STANDBYLOG' *.standby_file_management='auto' racdb1.thread=1 racdb2.thread=2 racdb2.undo_tablespace='UNDOTBS1' racdb1.undo_tablespace='UNDOTBS2' SQL> select force_logging from v$database; FOR --YES SQL> alter system set SEC_CASE_SENSITIVE_LOGON=FALSE scope=spfile sid='*'; System altered. [oracle@node1 dbs]$ orapwd file=$ORACLE_HOME/dbs/orapwracdb1 password=oracle force=y entries=5 ignorecase=y [oracle@node2 dbs]$ pwd /u01/app/oracle/product/11.2.0/db_1/dbs [oracle@node2 dbs]$ orapwd file=$ORACLE_HOME/dbs/orapwracdb2 password=oracle entries=5 force=y ignorecase=y

SQL> show parameter password NAME TYPE VALUE ------------------------------------ ----------- ---------------------remote_login_passwordfile string EXCLUSIVE SQL>

Create standby logfile on primary database ===================================

SQL> col member for a56 SQL> select group#,member,type from v$logfile order by 1; GROUP# MEMBER TYPE ----------------------------------------------------------------------1 +DATA/racdb/onlinelog/group_1.468.773206197 ONLINE 1 +DATA/racdb/onlinelog/group_1.467.773206195 ONLINE 2 +DATA/racdb/onlinelog/group_2.470.773206197 ONLINE 2 +DATA/racdb/onlinelog/group_2.469.773206197 ONLINE 3 +DATA/racdb/onlinelog/group_3.473.773206427 ONLINE 3 +DATA/racdb/onlinelog/group_3.474.773206427 ONLINE 4 +DATA/racdb/onlinelog/group_4.475.773206429 ONLINE 4 +DATA/racdb/onlinelog/group_4.476.773206429 ONLINE 8 rows selected. SQL> select group#,bytes from v$Log; GROUP# BYTES ---------- ---------1 5242880 2 5242880 3 5242880 4 5242880 SQL> alter database add standby logfile thread 1 group 5 size 5M, group 6 size 5M; Database altered. SQL> alter database add standby logfile thread 2 group 7 size 5M, group 8 size 5M; Database altered. SQL> select group#,member ,type from v$Logfile order by 1; GROUP# MEMBER TYPE ---------- -------------------------------------------------------- ---

1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8

+DATA/racdb/onlinelog/group_1.467.773206195 +DATA/racdb/onlinelog/group_1.468.773206197 +DATA/racdb/onlinelog/group_2.469.773206197 +DATA/racdb/onlinelog/group_2.470.773206197 +DATA/racdb/onlinelog/group_3.473.773206427 +DATA/racdb/onlinelog/group_3.474.773206427 +DATA/racdb/onlinelog/group_4.475.773206429 +DATA/racdb/onlinelog/group_4.476.773206429 +DATA/racdb/onlinelog/group_5.481.773209067 +DATA/racdb/onlinelog/group_5.482.773209067 +DATA/racdb/onlinelog/group_6.483.773209067 +DATA/racdb/onlinelog/group_6.484.773209069 +DATA/racdb/onlinelog/group_7.485.773209079 +DATA/racdb/onlinelog/group_7.486.773209079 +DATA/racdb/onlinelog/group_8.487.773209079 +DATA/racdb/onlinelog/group_8.488.773209081

ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE STANDBY STANDBY STANDBY STANDBY STANDBY STANDBY STANDBY STANDBY

16 rows selected.

Create pfile for standby database ===========================

SQL> create pfile='$ORACLE_HOME/dbs/initdrdb.ora' from spfile; File created. SQL>

Edit the required parameters remove the red colored lines for the standby database initdrdb.ora file edit/change the blue coloured lines as per standby database required.
racdb2.__db_cache_size=520093696 racdb1.__db_cache_size=520093696 racdb2.__java_pool_size=16777216 racdb1.__java_pool_size=16777216 racdb2.__large_pool_size=16777216 racdb1.__large_pool_size=16777216 racdb2.__pga_aggregate_target=587202560 racdb1.__pga_aggregate_target=587202560 racdb2.__sga_target=872415232 racdb1.__sga_target=872415232 racdb2.__shared_io_pool_size=0 racdb1.__shared_io_pool_size=0 racdb2.__shared_pool_size=301989888 racdb1.__shared_pool_size=301989888 racdb2.__streams_pool_size=0 racdb1.__streams_pool_size=0 *.audit_file_dest='/u01/app/oracle/admin/racdb/adump' *.audit_trail='db' *.cluster_database=true

*.compatible='11.2.0.0.0' *.control_files='+DATA/racdb/controlfile/current.466.773206191','+DATA/ racdb/controlfile/current.465.773206191' *.db_block_size=8192 *.db_create_file_dest='+DATA' *.db_domain='' *.db_file_name_convert='+DATA/drdb','+DATA/racdb' *.db_name='racdb' *.db_recovery_file_dest='+DATA' *.db_recovery_file_dest_size=4039114752 *.diagnostic_dest='/u01/app/oracle' *.dispatchers='(PROTOCOL=TCP) (SERVICE=racdbXDB)' racdb1.fal_client='racdb1' racdb2.fal_client='racdb2' *.fal_server='drdb' racdb2.instance_number=2 racdb1.instance_number=1 *.log_archive_config='dg_config=(racdb,drdb)' *.log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ONLINE_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=racdb' *.log_archive_dest_2='SERVICE=drdb LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=drdb' *.log_archive_dest_3='LOCATION=+DATA/racdb/STANDBYLOG VALID_FOR=(STANDBY_LOGFILES,STANDBY_ROLE) DB_UNIQUE_NAME=racdb' *.log_archive_dest_state_1='enable' *.log_archive_dest_state_2='defer' *.log_archive_dest_state_3='enable' *.log_archive_format='%t_%s_%r.dbf' *.log_archive_max_processes=7 *.log_archive_min_succeed_dest=2 *.memory_target=1449132032 *.open_cursors=300 *.processes=150 *.remote_listener='rac-scan:1521' *.remote_login_passwordfile='exclusive' *.sec_case_sensitive_logon=FALSE *.standby_file_management='auto' racdb1.thread=1 racdb2.thread=2 racdb2.undo_tablespace='UNDOTBS1' racdb1.undo_tablespace='UNDOTBS2'

************************************************************************************* after editing the initdrdb.ora file ************************************************************************************* db_cache_size=520093696 java_pool_size=16777216 large_pool_size=16777216 pga_aggregate_target=587202560 sga_target=872415232 shared_pool_size=301989888 streams_pool_size=0 *.audit_file_dest='/u01/app/oracle/admin/drdb/adump' *.audit_trail='db' *.compatible='11.2.0.0.0' *.control_files='+DATA/drdb/controlfile/control01.ctl'

*.db_block_size=8192 *.db_create_file_dest='+DATA' *.db_domain='' *.db_file_name_convert='+DATA/racdb/','+DATA/drdb/' *.db_name='racdb' db_unique_name='drdb' *.db_recovery_file_dest='+DATA' *.db_recovery_file_dest_size=4039114752 *.diagnostic_dest='/u01/app/oracle' *.dispatchers='(PROTOCOL=TCP) (SERVICE=drdbXDB)' fal_client='drdb' *.fal_server='racdb' *.log_archive_config='dg_config=(racdb,drdb)' *.log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ONLINE_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=drdb' *.log_archive_dest_2='SERVICE=racdb LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=racdb' *.log_archive_dest_3='LOCATION=+DATA/drdb/STANDBYLOG VALID_FOR=(STANDBY_LOGFILES,STANDBY_ROLE) DB_UNIQUE_NAME=drdb' *.log_archive_dest_state_1='enable' *.log_archive_dest_state_2='defer' *.log_archive_dest_state_3='enable' *.log_archive_format='%t_ %s_%r.dbf' *.log_archive_max_processes=7 *.log_archive_min_succeed_dest=2 *.open_cursors=300 *.processes=150 *.remote_listener='rac-scan:1521' *.remote_login_passwordfile='exclusive' *.standby_file_management='auto' undo_tablespace='UNDOTBS1' SEC_CASE_SENSITIVE_LOGON=FALSE
======================================================================= 1. Create required folder for drdb database and also add an entry in the /etc/oratab for drdb database. 2. Create a password file for drdb database (must have similar password like primary RAC database). 3. Add required tns entries in the tnsnames.ora and listener entry in the listener.ora file 4. Take a RMAN backup of target database and target database controlfile from any one of the RAC node 5. SCP the backup to the standby database host machine to create standby database. *********************************************************************** [oracle@node1 backup]$ pwd /u01/app/oracle/backup [oracle@node1 backup]$ . oraenv ORACLE_SID = [oracle] ? racdb1 The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1

is /u01/app/oracle [oracle@node1 backup]$ rman target / Recovery Manager: Release 11.2.0.1.0 - Production on Sun Jan 22 04:37:01 2012 Copyright (c) 1982, 2009, Oracle and/or its affiliates. reserved. connected to target database: RACDB (DBID=447762734) RMAN> configure channel device type disk format '/u01/app/oracle/backup/%U'; using target database control file instead of recovery catalog new RMAN configuration parameters: CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '/u01/app/oracle/backup/%U'; new RMAN configuration parameters are successfully stored RMAN> backup as compressed backupset database plus archivelog; RMAN> backup current controlfile for standby; RMAN> backup archivelog all; All rights

Creating directories ================

[oracle@node2 dbs]$ mkdir -p /u01/app/oracle/admin/drdb/adump [oracle@node2 dbs]$ mkdir -p /u01/app/oracle/backup [oracle@node2 dbs]$ . oraenv ORACLE_SID = [+ASM2] ? The Oracle base for ORACLE_HOME=/u01/app/11.2.0/grid is /u01/app/oracle [oracle@node2 dbs]$ asmcmd ASMCMD> ls DATA/ ASMCMD> cd data ASMCMD> mkdir DRDB ASMCMD> cd DRDB

ASMCMD> mkdir ARCHIVELOG CONTROLFILE DATAFILE ONLINELOG PARAMETERFILE STANDBYLOG TEMPFILE ASMCMD> ls ARCHIVELOG/ CONTROLFILE/ DATAFILE/ ONLINELOG/ PARAMETERFILE/ STANDBYLOG/ TEMPFILE/ ASMCMD>

Create password file for standby database

[oracle@node2 dbs]$ orapwd file=$ORACLE_HOME/dbs/orapwdrdb password=oracle entries=5 ignorecase=y [oracle@node2 dbs]$

Copy the backup to standby server

[oracle@node1 backup]$ ls -ltr total 290584 -rw-r----- 1 oracle oinstall 4157952 -rw-r----- 1 oracle oinstall 259842048 -rw-r----- 1 oracle oinstall 1114112 -rw-r----- 1 oracle oinstall 157184 -rw-r----- 1 oracle oinstall 18546688 -rw-r----- 1 oracle oinstall 13416448

Jan Jan Jan Jan Jan Jan

22 22 22 22 22 22

04:38 04:40 04:40 04:40 04:46 04:47

01n1cg5n_1_1 02n1cg5t_1_1 03n1cg9g_1_1 04n1cg9u_1_1 05n1cglv_1_1 06n1cgn8_1_1

[oracle@node1 backup]$ scp * oracle@node2:/u01/app/oracle/backup/ 01n1cg5n_1_1 100% 4061KB 4.0MB/s 00:00 02n1cg5t_1_1 100% 248MB 4.3MB/s 00:58 03n1cg9g_1_1 100% 1088KB 1.1MB/s 00:00 04n1cg9u_1_1 100% 154KB 153.5KB/s 00:00 05n1cglv_1_1 100% 18MB 2.0MB/s 00:09 06n1cgn8_1_1 100% 13MB 2.6MB/s 00:05 [oracle@node1 backup]$ pwd /u01/app/oracle/backup [oracle@node1 backup]$

Add an entry in the /etc/oratab file

vi /etc/oratab drdb:/u01/app/oracle/product/11.2.0/db_1:N

Add tns entry in all nodes

example:[oracle@node1 backup]$ cd /u01/app/oracle/product/11.2.0/db_1/network/admin/ [oracle@node1 admin]$ vi tnsnames.ora RACDB = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = racdb) ) ) DRDB = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = drdb) ) ) [oracle@node1 admin]$ tnsping racdb TNS Ping Utility for Linux: Version 11.2.0.1.0 - Production on 22-JAN2012 04:55:33 Copyright (c) 1997, 2009, Oracle. Used parameter files: Used TNSNAMES adapter to resolve the alias Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = racdb))) OK (40 msec) [oracle@node1 admin]$ tnsping drdb TNS Ping Utility for Linux: Version 11.2.0.1.0 - Production on 22-JAN2012 04:55:37 All rights reserved.

Copyright (c) 1997, 2009, Oracle. Used parameter files:

All rights reserved.

Used TNSNAMES adapter to resolve the alias Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = drdb))) OK (0 msec) [oracle@node2 admin]$ tnsping racdb TNS Ping Utility for Linux: Version 11.2.0.1.0 - Production on 22-JAN2012 04:56:52 Copyright (c) 1997, 2009, Oracle. Used parameter files: Used TNSNAMES adapter to resolve the alias Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = racdb))) OK (0 msec) [oracle@node2 admin]$ tnsping drdb TNS Ping Utility for Linux: Version 11.2.0.1.0 - Production on 22-JAN2012 04:56:56 Copyright (c) 1997, 2009, Oracle. Used parameter files: Used TNSNAMES adapter to resolve the alias Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = drdb))) OK (10 msec) [oracle@node2 admin]$ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ All rights reserved. All rights reserved.

Start the standby database in nomount mode

[oracle@node2 dbs]$ . oraenv ORACLE_SID = [+ASM2] ? drdb The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1 is /u01/app/oracle

[oracle@node2 dbs]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.1.0 Production on Sun Jan 22 05:00:18 2012 Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to an idle instance. SQL> startup nomount [oracle@node2 dbs]$ . oraenv ORACLE_SID = [drdb] ? The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1 is /u01/app/oracle [oracle@node2 dbs]$ rman target sys/oracle@RACDB auxiliary / Recovery Manager: Release 11.2.0.1.0 - Production on Sun Jan 22 05:33:25 2012 Copyright (c) 1982, 2009, Oracle and/or its affiliates. reserved. connected to target database: RACDB (DBID=447762734) connected to auxiliary database: RACDB (not mounted) RMAN> duplicate target database for standby nofilenamecheck; SQL> alter system set log_archive_dest_state_2=enable; System altered. SQL> SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION; Database altered. *********************************************************************** alert log file information +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Using STANDBY_ARCHIVE_DEST parameter default value as +DATA/drdb/standbylog ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=MEMORY; Sun Jan 22 05:52:53 2012 alter database recover managed standby database disconnect from session Attempt to start background Managed Standby Recovery process (drdb) Sun Jan 22 05:52:53 2012 MRP0 started with pid=29, OS id=5844 MRP0: Background Managed Standby Recovery process started (drdb) Serial Media Recovery started All rights

Managed Standby Recovery not using Real Time Apply Waiting for all non-current ORLs to be archived... All non-current ORLs have been archived. Media Recovery Waiting for thread 1 sequence 29 Completed: alter database recover managed standby database disconnect from session Sun Jan 22 05:54:16 2012 alter database recover managed standby database cancel Sun Jan 22 05:54:17 2012 MRP0: Background Media Recovery cancelled with status 16037 Errors in file /u01/app/oracle/diag/rdbms/drdb/drdb/trace/drdb_mrp0_5844.trc: ORA-16037: user requested cancel of managed recovery operation Recovery interrupted! Waiting for MRP0 pid 5844 to terminate Errors in file /u01/app/oracle/diag/rdbms/drdb/drdb/trace/drdb_mrp0_5844.trc: ORA-16037: user requested cancel of managed recovery operation MRP0: Background Media Recovery process shutdown (drdb) Managed Standby Recovery Canceled (drdb) Completed: alter database recover managed standby database cancel ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION Attempt to start background Managed Standby Recovery process (drdb) Sun Jan 22 05:56:07 2012 MRP0 started with pid=29, OS id=6135 MRP0: Background Managed Standby Recovery process started (drdb) Serial Media Recovery started Managed Standby Recovery starting Real Time Apply Waiting for all non-current ORLs to be archived... All non-current ORLs have been archived. Media Recovery Waiting for thread 1 sequence 29 Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION

Completed: ALTER DATABASE MOUNT Errors in file /u01/app/oracle/diag/rdbms/drdb/drdb/trace/drdb_arc1_6584.trc: ORA-01017: invalid username/password; logon denied Error 1017 received logging on to the standby -----------------------------------------------------------Check that the primary and standby are using a password file and remote_login_passwordfile is set to SHARED or EXCLUSIVE, and that the SYS password is same in the password files. returning error ORA-16191 -----------------------------------------------------------FAL[client, ARC1]: Error 16191 connecting to racdb for fetching gap sequence Errors in file /u01/app/oracle/diag/rdbms/drdb/drdb/trace/drdb_arc1_6584.trc: ORA-16191: Primary log shipping client not logged on standby

Errors in file /u01/app/oracle/diag/rdbms/drdb/drdb/trace/drdb_arc1_6584.trc: ORA-16191: Primary log shipping client not logged on standby ARC6: Archival started ARC0: STARTING ARCH PROCESSES COMPLETE

work around:====================================== Create password file again on both rac node database and also on standby database [oracle@node1 dbs]$ orapwd file=$ORACLE_HOME/dbs/orapwracdb1 password=oracle ignorecase=y force=y entries=5 [oracle@node2 dbs]$ orapwd file=$ORACLE_HOME/dbs/orapwracdb2 password=oracle ignorecase=y force=y entries=5 [oracle@node2 dbs]$ orapwd file=$ORACLE_HOME/dbs/orapwdrdb password=oracle ignorecase=y force=y entries=5

Add the following parameter on all databases

SEC_CASE_SENSITIVE_LOGON=FALSE SQL> show parameter SEC_CASE_SENSITIVE_LOGON NAME TYPE VALUE ------------------------------------ ----------- ---------------------sec_case_sensitive_logon boolean FALSE SQL> archive log list; Database log mode Automatic archival Archive destination Oldest online log sequence Next log sequence to archive Current log sequence SQL> shutdown immediate; ORA-01109: database not open Database dismounted. ORACLE instance shut down. Archive Mode Enabled USE_DB_RECOVERY_FILE_DEST 0 0 0

SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options [oracle@node2 dbs]$ vi initdrdb.ora [oracle@node2 dbs]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.1.0 Production on Sun Jan 22 05:59:26 2012 Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to an idle instance. SQL> startup mount ORACLE instance started. Total System Global Area Fixed Size Variable Size Database Buffers Redo Buffers Database mounted. 870457344 1339964 335547844 528482304 5087232 bytes bytes bytes bytes bytes

SQL> archive log list; Database log mode Automatic archival Archive destination Oldest online log sequence Next log sequence to archive Current log sequence

Archive Mode Enabled USE_DB_RECOVERY_FILE_DEST 0 0 0

SQL> show parameter SEC_CASE_SENSITIVE_LOGON=FALSE SQL> show parameter SEC_CASE_SENSITIVE_LOGON NAME TYPE VALUE ------------------------------------ ----------- ---------------------sec_case_sensitive_logon boolean FALSE SQL> archive log list; Database log mode Automatic archival Archive destination Oldest online log sequence Next log sequence to archive Current log sequence

Archive Mode Enabled USE_DB_RECOVERY_FILE_DEST 0 0 0

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION; Database altered. SQL> alter database recover managed standby database cancel; Database altered. SQL> archive log list; Database log mode Automatic archival Archive destination Oldest online log sequence Next log sequence to archive Current log sequence SQL> shutdown immediate; ORA-01109: database not open Database dismounted. ORACLE instance shut down. SQL> startup mount; ORACLE instance started. Total System Global Area Fixed Size Variable Size Database Buffers Redo Buffers Database mounted. SQL> archive log list; Database log mode Automatic archival Archive destination Oldest online log sequence Next log sequence to archive Current log sequence SQL> Archive Mode Enabled USE_DB_RECOVERY_FILE_DEST 37 0 38 870457344 1339964 335547844 528482304 5087232 bytes bytes bytes bytes bytes Archive Mode Enabled USE_DB_RECOVERY_FILE_DEST 0 0 0

Verification ==========
On primary database
SQL> create tablespace new datafile size 2m;

SQL> select name from v$tablespace; NAME -----------------------------SYSTEM SYSAUX UNDOTBS1 USERS TEMP UNDOTBS2 NEW 7 rows selected.

On standby database
SQL> archive log list; Database log mode Automatic archival Archive destination Oldest online log sequence Next log sequence to archive Current log sequence Archive Mode Enabled USE_DB_RECOVERY_FILE_DEST 39 0 40

SQL> select name from v$tablespace; NAME -----------------------------SYSTEM SYSAUX UNDOTBS1 USERS TEMP UNDOTBS2 6 rows selected. SQL> alter database recover managed standby database disconnect from session; Database altered.

On both RAC primary instances

SQL> alter system switch logfile; System altered. SQL> / System altered.

SQL> archive log list; Database log mode Automatic archival Archive destination Oldest online log sequence Next log sequence to archive Current log sequence Archive Mode Enabled USE_DB_RECOVERY_FILE_DEST 39 0 40

SQL> alter database recover managed standby database cancel; Database altered. SQL> archive log list; Database log mode Automatic archival Archive destination Oldest online log sequence Next log sequence to archive Current log sequence Archive Mode Enabled USE_DB_RECOVERY_FILE_DEST 39 0 40

SQL> select name from v$tablespace; NAME -----------------------------SYSTEM SYSAUX UNDOTBS1 USERS TEMP UNDOTBS2 NEW 7 rows selected. SQL> select name, db_unique_name, database_role from v$database; NAME DB_UNIQUE_NAME DATABASE_ROLE --------- ------------------------------ ---------------RACDB drdb PHYSICAL STANDBY