As root
# startx
# xhost +localhost
# su – oracle
$ export DISPLAY=:0
$/u01/app/oracle/database/runInstaller
October 19, 2008
March 17, 2008
RAC summary
Common commands for Oracle 10g RAC in RHEL 4 (Oracle Unbreakable Linux)
ps -ef|grep d.bin crs_stat -t crsctl check crs crs_start crs_stop olsnodes ocrcheck srvctl (Prerequisite: GSD daemon starts): srvctl -h|grep -i config dbca sqlplus cluvfy
CRS services
CRS - Cluster Readiness Service (fencing for Windows) CSS - Cluster Synchronization Service EVM - Event Management Service
Node Applications
GSD - Global Service Daemon (Cache fusion) ONS - Oracle Notification Service (extends EVM) VIP - Virtual IP LSNR - Listener
Storage
ASM + OCFS
Database
Instances
Services - for Load Balance and Transparent Application Failover (TAF)
Redo logs are on the shared disk; each instance has its own Redo logs (for clash recovery)
Different Undo tablespaces for different instances
Overview
In the past, it was not easy to become familiar with Oracle Real Application Clusters (RAC), due to the price of the hardware required for a typical production RAC configuration which makes this goal impossible.
Shared storage file systems, or even cluster file systems (e.g. OCFS2) are primarily used in a storage area network where all nodes directly access the storage on the shared file system. This makes it possible for nodes to fail without affecting access to the file system from the other nodes. Shared disk file systems are normally used in a high-availability cluster.
At the heart of Oracle RAC is a shared disk subsystem. All nodes in the cluster must be able to access all of the data, redo log files, control files and parameter files for all nodes in the cluster. The data disks must be globally available to allow all nodes to access the database. Each node has its own redo log and control files but the other nodes must be able to access them in order to recover that node in the event of a system failure.
Facts Sheet RAC
Here, we present some important Facts around the RAC Architecture.
Oracle RAC databases differ architecturally from single-instance Oracle databases in that each Oracle RAC database instance also has:
- At least one additional thread of redo for each instance
- An instance-specific undo tablespace (there are 2 UNDO Tablespaces for a 2-Node RAC)
All data files, control files, SPFILEs, and redo log files in Oracle RAC environments must reside on cluster-aware shared disks so that all of the cluster database instances can access these storage components.
Oracle recommends that you use one shared server parameter file (SPFILE) with instance-specific entries. Alternatively, you can use a local file system to store instance-specific parameter files (PFILEs).
http://www.akadia.com/services/ora_rac.html
Oracle Services
About Oracle Services
Oracle Database 10g introduces an automatic workload management facility, called services. A service represents the workload of applications with common attributes, performance thresholds, and priorities. A single service can represent an application, multiple applications or a subset of a single application. A single service can be associated with one or more instances of an Oracle RAC database, and a single instance can support multiple services. Services provide a single system image to manage competing applications, and they allow each workload to be managed as a single unit.
To manage workloads, you can define services that you assign to a particular application or to a subset of an application’s operations. You can also use services to manage the workload for different types of work. For example, online users can use one service while batch processing can use a different service and reporting can use yet another service type.
When a user or application connects to a database, Oracle recommends that you use a service for the connection. Oracle Database automatically creates one database service when the database is created. For many installations, this may be all you need. For more flexibility in the management of the workload using the database, Oracle Database enables you to create multiple services and specify which database instances offer the services.
Services are integrated with the Resource Manager, which enables you to restrict the resources that are used by a service within an instance. In addition, Oracle Scheduler jobs can run under a service, as opposed to a specific instance.
Configuring Services for High Availability
When you create a service, you define which instances typically support that service. These are known as the preferred instances for that service. You can also define other instances to support a service if the service’s preferred instance fails. These are known as available instances for a service.
When you specify a preferred instance for a service, the service runs on that instance during standard operation. Oracle Clusterware attempts to ensure that the service always runs on all the preferred instances that have been configured for a service. If the instance fails, the service is randomly relocated to one of the available instances. You can also manually relocate the service to an available instance. If you do not specify preferred or available instances when you create a service, then by default every instance in the Oracle RAC database is a preferred instance for that service.
If a service fails over to an available instance, the service is not moved back to its preferred instance automatically. However, you can automate the relocation of a service to its preferred instance by using a callout. For more information about callouts, see “About FAN Callouts”. An example callout script for relocating services back to their preferred instances is available in the Oracle Real Application Clusters Sample Code section on Oracle Technology Network at http://www.oracle.com/technology/sample_code/products/rac/index.html
You do not have to specify available instances for a service. However, if you configure a preferred instance for a service, but do not specify at least one available instance for the service, then the service does not relocate to another instance if the preferred instance fails.
You can also specify an instance as Not Used. This setting means the service does not run on the instance, even if the preferred instance for the service fails.
http://download.oracle.com/docs/cd/B19306_01/rac.102/b28759/configwlm.htm#CHDJIAJH
Time sync issue results in CRS UNKNOWN or OFFLINE state
If time difference between two-node RAC is over 30 seconds, it will result in abnormal CRS state or Oracle DB reboot repeatedly.
rac1-> ps -ef|grep -i d.bin root 4398 1 0 15:11 ? 00:00:00 /u01/app/oracle/product/10.2.0/crs_1/bin/crsd.bin reboot oracle 5259 4396 6 15:13 ? 00:00:00 /u01/app/oracle/product/10.2.0/crs_1/bin/evmd.bin oracle 5417 5369 6 15:13 ? 00:00:00 /u01/app/oracle/product/10.2.0/crs_1/bin/ocssd.bin
rac2-> crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.devdb.db application ONLINE ONLINE rac2 ora....b1.inst application ONLINE ONLINE rac1 ora....b2.inst application ONLINE ONLINE rac2 ora....SM1.asm application ONLINE ONLINE rac1 ora....C1.lsnr application ONLINE ONLINE rac1 ora.rac1.gsd application ONLINE UNKNOWN rac1 ora.rac1.ons application ONLINE ONLINE rac1 ora.rac1.vip application ONLINE ONLINE rac1 ora....SM2.asm application ONLINE ONLINE rac2 ora....C2.lsnr application ONLINE UNKNOWN rac2 ora.rac2.gsd application ONLINE UNKNOWN rac2 ora.rac2.ons application ONLINE ONLINE rac2 ora.rac2.vip application ONLINE ONLINE rac2
If many applications are UNKNOWN state, config NTP in RHEL 4 (Oracle Unbreakable Linux) by typing “system-config-time” command in terminal, Click “Network Time Protocol” and check “Enable Network Time Protocol” and select NTP server and click “OK” or date -s XX:XX:XX to modify time to sync two nodes.
Sometime you can see, some of the applications are UNKNOWN or OFFLINE.
The crs_stat command gives you the names of the applications, which you might need to shut down some applications manually, in order to shut the whole cluster down and restart it.
rac1-> crs_stat NAME=ora.devdb.db TYPE=application TARGET=ONLINE STATE=ONLINE on rac2 NAME=ora.devdb.devdb1.inst TYPE=application TARGET=ONLINE STATE=ONLINE on rac1 NAME=ora.devdb.devdb2.inst TYPE=application TARGET=ONLINE STATE=ONLINE on rac2 NAME=ora.rac1.ASM1.asm TYPE=application TARGET=ONLINE STATE=ONLINE on rac1 NAME=ora.rac1.LISTENER_RAC1.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on rac1 NAME=ora.rac1.gsd TYPE=application TARGET=ONLINE STATE=UNKNOWN on rac1 NAME=ora.rac1.ons TYPE=application TARGET=ONLINE STATE=ONLINE on rac1 NAME=ora.rac1.vip TYPE=application TARGET=ONLINE STATE=ONLINE on rac1 NAME=ora.rac2.ASM2.asm TYPE=application TARGET=ONLINE STATE=ONLINE on rac2 NAME=ora.rac2.LISTENER_RAC2.lsnr TYPE=application TARGET=ONLINE STATE=UNKNOWN on rac2 NAME=ora.rac2.gsd TYPE=application TARGET=ONLINE STATE=UNKNOWN on rac2 NAME=ora.rac2.ons TYPE=application TARGET=ONLINE STATE=ONLINE on rac2 NAME=ora.rac2.vip TYPE=application TARGET=ONLINE STATE=ONLINE on rac2 rac1-> crs_stop -all Attempting to stop `ora.rac1.ons` on member `rac1` Attempting to stop `ora.rac2.ons` on member `rac2` Attempting to stop `ora.devdb.db` on member `rac2` Stop of `ora.rac2.ons` on member `rac2` succeeded. Stop of `ora.rac1.ons` on member `rac1` succeeded. Stop of `ora.devdb.db` on member `rac2` succeeded. `ora.devdb.devdb2.inst` is already OFFLINE. Attempting to stop `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` Attempting to stop `ora.rac2.ASM2.asm` on member `rac2` Stop of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` succeeded. `ora.devdb.devdb1.inst` is already OFFLINE. Attempting to stop `ora.rac1.ASM1.asm` on member `rac1` Stop of `ora.rac2.ASM2.asm` on member `rac2` succeeded. Attempting to stop `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` Stop of `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` succeeded. Attempting to stop `ora.rac2.vip` on member `rac2` Stop of `ora.rac2.vip` on member `rac2` succeeded. Stop of `ora.rac1.ASM1.asm` on member `rac1` succeeded. Attempting to stop `ora.rac1.vip` on member `rac1` Stop of `ora.rac1.vip` on member `rac1` succeeded. CRS-0216: Could not stop resource 'ora.devdb.devdb1.inst'. CRS-0216: Could not stop resource 'ora.devdb.devdb2.inst'. rac1-> crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.devdb.db application OFFLINE OFFLINE ora....b1.inst application OFFLINE OFFLINE ora....b2.inst application OFFLINE OFFLINE ora....SM1.asm application OFFLINE OFFLINE ora....C1.lsnr application OFFLINE OFFLINE ora.rac1.gsd application ONLINE UNKNOWN rac1 ora.rac1.ons application OFFLINE OFFLINE ora.rac1.vip application OFFLINE OFFLINE ora....SM2.asm application OFFLINE OFFLINE ora....C2.lsnr application OFFLINE OFFLINE ora.rac2.gsd application ONLINE UNKNOWN rac2 ora.rac2.ons application OFFLINE OFFLINE ora.rac2.vip application OFFLINE OFFLINE
rac1-> crs_stop ora.rac1.gsd Attempting to stop `ora.rac1.gsd` on member `rac1` Stop of `ora.rac1.gsd` on member `rac1` succeeded. rac1-> crs_stop ora.rac2.gsd Attempting to stop `ora.rac2.gsd` on member `rac2` Stop of `ora.rac2.gsd` on member `rac2` succeeded.
rac1-> crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.devdb.db application OFFLINE OFFLINE ora....b1.inst application OFFLINE OFFLINE ora....b2.inst application OFFLINE OFFLINE ora....SM1.asm application OFFLINE OFFLINE ora....C1.lsnr application OFFLINE OFFLINE ora.rac1.gsd application OFFLINE OFFLINE ora.rac1.ons application OFFLINE OFFLINE ora.rac1.vip application OFFLINE OFFLINE ora....SM2.asm application OFFLINE OFFLINE ora....C2.lsnr application OFFLINE OFFLINE ora.rac2.gsd application OFFLINE OFFLINE ora.rac2.ons application OFFLINE OFFLINE ora.rac2.vip application OFFLINE OFFLINE
rac1-> crs_start -all Attempting to start `ora.rac1.vip` on member `rac1` Attempting to start `ora.rac2.vip` on member `rac2` Start of `ora.rac1.vip` on member `rac1` succeeded. Start of `ora.rac2.vip` on member `rac2` succeeded. Attempting to start `ora.rac1.ASM1.asm` on member `rac1` Attempting to start `ora.rac2.ASM2.asm` on member `rac2` Start of `ora.rac1.ASM1.asm` on member `rac1` succeeded. Attempting to start `ora.devdb.devdb1.inst` on member `rac1` Start of `ora.rac2.ASM2.asm` on member `rac2` succeeded. Attempting to start `ora.devdb.devdb2.inst` on member `rac2` Start of `ora.devdb.devdb1.inst` on member `rac1` succeeded. Attempting to start `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` Start of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` succeeded. Start of `ora.devdb.devdb2.inst` on member `rac2` succeeded. Attempting to start `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` Start of `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` succeeded. CRS-1002: Resource 'ora.rac1.ons' is already running on member 'rac1' CRS-1002: Resource 'ora.rac2.ons' is already running on member 'rac2' CRS-1002: Resource 'ora.devdb.db' is already running on member 'rac2' Attempting to start `ora.rac1.gsd` on member `rac1` Attempting to start `ora.rac2.gsd` on member `rac2` Start of `ora.rac1.gsd` on member `rac1` succeeded. Start of `ora.rac2.gsd` on member `rac2` succeeded. CRS-0223: Resource 'ora.devdb.db' has placement error. CRS-0223: Resource 'ora.rac1.ons' has placement error. CRS-0223: Resource 'ora.rac2.ons' has placement error.
rac1-> crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.devdb.db application ONLINE ONLINE rac2 ora....b1.inst application ONLINE ONLINE rac1 ora....b2.inst application ONLINE ONLINE rac2 ora....SM1.asm application ONLINE ONLINE rac1 ora....C1.lsnr application ONLINE ONLINE rac1 ora.rac1.gsd application ONLINE ONLINE rac1 ora.rac1.ons application ONLINE ONLINE rac1 ora.rac1.vip application ONLINE ONLINE rac1 ora....SM2.asm application ONLINE ONLINE rac2 ora....C2.lsnr application ONLINE ONLINE rac2 ora.rac2.gsd application ONLINE ONLINE rac2 ora.rac2.ons application ONLINE ONLINE rac2 ora.rac2.vip application ONLINE ONLINE rac2
March 14, 2008
Changing Network Addresses for Oracle 11g RAC
http://download-uk.oracle.com/docs/cd/B28359_01/rac.111/b28255/votocr.htm#BGBGDHFG
March 6, 2008
Enabling Archive Logs in a RAC Environment – (Oracle10g)
Whether a single instance or clustered database, Oracle tracks (logs) all changes to database blocks in online redolog files. In an Oracle RAC environment, each instance will have its own set of online redolog files known as a thread. Each Oracle instance will use its set (group) of online redologs in a circular manner. Once an online redolog fills, Oracle moves to the next one. If the database is in “Archive Log Mode”, Oracle will make a copy of the online redo log before it gets reused. A thread must contain at least two online redologs (or online redolog groups). The same holds true for a single instance configuration. The single instance must contain at least two online redologs (or online redolog groups).
The size of an online redolog file is completely independent of another intances’ redolog size. Although in most configurations the size is the same, it may be different depending on the workload and backup / recovery considerations for each node. It is also worth mentioning that each instance has exclusive write access to its own online redolog files. In a correctly configured RAC environment, however, each instance can read another instance’s current online redolog file to perform instance recovery if that instance was terminated abnormally. It is therefore a requirement that online redo logs be located on a shared storage device (just like the database files).
As already mentioned, Oracle writes to its online redolog files in a circular manner. When the current online redolog fills, Oracle will switch to the next one. To facilitate media recovery, Oracle allows the DBA to put the database into “Archive Log Mode” which makes a copy of the online redolog after it fills (and before it gets reused). This is a process known as archiving.
The Database Creation Assistant (DBCA) allows users to configure a new database to be in archive log mode, however most DBA’s opt to bypass this option. In cases like this where the database is in no archive log mode, it is a simple task to put the database into archive log mode. Note however that this will require a short database outage. From one of the nodes in the Oracle10g RAC configuration, use the following tasks to put a RAC enabled database into archive log mode. For the purpose of this article, I will use the node linux1 which runs the orcl1 instance:
- Login to one of the nodes (i.e. linux1) and disable the cluster instance parameter by setting cluster_database to FALSE from the current instance:
$ sqlplus "/ as sysdba" SQL> alter system set cluster_database=false scope=spfile sid='orcl1';
- Shutdown all instances accessing the clustered database:
$ srvctl stop database -d orcl
- Using the local instance, MOUNT the database:
$ sqlplus "/ as sysdba" SQL> startup mount
- Enable archiving:
SQL> alter database archivelog;
- Re-enable support for clustering by modifying the instance parameter cluster_database to TRUE from the current instance:
SQL> alter system set cluster_database=true scope=spfile sid='orcl1';
- Shutdown the local instance:
SQL> shutdown immediate
- Bring all instance back up using srvctl:
$ srvctl start database -d orcl
- (Optional) Bring any services (i.e. TAF) back up using srvctl:
$ srvctl start service -d orcl
- Login to the local instance and verify Archive Log Mode is enabled:
$ sqlplus "/ as sysdba" SQL> archive log list Database log mode Archive Mode Automatic archival Enabled Archive destination USE_DB_RECOVERY_FILE_DEST Oldest online log sequence 83 Next log sequence to archive 84 Current log sequence 84
After enabling Archive Log Mode, each instance in the RAC configuration can automatically archive redologs!
From http://www.quest-pipelines.com/pipelines/dba/tips.htm#august
Oracle 10g 2-node RAC password file not synchronized
Oracle 10g 10.2.0.3
2 nodes RAC
On node 1:
alter user sys identified by <pw>;
Launch OEM database control to login sys as sysdba to check the changed password.
Also check that changed password file timestamp modification.
