If time difference between two-node RAC is over 30 seconds, it will result in abnormal CRS state or Oracle DB reboot repeatedly.
rac1-> ps -ef|grep -i d.bin root 4398 1 0 15:11 ? 00:00:00 /u01/app/oracle/product/10.2.0/crs_1/bin/crsd.bin reboot oracle 5259 4396 6 15:13 ? 00:00:00 /u01/app/oracle/product/10.2.0/crs_1/bin/evmd.bin oracle 5417 5369 6 15:13 ? 00:00:00 /u01/app/oracle/product/10.2.0/crs_1/bin/ocssd.bin
rac2-> crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.devdb.db application ONLINE ONLINE rac2 ora....b1.inst application ONLINE ONLINE rac1 ora....b2.inst application ONLINE ONLINE rac2 ora....SM1.asm application ONLINE ONLINE rac1 ora....C1.lsnr application ONLINE ONLINE rac1 ora.rac1.gsd application ONLINE UNKNOWN rac1 ora.rac1.ons application ONLINE ONLINE rac1 ora.rac1.vip application ONLINE ONLINE rac1 ora....SM2.asm application ONLINE ONLINE rac2 ora....C2.lsnr application ONLINE UNKNOWN rac2 ora.rac2.gsd application ONLINE UNKNOWN rac2 ora.rac2.ons application ONLINE ONLINE rac2 ora.rac2.vip application ONLINE ONLINE rac2
If many applications are UNKNOWN state, config NTP in RHEL 4 (Oracle Unbreakable Linux) by typing “system-config-time” command in terminal, Click “Network Time Protocol” and check “Enable Network Time Protocol” and select NTP server and click “OK” or date -s XX:XX:XX to modify time to sync two nodes.
Sometime you can see, some of the applications are UNKNOWN or OFFLINE.
The crs_stat command gives you the names of the applications, which you might need to shut down some applications manually, in order to shut the whole cluster down and restart it.
rac1-> crs_stat NAME=ora.devdb.db TYPE=application TARGET=ONLINE STATE=ONLINE on rac2 NAME=ora.devdb.devdb1.inst TYPE=application TARGET=ONLINE STATE=ONLINE on rac1 NAME=ora.devdb.devdb2.inst TYPE=application TARGET=ONLINE STATE=ONLINE on rac2 NAME=ora.rac1.ASM1.asm TYPE=application TARGET=ONLINE STATE=ONLINE on rac1 NAME=ora.rac1.LISTENER_RAC1.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on rac1 NAME=ora.rac1.gsd TYPE=application TARGET=ONLINE STATE=UNKNOWN on rac1 NAME=ora.rac1.ons TYPE=application TARGET=ONLINE STATE=ONLINE on rac1 NAME=ora.rac1.vip TYPE=application TARGET=ONLINE STATE=ONLINE on rac1 NAME=ora.rac2.ASM2.asm TYPE=application TARGET=ONLINE STATE=ONLINE on rac2 NAME=ora.rac2.LISTENER_RAC2.lsnr TYPE=application TARGET=ONLINE STATE=UNKNOWN on rac2 NAME=ora.rac2.gsd TYPE=application TARGET=ONLINE STATE=UNKNOWN on rac2 NAME=ora.rac2.ons TYPE=application TARGET=ONLINE STATE=ONLINE on rac2 NAME=ora.rac2.vip TYPE=application TARGET=ONLINE STATE=ONLINE on rac2 rac1-> crs_stop -all Attempting to stop `ora.rac1.ons` on member `rac1` Attempting to stop `ora.rac2.ons` on member `rac2` Attempting to stop `ora.devdb.db` on member `rac2` Stop of `ora.rac2.ons` on member `rac2` succeeded. Stop of `ora.rac1.ons` on member `rac1` succeeded. Stop of `ora.devdb.db` on member `rac2` succeeded. `ora.devdb.devdb2.inst` is already OFFLINE. Attempting to stop `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` Attempting to stop `ora.rac2.ASM2.asm` on member `rac2` Stop of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` succeeded. `ora.devdb.devdb1.inst` is already OFFLINE. Attempting to stop `ora.rac1.ASM1.asm` on member `rac1` Stop of `ora.rac2.ASM2.asm` on member `rac2` succeeded. Attempting to stop `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` Stop of `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` succeeded. Attempting to stop `ora.rac2.vip` on member `rac2` Stop of `ora.rac2.vip` on member `rac2` succeeded. Stop of `ora.rac1.ASM1.asm` on member `rac1` succeeded. Attempting to stop `ora.rac1.vip` on member `rac1` Stop of `ora.rac1.vip` on member `rac1` succeeded. CRS-0216: Could not stop resource 'ora.devdb.devdb1.inst'. CRS-0216: Could not stop resource 'ora.devdb.devdb2.inst'. rac1-> crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.devdb.db application OFFLINE OFFLINE ora....b1.inst application OFFLINE OFFLINE ora....b2.inst application OFFLINE OFFLINE ora....SM1.asm application OFFLINE OFFLINE ora....C1.lsnr application OFFLINE OFFLINE ora.rac1.gsd application ONLINE UNKNOWN rac1 ora.rac1.ons application OFFLINE OFFLINE ora.rac1.vip application OFFLINE OFFLINE ora....SM2.asm application OFFLINE OFFLINE ora....C2.lsnr application OFFLINE OFFLINE ora.rac2.gsd application ONLINE UNKNOWN rac2 ora.rac2.ons application OFFLINE OFFLINE ora.rac2.vip application OFFLINE OFFLINE
rac1-> crs_stop ora.rac1.gsd Attempting to stop `ora.rac1.gsd` on member `rac1` Stop of `ora.rac1.gsd` on member `rac1` succeeded. rac1-> crs_stop ora.rac2.gsd Attempting to stop `ora.rac2.gsd` on member `rac2` Stop of `ora.rac2.gsd` on member `rac2` succeeded.
rac1-> crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.devdb.db application OFFLINE OFFLINE ora....b1.inst application OFFLINE OFFLINE ora....b2.inst application OFFLINE OFFLINE ora....SM1.asm application OFFLINE OFFLINE ora....C1.lsnr application OFFLINE OFFLINE ora.rac1.gsd application OFFLINE OFFLINE ora.rac1.ons application OFFLINE OFFLINE ora.rac1.vip application OFFLINE OFFLINE ora....SM2.asm application OFFLINE OFFLINE ora....C2.lsnr application OFFLINE OFFLINE ora.rac2.gsd application OFFLINE OFFLINE ora.rac2.ons application OFFLINE OFFLINE ora.rac2.vip application OFFLINE OFFLINE
rac1-> crs_start -all Attempting to start `ora.rac1.vip` on member `rac1` Attempting to start `ora.rac2.vip` on member `rac2` Start of `ora.rac1.vip` on member `rac1` succeeded. Start of `ora.rac2.vip` on member `rac2` succeeded. Attempting to start `ora.rac1.ASM1.asm` on member `rac1` Attempting to start `ora.rac2.ASM2.asm` on member `rac2` Start of `ora.rac1.ASM1.asm` on member `rac1` succeeded. Attempting to start `ora.devdb.devdb1.inst` on member `rac1` Start of `ora.rac2.ASM2.asm` on member `rac2` succeeded. Attempting to start `ora.devdb.devdb2.inst` on member `rac2` Start of `ora.devdb.devdb1.inst` on member `rac1` succeeded. Attempting to start `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` Start of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` succeeded. Start of `ora.devdb.devdb2.inst` on member `rac2` succeeded. Attempting to start `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` Start of `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` succeeded. CRS-1002: Resource 'ora.rac1.ons' is already running on member 'rac1' CRS-1002: Resource 'ora.rac2.ons' is already running on member 'rac2' CRS-1002: Resource 'ora.devdb.db' is already running on member 'rac2' Attempting to start `ora.rac1.gsd` on member `rac1` Attempting to start `ora.rac2.gsd` on member `rac2` Start of `ora.rac1.gsd` on member `rac1` succeeded. Start of `ora.rac2.gsd` on member `rac2` succeeded. CRS-0223: Resource 'ora.devdb.db' has placement error. CRS-0223: Resource 'ora.rac1.ons' has placement error. CRS-0223: Resource 'ora.rac2.ons' has placement error.
rac1-> crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.devdb.db application ONLINE ONLINE rac2 ora....b1.inst application ONLINE ONLINE rac1 ora....b2.inst application ONLINE ONLINE rac2 ora....SM1.asm application ONLINE ONLINE rac1 ora....C1.lsnr application ONLINE ONLINE rac1 ora.rac1.gsd application ONLINE ONLINE rac1 ora.rac1.ons application ONLINE ONLINE rac1 ora.rac1.vip application ONLINE ONLINE rac1 ora....SM2.asm application ONLINE ONLINE rac2 ora....C2.lsnr application ONLINE ONLINE rac2 ora.rac2.gsd application ONLINE ONLINE rac2 ora.rac2.ons application ONLINE ONLINE rac2 ora.rac2.vip application ONLINE ONLINE rac2
Thank you Jennifer ,
really this post was very much useful to me
because I couldn’t find this information else where
even in Oracle Manuals
so really interesting to know that:
If time difference between two-node RAC is over 30 seconds, it will result in abnormal CRS state or Oracle DB reboot repeatedly
Thank you again for saving me lots of time.
Best regards
Naji Ghanim
Comment by Naji Ghanim — July 14, 2008 @ 1:41 pm |
It’s really very use full info thanks it’s helped me to resolve same kind of issue with time differances in both RAC nodes.
Thanks a lot .
K.R.RAJU
Comment by K.R.RAJU — January 6, 2009 @ 2:09 pm |