Oracle 11gR2 RAC Votedisk and OCR Diskgroup Recovery

时间:2023-03-10 01:53:48
Oracle 11gR2 RAC Votedisk and OCR Diskgroup Recovery

check votedisk and OCR

[root@vzwc1 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3116
Available space (kbytes) : 259004
ID : 1177335321
Device/File Name : +SYSDG
Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded [root@vzwc1 ~]#
[root@vzwc1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 10a35374e0704fadbfdbced90a4dd17d (/dev/asm-diskg) [SYSDG]
2. ONLINE 0bd90c387c0b4fb9bf69e0e2feb7ab99 (/dev/asm-diskh) [SYSDG]
3. ONLINE 61ad9c6d57174fd6bf165513e4aef3f5 (/dev/asm-diski) [SYSDG]
4. ONLINE 1b8753ace6684fa2bfc948686b533416 (/dev/asm-diskj) [SYSDG]
5. ONLINE 71ecbbceeb9a4f7bbf7de7c862e6e049 (/dev/asm-diskk) [SYSDG]
Located 5 voting disk(s).
[root@vzwc1 ~]#
[root@vzwc1 ~]# ocrconfig -showbackup vzwc1 2013/09/20 19:45:02 /u01/app/11.2.0/grid/cdata/vzwc-cluster/backup00.ocr vzwc1 2013/09/20 15:45:02 /u01/app/11.2.0/grid/cdata/vzwc-cluster/backup01.ocr vzwc1 2013/09/20 15:45:02 /u01/app/11.2.0/grid/cdata/vzwc-cluster/day.ocr vzwc1 2013/09/20 15:45:02 /u01/app/11.2.0/grid/cdata/vzwc-cluster/week.ocr
PROT-25: Manual backups for the Oracle Cluster Registry are not available
[root@vzwc1 ~]#

Stop all nodes HAS

[root@vzwc1 ~]# crsctl stop has -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'vzwc1'
CRS-2673: Attempting to stop 'ora.crsd' on 'vzwc1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'vzwc1'
CRS-2673: Attempting to stop 'ora.CRSDG.dg' on 'vzwc1'
CRS-2673: Attempting to stop 'ora.SYSDG.dg' on 'vzwc1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'vzwc1'
CRS-2673: Attempting to stop 'ora.zwc.db' on 'vzwc1'
CRS-2673: Attempting to stop 'ora.cvu' on 'vzwc1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'vzwc1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'vzwc1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'vzwc1'
CRS-2677: Stop of 'ora.cvu' on 'vzwc1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'vzwc2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'vzwc1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'vzwc1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'vzwc1' succeeded
CRS-2673: Attempting to stop 'ora.vzwc1.vip' on 'vzwc1'
CRS-2677: Stop of 'ora.vzwc1.vip' on 'vzwc1' succeeded
CRS-2672: Attempting to start 'ora.vzwc1.vip' on 'vzwc2'
CRS-2676: Start of 'ora.cvu' on 'vzwc2' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'vzwc1' succeeded
CRS-2677: Stop of 'ora.zwc.db' on 'vzwc1' succeeded
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'vzwc1'
CRS-2673: Attempting to stop 'ora.FRADG.dg' on 'vzwc1'
CRS-2677: Stop of 'ora.scan1.vip' on 'vzwc1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'vzwc2'
CRS-2676: Start of 'ora.vzwc1.vip' on 'vzwc2' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'vzwc2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'vzwc2'
CRS-2677: Stop of 'ora.SYSDG.dg' on 'vzwc1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'vzwc2' succeeded
CRS-2677: Stop of 'ora.DATADG.dg' on 'vzwc1' succeeded
CRS-2677: Stop of 'ora.FRADG.dg' on 'vzwc1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'vzwc1' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'vzwc2'
CRS-2676: Start of 'ora.oc4j' on 'vzwc2' succeeded
CRS-2677: Stop of 'ora.CRSDG.dg' on 'vzwc1' succeeded
CRS-2679: Attempting to clean 'ora.CRSDG.dg' on 'vzwc1'
CRS-2681: Clean of 'ora.CRSDG.dg' on 'vzwc1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'vzwc1'
CRS-2677: Stop of 'ora.asm' on 'vzwc1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'vzwc1'
CRS-2677: Stop of 'ora.ons' on 'vzwc1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'vzwc1'
CRS-2677: Stop of 'ora.net1.network' on 'vzwc1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'vzwc1' has completed
CRS-2677: Stop of 'ora.crsd' on 'vzwc1' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'vzwc1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'vzwc1'
CRS-2673: Attempting to stop 'ora.crf' on 'vzwc1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'vzwc1'
CRS-2673: Attempting to stop 'ora.evmd' on 'vzwc1'
CRS-2673: Attempting to stop 'ora.asm' on 'vzwc1'
CRS-2677: Stop of 'ora.mdnsd' on 'vzwc1' succeeded
CRS-2677: Stop of 'ora.crf' on 'vzwc1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'vzwc1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'vzwc1' succeeded
CRS-2677: Stop of 'ora.asm' on 'vzwc1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'vzwc1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'vzwc1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'vzwc1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'vzwc1'
CRS-2677: Stop of 'ora.cssd' on 'vzwc1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'vzwc1'
CRS-2677: Stop of 'ora.gipcd' on 'vzwc1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'vzwc1'
CRS-2677: Stop of 'ora.gpnpd' on 'vzwc1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'vzwc1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@vzwc2 ~]# crsctl stop has -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'vzwc2'
CRS-2673: Attempting to stop 'ora.crsd' on 'vzwc2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'vzwc2'
CRS-2673: Attempting to stop 'ora.oc4j' on 'vzwc2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'vzwc2'
CRS-2673: Attempting to stop 'ora.vzwc1.vip' on 'vzwc2'
CRS-2673: Attempting to stop 'ora.cvu' on 'vzwc2'
CRS-2673: Attempting to stop 'ora.CRSDG.dg' on 'vzwc2'
CRS-2673: Attempting to stop 'ora.SYSDG.dg' on 'vzwc2'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'vzwc2'
CRS-2673: Attempting to stop 'ora.zwc.db' on 'vzwc2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'vzwc2'
CRS-2677: Stop of 'ora.cvu' on 'vzwc2' succeeded
CRS-2677: Stop of 'ora.vzwc1.vip' on 'vzwc2' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'vzwc2' succeeded
CRS-2673: Attempting to stop 'ora.vzwc2.vip' on 'vzwc2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'vzwc2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'vzwc2'
CRS-2677: Stop of 'ora.vzwc2.vip' on 'vzwc2' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'vzwc2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'vzwc2' succeeded
CRS-2677: Stop of 'ora.zwc.db' on 'vzwc2' succeeded
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'vzwc2'
CRS-2673: Attempting to stop 'ora.FRADG.dg' on 'vzwc2'
CRS-2677: Stop of 'ora.DATADG.dg' on 'vzwc2' succeeded
CRS-2677: Stop of 'ora.FRADG.dg' on 'vzwc2' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'vzwc2' succeeded
CRS-2677: Stop of 'ora.SYSDG.dg' on 'vzwc2' succeeded
CRS-2677: Stop of 'ora.CRSDG.dg' on 'vzwc2' succeeded
CRS-2679: Attempting to clean 'ora.CRSDG.dg' on 'vzwc2'
CRS-2681: Clean of 'ora.CRSDG.dg' on 'vzwc2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'vzwc2'
CRS-2677: Stop of 'ora.asm' on 'vzwc2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'vzwc2'
CRS-2677: Stop of 'ora.ons' on 'vzwc2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'vzwc2'
CRS-2677: Stop of 'ora.net1.network' on 'vzwc2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'vzwc2' has completed
CRS-2677: Stop of 'ora.crsd' on 'vzwc2' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'vzwc2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'vzwc2'
CRS-2673: Attempting to stop 'ora.evmd' on 'vzwc2'
CRS-2673: Attempting to stop 'ora.asm' on 'vzwc2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'vzwc2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'vzwc2'
CRS-2677: Stop of 'ora.crf' on 'vzwc2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'vzwc2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'vzwc2' succeeded
CRS-2677: Stop of 'ora.asm' on 'vzwc2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'vzwc2'
CRS-2677: Stop of 'ora.ctssd' on 'vzwc2' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'vzwc2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'vzwc2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'vzwc2' succeeded
CRS-2677: Stop of 'ora.cssd' on 'vzwc2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'vzwc2'
CRS-2677: Stop of 'ora.gipcd' on 'vzwc2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'vzwc2'
CRS-2677: Stop of 'ora.gpnpd' on 'vzwc2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'vzwc2' has completed
CRS-4133: Oracle High Availability Services has been stopped.

Using  "dd " damage +SYSDG diskgroup

[root@vzwc1 ~]# dd if=/dev/zero of=/dev/asm-diskg bs=1024k count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0156924 seconds, 66.8 MB/s
[root@vzwc1 ~]# dd if=/dev/zero of=/dev/asm-diskh bs=1024k count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00590076 seconds, 178 MB/s
[root@vzwc1 ~]# dd if=/dev/zero of=/dev/asm-diski bs=1024k count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00543799 seconds, 193 MB/s
[root@vzwc1 ~]# dd if=/dev/zero of=/dev/asm-diskj bs=1024k count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00634026 seconds, 165 MB/s
[root@vzwc1 ~]# dd if=/dev/zero of=/dev/asm-diskk bs=1024k count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00450158 seconds, 233 MB/s
[root@vzwc1 ~]#

Start  HAS on vzwc1

[root@vzwc1 ~]# crsctl start has
CRS-4123: Oracle High Availability Services has been started.

alertvzwc1.log

[grid@vzwc1 ~]$ tail -f $ORACLE_HOME/log/vzwc1/alertvzwc1.log
2013-09-20 21:48:29.405:
[cssd(32646)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/vzwc1/cssd/ocssd.log
2013-09-20 21:48:44.529:
[cssd(32646)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/vzwc1/cssd/ocssd.log
2013-09-20 21:48:59.553:
[cssd(32646)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/vzwc1/cssd/ocssd.log
2013-09-20 21:49:14.577:
[cssd(32646)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/vzwc1/cssd/ocssd.log
2013-09-20 21:49:29.596:
[cssd(32646)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/vzwc1/cssd/ocssd.log
2013-09-20 21:49:44.621:
[cssd(32646)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/vzwc1/cssd/ocssd.log
2013-09-20 21:49:59.644:
[cssd(32646)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/vzwc1/cssd/ocssd.log
2013-09-20 21:50:14.664:
[cssd(32646)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/vzwc1/cssd/ocssd.log

ocssd.log

[grid@vzwc1 ~]$ tail -f $ORACLE_HOME/log/vzwc1/cssd/ocssd.log
2013-09-20 21:50:54.417: [ CSSD][1086400832]clssgmDiscEndpcl: gipcDestroy 0x3f4e
2013-09-20 21:50:54.596: [ CSSD][1086400832]clssgmExecuteClientRequest(): type(37) size(80) only connect and exit messages are allowed before lease acquisition proc(0x13f88a0) client((nil))
2013-09-20 21:50:54.597: [ CSSD][1086400832]clssgmDeadProc: proc 0x13f88a0
2013-09-20 21:50:54.597: [ CSSD][1086400832]clssgmDestroyProc: cleaning up proc(0x13f88a0) con(0x3e62) skgpid ospid 32580 with 0 clients, refcount 0
2013-09-20 21:50:54.597: [ CSSD][1086400832]clssgmDiscEndpcl: gipcDestroy 0x3e62
2013-09-20 21:50:54.934: [ CSSD][1086400832]clssscSelect: cookie accept request 0xfc5240
2013-09-20 21:50:54.934: [ CSSD][1086400832]clssgmAllocProc: (0x12dfd60) allocated
2013-09-20 21:50:54.935: [ CSSD][1086400832]clssgmClientConnectMsg: properties of cmProc 0x12dfd60 - 1,2,3,4,5
2013-09-20 21:50:54.935: [ CSSD][1086400832]clssgmClientConnectMsg: Connect from con(0x3f8e) proc(0x12dfd60) pid(32580) version 11:2:1:4, properties: 1,2,3,4,5
2013-09-20 21:50:54.935: [ CSSD][1086400832]clssgmClientConnectMsg: msg flags 0x0000
2013-09-20 21:50:55.211: [ CSSD][1086400832]clssscSelect: cookie accept request 0x13f6bc0
2013-09-20 21:50:55.211: [ CSSD][1086400832]clssscevtypSHRCON: getting client with cmproc 0x13f6bc0
2013-09-20 21:50:55.211: [ CSSD][1086400832]clssgmRegisterClient: proc(4/0x13f6bc0), client(244/0x13f7fe0)
2013-09-20 21:50:55.211: [ CSSD][1086400832]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x13f6bc0) client(0x13f7fe0)
2013-09-20 21:50:55.211: [ CSSD][1086400832]clssgmDiscEndpcl: gipcDestroy 0x3fb4
2013-09-20 21:50:55.418: [ CSSD][1086400832]clssscSelect: cookie accept request 0x1384100
2013-09-20 21:50:55.418: [ CSSD][1086400832]clssscevtypSHRCON: getting client with cmproc 0x1384100
2013-09-20 21:50:55.418: [ CSSD][1086400832]clssgmRegisterClient: proc(3/0x1384100), client(246/0x1374320)
2013-09-20 21:50:55.418: [ CSSD][1086400832]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x1384100) client(0x1374320)
2013-09-20 21:50:55.418: [ CSSD][1086400832]clssgmDiscEndpcl: gipcDestroy 0x3fca
2013-09-20 21:50:56.213: [ CSSD][1086400832]clssscSelect: cookie accept request 0x13f6bc0
2013-09-20 21:50:56.213: [ CSSD][1086400832]clssscevtypSHRCON: getting client with cmproc 0x13f6bc0
2013-09-20 21:50:56.213: [ CSSD][1086400832]clssgmRegisterClient: proc(4/0x13f6bc0), client(245/0x14017b0)
2013-09-20 21:50:56.213: [ CSSD][1086400832]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x13f6bc0) client(0x14017b0)
2013-09-20 21:50:56.213: [ CSSD][1086400832]clssgmDiscEndpcl: gipcDestroy 0x3fe0
2013-09-20 21:50:56.420: [ CSSD][1086400832]clssscSelect: cookie accept request 0x1384100
2013-09-20 21:50:56.420: [ CSSD][1086400832]clssscevtypSHRCON: getting client with cmproc 0x1384100
2013-09-20 21:50:56.420: [ CSSD][1086400832]clssgmRegisterClient: proc(3/0x1384100), client(247/0x13f94e0)
2013-09-20 21:50:56.420: [ CSSD][1086400832]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x1384100) client(0x13f94e0)
2013-09-20 21:50:56.420: [ CSSD][1086400832]clssgmDiscEndpcl: gipcDestroy 0x3ff6
2013-09-20 21:50:57.215: [ CSSD][1086400832]clssscSelect: cookie accept request 0x13f6bc0
2013-09-20 21:50:57.215: [ CSSD][1086400832]clssscevtypSHRCON: getting client with cmproc 0x13f6bc0
2013-09-20 21:50:57.215: [ CSSD][1086400832]clssgmRegisterClient: proc(4/0x13f6bc0), client(246/0x148fb30)
2013-09-20 21:50:57.215: [ CSSD][1086400832]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x13f6bc0) client(0x148fb30)
2013-09-20 21:50:57.215: [ CSSD][1086400832]clssgmDiscEndpcl: gipcDestroy 0x400c
2013-09-20 21:50:57.422: [ CSSD][1086400832]clssscSelect: cookie accept request 0x1384100
2013-09-20 21:50:57.422: [ CSSD][1086400832]clssscevtypSHRCON: getting client with cmproc 0x1384100
2013-09-20 21:50:57.422: [ CSSD][1086400832]clssgmRegisterClient: proc(3/0x1384100), client(248/0x1407ae0)
2013-09-20 21:50:57.422: [ CSSD][1086400832]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x1384100) client(0x1407ae0)
2013-09-20 21:50:57.422: [ CSSD][1086400832]clssgmDiscEndpcl: gipcDestroy 0x4022
2013-09-20 21:50:58.217: [ CSSD][1086400832]clssscSelect: cookie accept request 0x13f6bc0
2013-09-20 21:50:58.217: [ CSSD][1086400832]clssscevtypSHRCON: getting client with cmproc 0x13f6bc0
2013-09-20 21:50:58.217: [ CSSD][1086400832]clssgmRegisterClient: proc(4/0x13f6bc0), client(247/0x13f91c0)
2013-09-20 21:50:58.217: [ CSSD][1086400832]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x13f6bc0) client(0x13f91c0)
2013-09-20 21:50:58.217: [ CSSD][1086400832]clssgmDiscEndpcl: gipcDestroy 0x4038
2013-09-20 21:50:58.424: [ CSSD][1086400832]clssscSelect: cookie accept request 0x1384100
2013-09-20 21:50:58.424: [ CSSD][1086400832]clssscevtypSHRCON: getting client with cmproc 0x1384100
2013-09-20 21:50:58.424: [ CSSD][1086400832]clssgmRegisterClient: proc(3/0x1384100), client(249/0x1409770)
2013-09-20 21:50:58.424: [ CSSD][1086400832]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x1384100) client(0x1409770)
2013-09-20 21:50:58.425: [ CSSD][1086400832]clssgmDiscEndpcl: gipcDestroy 0x404e
2013-09-20 21:50:59.218: [ CSSD][1086400832]clssscSelect: cookie accept request 0x13f6bc0
2013-09-20 21:50:59.218: [ CSSD][1086400832]clssscevtypSHRCON: getting client with cmproc 0x13f6bc0
2013-09-20 21:50:59.218: [ CSSD][1086400832]clssgmRegisterClient: proc(4/0x13f6bc0), client(248/0x13665a0)
2013-09-20 21:50:59.218: [ CSSD][1086400832]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x13f6bc0) client(0x13665a0)
2013-09-20 21:50:59.218: [ CSSD][1086400832]clssgmDiscEndpcl: gipcDestroy 0x4064
2013-09-20 21:50:59.426: [ CSSD][1086400832]clssscSelect: cookie accept request 0x1384100
2013-09-20 21:50:59.426: [ CSSD][1086400832]clssscevtypSHRCON: getting client with cmproc 0x1384100
2013-09-20 21:50:59.426: [ CSSD][1086400832]clssgmRegisterClient: proc(3/0x1384100), client(250/0x13fe550)
2013-09-20 21:50:59.426: [ CSSD][1086400832]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x1384100) client(0x13fe550)
2013-09-20 21:50:59.427: [ CSSD][1086400832]clssgmDiscEndpcl: gipcDestroy 0x407a
2013-09-20 21:50:59.603: [ CSSD][1086400832]clssgmExecuteClientRequest(): type(37) size(80) only connect and exit messages are allowed before lease acquisition proc(0x12dfd60) client((nil))
2013-09-20 21:50:59.604: [ CSSD][1086400832]clssgmDeadProc: proc 0x12dfd60
2013-09-20 21:50:59.604: [ CSSD][1086400832]clssgmDestroyProc: cleaning up proc(0x12dfd60) con(0x3f8e) skgpid ospid 32580 with 0 clients, refcount 0
2013-09-20 21:50:59.604: [ CSSD][1086400832]clssgmDiscEndpcl: gipcDestroy 0x3f8e
2013-09-20 21:50:59.701: [ GPNP][1089591616]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] get-profile call to url "ipc://GPNPD_vzwc1" disco "" [f=0 claimed- host: cname: seq: auth:]
2013-09-20 21:50:59.705: [ GPNP][1089591616]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2234] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote "ipc://GPNPD_vzwc1" disco ""
2013-09-20 21:50:59.705: [ CSSD][1089591616]clssnmReadDiscoveryProfile: voting file discovery string(/dev/asm-*)
2013-09-20 21:50:59.705: [ CSSD][1089591616]clssnmvDDiscThread: using discovery string /dev/asm-* for initial discovery
2013-09-20 21:50:59.705: [ SKGFD][1089591616]Discovery with str:/dev/asm-*: 2013-09-20 21:50:59.705: [ SKGFD][1089591616]UFS discovery with :/dev/asm-*: 2013-09-20 21:50:59.705: [ SKGFD][1089591616]Execute glob on the string /dev/asm-*
2013-09-20 21:50:59.705: [ SKGFD][1089591616]running stat on disk:/dev/asm-disko
2013-09-20 21:50:59.705: [ SKGFD][1089591616]running stat on disk:/dev/asm-diskn
2013-09-20 21:50:59.706: [ SKGFD][1089591616]running stat on disk:/dev/asm-diski
2013-09-20 21:50:59.706: [ SKGFD][1089591616]running stat on disk:/dev/asm-diskh
2013-09-20 21:50:59.706: [ SKGFD][1089591616]running stat on disk:/dev/asm-diskg
2013-09-20 21:50:59.707: [ SKGFD][1089591616]running stat on disk:/dev/asm-diskf
2013-09-20 21:50:59.707: [ SKGFD][1089591616]running stat on disk:/dev/asm-diskm
2013-09-20 21:50:59.707: [ SKGFD][1089591616]running stat on disk:/dev/asm-diske
2013-09-20 21:50:59.708: [ SKGFD][1089591616]running stat on disk:/dev/asm-diskl
2013-09-20 21:50:59.708: [ SKGFD][1089591616]running stat on disk:/dev/asm-diskk
2013-09-20 21:50:59.708: [ SKGFD][1089591616]running stat on disk:/dev/asm-diskj
2013-09-20 21:50:59.708: [ SKGFD][1089591616]running stat on disk:/dev/asm-diskd
2013-09-20 21:50:59.709: [ SKGFD][1089591616]running stat on disk:/dev/asm-diskc
2013-09-20 21:50:59.709: [ SKGFD][1089591616]running stat on disk:/dev/asm-diskb
2013-09-20 21:50:59.710: [ SKGFD][1089591616]Fetching UFS disk :/dev/asm-diskb: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]Fetching UFS disk :/dev/asm-diskc: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]Fetching UFS disk :/dev/asm-diskd: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]Fetching UFS disk :/dev/asm-diskj: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]Fetching UFS disk :/dev/asm-diskk: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]Fetching UFS disk :/dev/asm-diskl: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]Fetching UFS disk :/dev/asm-diske: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]Fetching UFS disk :/dev/asm-diskm: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]Fetching UFS disk :/dev/asm-diskf: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]Fetching UFS disk :/dev/asm-diskg: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]Fetching UFS disk :/dev/asm-diskh: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]Fetching UFS disk :/dev/asm-diski: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]Fetching UFS disk :/dev/asm-diskn: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]Fetching UFS disk :/dev/asm-disko: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]OSS discovery with :/dev/asm-*: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]Handle 0x131aa10 from lib :UFS:: for disk :/dev/asm-diskb: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]Handle 0x140cc30 from lib :UFS:: for disk :/dev/asm-diskc: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]Handle 0x13e55c0 from lib :UFS:: for disk :/dev/asm-diskd: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]Handle 0x1319680 from lib :UFS:: for disk :/dev/asm-diskj: 2013-09-20 21:50:59.710: [ SKGFD][1089591616]Handle 0x1319eb0 from lib :UFS:: for disk :/dev/asm-diskk: 2013-09-20 21:50:59.711: [ SKGFD][1089591616]Handle 0x1376c00 from lib :UFS:: for disk :/dev/asm-diskl: 2013-09-20 21:50:59.711: [ SKGFD][1089591616]Handle 0x1377430 from lib :UFS:: for disk :/dev/asm-diske: 2013-09-20 21:50:59.711: [ SKGFD][1089591616]Handle 0x1377c60 from lib :UFS:: for disk :/dev/asm-diskm: 2013-09-20 21:50:59.711: [ SKGFD][1089591616]Handle 0x13a12f0 from lib :UFS:: for disk :/dev/asm-diskf: 2013-09-20 21:50:59.711: [ SKGFD][1089591616]Handle 0x13a1b20 from lib :UFS:: for disk :/dev/asm-diskg: 2013-09-20 21:50:59.712: [ SKGFD][1089591616]Handle 0x13a2350 from lib :UFS:: for disk :/dev/asm-diskh: 2013-09-20 21:50:59.712: [ SKGFD][1089591616]Handle 0x13a2b80 from lib :UFS:: for disk :/dev/asm-diski: 2013-09-20 21:50:59.712: [ SKGFD][1089591616]Handle 0x1368fa0 from lib :UFS:: for disk :/dev/asm-diskn: 2013-09-20 21:50:59.712: [ SKGFD][1089591616]Handle 0x13697d0 from lib :UFS:: for disk :/dev/asm-disko: 2013-09-20 21:50:59.712: [ SKGFD][1089591616]Lib :UFS:: closing handle 0x131aa10 for disk :/dev/asm-diskb: 2013-09-20 21:50:59.712: [ SKGFD][1089591616]Lib :UFS:: closing handle 0x140cc30 for disk :/dev/asm-diskc: 2013-09-20 21:50:59.712: [ SKGFD][1089591616]Lib :UFS:: closing handle 0x13e55c0 for disk :/dev/asm-diskd: 2013-09-20 21:50:59.712: [ SKGFD][1089591616]Lib :UFS:: closing handle 0x1319680 for disk :/dev/asm-diskj: 2013-09-20 21:50:59.712: [ SKGFD][1089591616]Lib :UFS:: closing handle 0x1319eb0 for disk :/dev/asm-diskk: 2013-09-20 21:50:59.712: [ SKGFD][1089591616]Lib :UFS:: closing handle 0x1376c00 for disk :/dev/asm-diskl: 2013-09-20 21:50:59.712: [ SKGFD][1089591616]Lib :UFS:: closing handle 0x1377430 for disk :/dev/asm-diske: 2013-09-20 21:50:59.712: [ SKGFD][1089591616]Lib :UFS:: closing handle 0x1377c60 for disk :/dev/asm-diskm: 2013-09-20 21:50:59.712: [ SKGFD][1089591616]Lib :UFS:: closing handle 0x13a12f0 for disk :/dev/asm-diskf: 2013-09-20 21:50:59.712: [ SKGFD][1089591616]Lib :UFS:: closing handle 0x13a1b20 for disk :/dev/asm-diskg: 2013-09-20 21:50:59.712: [ SKGFD][1089591616]Lib :UFS:: closing handle 0x13a2350 for disk :/dev/asm-diskh: 2013-09-20 21:50:59.712: [ SKGFD][1089591616]Lib :UFS:: closing handle 0x13a2b80 for disk :/dev/asm-diski: 2013-09-20 21:50:59.712: [ SKGFD][1089591616]Lib :UFS:: closing handle 0x1368fa0 for disk :/dev/asm-diskn: 2013-09-20 21:50:59.712: [ SKGFD][1089591616]Lib :UFS:: closing handle 0x13697d0 for disk :/dev/asm-disko: 2013-09-20 21:50:59.712: [ CSSD][1089591616]clssnmvDiskVerify: Successful discovery of 0 disks
2013-09-20 21:50:59.712: [ CSSD][1089591616]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2013-09-20 21:50:59.712: [ CSSD][1089591616]clssnmvFindInitialConfigs: No voting files found
2013-09-20 21:50:59.712: [ CSSD][1089591616](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds
2013-09-20 21:50:59.929: [ CSSD][1086400832]clssscSelect: cookie accept request 0xfc5240
2013-09-20 21:50:59.930: [ CSSD][1086400832]clssgmAllocProc: (0x13f9bd0) allocated
2013-09-20 21:50:59.930: [ CSSD][1086400832]clssgmClientConnectMsg: properties of cmProc 0x13f9bd0 - 1,2,3,4,5
2013-09-20 21:50:59.930: [ CSSD][1086400832]clssgmClientConnectMsg: Connect from con(0x40e2) proc(0x13f9bd0) pid(32580) version 11:2:1:4, properties: 1,2,3,4,5
2013-09-20 21:50:59.930: [ CSSD][1086400832]clssgmClientConnectMsg: msg flags 0x0000
2013-09-20 21:51:00.220: [ CSSD][1086400832]clssscSelect: cookie accept request 0x13f6bc0
2013-09-20 21:51:00.220: [ CSSD][1086400832]clssscevtypSHRCON: getting client with cmproc 0x13f6bc0
2013-09-20 21:51:00.220: [ CSSD][1086400832]clssgmRegisterClient: proc(4/0x13f6bc0), client(249/0x131a120)
2013-09-20 21:51:00.220: [ CSSD][1086400832]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x13f6bc0) client(0x131a120)
2013-09-20 21:51:00.220: [ CSSD][1086400832]clssgmDiscEndpcl: gipcDestroy 0x4108
2013-09-20 21:51:00.428: [ CSSD][1086400832]clssscSelect: cookie accept request 0x1384100
2013-09-20 21:51:00.428: [ CSSD][1086400832]clssscevtypSHRCON: getting client with cmproc 0x1384100
2013-09-20 21:51:00.428: [ CSSD][1086400832]clssgmRegisterClient: proc(3/0x1384100), client(251/0x1319b60)
2013-09-20 21:51:00.428: [ CSSD][1086400832]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x1384100) client(0x1319b60)
2013-09-20 21:51:00.428: [ CSSD][1086400832]clssgmDiscEndpcl: gipcDestroy 0x411e

Useing "exclusive" start cluster

crsctl start crs -excl -nocrs

[root@vzwc1 ~]# crsctl stop has -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'vzwc1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'vzwc1'
CRS-2673: Attempting to stop 'ora.crf' on 'vzwc1'
CRS-2677: Stop of 'ora.mdnsd' on 'vzwc1' succeeded
CRS-2677: Stop of 'ora.crf' on 'vzwc1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'vzwc1'
CRS-2677: Stop of 'ora.gipcd' on 'vzwc1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'vzwc1'
CRS-2677: Stop of 'ora.gpnpd' on 'vzwc1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'vzwc1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@vzwc1 ~]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'vzwc1'
CRS-2676: Start of 'ora.mdnsd' on 'vzwc1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'vzwc1'
CRS-2676: Start of 'ora.gpnpd' on 'vzwc1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'vzwc1'
CRS-2672: Attempting to start 'ora.gipcd' on 'vzwc1'
CRS-2676: Start of 'ora.cssdmonitor' on 'vzwc1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'vzwc1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'vzwc1'
CRS-2672: Attempting to start 'ora.diskmon' on 'vzwc1'
CRS-2676: Start of 'ora.diskmon' on 'vzwc1' succeeded
CRS-2676: Start of 'ora.cssd' on 'vzwc1' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'vzwc1'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'vzwc1'
CRS-2672: Attempting to start 'ora.ctssd' on 'vzwc1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'vzwc1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'vzwc1'
CRS-2676: Start of 'ora.ctssd' on 'vzwc1' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'vzwc1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'vzwc1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'vzwc1'
CRS-2676: Start of 'ora.asm' on 'vzwc1' succeeded
[root@vzwc1 ~]#
[root@vzwc1 ~]# ps -ef|grep grid
grid 964 15251 0 21:54 pts/2 00:00:00 tail -f /u01/app/11.2.0/grid/log/vzwc1/alertvzwc1.log
root 1554 1 1 21:59 ? 00:00:01 /u01/app/11.2.0/grid/bin/ohasd.bin exclusive
grid 1682 1 0 22:00 ? 00:00:00 /u01/app/11.2.0/grid/bin/oraagent.bin
grid 1695 1 0 22:00 ? 00:00:00 /u01/app/11.2.0/grid/bin/mdnsd.bin
grid 1708 1 0 22:00 ? 00:00:00 /u01/app/11.2.0/grid/bin/gpnpd.bin
root 1722 1 0 22:00 ? 00:00:00 /u01/app/11.2.0/grid/bin/cssdmonitor
grid 1724 1 0 22:00 ? 00:00:00 /u01/app/11.2.0/grid/bin/gipcd.bin
root 1747 1 0 22:00 ? 00:00:00 /u01/app/11.2.0/grid/bin/cssdagent
grid 1771 1 0 22:00 ? 00:00:00 /u01/app/11.2.0/grid/bin/ocssd.bin -X
root 1846 1 1 22:00 ? 00:00:00 /u01/app/11.2.0/grid/bin/orarootagent.bin
root 1861 1 0 22:00 ? 00:00:00 /u01/app/11.2.0/grid/bin/octssd.bin
grid 2073 1 0 22:00 ? 00:00:00 asm_pmon_+ASM1
grid 2077 1 0 22:00 ? 00:00:00 asm_psp0_+ASM1
grid 2084 1 3 22:00 ? 00:00:00 asm_vktm_+ASM1
grid 2090 1 0 22:00 ? 00:00:00 asm_gen0_+ASM1
grid 2094 1 0 22:00 ? 00:00:00 asm_diag_+ASM1
grid 2098 1 0 22:00 ? 00:00:00 asm_ping_+ASM1
grid 2102 1 0 22:00 ? 00:00:00 asm_dia0_+ASM1
grid 2106 1 0 22:00 ? 00:00:00 asm_lmon_+ASM1
grid 2110 1 0 22:00 ? 00:00:00 asm_lmd0_+ASM1
grid 2114 1 0 22:00 ? 00:00:00 asm_lms0_+ASM1
grid 2120 1 0 22:00 ? 00:00:00 asm_lmhb_+ASM1
grid 2124 1 0 22:00 ? 00:00:00 asm_mman_+ASM1
grid 2128 1 0 22:00 ? 00:00:00 asm_dbw0_+ASM1
grid 2132 1 0 22:00 ? 00:00:00 asm_lgwr_+ASM1
grid 2136 1 0 22:00 ? 00:00:00 asm_ckpt_+ASM1
grid 2140 1 0 22:00 ? 00:00:00 asm_smon_+ASM1
grid 2144 1 0 22:00 ? 00:00:00 asm_rbal_+ASM1
grid 2148 1 0 22:00 ? 00:00:00 asm_gmon_+ASM1
grid 2152 1 0 22:00 ? 00:00:00 asm_mmon_+ASM1
grid 2156 1 0 22:00 ? 00:00:00 asm_mmnl_+ASM1
grid 2160 1 0 22:00 ? 00:00:00 asm_lck0_+ASM1
grid 2222 1 0 22:00 ? 00:00:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
grid 2236 1 0 22:00 ? 00:00:00 asm_fd00_+ASM1

Rebuild SYSDG diskgroup

[grid@vzwc1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Fri Sep 20 22:02:19 2013

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options SQL> create diskgroup SYSDG high redundancy disk '/dev/asm-diskg','/dev/asm-diskh','/dev/asm-diski','/dev/asm-diskj','/dev/asm-diskk'
2 attribute 'compatible.rdbms'='11.2','compatible.asm'='11.2'; Diskgroup created. SQL>

Recovery OCR

[root@vzwc1 ~]# ocrconfig -restore /u01/app/11.2.0/grid/cdata/vzwc-cluster/backup00.ocr
[root@vzwc1 ~]#
[root@vzwc1 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3116
Available space (kbytes) : 259004
ID : 1177335321
Device/File Name : +SYSDG
Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded [root@vzwc1 ~]#

Recovery votedisk

[grid@vzwc1 ~]$ crsctl query css votedisk
Located 0 voting disk(s).
[grid@vzwc1 ~]$
[grid@vzwc1 ~]$
[grid@vzwc1 ~]$ crsctl replace votedisk +SYSDG
Successful addition of voting disk 400bdba7f2e64f8fbf23f6806f25e4c6.
Successful addition of voting disk 76ff5698d65c4f0dbf7693abd7c33a4b.
Successful addition of voting disk 6410be3052a94ffdbfa2805c73cef4ea.
Successful addition of voting disk 76aedd7a81c74f75bf09bf5020bf340a.
Successful addition of voting disk 6968dea5a6be4f17bf0a6caa8a23d534.
Successfully replaced voting disk group with +SYSDG.
CRS-4266: Voting file(s) successfully replaced
[grid@vzwc1 ~]$
[grid@vzwc1 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 400bdba7f2e64f8fbf23f6806f25e4c6 (/dev/asm-diskg) [SYSDG]
2. ONLINE 76ff5698d65c4f0dbf7693abd7c33a4b (/dev/asm-diskh) [SYSDG]
3. ONLINE 6410be3052a94ffdbfa2805c73cef4ea (/dev/asm-diski) [SYSDG]
4. ONLINE 76aedd7a81c74f75bf09bf5020bf340a (/dev/asm-diskj) [SYSDG]
5. ONLINE 6968dea5a6be4f17bf0a6caa8a23d534 (/dev/asm-diskk) [SYSDG]
Located 5 voting disk(s).

Restart HAS

[root@vzwc1 ~]# crsctl start has
CRS-4123: Oracle High Availability Services has been started.
[root@vzwc1 ~]# crsctl check has
CRS-4638: Oracle High Availability Services is online
[root@vzwc1 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@vzwc1 ~]#
[root@vzwc1 ~]#
[root@vzwc1 ~]#
[root@vzwc1 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3128
Available space (kbytes) : 258992
ID : 1177335321
Device/File Name : +SYSDG
Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded [root@vzwc1 ~]#
[root@vzwc1 ~]# crscrl query css votedisk
-bash: crscrl: command not found
[root@vzwc1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 400bdba7f2e64f8fbf23f6806f25e4c6 (/dev/asm-diskg) [SYSDG]
2. ONLINE 76ff5698d65c4f0dbf7693abd7c33a4b (/dev/asm-diskh) [SYSDG]
3. ONLINE 6410be3052a94ffdbfa2805c73cef4ea (/dev/asm-diski) [SYSDG]
4. ONLINE 76aedd7a81c74f75bf09bf5020bf340a (/dev/asm-diskj) [SYSDG]
5. ONLINE 6968dea5a6be4f17bf0a6caa8a23d534 (/dev/asm-diskk) [SYSDG]
Located 5 voting disk(s).
[root@vzwc1 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRSDG.dg
ONLINE ONLINE vzwc1
ONLINE ONLINE vzwc2
ora.DATADG.dg
ONLINE ONLINE vzwc1
ONLINE ONLINE vzwc2
ora.FRADG.dg
ONLINE ONLINE vzwc1
ONLINE ONLINE vzwc2
ora.LISTENER.lsnr
ONLINE ONLINE vzwc1
ONLINE ONLINE vzwc2
ora.SYSDG.dg
ONLINE ONLINE vzwc1
ONLINE ONLINE vzwc2
ora.asm
ONLINE ONLINE vzwc1 Started
ONLINE ONLINE vzwc2 Started
ora.gsd
OFFLINE OFFLINE vzwc1
OFFLINE OFFLINE vzwc2
ora.net1.network
ONLINE ONLINE vzwc1
ONLINE ONLINE vzwc2
ora.ons
ONLINE ONLINE vzwc1
ONLINE ONLINE vzwc2
ora.registry.acfs
ONLINE ONLINE vzwc1
ONLINE ONLINE vzwc2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE vzwc1
ora.cvu
1 ONLINE ONLINE vzwc1
ora.oc4j
1 ONLINE ONLINE vzwc1
ora.scan1.vip
1 ONLINE ONLINE vzwc1
ora.vzwc1.vip
1 ONLINE ONLINE vzwc1
ora.vzwc2.vip
1 ONLINE ONLINE vzwc2
ora.zwc.db
1 ONLINE ONLINE vzwc1 Open
2 ONLINE ONLINE vzwc2 Open
[root@vzwc1 ~]#
[root@vzwc1 ~]#
[root@vzwc1 ~]# crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.CRSDG.dg ora....up.type 0/5 0/ ONLINE ONLINE vzwc1
ora.DATADG.dg ora....up.type 0/5 0/ ONLINE ONLINE vzwc1
ora.FRADG.dg ora....up.type 0/5 0/ ONLINE ONLINE vzwc1
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE vzwc1
ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE vzwc1
ora.SYSDG.dg ora....up.type 0/5 0/ ONLINE ONLINE vzwc1
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE vzwc1
ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE vzwc1
ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE vzwc1
ora.oc4j ora.oc4j.type 0/1 0/2 ONLINE ONLINE vzwc1
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE vzwc1
ora....ry.acfs ora....fs.type 0/5 0/ ONLINE ONLINE vzwc1
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE vzwc1
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE vzwc1
ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE vzwc1
ora.vzwc1.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.vzwc1.ons application 0/3 0/0 ONLINE ONLINE vzwc1
ora.vzwc1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE vzwc1
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE vzwc2
ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE vzwc2
ora.vzwc2.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.vzwc2.ons application 0/3 0/0 ONLINE ONLINE vzwc2
ora.vzwc2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE vzwc2
ora.zwc.db ora....se.type 0/2 0/1 ONLINE ONLINE vzwc1
[oracle@vzwc1 ~]$ sqlplus scott/tiger@zwc

SQL*Plus: Release 11.2.0.4.0 Production on Fri Sep 20 22:34:23 2013

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options SQL> show user
USER is "SCOTT"