Unable to start and stop database using SRVCTL fails with CRS-215

Visit the Below Website to access unlimited exam questions for all IT vendors and Get Oracle Certifications for FREE
http://www.free-online-exams.com

Problem:    Unable to start and stop database using SRVCTL fails with CRS-215

Symptoms:

Unable to start and stop the database using SRVCTL after

1. installed crs
2. installed oracle home 
3.move database from (db01& db02 ) to ub03 and db03
4. Registered services using srvctl 
5. UNable to start and stop the database using srvctl.
6. start & stop using sqlplus is fine.

Log files:

css4113.log:
2009-07-29 18:58:23.539: [ OCROSD][1]utsz:3: ioctl DKIOCGAPART failed. errno [25] ---> bug 7537050 this patch will be included in 10204 bundle5.

crsd.log:
2009-07-31 14:13:46.405: [ OCRUTL][50]u_freem: mem passed is null
2009-07-31 14:13:47.900: [ CRSRES][8152]32ora.SID.SID2.inst target set to OFFLINE before stop action
2009-07-31 14:13:47.900: [ CRSRES][8152]32StopResource: setting CLI values
2009-07-31 14:13:47.966: [ CRSRES][8152]32Attempting to stop `ora.SID.SID2.inst` on member `ServerName04`
2009-07-31 14:13:48.016: [ CRSRES][8154]32ora.SID.SID1.inst target set to OFFLINE before stop action
2009-07-31 14:13:48.025: [ CRSRES][8154]32StopResource: setting CLI values
2009-07-31 14:13:48.056: [ CRSRES][8154]32Attempting to stop `ora.SID.SID1.inst` on member `ServerName03`
2009-07-31 14:13:48.067: [ CRSEVT][8154]32CAAMonitorHandler :: 0:Action Script for resource 'ora.SID.SID1.inst' stdout redirection failed for `/u01/app/crs/product/10.2.0/crs/log/stopa02149.stdout` : No such file or directory
2009-07-31 14:14:20.830: [ CRSRES][8152]32Stop of `ora.SID.SID2.inst` on member `ServerName04` succeeded.
2009-07-31 14:23:55.782: [ CRSAPP][8154]32StopResource error for ora.SID.SID1.inst error code = 1
2009-07-31 14:23:55.792: [ CRSRES][8154][ALERT]32`ora.SID.SID1.inst` on member `ServerName03` has experienced an unrecoverable failure.
2009-07-31 14:23:55.792: [ CRSRES][8154]32Human intervention required to resume its availability.
2009-07-31 14:25:38.443: [ OCRUTL][46]u_freem: mem passed is null
2009-07-31 14:25:39.153: [ CRSRES][8182]32StopResource: setting CLI values
2009-07-31 14:25:39.175: [ CRSRES][8182]32Attempting to stop `ora.SID.SID1.inst` on member `ServerName03`
2009-07-31 14:25:39.186: [ CRSEVT][8182]32CAAMonitorHandler :: 0:Action Script for resource 'ora.SID.SID1.inst' stdout redirection failed for `/u01/app/crs/product/10.2.0/crs/log/stopa15676.stdout` : No such file or directory
2009-07-31 14:35:46.838: [ CRSAPP][8182]32StopResource error for ora.SID.SID1.inst error code = 1
2009-07-31 14:35:46.849: [ CRSRES][8182]32Stop of `ora.SID.SID1.inst` on member `ServerName03` succeeded.
2009-07-31 14:51:56.381: [ OCRUTL][54]u_freem: mem passed is null
[ OCRAPI][52]a_check_permission_int: Other doesn't have permission
2009-07-31 14:51:56.524: [ OCRSRV][52]proas_lookup_cache: User has no premission to operate on key [CRS.CUR]
[ OCRAPI][54]a_check_permission_int: Other doesn't have permission
2009-07-31 14:51:56.525: [ OCRSRV][54]proas_lookup_cache: User has no premission to operate on key [CRS.HIS]
[ OCRAPI][51]a_check_permission_int: Other doesn't have permission
2009-07-31 14:51:56.527: [ OCRSRV][51]proas_lookup_cache: User has no premission to operate on key [CRS.SEC]
[ OCRAPI][50]a_check_permission_int: Other doesn't have permission
2009-07-31 14:51:56.528: [ OCRSRV][50]proas_lookup_cache: User has no premission to operate on key [CRS.STAGE]

from srvctl_DB.log:
[Worker 0] [19:58:25:817] [HAOperationImpl.runCommand:1223] CRS cmd is: /u01/app/crs/product/10.2.0/bin/crs_start ora.SID.SID1.inst -f -c ServerName03 
[Worker 0] [19:58:25:818] [RuntimeExec.runCommand:74] Calling Runtime.exec() with the command 
[Worker 0] [19:58:25:818] [RuntimeExec.runCommand:76] /u01/app/crs/product/10.2.0/bin/crs_start 
[Worker 0] [19:58:25:818] [RuntimeExec.runCommand:76] ora.SID.SID1.inst 
[Worker 0] [19:58:25:819] [RuntimeExec.runCommand:76] -f 
[Worker 0] [19:58:25:819] [RuntimeExec.runCommand:76] -c 
[Worker 0] [19:58:25:819] [RuntimeExec.runCommand:76] ServerName03 
[Worker 0] [19:58:25:821] [RuntimeExec.runCommand:131] runCommand: Waiting for the process
[Thread-3] [19:58:25:822] [StreamReader.run:61] In StreamReader.run 
[Thread-2] [19:58:25:823] [StreamReader.run:61] In StreamReader.run 
[Thread-3] [19:58:26:130] [StreamReader.run:65] ERROR>Error creating output channel /u01/app/crs/product/10.2.0/crs/log/starta06453.stdout Permission denied
[Thread-3] [19:58:26:130] [StreamReader.run:65] ERROR>Error creating output channel /u01/app/crs/product/10.2.0/crs/log/starta06453.stderr Permission denied
[Thread-2] [19:58:26:279] [StreamReader.run:65] OUTPUT>Attempting to start `ora.SID.SID1.inst` on member `ServerName03`


[Thread-2] [20:18:42:384] [StreamReader.run:65] OUTPUT>`ora.SID.SID1.inst` on member `ServerName03` has experienced an unrecoverable failure.
[Thread-2] [20:18:42:386] [StreamReader.run:65] OUTPUT>Human intervention required to resume its availability.
[Thread-3] [20:18:42:419] [StreamReader.run:65] ERROR>CRS-0215: Could not start resource 'ora.SID.SID1.inst'.
[Thread-3] [20:18:42:420] [StreamReader.run:65] ERROR>
[Worker 0] [20:18:42:576] [RuntimeExec.runCommand:133] runCommand: process returns 115
[Worker 0] [20:18:42:577] [RuntimeExec.runCommand:147] RunTimeExec: output>

Solution:

Follow note

Note.781632.1 "Diagnosing CRS-215 Error When Starting Clusterware Resource"

References:

 Note 844272.1 

Note.781632.1 "Diagnosing CRS-215 Error When Starting Clusterware Resource"

Get Oracle Certifications for all Exams
Free Online Exams.com