ORA-00600: internal error code, arguments: [2103], [1], [0], [1], [900], [], [], []

Visit the Below Website to access unlimited exam questions for all IT vendors and Get Oracle Certifications for FREE
http://www.free-online-exams.com

Problem:ORA-00600: internal error code, arguments: [2103], [1], [0], [1], [900], [], [], []

10.2.0.3/HP-UX Itanium /11.31
2 node cluster.
Failed to write to control files from one of node

Symptoms:
On 2 node RAC config , from one of the node, all the sessions were hanging when trying to write to the controlfile
Controlfile is placed in the cluster filesystem. At the same time from the other node writes to the controlfile was happening without any issue.

After some time, the arc process got aborted and the arc process was unable to respwan thus caused all sessions from second node to hang.

As a work around I have bounced the second instance.

srvctl couldnt bring up the second instance after bouncing
found that crs_stat was hanging. As a workaround killed the crs daemon. After this srvctl could bring up the second instance

Log files:
Alert Log File:
Sat Dec 25 17:31:36 2010
GES: Potential blocker (pid=12884) on resource CF-00000000-00000000;
enqueue info in file /u01/app/admin/prdsbl/bdump/prdsbl2_lmd0_10853.trc and DIAG trace file
Killing enqueue blocker (pid=12884) on resource CF-00000000-00000000
by killing session 170.21747


Sat Dec 25 17:31:36 2010
Errors in file /u01/app/admin/prdsbl/bdump/prdsbl2_arc0_11225.trc:
ORA-00600: internal error code, arguments: [2103], [1], [0], [1], [900], [], [], []
Sat Dec 25 17:31:36 2010

/* ORA 600 triggered */
Sat Dec 25 17:31:36 2010
Errors in file /u01/app/admin/prdsbl/bdump/prdsbl2_arc0_11225.trc:
ORA-00600: internal error code, arguments: [2103], [1], [0], [1], [900], [], [], []
Sat Dec 25 17:31:36 2010
Errors in file /u01/app/admin/prdsbl/bdump/prdsbl2_arc1_11231.trc:
ORA-00600: internal error code, arguments: [2103], [1], [0], [1], [900], [], [], []
Sat Dec 25 17:31:38 2010
Trace dumping is performing id=[cdmp_20101225173138]
Sat Dec 25 17:31:38 2010
Errors in file /u01/app/admin/prdsbl/bdump/prdsbl2_arc0_11225.trc:
ORA-00600: internal error code, arguments: [2103], [1], [0], [1], [900], [], [], []
Sat Dec 25 17:31:38 2010
Errors in file /u01/app/admin/prdsbl/bdump/prdsbl2_arc1_11231.trc:
ORA-00600: internal error code, arguments: [2103], [1], [0], [1], [900], [], [], []


Solution:



There could be two reasons.


1. Someone is holding CF enqueue on node1 and not allowing others. But this looks very remote condition because if someone is holding CF enqueue then no one on entire database should be able to get the CF enqueue. But here we are able to access (read/write) controlfile from node1.


2. I/O is blocked at the OS level. This looks the most probable cause as I/O calls were failing at the time of problem. This got resolved after instance restart. Was there any node reboot performed at this time?

Workaround is to bounce the instance.

Get Oracle Certifications for all Exams
Free Online Exams.com