cannot write: Text file busy, Out of memory. Root102.sh fails and crashes the system on 10.2.0.5 crs, ONS GOES OFFLINE WHEN CRS IS UPGRADED FROM 10201-10202-10205

Visit the Below Website to access unlimited exam questions for all IT vendors and Get Oracle Certifications for FREE
http://www.free-online-exams.com

Problem: cannot write: Text file busy, Out of memory. Root102.sh fails and crashes the system on 10.2.0.5 crs, ONS GOES OFFLINE WHEN CRS IS UPGRADED FROM 10201-10202-10205


Symptoms:
Two Node RAC
CRS upgrades to 10205. OUI succesfully completed.
While running root102.sh on one node hangs and crashes the system.

crsctl query crs softwareversion
CRS software version on node [ServerNAme] is [10.2.0.4.0] NOT 10.2.0.5 !!!

/u01/app/crs/product/10g/OPatch/opatch lsinventory
Exception in thread "main" java.lang.NoClassDefFoundError: OsInfo
Invoking OPatch 10.2.0.4.9

Oracle Interim Patch Installer version 10.2.0.4.9
Copyright (c) 2009, Oracle Corporation. All rights reserved.


Oracle Home : /u01/app/crs/product/10g
Central Inventory : /u01/app/oraInventory
from : /var/opt/oracle/oraInst.loc
OPatch version : 10.2.0.4.9
OUI version : 10.2.0.5.0
OUI location : /u01/app/crs/product/10g/oui
Log file location : /u01/app/crs/product/10g/cfgtoollogs/opatch/opatch2010-10-26_09-02-15AM.log

Patch history file: /u01/app/crs/product/10g/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /u01/app/crs/product/10g/cfgtoollogs/opatch/lsinv/lsinventory2010-10-26_09-02-15AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (2):

Oracle Clusterware 10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 4 10.2.0.5.0
There are 2 products installed in this Oracle Home.


There are no Interim patches installed in this Oracle Home.
Log files:
*
After applying 10205, the ons is failed to start with the below error.
I could see that in ocr.config file the port is different.

I had commented the below to avoid the ocr registry check for starting ons.

Ocr registered port for ons is 4948 and in ons.config is 6106. We have not done any changes in the port configuration.


useocr=on


Number of configuration nodes retrieved: 2
0: {node = pdppdb03, port = 4948}
Remote port for local node in local config does not match that from OCR.
Remote port for local node in local config does not match that from OCR.
Number of configuration nodes retrieved: 2
0: {node = pdppdb03, port = 4948}
Remote port for local node in local config does not match that from OCR.
onsctl: ons failed to start
**
/u01/app/crs/product/10g/install/root102.sh

mv: /u01/app/crs/product/10g/./lib/libnnz10.so: cannot write: Text file busy

mv: /u01/app/crs/product/10g/./lib/libclntsh.so.10.1: cannot write: Text file busy

mv: /u01/app/crs/product/10g/./lib/libocr10.so: cannot write: Text file busy

mv: /u01/app/crs/product/10g/./lib/libocrb10.so: cannot write: Text file busy

mv: /u01/app/crs/product/10g/./lib/libocrutl10.so: cannot write: Text file busy

mv: /u01/app/crs/product/10g/./lib/libhasgen10.so: cannot write: Text file busy

Completed patching clusterware files to /u01/app/crs/product/10g

rmdir: /u01/app/crs/product/10g/install/patch102/lib: Directory not empty

rmdir: /u01/app/crs/product/10g/install/patch102: Directory not empty

Relinking some shared libraries.

Out of memory.



Stop.

Out of memory.



Stop.

Relinking of patched files is complete.

Copying reboot_toc to /var/opt/oracle/bin for HPUX

WARNING: directory '/u01/app/crs/product' is not owned by root

WARNING: directory '/u01/app/crs' is not owned by root

WARNING: directory '/u01/app' is not owned by root

Preparing to recopy patched init and RC scripts


Solution:



Ocr registered port for ons is 4948 and in ons.config is 6106

The CRS port was held by other process.

The workaround is:
Workaround is to sync the OCR with the port in ons.config. Once this is done, ons will be back online

Can you try to sync the OCR with the port in ons.config to check if we can bring the ONS online or not?

There is no way we can have the ons.config file automaitcally change.

As per document Doc ID 759895.1 you can do either of the following :

Edit your crs1_ons_config_current.config to have remoteport=4948 that matches OCR registered ONS ports
-or-
Change the OCR using racgons commands (options, remove_config & add-config)to change the ONS port from 4948 to 6200.

Found several internal bugs logged for this issue with 10.2.0.5 and ONS port getting changed problems.

Bug 8925749: 10205 LINUX: ONS GOES OFFLINE WHEN CRS IS UPGRADED FROM 10201-10202-10205
Bug 9653027: HPI-SG-10205: OUTPUT OF "ONSCTL DEBUG" DOESN'T SHOW ANY "SERVER CONNECTIONS"
Bug 9861426: 10.2.0.5.0_LINUX.Z64_100418.100622:-ONS RESOURCES ARE OFFLINE AFTER PATCHING CRS

As per internal updates given in bugs this issue was identified during internal testing ofpatchset install and so was decided to cite itin release notes as known issues.
Workaround remains same as suggested before.
or
Simply modify the ons.configfile as follows:
Replace the line remoteport=value with useocr=on
This workaround also applies when OCR is enabled, by default, for existinginstallations.

You can stop just ONS resource with crs_stop ; then modify ONS (ons.config or racgons) and start ONS back with crs_start ;


References:

Doc ID 759895.1

Get Oracle Certifications for all Exams
Free Online Exams.com