Monday, June 27, 2011

RAC Troubleshooting: SERVICES ARE GETTING TERMINATED UNEXPECTEDLY

Visit the Below Website to access unlimited exam questions for all IT vendors and Get Oracle Certifications for FREE
http://www.free-online-exams.com

RAC Troubleshooting:  SERVICES ARE GETTING TERMINATED UNEXPECTEDLY on RAC Database

Here what you have to do to troubleshoot RAC services termination unexpectedly

- Check all logs under $ORACLE_HOME/log/<hostname>/racg/*

- /var/log/syslog.log

- Check  /tmp/node1status.txt ,/tmp/node2status.txt after running the following commands :

Node-1
-------
$script /tmp/node1status.txt
$hostname
$crsctl check crs
$crs_stat
$crs_stat -t
$crs_stat -ls
$crs_stat -p
$ps -ef | grep d.bin
$ps -ef | grep crs
$ps -ef | grep oprocd
$olsnodes -n
$olsnodes -v
$/sbin/ifconfig -a
$cat /etc/hosts
$ls -l /etc/hosts
$ls -ld /var/tmp/.oracle
$ls -ld /tmp/.oracle
$srvctl config nodeapps -n <nodename1> -a -g -s -l
$srvctl config nodeapps -n <nodename2> -a -g -s -l
$srvctl config database -d <db_name> -a
$srvctl config service -d <db_name> -a
$srvctl config asm -n <node name>
$ocrcheck
$crsctl query css votedisk
$crsctl query crs activeversion
$crsctl query crs softwareversion
$oifcfg getif
$oifcfg iflist
$olsnodes -p 
$ocrdump run this as root user
$ifconfig -a 
$/etc/hosts 
$exit;

Node-2
--------
$hostname
$script /tmp/node2status.txt
$crsctl check crs
$ocrcheck
$crsctl query crs activeversion
$crsctl query crs softwareversion
$oifcfg getif
$oifcfg iflist
$ps -ef | grep d.bin
$ps -ef | grep crs
$ps -ef | grep oprocd
$/sbin/ifconfig -a
$cat /etc/hosts
$ls -l /etc/hosts
$ls -ld /var/tmp/.oracle
$ls -ld /tmp/.oracle
$exit;
Get Oracle Certifications for all Exams
Free Online Exams.com

Sunday, June 26, 2011

R12: How to change the hostname/IP address of an Database Tier using AutoConfig

Visit the Below Website to access unlimited exam questions for all IT vendors and Get Oracle Certifications for FREE
http://www.free-online-exams.com

Problem Description: R12: How to change the hostname/IP address of Apps  Database Tier using AutoConfig

Solution:

Sometimes we have a need of changing the hostname/domainname/port on database tier. Oracle recommends the following set of action rather than simply changing the context-file and running autoconfig.

1. Deregister the current database server
perl /appsutil/bin/adgentns.pl appspass=APPSpwd \
contextfile=$CONTEXT -removeserver
2. Update the AutoConfig Context files
DB Tier: Create a new context file using the following syntax : 
cd $ORACLE_HOME/appsutil/bin
perl adclonectx.pl contextfile=$CONTEXT_FILE.
3. Shutdown the Applications Tier Services
4. Reseed the Net Services Topology Model
Run AutoConfig on the Database Tier
./adconfig.sh contextfile=$ORACLE_HOME/appsutil/$SID_new_hostname>.xml
Run AutoConfig on the Applications Tier(s)
./adautocfg.sh appspass=appspasswd
5. Shutdown the Database Listener
6. Shutdown the Database 
7. Change the machine hostname 
edit the /etc/hosts file to update the new hostname/domainname.

8. Start the Database
9. Start the DB TNS Listener
10. Start the Applications Tier Services



References:

Interesting links around Autoconfig in the E-Business-Suite [ID 949160.1]
@Note 840828.1 - 'R12: How to change the hostname/IP address of an Applications Tier using AutoConfig'
@Note 840847.1 - 'R12: How to change the hostname and/or port of the Database Tier using AutoConfig'
Get Oracle Certifications for all Exams
Free Online Exams.com

Can't locate strict.pm in @INC (@INC while running adpreclone.pl dbTier

Visit the Below Website to access unlimited exam questions for all IT vendors and Get Oracle Certifications for FREE
http://www.free-online-exams.com

Running adpreclone.pl on the Database tier gave the below error:

oratest@erptest-db $ perl adpreclone.pl dbTier
Can't locate strict.pm in @INC (@INC contains: /oracle/11.2.0/perl/lib/5.8.3 /oracle/11.2.0/perl/lib/site_perl/5.8.3 /oracle/11.2.0/appsutil/perl ../lib/5.10.0/sun4-solaris-thread-multi-64 ../lib/5.10.0 ../lib/site_perl/5.10.0/sun4-solaris-thread-multi-64 ../lib/site_perl/5.10.0 ../lib/5.10.0 ../lib/5.10.0/sun4-solaris-thread-multi-64 ../lib/site_perl .) at adpreclone.pl line 34.
BEGIN failed--compilation aborted at adpreclone.pl line 34.

After upgrading you database from 10g to 11g , perl utility has been upgraded from 5.8.3 to 5.10.0

So in order to resolve this issue, you need to update the database main environment file :
cd $ORACLE_HOME
vi <SID>_hostname.env
and update the
PERL5LIB with the right location of your perl libraries.
from
$ORACLE_HOME/perl/lib/5.8.3 to $ORACLE_HOME/perl/lib/5.10.0

However


If you are using autoconfig, you have to
Login to the Application Manager then click on site map
From the Administration tap , click on AutoConfig from the system configuration area
Click on Edit Parameters for the Database section
From the search , search for title:PERL5LIB and click save
Then run autoconfig for the database.


Get Oracle Certifications for all Exams
Free Online Exams.com

AutoPatch error:ATTENTION: All workers either have failed or are waiting:FAILED: file xxxx on worker x.

Visit the Below Website to access unlimited exam questions for all IT vendors and Get Oracle Certifications for FREE
http://www.free-online-exams.com

AutoPatch error:ATTENTION: All workers either have failed or are waiting:FAILED: file xxxx on worker x.

Whatever listed below is from my own experience for cases and issues I faced while I am working as an Apps DBA .
It does not nessassary to apply on all cases just it will give you a hint or clu to where you should focus.


Here what you are going to do if you face the below error while applying patches using adpatch

ATTENTION: All workers either have failed or are waiting:
FAILED: file xxxx on worker x.
ATTENTION: Please fix the above failed worker(s) so the manager can continue


Check the worker ID log file ( adworkerxx.log ) from $APPL_TOP/admin/$SID/log , the exact issue should be mentioned in it, take an action and resolve the issue then do the following to proceed with the adpatch:


Run adctrl
Choose option 1 "Show Worker Status" which should show the status failed.
Then choose option "2. Tell worker to restart a failed job". Enter the number of worker which you want to restart.
The status will change to "fixed,restart" and the patch will progress.

Get Oracle Certifications for all Exams
Free Online Exams.com

AutoPatch error:adfrjp(): Error running jobs in workers" After your adpatch session disconnects inproperly

Visit the Below Website to access unlimited exam questions for all IT vendors and Get Oracle Certifications for FREE
http://www.free-online-exams.com

Whatever listed below is from my own experience for cases and issues I faced while I am working as an Apps DBA .

It does not nessassary to apply on all cases just it will give you a hint or clu to where you should focus.


While I was trying to reapply a patch using adpatch after my old session got disconnected inproperly ( This includes you close your session by mistake, your PC reboots , network issues),

I ran the adpatch again and select "YES" for continue from previous run so it will resume from the last point befire it got terminated by using the *.rf9 files in $APPL_TOP/admin/$sid/restart


Thats would be in an ideal situation,however sometimes it fails  throwing the following error:

"AutoPatch error:
The worker should not have status 'Running' or 'Restarted' at this point.


Telling workers to quit...


All workers have quit.


AutoPatch error:
adfrjp(): Error running jobs in workers"


What is caused it is an "inconsistency between fnd_install_processes, ad_deferred_jobs and .rf9 files".

What to do next is to use adctrl utility  and select option "1. Show worker status" ==> Show Running
it you try  to quit the worker by hitting option "3. Tell worker to quit" , you will get "Control code already set to 'Quit' for worker xxx".


Here what you going to do to get on the top of this issue:
Hit on option "4. Tell manager that a worker failed its job" from the adctrl utility and type the worker id you saw above to change his status from "Running" to "Failed"
Then tell this gentelman worker to restart his failed job by hitting "2. Tell worker to restart a failed job"


and hopefully will get your issue resolved and it will proceed with your adpatch

Get Oracle Certifications for all Exams
Free Online Exams.com

Friday, June 24, 2011

adrelink is not running

Visit the Below Website to access unlimited exam questions for all IT vendors and Get Oracle Certifications for FREE
http://www.free-online-exams.com
Problem:

Running adrelink gave the below without actually run:

adrelink.sh force=y "ad adpatch"

Usage: adrelink force={y|n} [<optional args>] <targets>
or: adrelink force={y|n} [<optional args>] filelist=<file>

where <targets> = { <product module> [<product module>] ... }
and <product module> = "<product> <module name>"
- <product> should be in lower case
- <module name> should exactly match the executable name

Valid <optional args> are:

envfile=adsetenv.sh
Used only by the 'adsetup' script
link_debug={y|n}
Will we link executables with debug or not? Default is 'n'
backup_mode={none|all|file}
Which executables will we back up when linking with force=y?
Default is 'file'. Meanings are:
- none: do not back up any executables
- all : back up all executables
- file: do what $APPL_TOP/admin/adlinkbk.txt says to do

Type 'adrelink examples' to see some examples of running adrelink.


Solution:


You should use tr utility from  /usr/bin not from this /usr/xpg4/bin/tr as you are running version < 12.1.1
Get Oracle Certifications for all Exams
Free Online Exams.com

adlibout: error: Can't write to lib directory for product ad While applying patches using adpatch

Visit the Below Website to access unlimited exam questions for all IT vendors and Get Oracle Certifications for FREE
http://www.free-online-exams.com
Problem:

While applying patch Patch 7461070 the below error generated:

---ad----------------------------------------
Extracting object modules for product ad...

adlibout: error: Can't write to lib directory for product ad
Directory name is /lib
Directory permissions are drwxr-xr-x

Done extracting requested object modules for product ad.

End of adlibout session
Date/time is Monday 20 June 2011 11:54:22 AM GST
**********************************************************

An error occurred while extracting files from library.
Continue as if it were successful [No] :


Solution:

You should use tr utility from  /usr/bin not from this /usr/xpg4/bin/tr as you are running version < 12.1.1
Get Oracle Certifications for all Exams
Free Online Exams.com