Databases shows status is down “There has been a user-initiated shutdown”

Visit the Below Website to access unlimited exam questions for all IT vendors and Get Oracle Certifications for FREE
http://www.free-online-exams.com

Problem:    databases shows status is down “There has been a user-initiated shutdown”

Symptoms:

After Upgrading EMGC to 10.2.0.5, some databases shows status is down with "There has been a user-initiated shutdown." while databases are really UP

Log files:

emagent_perl.trc:
-------------------------
maxRowCount.pl: Mon Jun 15 15:29:04 2009: ERROR: Max Count Value not set properly in file /u01/app/oracle/product/agent10g/sysman/config/esa/database.properties line no. 68

emdctl.trc
-----------------

2008-08-04 13:49:40,163 Thread-1 ERROR main: nmectla_agentctl: Error connecting to http://jctpdb01.dubaiworld.ae:3872/emd/main/. Returning status code 1
2

2008-12-02 07:02:48,343 Thread-1 ERROR http: 6: Unable to initialize ssl connection with server, aborting connection attempt

emoms.trc:
----------------
2009-06-14 16:46:41,588 [EMUI_16_46_41_/console/targets$ctxType=Hosts] ERROR target.TargetData getTargets.958 - ORA-20001: Error in get_metrics_summary: ORA-28112: failed to execute policy function
ORA-06512: at "SYSMAN.EMD_MNTR", line 9666
ORA-06512: at line 1

java.sql.SQLException: ORA-20001: Error in get_metrics_summary: ORA-28112: failed to execute policy function
ORA-06512: at "SYSMAN.EMD_MNTR", line 9666
ORA-06512: at line 1

2009-06-15 15:20:10,871 [EMUI_15_20_10_/console/database/instance/sitemap] ERROR svlt.PageHandler handleRequest.640 - java.lang.NullPointerException
java.lang.NullPointerException

2009-06-15 15:20:10,855 [EMUI_15_20_10_/console/database/instance/sitemap] ERROR perf.sitemap logp.251 - Failed to initialize object: ORA-28112: failed to execute policy function
ORA-06512: at "SYSMAN.EMD_DATABASE", line 1166
ORA-06512: at line 1

2009-06-15 15:00:17,967 [AJPRequestHandler-ApplicationServerThread-957] ERROR eml.OMSHandshake processFailure.843 - OMSHandshake failed.(AGENT URL = https://pebpbp02.dubaiworld.ae:3872/emd/main/)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28112: failed to execute policy function

2009-06-15 14:43:22,151 [EMUI_14_43_22_/console/database/instance/instanceActivity] ERROR perf.instAct logp.251 - oracle.sysman.emo.perf.metric.eng.MetricCollectorTimeoutException

emagent.trc
-------------------
2009-06-15 15:29:03,246 Thread-20 ERROR : (nmecmgr.c,3207):Memory 0x0 encountered, expect struct_id=11011

Solution:

Hitting a Bug 5890966 for DB versions 10.2.0.2 and 10.2.0.3, so we need to apply one-off patch for the DB and another one for the agent

-- To implement the solution, please execute the following steps::
1.Download the needed patch:
- There are patches for Bug 5890966 for DB versions 10.2.0.2 and 10.2.0.3.
- download the one-off Patch 5890966 or upgrade to 10.2.0.4 version for the
target database.
- download Patch 7012242 for the agent.
- Check if there are any errors while accessing the job activity tab in grid
control.
2.Stop the OMS
3.stop the agent
4 .Follow the instructions the in Readme and apply the patchs
5.Restart all EM components and monitor again the error.

References:

    

Get Oracle Certifications for all Exams
Free Online Exams.com

Agent restarts frequently. draws alot of memory/VM on version 10.2.0.4

Visit the Below Website to access unlimited exam questions for all IT vendors and Get Oracle Certifications for FREE
http://www.free-online-exams.com

Problem Description
Agent restarts frequently. draws alot of memory/VM on version 10.2.0.4

Check AGENT_HOME/sysman/log

emagent.trc

2009-06-02 14:02:47,116 Thread-3020 ERROR fetchlets.dms: Error allocating memory for response body
2009-06-02 14:02:47,344 Thread-3020 ERROR fetchlets.dms: Error allocating memory for response body
2009-06-02 14:02:47,441 Thread-3020 ERROR engine: [oc4j,psoa.jdpuap02.dubaiworld.ae_home,oc4j_jms_raw] : nmeegd_GetMetricData failed : Result has repeating key value : jdpuap02.dubaiworld.ae/oc4j_soa:OC4J:12505//JMS
2009-06-02 14:02:47,441 Thread-3020 WARN collector: <nmecmc.c> Error exit. Error message: Result has repeating key value : jdpuap02.dubaiworld.ae/oc4j_soa:OC4J:12505//JMS
2009-06-02 14:02:47,562 Thread-3020 ERROR fetchlets.dms: Error allocating memory for response body
2009-06-02 14:02:47,685 Thread-3020 ERROR fetchlets.dms: Error allocating memory for response body

emagent.nohup

(pid=13735): emagent started successfully
URLTiming: Using SunX509
(pid=13735): signal handler called due to abnormal condition; dumping core due to signal SIGSEGV
----- Tue Jun 2 14:03:55 2009::Checking status of EMAgent : 13735 -----
----- Tue Jun 2 14:03:58 2009::Checking status of EMAgent : 13735 -----
----- Tue Jun 2 14:03:58 2009::EMAgent exited at Tue Jun 2 14:03:58 2009 with signal 6 -----
----- Tue Jun 2 14:03:58 2009::EMAgent has exited due to an internal error -----
----- Tue Jun 2 14:03:58 2009:: - checking for corefile at /u01/app/appsoau/product/grid_agent/agent10g/sysman/emd -----
----- Tue Jun 2 14:03:58 2009::EMAgent coredump found at /u01/app/appsoau/product/grid_agent/agent10g/sysman/emd/core
----- Tue Jun 2 14:03:58 2009::Core file moved to /u01/app/appsoau/product/grid_agent/agent10g/sysman/emd/core_EMAgent_1243937038
----- Tue Jun 2 14:04:00 2009::EMAgent will be restarted due to core dump(via signal 6). -----
----- Tue Jun 2 14:04:00 2009::Restarting EMAgent. -----
----- Tue Jun 2 14:04:00 2009::Agent Launched with PID 18989 at time Tue Jun 2 14:04:00 2009 -----
(pid=18989): starting emagent version 10.2.0.4.0
(pid=18989): emagent started successfully
URLTiming: Using SunX509

 Agent is leaking some memory, hence triggering an unstable state and restarting

The only workaround id to upgrade the agent to 10.2.0.5

Get Oracle Certifications for all Exams
Free Online Exams.com