emagent consuming very high CPU affecting the performance

Visit the Below Website to access unlimited exam questions for all IT vendors and Get Oracle Certifications for FREE
http://www.free-online-exams.com

Problem: emagent consuming very high CPU affecting the performance

Log files:
emagent.nohup
------------

--- Standalone agent
----- Sun Jul 18 14:31:45 2010::Agent Launched with PID 4583 at time Sun Jul 18 14:31:45 2010 -----
----- Sun Jul 18 14:31:45 2010::Time elapsed between Launch of Watchdog process and execing EMAgent is 2 secs -----
(pid=4583): starting emagent version 10.2.0.5.0
(pid=4583): emagent started successfully
----- Sun Jul 18 14:34:19 2010::Checking status of EMAgent : 4583 -----
(pid=4583): emagent now exiting normally
----- Sun Jul 18 14:34:28 2010::Checking status of EMAgent : 4583 -----
----- Sun Jul 18 14:34:28 2010::EMAgent exited at Sun Jul 18 14:34:28 2010 with return value 0. -----
----- Sun Jul 18 14:34:28 2010::EMAgent was shutdown normally. -----
----- Sun Jul 18 14:34:28 2010::Exiting watchdog loop
-----
--- Standalone agent
----- Sun Jul 18 14:34:44 2010::Agent Launched with PID 6598 at time Sun Jul 18 14:34:44 2010 -----
----- Sun Jul 18 14:34:44 2010::Time elapsed between Launch of Watchdog process and execing EMAgent is 1 secs -----
(pid=6598): starting emagent version 10.2.0.5.0
(pid=6598): emagent started successfully
URLTiming: Using SunX509
(pid=6598): emagent now exiting normally


emagent.trc
-----------
2010-07-18 17:34:32,339 Thread-20274 WARN fetchlets.OJMX: Error communicating with SOAP server using URL: http://jdtpap01.dubaiworld.ae:12508/JMXSoapAdapter/JMXSoapAdapter
2010-07-18 17:34:32,339 Thread-20274 WARN fetchlets.OJMX: SOAP test failed, error:4294967295, host:port:oracle_home:oc4jInstanceName:jvmId:mgmtWebSite is: jdtpap01.dubaiworld.ae::/u01/app/oracle/products/OAS/mid:customs:1:default-web-site
2010-07-18 17:34:32,339 Thread-20274 ERROR engine: [oc4jjvm,DT_PROD_APP1.jdtpap01.dubaiworld.ae_customs_JVM_1,MemoryPools] : nmeegd_GetMetricData failed : Error communicating with server.
2010-07-18 17:34:32,339 Thread-20274 WARN collector: Error exit. Error message: Error communicating with server.
2010-07-18 17:34:32,353 Thread-20274 WARN fetchlets.OJMX: Got HTTP Status: 401
2010-07-18 17:34:32,353 Thread-20274 WARN fetchlets.OJMX: Error communicating with SOAP server using URL: http://jdtpap01.dubaiworld.ae:12508/JMXSoapAdapter/JMXSoapAdapter
2010-07-18 17:34:32,353 Thread-20274 WARN fetchlets.OJMX: SOAP test failed, error:4294967295, host:port:oracle_home:oc4jInstanceName:jvmId:mgmtWebSite is: jdtpap01.dubaiworld.ae::/u01/app/oracle/products/OAS/mid:customs:1:default-web-site
2010-07-18 17:34:32,353 Thread-20274 ERROR engine: [oc4jjvm,DT_PROD_APP1.jdtpap01.dubaiworld.ae_customs_JVM_1,GarbageCollectors] : nmeegd_GetMetricData failed : Error communicating with server.
2010-07-18 17:34:32,353 Thread-20274 WARN collector: Error exit. Error message: Error communicating with server.
2010-07-18 17:34:58,199 Thread-20322 WARN fetchlets.OJMX: Got HTTP Status: 401
2010-07-18 17:34:58,200 Thread-20322 WARN fetchlets.OJMX: Error communicating with SOAP server using URL: http://jdtpap01.dubaiworld.ae:12507/JMXSoapAdapter/JMXSoapAdapter
2010-07-18 17:34:58,200 Thread-20322 WARN fetchlets.OJMX: SOAP test failed, error:4294967295, host:port:oracle_home:oc4jInstanceName:jvmId:mgmtWebSite is: jdtpap01.dubaiworld.ae::/u01/app/oracle/products/OAS/mid:mdtwebserv:1:default-web-site
2010-07-18 17:34:58,200 Thread-20322 ERROR engine: [oc4jjvm,DT_PROD_APP1.jdtpap01.dubaiworld.ae_mdtwebserv_JVM_1,Threads_Raw] : nmeegd_GetMetricData failed : Error communicating with server.
2010-07-18 17:34:58,200 Thread-20322 ERROR engine: [DT_PROD_APP1.jdtpap01.dubaiworld.ae_mdtwebserv_JVM_1,oc4jjvm] Failure in step [0] : Threads_Raw. Error communicating with server.. Stopping execution
2010-07-18 17:34:58,200 Thread-20322 ERROR engine: [DT_PROD_APP1.jdtpap01.dubaiworld.ae_mdtwebserv_JVM_1,oc4jjvm] : Threads : nmeeam_GetMetricData failed
2010-07-18 17:34:58,200 Thread-20322 ERROR engine: [oc4jjvm,DT_PROD_APP1.jdtpap01.dubaiworld.ae_mdtwebserv_JVM_1,Threads] : nmeegd_GetMetricData failed : Error communicating with server.
2010-07-18 17:34:58,200 Thread-20322 WARN collector: Error exit. Error message: Error communicating with server.
2010-07-18 17:34:58,213 Thread-20322 WARN fetchlets.OJMX: Got HTTP Status: 401
2010-07-18 17:34:58,213 Thread-20322 WARN fetchlets.OJMX: Error communicating with SOAP server using URL: http://jdtpap01.dubaiworld.ae:12507/JMXSoapAdapter/JMXSoapAdapter
2010-07-18 17:34:58,213 Thread-20322 WARN fetchlets.OJMX: SOAP test failed, error:4294967295, host:port:oracle_home:oc4jInstanceName:jvmId:mgmtWebSite is: jdtpap01.dubaiworld.ae::/u01/app/oracle/products/OAS/mid:mdtwebserv:1:default-web-site
2010-07-18 17:34:58,213 Thread-20322 ERROR engine: [oc4jjvm,DT_PROD_APP1.jdtpap01.dubaiworld.ae_mdtwebserv_JVM_1,Threads_Raw] : nmeegd_GetMetricData failed : Error communicating with server.
2010-07-18 17:34:58,213 Thread-20322 WARN collector: Error exit. Error message: Error communicating with server.
2010-07-18 17:34:58,226 Thread-20322 WARN fetchlets.OJMX: Got HTTP Status: 401
2010-07-18 17:34:58,226 Thread-20322 WARN fetchlets.OJMX: Error communicating with SOAP server using URL: http://jdtpap01.dubaiworld.ae:12507/JMXSoapAdapter/JMXSoapAdapter
2010-07-18 17:34:58,226 Thread-20322 WARN fetchlets.OJMX: SOAP test failed, error:4294967295, host:port:oracle_home:oc4jInstanceName:jvmId:mgmtWebSite is: jdtpap01.dubaiworld.ae::/u01/app/oracle/products/OAS/mid:mdtwebserv:1:default-web-site
2010-07-18 17:34:58,227 Thread-20322 ERROR engine: [oc4jjvm,DT_PROD_APP1.jdtpap01.dubaiworld.ae_mdtwebserv_JVM_1,DeadlockedThreads] : nmeegd_GetMetricData failed : Error communicating with server.
2010-07-18 17:34:58,227 Thread-20322 WARN collector: Error exit. Error message: Error communicating with server.
2010-07-18 17:35:12,361 Thread-4 WARN http: nmehl_httpListener: signaled to exit from emctl
2010-07-18 17:35:13,371 Thread-4 WARN : Signalled to Exit Normally. Signaled to exit from emctl


Solution:



1. Set the below parameters in the AGENT_HOME/bin/commonenv file:
# Agent memory growth and monitoring
EMAGENT_RECYCLE_MAXMEMORY=1700 #recycle at max memory over 1700 MB
export EMAGENT_RECYCLE_MAXMEMORY
EMAGENT_MAXMEM_INCREASE=10 #recycle if memory grows over 10 MB during check period
export EMAGENT_MAXMEM_INCREASE
EMAGENT_MEMCHECK_HOURS=240 #check memory every 240 hours
export EMAGENT_MEMCHECK_HOURS
+If this file doesn't exist, please create commonenv file using any text editor.
This should be able to make the agent stable.
2. Restart the agent and check if agent still consumes high CPU
2. Apply the patch 6608425 and increase collection interval for oc4jjvm metrics as per note 430366.1
References:


Bug 9270034: AGENT CONSUMES CPU RESOURCES (IN SPIKES) MONITORING APP-TIER TARGETS

Get Oracle Certifications for all Exams
Free Online Exams.com