Sunday, October 16, 2011

OAS 10.1.3.4 HTTP Server not releasing the connections under load The Oracle HTTP Server is continually opening up httpd processes and never terminating them. As a result memory utilization increases and there is performance degradation

Visit the Below Website to access unlimited exam questions for all IT vendors and Get Oracle Certifications for FREE
http://www.free-online-exams.com

Problem: OAS 10.1.3.4 HTTP Server not releasing the connections under load
The Oracle HTTP Server is continually opening up httpd processes and never
terminating them. As a result memory utilization increases and there is
performance degradation

To return service back to normal the HTTP Server has to be stopped and restarted
Symptoms:
After upgraded OAS from 10.1.3.0 to 10.1.3.4 and performed load test before go-live in productions. During load test ,http server is not releasing the connections.

1- Open http://ServerName:1880/server-status

2- From Backend use Apache Bench utility

$ORACLE_HOME/Apache/Apache/bin/ab -n 300 -c 10 http://pdtuap01.dubaiworld.ae:1880/

3- Monitor server-status, you will find that web server is not releasing the connections it will be in Working status for ever until the Apache is restarted.

Different scenarios:

1. Installed OAS 10.1.3.0 and tested with "ab". Issue exists (connection not releasing)
2. Installed 10.1.3.0.0 and applied 10.1.3.5 --- Issue exists
3. Installed 10.1.3.0.0 and applied 10.1.3.4 --- Issue exists -- This is our current production environment
4. Installed 10.1.3.0.0 and applied patch 5302846 --- Issue doesn't exists
5. Installed 10.1.3.0.0 and applied patch 5302846 then applied 10.1.3.5 --- Issue doesn't exists

Log files:
Apache error log file shows this:
--------------
[Thu May 27 12:39:30 2010] [error] mod_oc4j: uri=/webapp, application=application://datatags deployed already, ignore application=datatags
[Thu May 27 12:39:30 2010] [error] mod_oc4j: uri=/.rmiTunnel, application=application://default deployed already, ignore application=default
[Thu May 27 12:39:30 2010] [error] mod_oc4j: uri=/.rmiTunnel, application=application://default deployed already, ignore application=default
[Thu May 27 12:39:30 2010] [error] mod_oc4j: uri=/j2ee, application=application://default deployed already, ignore application=default
[Thu May 27 12:39:30 2010] [error] mod_oc4j: uri=/j2ee, application=application://default deployed already, ignore application=default
[Thu May 27 12:39:30 2010] [error] mod_oc4j: uri=/jmsrouter, application=application://default deployed already, ignore application=default
[Thu May 27 12:39:30 2010] [error] mod_oc4j: uri=/jmsrouter, application=application://default deployed already, ignore application=default
[Thu May 27 12:39:30 2010] [error] mod_oc4j: uri=/JMXSoapAdapter, application=application://system deployed already, ignore application=system
[Thu May 27 12:39:30 2010] [error] mod_oc4j: uri=/JMXSoapAdapter, application=application://system deployed already, ignore application=system
[Thu May 27 12:39:30 2010] [error] mod_oc4j: uri=/dmsoc4j, application=application://system deployed already, ignore application=system
[Thu May 27 12:39:30 2010] [error] mod_oc4j: uri=/dmsoc4j, application=application://system deployed already, ignore application=system

Solution:


Solution based on testing of patch application after upgrading to 10.1.3.4. cause based on details in bug 5223139.
This is a known issue (bug 5223139 HTTP SERVER DOES NOT CLEAR DOWN HTTPD PROCESSES ON SOLARI) which is addressed / fixed in merge patch 5302846

According to Note 392038.1 "Under Load 10.1.3 OHS Unexpectedly Hangs or Performance Degrades, Server-Status Shows Processes In 'W' Status" the 10.1.3.3 and greater contain patch 5302846. However this does not appear to be the case for HP-UX versions. Therefore in order to apply the patch to the 10.1.3.4 instance follow these steps to manually apply the patch:

1. Make a backup copy of the patch directory 5302846

2. Navigate to .../5302846/etc/config

3. edit the two files (actions and inventory) and change the references for 10.1.3.0.0 to 10.1.3.4.0:

actions


inventory



4. Apply the patch to your 10.1.3.4.0 instance
2. In the ulimit setting change the following:
data(kbytes) 1048576
stack(kbytes) 262144
nofiles(descriptors) 2048
to unlimited.

The first two entries in the etc/hosts file should be as follows:
------------------
127.0.0.1 localhost.localdomain localhost

3. Tune HTTP_Server as per the tuning guide in the url:
http://download.oracle.com/docs/cd/B14099_19/core.1012/b14001/opthttp.htm#i1005739
4. edit httpd.conf
Change

KeepAlive On
to
KeepAlive off

- Restart HTTP Server
References:
Note:392038.1 "Under Load 10.1.3 OHS Unexpectedly Hangs or Performance Degrades, Server-Status Shows Processes In 'W' Status"

Note 280165.1 Troubleshooting Oracle HTTP Server Using /server-status
(mod_status)
Get Oracle Certifications for all Exams
Free Online Exams.com

No comments: