Notification of a possible issue with Linux kernel causing high CPU on EM Agents

I wanted to push out a short note in case you are unaware but we’re recently published a blog entry about a possible issue with customer running on Linux after the introduction of the leap-second last weekend 30-Jun-2012.  We’re seeing some reports of customers where the EM Agent (which is Java based now in EM12c) is observed to consume high levels of CPU on hosts where they are running.

We’ve identified this as a bug which was discovered in the Linux kernel which seems to largely affect Java JVM’s causing them to exhibit high levels of CPU.  Not all customers will be affected but initial investigations show that systems which are moderate to higher loaded seem to be more at risk of seeing the problem.
For more details we’ve got a blog post that expands on the details and provides links to the necessary MOS resources to work through this issue.

MEM Utilization on AIX always showing 100%

Mem Util on AIX always shows a value of 100%, which is incorrect.

This is due to a unpublished bug 8526608

MOS Note Performance Tab Memory Utilization On Hosts Targets( monitored by 10.2.0.5 agent ) Shows 100% (Doc ID 849783.1)

This is also true for 11.1 Agents I noticed

I performed the following:

1. cd <agent_home>/sysman/admin/metadata
2. cp host.xml host.xml_20090623
3. edit host.xml and change
.
<ColumnDescriptor NAME=”freeMem” TYPE=”NUMBER” IS_KEY=”FALSE”
TRANSIENT=”TRUE” COMPUTE_EXPR=”(freeMemRaw / 1024.0)” HELP=”NO_HELP”>
.
to
.
<ColumnDescriptor NAME=”freeMem” TYPE=”NUMBER” IS_KEY=”FALSE”
TRANSIENT=”TRUE” COMPUTE_EXPR=”(freeMemRaw )” HELP=”NO_HELP”>
.
.
4. restart agent.

For me this solved the problem

Installing Agent 10.2.0.5.0 and upgrading to 10.2.0.5.2 using OEM Grid Control

After installing OEM GC 10.2.0.5.2 (see my previous post), I did an installation of a new Agent on a different Host, pushing it from the OMS.

After installing the 10.2.0.5.0 Agent, I upgraded the Agent to 10.2.0.5.2 by using the OEM Procedure “Patch Agent”

I have created a document describing all steps I went through

Installing Agent 10.2.0.5.0 and upgrading to 10.2.0.5.2 using OEM Grid Control

Deployment of an Agent – Preparation

To be able to do deployment of an agent with EM the following preparations must be taken care of:

Download the Mass Agent deployment zip file from otn.oracle.com
Oracle Enterprise Manager 10g Grid Control Management Agents (10.2.0.4) for Windows, Linux x86, and Solaris SPARC

On each OMS
Unzip the zip file (Linux_x86_64_Grid_Control_agent_download_10_2_0_4_0.zip) to $OMS_HOME//sysman/agent_download/10.2.0.4.0/.

$ unzip Linux_x86_64_Grid_Control_agent_download_10_2_0_4_0.zip -d /u01/app/oracle/product/10.2.0/oms10g/sysman/agent_download/10.2.0.4.0

$ pwd
/u01/app/oracle/product/10.2.0/oms10g/sysman/agent_download/10.2.0.4.0/linux
oracle@lsrv0004.rf.rabobank.nl:OMS:/u01/app/oracle/product/10.2.0/oms10g/sysman/agent_download/10.2.0.4.0/linux_x64
$ ls -al
total 64
total 64
drwxrwxr-x 8 oracle dba 4096 Apr 11 06:24 .
drwxrwx— 4 oracle dba 4096 Apr 17 09:05 ..
drwxrwxr-x 2 oracle dba 4096 Apr 11 06:24 addons
drwxrwxr-x 4 oracle dba 4096 Apr 11 07:16 agent
drwxrwxr-x 2 oracle dba 4096 Apr 11 06:24 agentdeploy
-rw-r–r– 1 oracle dba 28608 Sep 3 2007 agentDownload.linux_x64
-rw-r–r– 1 oracle dba 165 Jul 21 2006 agent_scripts.jar
drwxrwxr-x 2 oracle dba 4096 Apr 11 06:24 oui
drwxrwxr-x 2 oracle dba 4096 Apr 11 07:00 prereqs
drwxrwxr-x 2 oracle dba 4096 Apr 11 06:24 response

oracle@omsserver1:/u01/app/oracle/product/10.2.0/oms10g/sysman/agent_download/10.2.0.4.0/linux_x64
Modification of configuration files
On each OMS server the following checks should be made:

Check that the file $OMS_OH/sysman/agent_download/10.2.0.4.0/agent_download.rsp contains the correct settings.
#——————————————————————————-
#Name : s_OMSHost
#Datatype : String
#Description: Fully-qualified hostname where the OMS resides
# Example: “foo.us.oracle.com”
#Component : oracle.sysman.top.agent
#——————————————————————————-
s_OMSHost=”omserver1″

#——————————————————————————-
#Name : s_OMSPort
#Datatype : String
#Description: Port number used to communicate with the Management Service
# If port 4889 is in use by another application, please reset.
#Component : oracle.sysman.top.agent
#——————————————————————————-
s_OMSPort=”4889″

Normally these values were set to the correct values, during installation of the OMS.

Check that the file $OMS_OH/sysman/prov/resources/userPaths.properties contains the correct settings:

SSH_PATH=/usr/bin/ssh
SCP_PATH=/usr/bin/scp

Next you need to setup the ssh configuration, allowing to ssh from you OMS server to the servers you want to deploy to:

$ cd $OMS_HOME/sysman/prov/resources/scripts
$ ./sshConnectivity.sh -hosts “