Reducing Downtime While Patching Multi-OMS Environments

Enterprise Manager has now been released for a few weeks, as well as the OMS Bundle patches (also known as System patches). If you plan to apply these bundle patches to your OMS, and you are concerned about the downtime, then, you can reduce the downtime by referring to this whitepaper that contains patching instructions to reduce downtime.

Check this article on Oracle’s blog on EM Reducing Downtime While Patching Multi-OMS Environments

Optimize Oracle WebLogic performance/manageability: Oracle EM12c

Screen Shot 2012-10-01 at 9.34.03 AM

For those of you who are still not aware; the days of EM being just a nice Database monitoring tool are way behind us. Curious to know what managing the complete stack means, might have persuade you to join this session by Nicole Haba (Oracle) and Ishaq Mohammed (Pacific Gas & Electric).


Nicole starts with pointing out some of the challenges we have when trying to monitor and manage the Middleware Stack.

These include:

  • Monitoring & Diagnostics
    • Multiple Domain Discovery
    • Out-of-Box Monitoring
    • Metric Extensions
    • Administration Groups & Template Collections
    • Incident & Problem Management
    • Business Application Management
    • Middleware Diagnostics Advisor
    • JVM Diagnostics
    • Log Viewer
  • Administration
    • Credential Management
    • Record WLST scripts
    • System MBean Browser
    • Configure Log File Settings
    • Change Center
    • JDBC Data Source Management
    • Process Control
    • Selective Tracking
  • Configuration Management
    • Automated Asset Directory
    • Compare Configurations
    • Search Configurations
    • Compliance Management
    • Custom Configurations
    • Synchronize Configurations
    • Config Change History
  • Lifecycle Management
    • Cloning Domains
    • Patch Automation

Nicole introduces some Hints and Tips… :

Like: How can we discover multiple domains in EM? Simply create a csv file containing the specs of all domains to be discovered and use EMCLI to perform the discovery, using the csv as input parameter file.


Using the Middleware Diagnostics Advisor to identify and resolve performance issues


I must admit after conducting several EM12c Bootcamps, I am a MDA fan myself…

Enforcing standards by cloning WebLogic Domains. Exactly as we are used to working with “Golden Images” of Database Homes, the same goes for WebLogic domains.

Uploading binaries to the EM Software Library (a central location in EM to store images like OS, Database, Middleware and Applications) to allow provisioning of these to new servers and by that enforcing standardization.


Next Ishaq Mohammed gives us an insight of the challenges at Pacific Gas & Electric and the way they implemented EM12c to monitor and manage the middleware stack.


Several challenges motivated PG&E to start EM12c implementation, including:

  • 20% increase of WebLogic landscape over next 12 months with same number of administrators
  • Unawareness of performance problems until end users complain
  • Manual deployments of software being much too time-consuming, error prone, leading to inconsistent  non-standard configurations

Key Use cases included:

  • Administration Groups and Template Collections – to enforce standard monitoring settings
  • Middleware Diagnostics Advisor – To allow for fast root cause analysis
  • JVM Diagnostics – For application to disk drilldown
  • Provisioning Profiles and Cloning WebLogic Domains – to enforce standard domain deployment



Using Provisioning Profiles and Cloning of WebLogic Domains allowed for SSL-Enabled WebLogic Domains with OSB deployed as well as SSL-Enabled Weblogic Domain with WebLogic Portal deployed.

It also supports the differentiation of provisioning profiles in case of dev, test or production situations.

Customized Deployment Procedures were created to provision OSB domains and WebLogic Portal domains.

Initially the EM console has been used for launching deployment procedures, to be later replaced with EMCLI.

Ishaq concludes with some final steps finalise the project including: the creation of custom reports using BI Publisher, the patching of WebLogic Servers across multiple domains, the monitoring to Service Level Agreements and the implementation of Business Transaction Management (BTM).

I must say a very interesting customer story appreciated by many in the audience.

Installing Patch Set Update using EM12c

The integration between EM12c and My Oracle Support provides us with a constant inside on available and relevant patches. Based on this integration we can access all patch information in MOS instantly from Enterprise Manager and create a Patch Plan, Analyse included Patches and start the actual deployment to multiple Targets.

In this paper we will see that the installation of a Patch Set Update using EM12c is as easy as 1,2,3

Installing Patch Set Update using EM12c

Integration between OEM Console and My Oracle Support

First announced at Oracle Open World 2009, the integration between Oracle Enterprise Manager and the My Oracle Support website.

This new features allows you to track down Patch recommendations, create Patch Plans, validate Patch Plans for possible conflicts and many more.

Read the Paper I published on this subject

Integration of Enterprise Manager Grid Control and My Oracle Support

Organizations that are facing strict Security Compliance rules and are dealing with 24/7 demands, will find this feature extremely helpful to allow there data enter staff to prepare for necessary patching with a minimum of outage.

One of the major frustrations this feature is dealing with is the situation where administrators have been investing an enormous amount of time in debugging issues, some other customers already dealt with months ago and found a solution for.

Using the Recommended Patching feature helps you to avoid situations like this.

New Recommended Bundle Patch (APR 2010) – 9405592 for Patch Automation on EM

This one I received from Oracle today.

New Recommended Bundle Patch 9405592 is available for download from My Oracle Support now. This patch primarily enhances the Patching functionality offered by Oracle Enterprise Manager Grid Control.

This patch is  cumulative and is a superset of the previously released bundles # 9132461, #8992470, and #8653501 that were released previously, and
therefore, includes all the features that were introduced as part of those Recommended Bundle Patches.

For more information, refer to Comprehensive Overview of Recommended Bundle Patch 9405592 under
support note – OMS and Agent Patches required for setting up Provisioning, Patching and Cloning in to GC [ID 427577.1]


#1 If I had applied the previous recommended patches, do I need to rollback before applying this?
Yes, if you had applied any of the patches (# 9132461, #8992470 and #8653501) you would need to rollback the patch and apply this.
For rollback instructions, refer to the patch README from the support note 427577.1

#2 I recently applied the patch 9132461, do I still need the new patch?
The new patch contains additional bug fixes.  (
For more info see, Comprehensive Overview of Recommended Bundle Patch 9405592)
Augmented Verification and Support for Oracle Database 9i Release 2 ( and Oracle  Databases on Microsoft Windows Platform
Bug fixes resolving issues with patching CPUs on Databases running on Windows platforms.
– Key Bug fixes identified at the various customers
Oracle strongly recommends you to apply the latest patch to make sure your do not encounter these issues and you are at the current patch levels for faster issue resolution through support

#3 Can I apply this patch on top of PSU3 for Enterprise Manager ?
Yes, this patch does NOT conflict with PSU3 and can be applied over it.

#4 Is there any known conflicts?
If you had applied the patch 8573971, it would conflict with this patch(9405592). You would need to rollback the patch 8573971 and apply this Bundle.
Apply the overlay patch –
9583322 to get the fixes of the rolled back patch 8573971. Note: The overlay patch is currently unavailable, it will be made available in few days.

Installing Agent and upgrading to using OEM Grid Control

After installing OEM GC (see my previous post), I did an installation of a new Agent on a different Host, pushing it from the OMS.

After installing the Agent, I upgraded the Agent to by using the OEM Procedure “Patch Agent”

I have created a document describing all steps I went through

Installing Agent and upgrading to using OEM Grid Control

A word of caution to all you folks about uptake on the PSU1 patchset

A word of caution to all you folks about uptake on the PSU1 patchset, if you intend to use your GC to monitor and manage 11gR2 databases using the recently released 11.2 DB support patch (8825226) please be aware that PSU1 and this patches conflict so you can’t have both on your GC simultaneously for the time being.

Support/Dev are working on releasing a version of the 11.2 DB support bundle that will sit on top of + PSU1 but for the moment that is not available.

Based on a comment of Andrew Bulloch (Oracle)

A word of caution to all you folks about uptake on the PSU1 patchset, if you intend to use your GC to monitor and manage 11gR2 databases using the recently released 11.2 DB support patch (8825226) please be aware that PSU1 and this patches conflict so you can’t have both on your GC simultaneously for the time being.

Support/Dev are working on releasing a version of the 11.2 DB support bundle that will sit on top of + PSU1 but for the moment that is not available.

Installation of Patch 8864918 succesful

After upgrading to the latest OPatch version. I managed to install the PSU1 successfully.

There was however 1 issue:
After completing the opatch apply, you need to run the post_deploy_scripts.sql.

This script ends with a message saying the SYSMAN schema has one INVALID object. After querying the repository it seems that Package MGMT_JOB_PARAM_RECORD is invalid.

Metalink Note 842747.1 explains how to solve this problem.
After this all objects in the SYSMAN schema are VALID.

Please read the README.txt for instructions to install this Patch.