Exadata Patching - Compute Node

In earlier posts we have seen how to patch infiniband switch and Cell Server. 

Well, now let's continue with Database Server patching which is pending since long time.

Patching on Database server can be performed serially or in parallel using DCLI utility.

dbnodeupdate.sh utility is used to perform the database server patching

What dbnodeupdate.sh utility does?

• Stop/unlock/disable CRS for host restart
• Perform LVM snapshot backup of / filesystem
• Mount yum ISO image and configure yum
• Apply OS updates via yum
• Relink all Oracle homes for RDS protocol
• Lock GI home and enable CRS upon host restart

Patching:

What Should Be The Order of Exadata Machine Patching?

In earlier post we have given overview for downloading and extract patch on database system.

Now lets have an overview of patching order for Oracle Exadata Machine with list of utility which need to be used to apply it, as it is not compulsory to maintain this order but usually we should follow it to mitigate risk.

We have give the patch location of particular component based on patch number 19625719 (QFSDP).

---------------------------------------------------------------------------------------------

1. InfiniBand Switch 

• It will be available with storage server patches
• First apply it on spine switch than leaf switches

Exadata Patching - Download, Extract, Prepare

Let's go through the Exadata machine patching procedure in bullet points based on our patching experience.

Initially planned to write a article for entire patching activity for all database machine components in single post but it will look bit nasty so thought to write a post for each database machine component separately which would be easy to understand.

Exadata patching activity is very easy task if you plan it well.

Well, let's start from downloading the bundle patch and extract it to compute node. 

1. First go to the Oracle Document ID 888828.1, this is the first read requirement for any exadata database machine patching activity.

DB Node Patching Error : Total amount of free space in the volume group is less than 2GB

Faced below issue while patching on Exadata DB node.

While running dbnodeupdate.sh we got below warning.

./dbnodeupdate.sh -u -l /u01/19625719/Infrastructure/11.2.3.3.1/ExadataDatabaseServer/p18876946_112331_Linux-x86-64.zip -v

Continue ? [Y/n]

y

(*) 2015-02-03 19:33:28: Unzipping helpers (/u01/19625719/Infrastructure/ExadataDBNodeUpdate/3.60/dbupdate-helpers.zip) to /opt/oracle.SupportTools/dbnodeupdate_helpers

(*) 2015-02-03 19:33:28: Initializing logfile /var/log/cellos/dbnodeupdate.log

(*) 2015-02-03 19:33:29: Collecting system configuration details. This may take a while...

(*) 2015-02-03 19:33:56: Validating system details for known issues and best practices. This may take a while...

(*) 2015-02-03 19:33:56: Checking free space in /u01/19625719/Infrastructure/11.2.3.3.1/ExadataDatabaseServer/iso.stage.030215193245

(*) 2015-02-03 19:33:57: Unzipping /u01/19625719/Infrastructure/11.2.3.3.1/ExadataDatabaseServer/p18876946_112331_Linux-x86-64.zip to /u01/19625719/Infrastructure/11.2.3.3.1/ExadataDatabaseServer/iso.stage.030215193240, this may take a while

(*) 2015-02-03 19:34:08: Original /etc/yum.conf moved to /etc/yum.conf.030215193240, generating new yum.conf

(*) 2015-02-03 19:34:08: Generating Exadata repository file /etc/yum.repos.d/Exadata-computenode.repo

Warning: Total amount of free space in the volume group is less than 2GB. Unable to make a snapshot

Warning: Backup functionality will not be provided by dbnodeupdate.sh and need to be made separately

Issue:

Exadata Bundle Patch Bug on 11.2.0.3.18ExadataDatabase

One of our team member has shared resolution of bug while patching on Exadata storage server.

Below is the full story

System landscape:

Exadata Machine = Exadata 1/8th Rack X3-2 machine

Platform = Linux x86-64 

Product = Oracle Database 

Version = 11.2.0.3.18ExadataDatabase 

Here we are doing prechek to apply bundle patch on Exadata. ((Quarterly Full Stack Download Patch (QFSDP) Oct 2014)

Go to the patch directory and first execute cleanup to clean previous patch manager utility.

Here patch directory for storage cell is "/bpatch/19625719/Infrastructure/11.2.3.3.1/ExadataStorageServer_InfiniBandSwitch/patch_11.2.3.3.1.140708/"

#./patchmgr -cells /root/cellgroup -cleanup                                                     
2014-12-06 12:01:05 +0800        :Working: DO: Cleanup ...
2014-12-06 12:01:45 +0800        :SUCCESS: DONE: Cleanup

Executing pre-check through patchmgr utility 

#./patchmgr -cells ~/cellgroup -patch_check_prereq -rolling

2014-12-06 12:09:47 +0800        :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...
2014-12-06 12:10:24 +0800        :SUCCESS: DONE: Initialize files, check space and state of cell services.
2014-12-06 12:10:24 +0800        :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes ...
2014-12-06 12:10:38 +0800 Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes.
2014-12-06 12:10:40 +0800        :SUCCESS: DONE: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction.
2014-12-06 12:10:40 +0800        :Working: DO: Check prerequisites on all cells. Up to 2 minutes ...
2014-12-06 12:12:36 +0800        :SUCCESS: DONE: Check prerequisites on all cells.
2014-12-06 12:12:36 +0800        :Working: DO: Execute plugin check for Patch Check Prereq ...
2014-12-06 12:12:36 +0800 :INFO: Patchmgr plugin start: Prereq check for exposure to bug 17854520 v1.1. Details in logfile /bpatch/19625719/Infrastructure/11.2.3.3.1/ExadataStorageServer_InfiniBandSwitch/patch_11.2.3.3.1.140708/patchmgr.stdout.
2014-12-06 12:12:36 +0800 :INFO: This plugin checks dbhomes across all nodes with oracle-user ssh equivalence, but only for those known to the local system. dbhomes that exist only on remote nodes must be checked manually.
2014-12-06 12:12:55 +0800 :WARNING: ACTION REQUIRED: Required fix for bug 17854520 not found in dbhome /u01/app/oracle/product/11.2.0.3/dbhome_1 on dbadm01. Install the fix before performing rolling cell patching.
2014-12-06 12:12:55 +0800 :WARNING: ACTION REQUIRED: Required fix for bug 17854520 not found in dbhome /u01/app/oracle/product/11.2.0.3/dbhome_1 on dbadm02. Install the fix before performing rolling cell patching.
2014-12-06 12:12:55 +0800 :WARNING: Patchmgr plugin complete: Prereq check failed - fix for bug 17854520 is required
2014-12-06 12:12:56 +0800        :FAILED: Details in files <cell_name>.log /bpatch/19625719/Infrastructure/11.2.3.3.1/ExadataStorageServer_InfiniBandSwitch/patch_11.2.3.3.1.140708/patchmgr.stdout, /bpatch/19625719/Infrastructure/11.2.3.3.1/ExadataStorageServer_InfiniBandSwitch/patch_11.2.3.3.1.140708/patchmgr.stderr
2014-12-06 12:12:56 +0800        :FAILED: DONE: Execute plugin check for Patch Check Prereq.
[ERROR] Patch prerequisite checks failed. Please run cleanup before retrying.

As per above output, pre-check has not been completed successfully and we need to apply bug fix 17854520 on Grid and RDBMS home.


Now let's do the exercise to resolve this issue.

Things To Do Before Applying Bundle Patch On Exadata

One of our reader asked us, what we should take care before applying the bundle patch on Exadata, so thought to write this article where we have mentioned some points based on our patching experience, we can say it's part of bundle patch pre-preparation.

Missing of single point can lead you to difficult stage which we can eliminate by planing better.

               "Always plan for worst to do the best"

• Make sure that you have backup plan for Exadata box connectivity in case you are doing things remotely
• Install VNC client on your workstation
• Install VNC server on Exadata in case if you required GUI. You can also use Xmanager and Xming to take GUI from your workstation

Exadata Patching Strategy

Let's have an overview of Exadata Machine patching.

1. Check latest patches available on Oracle note ID - 888828.1 
2. Download the required patches
3. Review the patch README file
4. Make step by step plan to apply patch for all the component
5. Run Exachk utility before the patch application, analyse the report and correct wherever it's required.
6. Automate the patch application process based on rolling or non-rolling method
7. Apply the patch