Exadata Quick Reference 3: Oracle Exadata Patching Commands

Compute Node Patching: 

Patching on Database server can be performed serially or in parallel using DCLI utility.

dbnodeupdate.sh utility is used to perform the database server patching

What dbnodeupdate.sh utility does?

• Stop/unlock/disable CRS for host restart
• Perform LVM snapshot backup of / filesystem
• Mount yum ISO image and configure yum
• Apply OS updates via yum
• Relink all Oracle homes for RDS protocol
• Lock GI home and enable CRS upon host restart

Top 5 Critical Factors To Determine Database Recovery Objectives

Recovery of the database is the critical part of the backup and recovery strategy. Any database backup strategy is mainly governed by recovery requirements.

Following are the top critical factor which plays very important role while planing the backup and recovery strategy for Oracle database backup.

How To Change GRUB Password On Exadata Database Server?

Let's first understand what is GRUB?

GRUB is Grand Unified Boot loader. It is a boot loader package from the GNU Project. GRUB is the reference implementation of the Free Software Foundation's Multiboot Specification, which provides a user the choice to boot one of multiple operating systems installed on a computer or select a specific kernel configuration available on a particular operating system's partitions.

CELLSRVSTAT in Exadata Cell Storage

Cellsrvstat is very useful utility to get the cell level statastics for all the logical components of  cell like memory,io,smartio,flashcache etc...

Cellsrvstat is use to get quick cell level statistics from particular cell storage. Each CELL contains tool like cellsrvstat which gives brief information about CELL level statistics. It also helps you to get information of offloading and storage index.

It gives brief statistics for following sections.

io                  Input/Output related stats
mem              Memory related stats
exec               Execution related stats
net                 Network related stats
smartio           SmartIO related stats
flashcache       FlashCache related stats
ffi                   FFI related stats
lio                  LinuxBlockIO related stats

Simply running the utility from the command prompt, without any additional parameters or qualifiers, produces the output. You can also restrict the output of cellsrvstat by using the -stat_group parameter to specify which group, or groups, you want to monitor.

Help from cellsrvstat

[root@cell01 ~]# cellsrvstat -h

Integrate Oracle Support with OEM12c

Procedure to integrate Oracle Support with OEM12c

1. First set the proxy if any, otherwise you can keep as it is like no proxy

Go to


Setup -> Proxy Settings -> My Oracle Support


Click to enlarge

Provide proxy settings:

Test the proxy setting:

Apply Proxy:

2. Apply your oracle credentials which you want to integrarte with EM

Oracle Block Change Tracking (BCT)

Block change tracking records the modified blocks since last backup and stores this logs in block change tracking file.

During backups,  RMAN uses this log file to identify the specific blocks (changed block) that must be backed up.

Benifits: BCT improves RMAN backup performance as it would not required to scan whole datafiles to detect changed blocks. It will only see the changed block through block change tracking files and take the back up that clock.

Whenever data blocks change, the Change Tracking Writer (CTWR) background process tracks the changed blocks in a private area of memory. 

When a commit is issued against the data block, the block change tracking information is copied to a shared area in Large Pool called the CTWR buffer.

During the checkpoint, the CTWR process writes the information from the CTWR RAM buffer to the block change-tracking file.

To see the status of BCT Status

How to change OEM 12c Agent Port ?

Whenever we install EM agent, it will use default port at time of installation on target host. If we need to modify the port then we can follow below steps to change the port number.

Stop the EM Agent
$ $AGENT_HOME/bin/emctl stop agent

Modify the $AGENT_HOME/sysman/config/emd.properties file in the EM Agent home directory:

For example:
EMD_URL=http://example.com:3872/emd/main

Modify the <port number> in the EMD_URL property so the EM Agent uses a new unused port on the target host.

EMD_URL=http://example.com:3883/emd/main

Start the EM Agent.

$ $AGENT_HOME/bin/emctl start agent

Check the EM status

Exadata Patching Strategy

Let's have an overview of Exadata Machine patching.

1. Check latest patches available on Oracle note ID - 888828.1 
2. Download the required patches
3. Review the patch README file
4. Make step by step plan to apply patch for all the component
5. Run Exachk utility before the patch application, analyse the report and correct wherever it's required.
6. Automate the patch application process based on rolling or non-rolling method
7. Apply the patch

Resource Control in Exadata

We can check enabled resources (CPUs, RAM) with below command on exadata.

Sample output from one storage cell of Exadata 1/8th Rack

[root@cell01 ~]# /opt/oracle.SupportTools/resourcecontrol -show

[INFO] Validated hardware and OS. Proceed.

[SHOW] Number of cores active per socket: 3

[SHOW] Number of harddisks active - 6.

[SHOW] Number of flashdisks active - 8.

1/8th Exadata Rack comes with same hardware as quarter rack, but only half of the resources would be enabled on each 1/8th storage cell So, if in future if we need to upgrade 1/8th rack to quarter rack then we can use this utility to enable rest all the resources.

Above example shows all the enabled resources on one storage cell.

Exadata Monitoring Command

Additional monitoring commands which are required with usual DBA skill set on DB and CELL server.

Database Server

    Linux: top, mpstat, vmstat, iostat, fdisk, ustat, sar, sysinfo

    Exadata: dcli

    ASM: asmcmd, asmca

    Clusterware: crsctl, srvctl

Cell Server

    Linux: top, mpstat, vmstat, iostat, fdisk, ustat, sar, sysinfo

    Cell management: cellcli, cellsrvstat


                                    Exadata Monitoring Tools

Exadata OneCommand Utility Steps

Onecommand:

• Onecommand utility is used to configure the Exadata machine based on given information through OEDA by customer.
• Also whatever steps are performed by oncecommand can be changed based on the customer's required configuration and environment.
• All the steps runs sequentially and each step must be completed before execution of next step.
• Steps can be run individually as well as in one shot with single command.

Below are the steps which need to be executed through onecommand utility. 

Step 0: Validate the environment.

Step 1: Create work directory

Step 2: Unzip files

Step 3: Setup SSH for the root user

Step 4: Update the /etc/hosts file

Step 5: Create the cellip.ora and cellinit.ora files

Step 6: Validate the InfiniBand network

Step 7: Update the cell software

Step 8: Validate the cells

Step 9: Check RDS using the ping command

Step 10: Run CALLIBRATE on the cells

Exadata Storage Index Part-2 (Demo)

Continue from previous blog.....

In previous blog we have explained storage index and it's features. Now we will check real time performance benefits of storage index.

we have flush shared pool and buffer cache to determine exact value and benefits given by Storage Index.

Below query will show that how much physical IO bytes has been saved by storage Index.

SQL> select name, value from v$mystat s, v$statname n where n.statistic# = s.statistic# and name like '%storage%';

NAME                                                                  VALUE

---------------------------------------------------------------- ----------

cell physical IO bytes saved by storage index                             0

Enabling the storage index

SQL> alter session set "_kcfis_storageidx_disabled"=false;

Session altered.

Executing command on employee table

SQL> select ID from EMPLOYEE where ID is null;

no rows selected

Now we will check how much bytes has been saved by storage index with below query.

SQL> select name, value from v$mystat s, v$statname n where n.statistic# = s.statistic# and name like '%storage%';

NAME                                                                  VALUE

---------------------------------------------------------------- ----------

cell physical IO bytes saved by storage index                      48684323

Above value shows that we have saved 48684323 IO bytes saved with storage index while executing the query.

Exadata X4 series launched

                      http://www.oracle.com/us/corporate/press/2079925

Exadata Installation Activities

Implementation of Exadata done by Oracle ACS team so many time we are curious to know that what exactly they are doing so here we have listed down the list of activities which can be done by Oracle in various phases.

1.  Configuration worksheet

2.  Pre-delivery Survey

(ACS Team Activity)

3.  Generic Config Files based on Configuration Worksheet

4.  Run checkip.sh

(Oracle H/W Team Activity)

5.  Power on and validate all the components

6.  Configure KVM

7.  Configure Infiniband Swithces

8.  Configure Cisco Switch

9.  Configure IP to PDUs

10. Storage Cell validation

11. Compute Node validation

(ACS Team Activity)

12. Transfer configuration files from USB to DB node

13. Execute firstboot and applyconfig.sh

14. Stage Oracle Software on Node 1

15. Run OneCommand

                                           ASR Installation

Location of Log files in Exadata

On the cell nodes

============

1. Cell alert.log file

/opt/oracle/cell/log/diag/asm/cell/{node name}/trace/alert.logorif the CELLTRACE parameter is set just do cd $CELLTRACE

2. MS logfile

/opt/oracle/cell/log/diag/asm/cell/{node name}/trace/ms-odl.log.

orif the CELLTRACE parameter is set just do cd $CELLTRACE

3. OS watcher output data

/opt/oracle.oswatcher/osw/archive/

To get OS watcher data of specific date :

#cd /opt/oracle.oswatcher/osw/archive

#find . -name '*12.01.13*' -print -exec zip /tmp/osw_`hostname`.zip {} ;

where 12- year 01- Month 13-day

4. Os message logfile

/var/log/messages

5. VM Core files

/var/log/oracle/crashfiles

More details can be found in the following note:

Where / How to find OS crashcore file in Exadata Systems [Linux] (Doc ID 1389225.1)

6. SunDiag output files.

/tmp/sundiag_.tar.bz2

7. Cell patching issues related logfiles:

/var/log/cellos

The major logfile of patch application output you will find in the db node from where you are patching in the location /tmp/<cell version>/patchmgr.stdout and patchmgr.err

8. Disk controller firmware logs:

/opt/MegaRAID/MegaCli/Megacli64 -fwtermlog -dsply -a0

Download Free Oracle Exadata Certification Exam Dumps

On the Database Nodes

================

1. Database alert.log

$ORACLE_BASE/diag/rdbms/{DBNAME}/{sid}/trace/alert_{sid}.log

Ex: /u01/app/oracle/diag/rdbms/dbfs/DBFS2/trace/alert_DBFS2.log

2. ASM alert.log

$ORACLE_BASE/diag/asm/+asm/+ASM{instance number}/trace/ alert_+ASM {instance number}.log

Ex: /u01/app/oracle/diag/asm/+asm/+ASM2/trace/alert_+ASM2.log

3. Clusterware CRS alert.log

$GRID_HOME/log/{node name}/alert{node name}.log

Ex: /u01/app/11.2.0/grid/log/dmorldb02/alertdmorldb02.log

4. Diskmon logfiles

$GRID_HOME/log/{node name}/diskmon/diskmon.lo*

Ex: /u01/app/11.2.0/grid/log/dmorldb02/diskmon/diskmon.log

5. OS Watcher output files

/opt/oracle.oswatcher/osw/archive/

To get OS watcher data of specific date :

#cd /opt/oracle.oswatcher/osw/archive

#find . -name '*12.01.13*' -print -exec zip /tmp/osw_`hostname`.zip {} ;

where 12- year 01- Month 13-day

6. Os message logfile

/var/log/messages

7. VM Core files for Linux

/u01/crashfiles

More details can be found in the following note:Where / How to find OS crashcore file in Exadata Systems [Linux] (Doc ID 1389225.1)

8. Disk controller firmware logs:

/opt/MegaRAID/MegaCli/Megacli64 -fwtermlog -dsply -a0


What are the Exadata Installation Activities ?

ASR Installation and Registration procedure with Oracle

Plateforms:

RHEL 3 or latter x86
SUSE Linux 9 or latter x86
OEL 4 or latter x86

Software Requirement (08/10/2013) :

1. Oracle Automated Service Manager (OSAM) Package (Patch Number p17270592_150_Linux-x86-64.zip)
2. Oracle Automated Service Request (ASR) Package (Patch Number p17199302_45_Linux-x86-64.zip)
3. Service Tools Bundle (STB) (Patch Number p12757884_10000_Generic.zip)

Here we have taken rpm and patch for testing purpose, it's always best practice to use latest RPMs and patch.

All the installation must be done by super User

1. Install STB on ASR Manager Server

          #rpm -i sun-hardware-reg-1.0.0-1.i386.rpm
          #rpm -i sun-servicetag-1.1.5-1.i386.rpm

   2. Verify created service tag execute below command

         #/opt/sun/servicetag/bin/stclient -x

  3. Install OSAM on ASR Manager Server

        #rpm -i SUNWsasm-1.5.0-112.rpm

  4. Install ASR Package on ASR Manager Server

       #rpm -ivh SUNWswasr-4.5-20130703104534.rpm

  5. Add asr path in .bash_profile file:

      PATH=$PATH:/opt/SUNWswasr/bin/asr
      export PATH

Register ASR Manager :

DBFS on Oracle Exadata

Follow below steps to create DBFS

Create the mount point for DBFS

[root@db01 ]# mkdir /dbfs_dir

Login with Oracle user

[root@db1 ]# su - oracle

Set the Oracle Database environment variable to connect the your database.

Connect Database as sysdba

[oracle@db01 ~]$ sqlplus  / as sysdba

Create tablespace on ASM disk group

SQL> create tablespace dbfsts datafile '+DATA' size 1G;

SQL> create user dbfs_usr identified by Dbfs_pwd1 default tablespace dbfsts quota unlimited on dbfsts;

SQL> grant create session, create table, create procedure, dbfs_role to dbfs_usr;

You can use below syntax to create bigfile tablespace.

create bigfile tablespace dbfsts datafile '+DATA' size 2048g autoextend off nologging extent management local;

Go to admin directory of oracle binaryand run the below procedure

$cd $ORACLE_HOME/rdbms/admin

$sqlplus dbfs_usr/Dbfs_pwd1

SQL> start dbfs_create_filesystem_advanced dbfsts FS nocompress nodeduplicate noencrypt non-partition

Above procedure should run without errors.

TEST DBFS SETUP

[oracle@db01 ~]$ echo Dbfs_pwd1 > pwd.txt

[oracle@db01 ~]$ nohup $ORACLE_HOME/bin/dbfs_client dbfs_usr@test /dbfs_dir < pwd.txt &

- 'test' is database name

- '&' to run the process in background

- '/dbfs_dir' is a directory on which DBFS will be mounted

LOGIN to Oracle USER account from another terminal and check the DBFS file system

[oracle@db01 ~]$ cd /dbfs_dir/FS

[oracle@db01 ~]$ touch test.txt

[oracle@db01 ~]$ ls -ltr test.txt

LOGIN to root USER account from another terminal and check whether you can get acces of the DBFS file system

[root@db01 ]# cd /dbfs_dir/FS

Permission Denied

To unmount the file system issue the following command from the "root" OS user.

[root@db01 ]# fusermount -u /dbfs_dir

To allow other users ( example root to access the filesystem )

[oracle@dbs1 ]$ su - root

Password :

[root@db01 ]# echo user_allow_other > /etc/fuse.conf

change permission of /etc/fuse.conf to 644

[root@db01 ]#chmod 644 /etc/fuse.conf

[root@db01 ]# su - oracle

[oracle@db01 ~]$ nohup $ORACLE_HOME/bin/dbfs_client dbfs_usr@test -o direct_io,allow_root /dbfs_dir < pwd.txt &

You can check nohup file for any error logs.

Login to root user account from another terminal and check the accessibility of DBFS

[root@db01 ]# cd /dbfs_dir/FS

[root@db01 ]# ls -lt test.txt

To unmount the file system issue the following command from the "root" OS user.

# fusermount -u /dbfs_dir

Hot to Verify InfiniBand Fabric Topology ?

Verifying infiniband topology whether it is configured current or not. If it not gives the desired output as per below then it will cause the InfiniBand network to operate at degraded efficiency, intermittently, or fail to operate.

# /opt/oracle.SupportTools/ibdiagtools/verify-topology -t fattree

[ DB Machine InfiniBand Cabling Topology Verification Tool ]
Is every external switch connected to every internal switch..........[SUCCESS]
Are any external switches connected to each other....................[SUCCESS]
Are any hosts connected to spine switch..............................[SUCCESS]
Check if all hosts have 2 CAs to different switches..................[SUCCESS]
Leaf switch check: cardinality and even distribution.................[SUCCESS]
Check if each rack has an valid internal ring........................[SUCCESS]

If anything other than "SUCCESS" is reported, investigate and correct the condition.

OEM11g / OMS Auto Start and Stop Script for Linux

After installation OEM 11g mostly we required is to start the OEM services even after server reboot so here you will see how to make a OEM start-up script and add into the OS start-up sequence.

This is purely to avoid manual operation of OEM start-up if in any case we reboot the server.

1. Create dbora (service name) file as below: 

 vi /etc/init.d/dbora

#Run-level Startup script for the Oracle Instance and Listener

#

# chkconfig: 345 91 19

# description: Startup/Shutdown Oracle listener and instance

ORACLE_HOME="/oem/oracle/app/oracle/product/11.2.0/dbhome_1/"

ORACLE_OWNR="oracle"

OMS_ORACLE_HOME="/oem/Middleware/oms11g/"

OEM_AGENT_HOME="/oem/Middleware/agent11g/"

# if the executables do not exist -- display error

Power Off Sequence for Exadata Stack

Poweroff Database Servers

For single server power off /Restart

1. First stop the oracle cluster with below command

#$GRID_HOME/grid/bin/crsctl stop cluster

2. If it fails to stop the cluster then use -f option to stop it forcefully.
Shutdown the server with below command

#shutdown -h -y now

To restart the server execute below command

#shutdown -r -y now      or      #reboot

3. Remove power cable from the server

For multiple servers poweroff/restart with dcli utility

1. First stop the oracle cluster with below command

#$GRID_HOME/grid/bin/crsctl stop cluster -all

2. Shutdown all the servers with dcli utility

#dcli -g dbs_group -l root shutdown -h -y now

---dbs_group is the file which contains the list of all the exadata servers.

To restart the server execute below command

#dcli -g dbs_group -l root shutdown -r -y now    or  #dcli -g dbs_group -l root reboot

----dbs_group is the file which contains the list of all the exadata servers.

3. Remove power cables from the servers

Poweroff Storage Servers

Reat the same procedure of server to power off the exadata storage servers except to stopping the crsctl.

Note: We have to use cell_group file instead of dbs_group file to shutdown the storage servers.

Poweroff Network Equipments

We can directly remove power cables from the network switches as network equipments don't have power switches.