Exadata Quick Reference 6: Important Oracle Exadata Configuration Files

Oracle Exadata and Storage cell having multiple configuration files which define the Exadata configuration. We have listed down the important configuration files for Compute and Storage node which can be used as quick reference for any kind of troubleshooting during Exadata support.

Exadata Compute Node:

/opt/oracle - contains dbserver software

Exadata Monitoring Command

Additional monitoring commands which are required with usual DBA skill set on DB and CELL server.

Database Server

    Linux: top, mpstat, vmstat, iostat, fdisk, ustat, sar, sysinfo

    Exadata: dcli

    ASM: asmcmd, asmca

    Clusterware: crsctl, srvctl

Cell Server

    Linux: top, mpstat, vmstat, iostat, fdisk, ustat, sar, sysinfo

    Cell management: cellcli, cellsrvstat


                                    Exadata Monitoring Tools

Exadata OneCommand Utility Steps

Onecommand:

• Onecommand utility is used to configure the Exadata machine based on given information through OEDA by customer.
• Also whatever steps are performed by oncecommand can be changed based on the customer's required configuration and environment.
• All the steps runs sequentially and each step must be completed before execution of next step.
• Steps can be run individually as well as in one shot with single command.

Below are the steps which need to be executed through onecommand utility. 

Step 0: Validate the environment.

Step 1: Create work directory

Step 2: Unzip files

Step 3: Setup SSH for the root user

Step 4: Update the /etc/hosts file

Step 5: Create the cellip.ora and cellinit.ora files

Step 6: Validate the InfiniBand network

Step 7: Update the cell software

Step 8: Validate the cells

Step 9: Check RDS using the ping command

Step 10: Run CALLIBRATE on the cells

Remove Exadata Flash Grid disk and configure as normal Flash

In previous blog we have created Grid Disk on Flash Disk. Here we are removing Flash Grid Disk and configuring normal Flash Cache.

Step-1
Remove or move database or any contents to some another ASM Disk group

Step-2
Drop ASM disk group which was create on flash disk

Step-3
Drop all the flash grid disk

CellCLI> drop griddisk all prefix=flashdisk force
GridDisk FlashDisk_FD_00_exa1cel01 successfully dropped
GridDisk FlashDisk_FD_01_exa1cel01 successfully dropped
GridDisk FlashDisk_FD_02_exa1cel01 successfully dropped
GridDisk FlashDisk_FD_03_exa1cel01 successfully dropped
GridDisk FlashDisk_FD_04_exa1cel01 successfully dropped
GridDisk FlashDisk_FD_05_exa1cel01 successfully dropped
GridDisk FlashDisk_FD_06_exa1cel01 successfully dropped
GridDisk FlashDisk_FD_07_exa1cel01 successfully dropped

Step-4

Configure Exadata Flash Disk as Grid Disk

We can use Flash Cache as Grid Disk to create the ASM disk group and can keep the database on flash disk but this is not reccomended solution as flash cache performs better to cache frequently access data rather than creating database on it.

If you want put your database on Flash cache than below are the steps to configure FLASH CACHE as ASM disk.

Steps:

1. Drop existing flash cache

2. Create flashcache with required size

3. Create grid disk from Flash Cache

4. Create ASM disk group on top of the Flash Griddisk

5. Put your database on ASM disk group which has been configured from Flash grid disk

Step-1

CellCLI>drop flashcache

If it gives error like below

CELL-02769: Cannot complete flash cache drop on: FD_03_ex01cceladm03. 

Received error: CELL-02732: There may be data not synchronized to the grid disks (dirty data) in flash cache.

Use ALTER FLASHCACHE to flush the dirty data and to stop caching, and then retry the command.

This error comes because flash cache mode has been configured as writeback hence some of the dirty data are still not written to the grid disks.

To overcome from this error flush all the flashcache

CellCLI>  alter flashcache all flush

Flash cache exa1cel01_FLASHCACHE altered successfully

CellCLI> drop flashcache 

Flash cache exa1cel01_FLASHCACHE successfully dropped 

Step-2

How to enable Exadata writeback Flashcache on cell storage ? (WriteThrough to WriteBack)

Steps to enabling the Flashcache mode:

This can be done in rolling as well as non-rolloing manner.

First check flashcache mode with below command on all cell servers with the use of dcli utility on DB node. It can be check on cell each server also with cellcli utility.

dbnode1#dcli -l root -g cellgroup "cellcli -e list cell detail" | grep "flashCacheMode"

10.156.110.10: flashCacheMode:          WriteThrough
10.156.110.11: flashCacheMode:          WriteThrough
10.156.110.12: flashCacheMode:          WriteThrough
10.156.110.13: flashCacheMode:          WriteThrough
10.156.110.14: flashCacheMode:          WriteThrough
10.156.110.15: flashCacheMode:          WriteThrough
10.156.110.16: flashCacheMode:          WriteThrough

Step-1
Also we can find flashcache mode in each cell by executing below command:

cellcli>list cell detail

flashCacheMode: WriteThrough     --->Currently flashCacheMode configured as WriteThrough

Step-2
Now drop flashcache on each cell server: (If you are following rolling manner then do one by one on each cell server)

cellcli> drop flashcache

Step-3
Be sure asmdeactivationoutcome is YES before disabling grid disk

CellCLI> list griddisk attributes name,asmmodestatus,asmdeactivationoutcome

Step-4

Cellcli - commands

Export 

CellCLI> export celldisk 

Import 

CellCLI> import celldisk 

Assign 

CellCLI> assign key 

Calibrate 

CellCLI> calibrate
CellCLI> calibrate force 

Set 

CellCLI> help set 
Usage: SET <variable> <value> 
Purpose: Sets a variable to alter the CELLCLI environment settings for your current session. 
Arguments: variable and value represent one of the following clauses: 
DATEFORMAT { STANDARD | LOCAL } 
ECHO { ON | OFF } 

Cellcli - Alter

CellCLI> alter cell shutdown services rs --> To shutdown the Restart Server service 
CellCLI> alter cell shutdown services MS --> To shutdown the Management Server service 
CellCLI> alter cell shutdown services CELLSRV --> To shutdown the Cell Services 
CellCLI> alter cell shutdown services all -->To shutdown the RS, CELLSRV and MS services 
CellCLI> alter cell restart services rs 
CellCLI> alter cell restart services all 
CellCLI> alter cell led on
CellCLI> alter cell led off 
CellCLI> alter cell validate mail
CellCLI> alter cell validate configuration 
CellCLI> alter cell smtpfromaddr='cell07@orac.com' 
CellCLI> alter cell smtpfrom='Exadata Cell 07' 
CellCLI> alter cell smtptoaddr='satya@orac.com' 
CellCLI> alter cell emailFormat='text' 
CellCLI> alter cell emailFormat='html' 
CellCLI> alter cell validate snmp type=ASR - Automatic Service Requests (ASRs) 
CellCLI> alter cell snmpsubscriber=((host='snmp01.orac.com,type=ASR')) 
CellCLI> alter cell restart bmc - BMC, Baseboard Management Controller, controls the compoments of the cell. 
CellCLI> alter cell configure bmc 
CellCLI> alter physicaldisk 34:2,34:3 serviceled on
CellCLI> alter physicaldisk 34:6,34:9 serviceled off 
CellCLI> alter physicaldisk harddisk serviceled on 
CellCLI> alter physicaldisk all serviceled on 
CellCLI> alter lun 0_10 reenable 
CellCLI> alter lun 0_04 reenable force 
CellCLI> alter celldisk FD_01_cell07 comment='Flash Disk' 
CellCLI> alter celldisk all harddisk comment='Hard Disk' 
CellCLI> alter celldisk all flashdisk comment='Flash Disk' 
CellCLI> alter griddisk RECO_CD_10_cell06 comment='Used for Reco'
CellCLI> alter griddisk all inactive 
CellCLI> alter griddisk RECO_CD_11_cell12 inactive 
CellCLI> alter griddisk RECO_CD_08_cell01 inactive force 
CellCLI> alter griddisk RECO_CD_11_cell01 inactive nowait 
CellCLI> alter griddisk DATA_CD_00_CELL01,DATA_CD_02_CELL01,...DATA_CD_11_CELL01 inactive 
CellCLI> alter griddisk all active 
CellCLI> alter griddisk RECO_CD_11_cell01 active 
CellCLI> alter griddisk all harddisk comment='Hard Disk' 
CellCLI> alter ibport ibp2 reset counters
CellCLI> alter iormplan active 
CellCLI> alter quarantine 
CellCLI> alter threshold DB_IO_RQ_SM_SEC.PRODB comparison=">", critical=100 
CellCLI> alter alerthistory

Cellcli - Create

CellCLI> CREATE CELL [cellname] [realmname=realmvalue,] [interconnect1=ethvalue,] [interconnect2=ethvalue,][interconnect3=ethvalue,] [interconnect4=ethvalue,] 
( ([ipaddress1=ipvalue,] [ipaddress2=ipvalue,] [ipaddress3=ipvalue,] [ipaddress4=ipvalue,]) | ([ipblock=ipblkvalue, cellnumber=numvalue]) ) --- To configure the Oracle Exadata cell network and starts services. 

CellCLI> create celldisk all harddisk
CellCLI> create celldisk all 
CellCLI> create celldisk all harddisk interleaving='normal_redundancy' 
interleaving -- none(default), normal_redundancy or high_redundancy
CellCLI> create celldisk all flashdisk 
CellCLI> create griddisk RECO_CD_11_cell01 celldisk=CD_11_cell01 
CellCLI> create griddisk RECO_CD_11_cell01 celldisk=CD_11_cell01 size=100M 
CellCLI> create griddisk all prefix RECO 
CellCLI> create griddisk all flashdisk prefix FLASH 
CellCLI> create griddisk all harddisk prefix HARD 
CellCLI> create griddisk all harddisk prefix='data', size='270g'
CellCLI> create griddisk all prefix='data', size='300g'
CellCLI> create griddisk all prefix='redo', size='150g' 
CellCLI> create griddisk all harddisk prefix=systemdg 
CellCLI> create flashcache celldisk='FD_00_cell01' 
CellCLI> create flashcache celldisk='FD_13_cell01,FD_00_cell01,FD_10_cell01,FD_02_cell01,FD_06_cell01, FD_12_cell01,FD_05_cell01,FD_08_cell01,FD_15_cell01,FD_14_cell01,FD_07_cell01,FD_04_cell01,FD_03_cell01,FD_11_cell01,FD_09_cell01,FD_01_cell01' 

CellCLI> create flashcache all 
CellCLI> create flashcache all size=365.25G 
CellCLI> create key 
CellCLI> create quarantine 
CellCLI> create threshold cd_io_errs_min.prodb comparison=">", critical=10 
CellCLI> create threshold CD_IO_ERRS_MIN warning=1, comparison='>=', occurrences=1, observation=1 

Cellcli - Help

CellCLI> help

HELP [topic]

    Available Topics:

        ALTER

        ALTER ALERTHISTORY

        ALTER CELL

        ALTER CELLDISK

        ALTER GRIDDISK
        ALTER IBPORT

        ALTER IORMPLAN

        ALTER LUN
        ALTER PHYSICALDISK
        ALTER QUARANTINE

        ALTER THRESHOLD

        ASSIGN KEY

        CALIBRATE

        CREATE

        CREATE CELL

        CREATE CELLDISK

        CREATE FLASHCACHE

        CREATE GRIDDISK

        CREATE KEY
        CREATE QUARANTINE

        CREATE THRESHOLD

        DESCRIBE

        DROP

        DROP ALERTHISTORY

        DROP CELL

        DROP CELLDISK

        DROP FLASHCACHE

        DROP GRIDDISK
        DROP QUARANTINE

        DROP THRESHOLD

        EXPORT CELLDISK

        IMPORT CELLDISK

        LIST

        LIST ACTIVEREQUEST

        LIST ALERTDEFINITION

        LIST ALERTHISTORY

        LIST CELL

        LIST CELLDISK

        LIST FLASHCACHE

        LIST FLASHCACHECONTENT
        LIST GRIDDISK

        LIST IBPORT
        LIST IORMPLAN

        LIST KEY

        LIST LUN

        LIST METRICCURRENT

        LIST METRICDEFINITION

        LIST METRICHISTORY

        LIST PHYSICALDISK
        LIST QUARANTINE

        LIST THRESHOLD

        SET

        SPOOL

        START

What is cellcli ?

What is cellcli utility?

Full form – Cell Command Line Interface

Cellcli used to manage the cell through the cell command line utility.

To invoke the cellcli utility, login to the cell storage through root, celladmin or cellmonitor and type cellcli.

Exadata Interview Questions - Part 1

Get an e-book of Exadata Interview Questions and Answers to stay miles ahead of your competition..!!!

• What is Exadata?
• Key components of Exadata
• Features of Exadata
• Exadata Sizing
• Difference between ordinary database setup and Exadata
• What is storage index and how it works
• Use of Flash Cache
• Background services of Cell Server
• What is cellcli?
• How to replace faulty HDD in Exadata?
• How to remove and add cell storage to running system?
• How to migrate database from normal setup to Exadata ?
• Exdata Shutdown procedure
• Which OS is supported by Exadata?
• What is ASR?
• How to integrate Exadata with OEM 12c?
• How to upgrade firmware of Exadata components?
• How to upgrade Exadata?
• Where we can define which cell storage can be used by particular database server?
• How to check Exadata health check?
• What is EHCC?
• Options to configure backup and recovery for Exadata
• What is offloading and how it works?
• What is cellcli and dcli?

Get Questions with answers HERE....

You can follow our Exadata Certification Question Bank to achieve maximum success in your Exadata certification.