Useful Files in Solaris

Message log:  /var/adm/messages

Tape driver configuration file:  /kernel/drv/st.conf

Disk driver configuration file:  /kernel/drv/sd.conf

File defining filesystems to mount:  /etc/vfstab

Kernel configuration parameters:  /etc/system

Names and IP addr of machines:  /etc/hosts

Default Port Numbers:  /etc/services

                                    Analyze Linux Server Performance

Oracle Exadata IORM

IORM is the great feature come with Exadata which gives us to control the resources for multiple databases as well as different workloads on same database . Just given overview regarding IORM.

IORM is the extension of DBRM. IORM does not directly improve efficiency of operation by reducing the amount of resources required but  I/O Resource Manager contributes to improved efficiency by ensuring that in a scenario where there is more demand for a resource than can be simultaneously supplied by the Exadata Storage Server, the most critical operations will deliver a consistent quality of service.

We can priorotize the storage I/O to the Database based on the criticality of the database. IORM deliver consistent performance quality. IORM guarantee that particular DB will get percentage of resources at any time.

When bandwidth is dividing in different workload in same database, it's called intradatabase resource management. In this case IORM does not come in the picture as whole resource utilization can be configured on compute node.

When bandwidth is dividing for different databases, it's called interdatabase IORM. In this scenario IORM need to be configured on cell level.

IORM Architecture:

Integrate Oracle Support with OEM12c

Procedure to integrate Oracle Support with OEM12c

1. First set the proxy if any, otherwise you can keep as it is like no proxy

Go to


Setup -> Proxy Settings -> My Oracle Support


Click to enlarge

Provide proxy settings:

Test the proxy setting:

Apply Proxy:

2. Apply your oracle credentials which you want to integrarte with EM

Oracle Block Change Tracking (BCT)

Block change tracking records the modified blocks since last backup and stores this logs in block change tracking file.

During backups,  RMAN uses this log file to identify the specific blocks (changed block) that must be backed up.

Benifits: BCT improves RMAN backup performance as it would not required to scan whole datafiles to detect changed blocks. It will only see the changed block through block change tracking files and take the back up that clock.

Whenever data blocks change, the Change Tracking Writer (CTWR) background process tracks the changed blocks in a private area of memory. 

When a commit is issued against the data block, the block change tracking information is copied to a shared area in Large Pool called the CTWR buffer.

During the checkpoint, the CTWR process writes the information from the CTWR RAM buffer to the block change-tracking file.

To see the status of BCT Status

How to change OEM 12c Agent Port ?

Whenever we install EM agent, it will use default port at time of installation on target host. If we need to modify the port then we can follow below steps to change the port number.

Stop the EM Agent
$ $AGENT_HOME/bin/emctl stop agent

Modify the $AGENT_HOME/sysman/config/emd.properties file in the EM Agent home directory:

For example:
EMD_URL=http://example.com:3872/emd/main

Modify the <port number> in the EMD_URL property so the EM Agent uses a new unused port on the target host.

EMD_URL=http://example.com:3883/emd/main

Start the EM Agent.

$ $AGENT_HOME/bin/emctl start agent

Check the EM status

Exadata Patching Strategy

Let's have an overview of Exadata Machine patching.

1. Check latest patches available on Oracle note ID - 888828.1 
2. Download the required patches
3. Review the patch README file
4. Make step by step plan to apply patch for all the component
5. Run Exachk utility before the patch application, analyse the report and correct wherever it's required.
6. Automate the patch application process based on rolling or non-rolling method
7. Apply the patch

Resource Control in Exadata

We can check enabled resources (CPUs, RAM) with below command on exadata.

Sample output from one storage cell of Exadata 1/8th Rack

[root@cell01 ~]# /opt/oracle.SupportTools/resourcecontrol -show

[INFO] Validated hardware and OS. Proceed.

[SHOW] Number of cores active per socket: 3

[SHOW] Number of harddisks active - 6.

[SHOW] Number of flashdisks active - 8.

1/8th Exadata Rack comes with same hardware as quarter rack, but only half of the resources would be enabled on each 1/8th storage cell So, if in future if we need to upgrade 1/8th rack to quarter rack then we can use this utility to enable rest all the resources.

Above example shows all the enabled resources on one storage cell.

Exadata Monitoring Command

Additional monitoring commands which are required with usual DBA skill set on DB and CELL server.

Database Server

    Linux: top, mpstat, vmstat, iostat, fdisk, ustat, sar, sysinfo

    Exadata: dcli

    ASM: asmcmd, asmca

    Clusterware: crsctl, srvctl

Cell Server

    Linux: top, mpstat, vmstat, iostat, fdisk, ustat, sar, sysinfo

    Cell management: cellcli, cellsrvstat


                                    Exadata Monitoring Tools

Exadata OneCommand Utility Steps

Onecommand:

• Onecommand utility is used to configure the Exadata machine based on given information through OEDA by customer.
• Also whatever steps are performed by oncecommand can be changed based on the customer's required configuration and environment.
• All the steps runs sequentially and each step must be completed before execution of next step.
• Steps can be run individually as well as in one shot with single command.

Below are the steps which need to be executed through onecommand utility. 

Step 0: Validate the environment.

Step 1: Create work directory

Step 2: Unzip files

Step 3: Setup SSH for the root user

Step 4: Update the /etc/hosts file

Step 5: Create the cellip.ora and cellinit.ora files

Step 6: Validate the InfiniBand network

Step 7: Update the cell software

Step 8: Validate the cells

Step 9: Check RDS using the ping command

Step 10: Run CALLIBRATE on the cells

Exadata Certification Sample Questions & Answers (1Z0-536)

I have given some sample questions for Exadata Certification. I hope it will help you to prepare for certification. 


When do you specify the type of interleaving for a grid disk?

A. When you create the cell disk
B. When you create the grid disk
C. Through ASM
D. When you create a tablespace
Answer: B

Why would you reduce the default disk_repair_time parameter?

A. To extend the availability of your ASM disk groups.
B. To reduce the need for high redundancy
C. To reduce the amount of data collected for a fast resynch
D. To increase performance of anExadata Storage Server.
Answer: C


What benefit is provided by column filtering?

A. The Exadata Storage Server can select rows based on column values listed in a SQL predicate
B. Storage indexes are built on columns for use in filtering
C. Only necessary columns are returned to the database server
D. Column filtering is a marketing term, not a real benefit
Answer: C

Which types of data are most likely to be cached In the Exadata SmartFlash Cache?

A. Results of random reads.
B. Results of table scans.
C. Write to a mirror
D. Redo data
E. All data is cached In the Flash Cache
Answer: A

What does the role attribute of a DB plan Indicate?

A. The role specified for the category
B. The role specified for the user
C. The role specified In a Data Guard environment
D. The role specified for the application
Answer: A

Your customer has designated a number of database objects to be kept persistently in the Exadata Smart Flash Cache. What happens if the total size of these objects is greater than 80% of the size of the available Exadata Smart Flash Cache?

A. Nothing
B. Exadata Smart Flash Cache expands the allocation to hold the objects.
C. Not all objects will be stored in Exadata Smart Flash Cache.
D. Overall performance is increased as more objects fit into Exadata Smart Flash Cache.
Answer: C

Which three attributes are likely to result in data being cached in the Exadata Smart Flash Cache?

A. CELL_FLASH_CACHE attribute on the data object
B. CACHE hint In the SQL statement
C. Data from a table scan
D. Small data less than 128 KBs
E. Control file I/Os
Answer: A,C,E

How many IP addresses you need to assign for each Exadata storage cell?

Exadata Installation Activities

Implementation of Exadata done by Oracle ACS team so many time we are curious to know that what exactly they are doing so here we have listed down the list of activities which can be done by Oracle in various phases.

1.  Configuration worksheet

2.  Pre-delivery Survey

(ACS Team Activity)

3.  Generic Config Files based on Configuration Worksheet

4.  Run checkip.sh

(Oracle H/W Team Activity)

5.  Power on and validate all the components

6.  Configure KVM

7.  Configure Infiniband Swithces

8.  Configure Cisco Switch

9.  Configure IP to PDUs

10. Storage Cell validation

11. Compute Node validation

(ACS Team Activity)

12. Transfer configuration files from USB to DB node

13. Execute firstboot and applyconfig.sh

14. Stage Oracle Software on Node 1

15. Run OneCommand

                                           ASR Installation

Location of Log files in Exadata

On the cell nodes

============

1. Cell alert.log file

/opt/oracle/cell/log/diag/asm/cell/{node name}/trace/alert.logorif the CELLTRACE parameter is set just do cd $CELLTRACE

2. MS logfile

/opt/oracle/cell/log/diag/asm/cell/{node name}/trace/ms-odl.log.

orif the CELLTRACE parameter is set just do cd $CELLTRACE

3. OS watcher output data

/opt/oracle.oswatcher/osw/archive/

To get OS watcher data of specific date :

#cd /opt/oracle.oswatcher/osw/archive

#find . -name '*12.01.13*' -print -exec zip /tmp/osw_`hostname`.zip {} ;

where 12- year 01- Month 13-day

4. Os message logfile

/var/log/messages

5. VM Core files

/var/log/oracle/crashfiles

More details can be found in the following note:

Where / How to find OS crashcore file in Exadata Systems [Linux] (Doc ID 1389225.1)

6. SunDiag output files.

/tmp/sundiag_.tar.bz2

7. Cell patching issues related logfiles:

/var/log/cellos

The major logfile of patch application output you will find in the db node from where you are patching in the location /tmp/<cell version>/patchmgr.stdout and patchmgr.err

8. Disk controller firmware logs:

/opt/MegaRAID/MegaCli/Megacli64 -fwtermlog -dsply -a0

Download Free Oracle Exadata Certification Exam Dumps

On the Database Nodes

================

1. Database alert.log

$ORACLE_BASE/diag/rdbms/{DBNAME}/{sid}/trace/alert_{sid}.log

Ex: /u01/app/oracle/diag/rdbms/dbfs/DBFS2/trace/alert_DBFS2.log

2. ASM alert.log

$ORACLE_BASE/diag/asm/+asm/+ASM{instance number}/trace/ alert_+ASM {instance number}.log

Ex: /u01/app/oracle/diag/asm/+asm/+ASM2/trace/alert_+ASM2.log

3. Clusterware CRS alert.log

$GRID_HOME/log/{node name}/alert{node name}.log

Ex: /u01/app/11.2.0/grid/log/dmorldb02/alertdmorldb02.log

4. Diskmon logfiles

$GRID_HOME/log/{node name}/diskmon/diskmon.lo*

Ex: /u01/app/11.2.0/grid/log/dmorldb02/diskmon/diskmon.log

5. OS Watcher output files

/opt/oracle.oswatcher/osw/archive/

To get OS watcher data of specific date :

#cd /opt/oracle.oswatcher/osw/archive

#find . -name '*12.01.13*' -print -exec zip /tmp/osw_`hostname`.zip {} ;

where 12- year 01- Month 13-day

6. Os message logfile

/var/log/messages

7. VM Core files for Linux

/u01/crashfiles

More details can be found in the following note:Where / How to find OS crashcore file in Exadata Systems [Linux] (Doc ID 1389225.1)

8. Disk controller firmware logs:

/opt/MegaRAID/MegaCli/Megacli64 -fwtermlog -dsply -a0


What are the Exadata Installation Activities ?

ASR Installation and Registration procedure with Oracle

Plateforms:

RHEL 3 or latter x86
SUSE Linux 9 or latter x86
OEL 4 or latter x86

Software Requirement (08/10/2013) :

1. Oracle Automated Service Manager (OSAM) Package (Patch Number p17270592_150_Linux-x86-64.zip)
2. Oracle Automated Service Request (ASR) Package (Patch Number p17199302_45_Linux-x86-64.zip)
3. Service Tools Bundle (STB) (Patch Number p12757884_10000_Generic.zip)

Here we have taken rpm and patch for testing purpose, it's always best practice to use latest RPMs and patch.

All the installation must be done by super User

1. Install STB on ASR Manager Server

          #rpm -i sun-hardware-reg-1.0.0-1.i386.rpm
          #rpm -i sun-servicetag-1.1.5-1.i386.rpm

   2. Verify created service tag execute below command

         #/opt/sun/servicetag/bin/stclient -x

  3. Install OSAM on ASR Manager Server

        #rpm -i SUNWsasm-1.5.0-112.rpm

  4. Install ASR Package on ASR Manager Server

       #rpm -ivh SUNWswasr-4.5-20130703104534.rpm

  5. Add asr path in .bash_profile file:

      PATH=$PATH:/opt/SUNWswasr/bin/asr
      export PATH

Register ASR Manager :

DBFS on Oracle Exadata

Follow below steps to create DBFS

Create the mount point for DBFS

[root@db01 ]# mkdir /dbfs_dir

Login with Oracle user

[root@db1 ]# su - oracle

Set the Oracle Database environment variable to connect the your database.

Connect Database as sysdba

[oracle@db01 ~]$ sqlplus  / as sysdba

Create tablespace on ASM disk group

SQL> create tablespace dbfsts datafile '+DATA' size 1G;

SQL> create user dbfs_usr identified by Dbfs_pwd1 default tablespace dbfsts quota unlimited on dbfsts;

SQL> grant create session, create table, create procedure, dbfs_role to dbfs_usr;

You can use below syntax to create bigfile tablespace.

create bigfile tablespace dbfsts datafile '+DATA' size 2048g autoextend off nologging extent management local;

Go to admin directory of oracle binaryand run the below procedure

$cd $ORACLE_HOME/rdbms/admin

$sqlplus dbfs_usr/Dbfs_pwd1

SQL> start dbfs_create_filesystem_advanced dbfsts FS nocompress nodeduplicate noencrypt non-partition

Above procedure should run without errors.

TEST DBFS SETUP

[oracle@db01 ~]$ echo Dbfs_pwd1 > pwd.txt

[oracle@db01 ~]$ nohup $ORACLE_HOME/bin/dbfs_client dbfs_usr@test /dbfs_dir < pwd.txt &

- 'test' is database name

- '&' to run the process in background

- '/dbfs_dir' is a directory on which DBFS will be mounted

LOGIN to Oracle USER account from another terminal and check the DBFS file system

[oracle@db01 ~]$ cd /dbfs_dir/FS

[oracle@db01 ~]$ touch test.txt

[oracle@db01 ~]$ ls -ltr test.txt

LOGIN to root USER account from another terminal and check whether you can get acces of the DBFS file system

[root@db01 ]# cd /dbfs_dir/FS

Permission Denied

To unmount the file system issue the following command from the "root" OS user.

[root@db01 ]# fusermount -u /dbfs_dir

To allow other users ( example root to access the filesystem )

[oracle@dbs1 ]$ su - root

Password :

[root@db01 ]# echo user_allow_other > /etc/fuse.conf

change permission of /etc/fuse.conf to 644

[root@db01 ]#chmod 644 /etc/fuse.conf

[root@db01 ]# su - oracle

[oracle@db01 ~]$ nohup $ORACLE_HOME/bin/dbfs_client dbfs_usr@test -o direct_io,allow_root /dbfs_dir < pwd.txt &

You can check nohup file for any error logs.

Login to root user account from another terminal and check the accessibility of DBFS

[root@db01 ]# cd /dbfs_dir/FS

[root@db01 ]# ls -lt test.txt

To unmount the file system issue the following command from the "root" OS user.

# fusermount -u /dbfs_dir

Hot to Verify InfiniBand Fabric Topology ?

Verifying infiniband topology whether it is configured current or not. If it not gives the desired output as per below then it will cause the InfiniBand network to operate at degraded efficiency, intermittently, or fail to operate.

# /opt/oracle.SupportTools/ibdiagtools/verify-topology -t fattree

[ DB Machine InfiniBand Cabling Topology Verification Tool ]
Is every external switch connected to every internal switch..........[SUCCESS]
Are any external switches connected to each other....................[SUCCESS]
Are any hosts connected to spine switch..............................[SUCCESS]
Check if all hosts have 2 CAs to different switches..................[SUCCESS]
Leaf switch check: cardinality and even distribution.................[SUCCESS]
Check if each rack has an valid internal ring........................[SUCCESS]

If anything other than "SUCCESS" is reported, investigate and correct the condition.

OEM11g / OMS Auto Start and Stop Script for Linux

After installation OEM 11g mostly we required is to start the OEM services even after server reboot so here you will see how to make a OEM start-up script and add into the OS start-up sequence.

This is purely to avoid manual operation of OEM start-up if in any case we reboot the server.

1. Create dbora (service name) file as below: 

 vi /etc/init.d/dbora

#Run-level Startup script for the Oracle Instance and Listener

#

# chkconfig: 345 91 19

# description: Startup/Shutdown Oracle listener and instance

ORACLE_HOME="/oem/oracle/app/oracle/product/11.2.0/dbhome_1/"

ORACLE_OWNR="oracle"

OMS_ORACLE_HOME="/oem/Middleware/oms11g/"

OEM_AGENT_HOME="/oem/Middleware/agent11g/"

# if the executables do not exist -- display error

Power Off Sequence for Exadata Stack

Poweroff Database Servers

For single server power off /Restart

1. First stop the oracle cluster with below command

#$GRID_HOME/grid/bin/crsctl stop cluster

2. If it fails to stop the cluster then use -f option to stop it forcefully.
Shutdown the server with below command

#shutdown -h -y now

To restart the server execute below command

#shutdown -r -y now      or      #reboot

3. Remove power cable from the server

For multiple servers poweroff/restart with dcli utility

1. First stop the oracle cluster with below command

#$GRID_HOME/grid/bin/crsctl stop cluster -all

2. Shutdown all the servers with dcli utility

#dcli -g dbs_group -l root shutdown -h -y now

---dbs_group is the file which contains the list of all the exadata servers.

To restart the server execute below command

#dcli -g dbs_group -l root shutdown -r -y now    or  #dcli -g dbs_group -l root reboot

----dbs_group is the file which contains the list of all the exadata servers.

3. Remove power cables from the servers

Poweroff Storage Servers

Reat the same procedure of server to power off the exadata storage servers except to stopping the crsctl.

Note: We have to use cell_group file instead of dbs_group file to shutdown the storage servers.

Poweroff Network Equipments

We can directly remove power cables from the network switches as network equipments don't have power switches.

How to find serial number of Exadata Server/Storage

Execute below command on operating system to find the serial number of DB node and Storage cell.

# ipmitool sunoem cli "show /SP system_identifier"


Example:

[root@db01 ~]#  ipmitool sunoem cli "show /SP system_identifier"
Connected. Use ^D to exit.
-> show /SP system_identifier

  /SP
    Properties:
        system_identifier = Exadata Database Machine X3-2 AK00097035


-> Session closed
Disconnected

You can follow our Exadata Certification Question Bank to achieve maximum success in your Exadata certification exam.

User Management in Exadata

User Management

#pam_tally2 -- Check the failed user login

#pam_tally2 -r -u username  ---Reset user user

######Check user pwd policy

#chage -l username

Cell Storage Processes

There are three processes will be running on the cell stoarge servers.

cellsrv - Cell Server

It is mutlithreaded process. The primary storage software components. Responsible for disk I/O with the use of iDB protocol.It communicates with Oracle Database to serve simple block requests like database buffer cache reads and Sart Scan requests like tablespaces with projections and filters. CELLSRV also implements I/ORM and collect statistics relating to its operation

MS - Management Server

This is the intermediater between command line interface and cell storage. It provide the interface of storage for configuration and administration purpose. It works with cellcli.It is also responsible to send alerts and collect statistics.

RS - Restart Server

Restart service will monitor the cellsrv and MS processes. It restarts them if necessary.