Wait Events in RAC

Session 362
Arup Nanda
Longtime Oracle DBA
arup.blogspot.com

What is This?
RAC Performance Tuning
I teach a course Performance Tuning in RAC
Wait events are useful for understanding the bottlenecks
All single instance wait events are applicable to RAC
RAC has some special cases
This session talks about those RAC-specific wait events
This is just a subset of the events; not a comprehensive
list

Wait Events in RAC

Whats a Wait?
A process in Oracle can only be in three states
Doing something Useful (consuming CPU) .. U
Idle, waiting for some work to be assigned .. I
Waiting for something, e.g.
.. W
a block from disk
a lock
a latch (could be waiting on CPU)

Response time = U + I + W
We must accurately measure each component time
before we decide what and how to tune

Wait Events in RAC

Cluster Coordination
Buffer Cache

Buffer Cache

SCN1

SCN2

DBWR

DBWR

LMS

Checkpo
int!

LMS

Node 1

Checkpo
int!

Node 2

Database
DBWR must get a lock on the database block
before writing to the
disk.
This is called a
RAC
for Beginners
Block Lock.

Cache Fusion
Will get it via
interconnect

session1

Wants to
modify it

Instance 1

session2

Instance 2
5

Has
modified
it

Wait Events in RAC

Checking for Buffers

How exactly is this check
performed?

By checking for a lock on the

block
The request comes to the Grant
Queue of the block
Someone checks that no other
instance has any lock
Instance 1 can read from the
disk
i.e. Instance 1 is granted the
lock
Wait Events in RAC

Block
Grant
Queue

Convert
Queue

SID1

SID5

SID2

SID6

SID3

SID7

Master Instance
Only one instance holds the grant and
convert queues of a specific block
This instance is called Master Instance of
that block
Grant
Queue
Master instance varies for each block
SID1
The memory structure that shows the
SID2
master instance of a buffer is called Global
SID3
Resource Directory (GRD)
That is replicated across all instances
The requesting instance must check the
GRD to find the master instance
Then make a request to the master
instance for the lock
Wait Events in RAC

Block
Convert
Queue
SID5
SID6
SID7

Scenario 1

Sessi
on

Instance 1

Instance 2

D
B

Session connected to Instance 1 wants to select a block

from the table
Activities by Instance 1
1. Check its own buffer cache to see if the block exists
1. If it is found, can it just use it?
2. If it not found, can it select from the disk?

2. If not, then check the other instances

How will it know which copy of the block is the best
source?
Wait Events in RAC

Cache Fusion
Buffer Cache

Buffer Cache

SMON

SMON

LMS
Node 1

LMS
Node 2

When node 2 wants a buffer, it sends a

message to the other instance. The
message is sent to the LMS (Lock
Management Server) of the other
instance. LMS then sends the buffer to the
Wait LMS
Events inis
RACalso called Global
other instance.

messa
ge
buffer

Grant Scenario 2
1. Check its buffer cache to see if the block exists
2. And the buffer is found. Can Instance1 use it?
Not really. The buffer may be old; it may have been changed

3.
3.
4.
5.

LMS of node1 sends message to master of the buffer

Master checks the GES and doesnt sees any lock
Instance 1 is granted the global block lock
No buffer actually gets transferred

Wait Events in RAC

10

Grant Scenario 3
Instance 1 is the master
Then it doesnt have to make a request for the grant
In summary, here are the possible scenarios when
Instance1 requests a buffer
Instance1 is the master; so no more processing is
required
No one has the lock on the buffer, the grant is made
by the master immediately
Another instance has the buffer in an incompatible
mode. It has to be changed.

Wait Events in RAC

11

Buffer States and Locks

Buffers can be gotten in two states
Current when the intention is to modify
Shared Current most recent copy. One copy per
instance. Same as disk
Exclusive Current only one copy in the entire
cluster. No shared current present

CR when the intention is to only select

Locks facilitate the state enforcement
XCUR for Exclusive Current
SCUR for Shared Current
No locking for CR
Wait Events in RAC

12

Placeholder Event
When the buffer is first requested, the session does not
know which of the three paths it will go on to
Therefore it is assigned a placeholder event
This event is known as gc cr block request (for
Consistent Read requests)
If the request is made for the buffer in Current mode, the
event is gc current block request
When one of the three options is chosen, the appropriate
event replaces the placeholder event

Wait Events in RAC

13

Grant Event
If the session merely requests a grant from the master:
It waits with the gc cr|current grant 2-way event
gc cr grant 2-way, for requests for buffers in
Consistent Read mode
gc current grant 2-way, for requests for buffers in
Current mode

Wait Events in RAC

14

gc current|cr grant 2-way

Block

gc file
current
block read
grant
2-way
request
db
scattered

Grant
Queue

Session

re
q
ue
s
t

LMS
GRD

GES

a
gr

LMS
Instance 2

Instance 1

Database

Wait Events in RAC

15

Conver
t Queue

SID
1

SID
5

SID
2

SID
6

SID
3

SID
7

ed
t
n

Block Event
After the request is made
and assuming that the buffer is not in the local cache
the buffer may be found in another instance
Requestor requests the buffer from the holding
instance
Possibilities:
Holder is also the master of that buffer
Holder is not the master
If the holder is the master, the requesting session waits
with the event gc cr|current block 2-way

Wait Events in RAC

16

gc current|cr block 2-way

Block

gc
gccurrent
currentblock
block2-way
request

Grant
Queue

Session

LMS
GRD

LMS

GES

Instance 2

Instance 1

Database

Wait Events in RAC

17

Conver
t Queue

SID
1

SID
5

SID
2

SID
6

SID
3

SID
7

gc current|cr block 3-way

Block

gc current block 3-way

Grant
Queue

Session

Instance 2

Instance 1

Requestor
Wait Events in RAC

Instance 3
18

Conver
t Queue

SID
1

SID
5

SID
2

SID
6

SID
3

SID
7

Master

Holder

Grant vs- Get Events

Action

Wait Event is

Block is requested by a session

Placeholder event gc cr|current

block request

Buffer lock is requested from the

master

gc cr|current grant 2-way

Buffer lock is granted, get from

the disk

db file scattered|sequential read

Buffer is requested from the

holder which is the same as the
master

gc cr|current block 2-way

Buffer is requested from the

holder which is not the master

gc cr|current block 3-way

1. There is no 3-way grant event, since the request is made

to the master
2. There is no 4-way block event; since there will a maximum
of 2 hops: requestor master holder
Wait Events in RAC

19

Interpretation
Scenario

Interpretation

Tuning Implication

The grant waits are

very high compared to
block waits

The requested blocks

are in the current
instance; but the
master is another
instance

Manually remaster the

object?

The 2-way block waits

The cache fusion is
are very high compared taking place as
to 3-way
expected, and most of
the blocks are held and
mastered at the same
instance. But most of
the blocks are not
found in the local
instance.

Faster interconnect
Shorten the run queue

The 3-way block waits

are high compared to
2-way

Manually remaster the

object
Application
partitioning

Cache fusion is taking

place; but the master
and the holder are
usually
Wait
Eventsdifferent.
in RAC
20

gc current/cr block lost

Lost blocks while being transferred to
the remote instance [in the
interconnect]
Cause: may not be the network itself
Cause #1 Network
Network traffic drops the packets
Confirm from the ifconfig -a output
If packets are dropping, this could
be a cause
Why?
Bad network configuration
CPU used for network
processing
Cause #2 CPU
LMS process is CPU starved
Wait Events in RAC

gc
gccurrent
currentblock
block2-way
request
SID
1

Session
Session

LMS
LMS
GR
GR
D
D

Block

SID
5

SID
2

SID
6

SID
3

SID
7

LMS
LMS
Instance
Instance 2
2

GES
GES

Instance
Instance 1
1

G
r
a
n
t
Q
u
e
u
e

Database

21

C
o
n
v
e
r
t
Q
u
e
u
e

gc current/cr block busy

What it Means
Session wants a block from the remote instance
Remote instance delays preparing the block to
fulfill the request [CR or current] read
Cause:
Local delay on the remote instance
Most likely: an I/O bottleneck on the remote
instance
block is being accessed by some session which is CPU
starved
LGWR has not written a buffer to redo yet

Less likely: CPU starvation

Wait Events in RAC

22

gc current/cr block congested

Meaning
The instance has requested block from remote instance.
Remote instance has prepared the block and shipped it
but it has not reached the requesting instance within 1 ms.

What could be cause network bottleneck?

Not necessarily. Causes:
Long run queues, causing the LMS process to be
delayed in processing the incoming block
Heavy paging due to memory deficiency. This causes
the blocks to be paged in before being processed.

Wait Events in RAC

23

Putting it All Together

Every Oracle process is either
Doing some productive work
Waiting for some work to be given (idle)
Waiting for some resource
Understand the reason for the wait
Devise a plan accordingly
RAC related wait events are manifestations of these
issues mostly
Caused by
Network issues
LMS being overloaded
Blocks busy
Wait Events in RAC

24

Thank You!
Session 362
More Information:
Blog: arup.blogspot.com
100 Things You Probably Didn't Know About Oracle
Database http://bit.ly/evr05e

Twitter: arupnanda
Wait Events in RAC

25