Automatic Storage Management

Julian Dyke Independent Consultant Web Version - December 2008

1

2008 Julian Dyke

juliandyke.com

Objectives

1.

Understand how Oracle database files are stored in ASM Calculate how long ASM rebalance operations will take

2.

2008 Julian Dyke

juliandyke.com

Agenda

ASM Instances ASM Disk Groups Metadata Extent Distribution Rebalancing Redundancy

2008 Julian Dyke

juliandyke.com

ASM Instances

2008 Julian Dyke

juliandyke.com

ASM Single Instance Architecture

OCSSD Daemon Only Oracle Clusterware ASM Instance RDBMS Instance

Server
Dedicated Storage

2008 Julian Dyke

juliandyke.com

ASM Single Instance Background Processes

Oracle 11.1

DBW0 LGWR CKPT

VKTM

PSP0 Fixed Area

MMAN

RBAL GMON X000

Variable Area SMON PMON ASM Cache DIAG DIA0

2008 Julian Dyke

juliandyke.com

ASM RAC Architecture

Public Network Private Network

Oracle Clusterware ASM Instance RDBMS Instance

Oracle Clusterware ASM Instance RDBMS Instance

Oracle Clusterware ASM Instance RDBMS Instance

Oracle Clusterware ASM Instance RDBMS Instance

Node 1
Storage Network

Node 2

Node 3

Node 4

Shared Storage

2008 Julian Dyke

juliandyke.com

ASM RAC Architecture

CLUSTERWARE +ASM1 Clusterware ASM Instances RDBMS Instances CLUSTERWARE +ASM2

PROD1

TEST1

PROD2

TEST2

PROD TEST Database Files

8

2008 Julian Dyke

juliandyke.com

ASM RAC Instance Background Processes

Oracle 11.1 LMON DBW0 LGWR CKPT SMON PMON MARK ASM Cache Variable Area LMS0 VKTM LMD0 PSP0 Fixed Area LCK0 MMAN DIAG RBAL GMON X000 DIAG DIA0 KATE

2008 Julian Dyke

juliandyke.com

ASM Disk Groups

10

2008 Julian Dyke

juliandyke.com

ASM Disk Groups and Disks

Disk Group 1 Disk Group 2 Disk Group 3

Disk 1

Disk 2

Disk 5

Disk 3

Disk 4

Disk 6

Disk 7

11

2008 Julian Dyke

juliandyke.com

ASM Disk Groups and Disks

Disk Group 1 Disk Group 2 Disk Group 3

Disk 1

Disk 4

Disk 2

Disk 5

Disk 6

Disk 7

Disk 3

12

2008 Julian Dyke

juliandyke.com

ASM Disk Groups, Disks and Database Files

Disk Group 1 Disk Group 2 Disk Group 3

File 3

File 2 File 1

File 4

File 5

File 5 Disk 1 Disk 4 Disk 2

File 6

Disk 3

Disk 5

Disk 6

Disk 7

13

2008 Julian Dyke

juliandyke.com

File Extents versus Allocation Units

File Extent Logical unit of ASM file Map to allocation units One to many mapping Allocation Unit Physical unit of ASM disk Oracle 10.2 and below Always 1MB Can be increased using _asm_ausize Oracle 11.1 and above Variable size 1MB, 2MB ,4MB, 8MB, 16MB, 32MB, 64MB

14

2008 Julian Dyke

juliandyke.com

X$KFFXP

Maps file extents to allocation units Only populated in ASM instance Columns include
Column Name GROUP_KFFXP NUMBER_KFFXP COMPOUND_KFFXP INCARN_KFFXP PXN_KFFXP XNUM_KFFXP LXN_KFFXP DISK_KFFXP AU_KFFXP SIZE_KFFXP Description Disk Group Number File Number Disk Group Number || File Number Incarnation Number Physical Extent Number (within file) Logical Extent Number (within file) 0=primary, 1=first mirror, 2=second mirror Disk Number Allocation Unit Number (within disk) Size (# allocation units)

15

2008 Julian Dyke

juliandyke.com

ASM Metadata

Metadata is stored in first 256 files in ASM disk group Space is initially allocated when disk group is created Can be subsequently extended Metadata allocation units are divided into blocks Each block is 4096 bytes Block size specifed using _asm_blksize
File# Description Metadata Header File Directory Disk Directory Active Change Directory Continuing Operations Directory Template Directory Alias Directory Attribute directory (optional) Staleness registry (optional)

Metadata files include

0 1 2 3 4 5 6 9 12

16

2008 Julian Dyke

juliandyke.com

ASM Metadata
Metadata Header Partner Status Table File Directory Disk Directory Active Change Directory Continuing Operations Directory Template Directory Alias Directory Metadata Header Disk Header Free Space Table Allocation Table

17

2008 Julian Dyke

juliandyke.com

ASM Metadata

Initial Allocation (Single Instance)

File# AU 0 Description 0Disk Header, Free Space Table, Allocation Table 1Partner Status Table 1 2 3 4 File Directory Disk Directory Active Change Directory Continuing Operations Directory # AUs 1 1 1 1 42 2 1 1

5 Template Active Change DirectoryDirectory Records changes to metadata 6 Alias Directory Used during recovery of instance or operation failures Continuing Operations Directory Maintains state of active operations

18

2008 Julian Dyke

juliandyke.com

ASM Metadata Block Types

Type 1 2 3 4 5 6 7 8 9 10 11 12 Description KFBTYP_DISKHEAD KFBTYP_FREESPC KFBTYP_ALLOCTBL KFBTYP_FILEDIR KFBTYP_LISTHEAD KFBTYP_DISKDIR KFBTYP_ACDC KFBTYP_CHNGDIR KFBTYP_CODBGO KFBTYP_TMPLTDIR KFBTYP_ALIASDIR KFBTYP_INDIRECT Type 13 14 15 16 17 18 19 20 21 22 23 Description KFBTYP_PST_NONE KFBTYP_HASHNODE KFBTYP_COD_RBO KFBTYP_COD_DATA KFBTYP_PST_META KFBTYP_PST_DTA KFBTYP_HBEAT KFBTYP_SR KFBTYP_STALEDIR KFBTYP_VOLUMEDIR KFBTYP_ATTRDIR

19

2008 Julian Dyke

juliandyke.com

KFED Utility

In Oracle 10.2 and above the kfed utility can be used to inspect and edit the contents of ASM blocks
[oracle@server3 ~]$ $ORACLE_HOME/bin/kfed -h as/mlib ASM Library [asmlib='lib'] aun/um AU number to examine or update [AUNUM=number] aus/z Allocation Unit size in bytes [AUSZ=number] blkn/um Block number to examine or update [BLKNUM=number] blks/z Metadata block size in bytes [BLKSZ=number] ch/ksum Update checksum before each write [CHKSUM=YES/NO] cn/t Count of AUs to process [CNT=number] d/ev ASM device to examine or update [DEV=string] o/p KFED operation type [OP=READ/WRITE/MERGE/NEW/FORM/FIND/STRUCT] p/rovnm Name for provisioning purposes [PROVNM=string] s/eek AU number to seek to [SEEK=number] te/xt File name for translated block text [TEXT=string] ty/pe ASM metadata block type number [TYPE=number]

This utility should only be used under the guidance of Oracle Support

20

2008 Julian Dyke

juliandyke.com

KFED Utility

For example to dump blocks in aliases directory in DISKGROUP1 Find group number SELECT group_number FROM v$asm_diskgroup WHERE name = 'DISKGROUP1'; Alias directory is stored in file number 6 SELECT disk_kffxp, au_kffxp FROM x$kffxp WHERE group_kffxp = 1 AND number_kffxp = 6 AND lxn_kffxp = 0; Disk 0 Find disk name SELECT path FROM v$asm_disk WHERE group_number = 1 AND disk_number = 0; Path /dev/oracleasm/disks/VOL1 Allocation Unit 49

21

2008 Julian Dyke

juliandyke.com

KFED Utility

Example (continued) Allocation unit is 1MB Block size is 4096 Therefore there are 256 blocks per allocation unit Starting block offset = 256 * 49 = 12544 for (( f = 12544 ; f < 12544 + 256 ; f++ )) do kfed op=read blkn=$f dev='/dev/oracleasm/disks/VOL1' > blk${f} done

22

2008 Julian Dyke

juliandyke.com

Extent Distribution

23

2008 Julian Dyke

juliandyke.com

Extent Distribution

Creating a disk group: CREATE DISKGROUP diskgroup1 EXTERNAL REDUNDANCY DISK '/dev/oracleasm/disks/VOL1';

Dropping a disk group: DROP DISKGROUP diskgroup1 INCLUDING CONTENTS

24

2008 Julian Dyke

juliandyke.com

Extent Distribution 1 disk

Metadata

0 1 2 3 4 5 6 7

Disk 0 Metadata Data

25

2008 Julian Dyke

juliandyke.com

Extent Distribution 2 disks

1 3 5 7 9 11 13 15

Metadata

0 2 4 6 8 10 12 14

Disk 0 Metadata

Disk 1 Data

26

2008 Julian Dyke

juliandyke.com

Extent Distribution 4 disks

2 6 10 14 18 22 26 30 1 5 9 13 17 21 25 29 3 7 11 15 19 23 27 31

Metadata

0 4 8 12 16 20 24 28

Disk 0

Disk 1 Metadata

Disk 2 Data

Disk 3

27

2008 Julian Dyke

juliandyke.com

Extent Distribution 1 large disk - 1 small disk

2 5 8 11

Metadata

0 1 3 4 6 7 9 10

Disk 0 Metadata

Disk 1 Data

28

2008 Julian Dyke

juliandyke.com

Extent Distribution 1 large disk - 3 small disks

2 7 12 17 1 6 11 16 4 9 14 19

Metadata

0 3 5 8 10 13 15 18

Disk 0

Disk 1 Metadata

Disk 2 Data

Disk 4

29

2008 Julian Dyke

juliandyke.com

Extent Distribution 2 large disks - 2 small disks

1 4 7 10 13 16 19 22 3 9 15 21 5 11 17 23

Metadata

0 2 6 8 12 14 18 20

Disk 0

Disk 1 Metadata

Disk 2 Data

Disk 4

30

2008 Julian Dyke

juliandyke.com

Rebalancing

31

2008 Julian Dyke

juliandyke.com

Extent Distribution

Adding a disk: ALTER DISKGROUP diskgroup1 ADD DISK '/dev/oracleasm/disks/VOL2' REBALANCE POWER 0;

Dropping a disk: ALTER DISKGROUP diskgroup1 DROP DISK 'DISKGROUP1_0002' REBALANCE POWER 0;

Rebalancing a disk group: ALTER DISKGROUP diskgroup1 REBALANCE POWER 1;

32

2008 Julian Dyke

juliandyke.com

Rebalancing Adding disks - 1 disks to 2 disks

0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 1 3 5 7 1 3 5 7

Disk 0 Metadata

Disk 1 Data

33

STOP

2008 Julian Dyke

juliandyke.com

Rebalancing Adding disks - 1 disks to 4 disks

0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 2 6 2 6 1 5 1 5 0 4 0 4

Disk 0

Disk 1 Metadata

Disk 2 Data

Disk 3

34

STOP

2008 Julian Dyke

juliandyke.com

Rebalancing Adding disks - 2 disks to 3 disks

0 1 2 3 4 5 1 3 5 7 9 11 13 15 17 0 2 1 4 6 8 4 10 12 14 16 0 3 2 5 8 11 14 17

Disk 0

Disk 1 Metadata Data

Disk 2

35

2008 Julian Dyke

juliandyke.com

Rebalancing Adding disks - 2 disks to 4 disks

0 1 2 3 4 5 6 7 1 3 5 7 9 11 13 15 0 2 2 4 6 6 8 10 12 14 1 5 1 5 9 13 0 4 0 4 8 12

Disk 0

Disk 1 Metadata

Disk 2 Data

Disk 3

36

STOP

2008 Julian Dyke

juliandyke.com

Rebalancing Dropping disks - 3 disks to 1 disk

0 3 0 3 6 1 2 4 5 1 2 4 5 7 8 1 4 1 4 7 2 5 2 5 8

Disk 0

Disk 1 Metadata Data

Disk 2

37

STOP

2008 Julian Dyke

juliandyke.com

Rebalancing Moving disks - 2 disks to 2 disks

0 1 2 3 4 5 6 7 1 3 5 7 9 11 13 15 0 2 4 6 8 10 12 14 0 2 4 6 0 2 4 6 8 10 12 14

Disk 0

Disk 1 Metadata Data

Disk 2

38

STOP

2008 Julian Dyke

juliandyke.com

Rebalancing V$ASM_OPERATION

Contains details of ongoing rebalance operations

Column Name GROUP_NUMBER OPERATION STATE POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTE ERROR_CODE Data Type NUMBER CHAR(5) VARCHAR2(4) NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER VARCHAR2(44)

Estimate of remaining time

39

2008 Julian Dyke

juliandyke.com

Rebalancing Power Limit

Power limit can be 0 to 11 0 disables rebalance operation 1 to 11 specifies number of ARBn background processes used for rebalance In Oracle 10.2 RBAL manages rebalance operation Each ARBn background process is allocated a range of 128 allocation units to rebalance When complete another range is requested AD lock is taken while an allocation unit is being rebalanced Rebalance operations take much longer than theoretically necessary. Possible reasons include: Locking GES updates with other ASM instances Updates to RDBMS instance

40

2008 Julian Dyke

juliandyke.com

Rebalancing Summary

EST_MINUTES column of V$ASM_OPERATION is reasonably accurate Allow a few minutes for SAN cache to stabilize Check regularly for changes to estimate ASM rebalance operations do not affect workload Locks are only taken briefly Lock mechanism has changed in Oracle 11.1 SAN cache and I/O performance will be affected In Oracle 10.2 rebalancing is fastest if Other ASM instances are shutdown RDBMS instance is shutdown Estimated completion time will be affected by: Use of SAN cache and I/O by rest of workload Rate of change by applications to blocks in ASM files being rebalanced

41

2008 Julian Dyke

juliandyke.com

Redundancy

42

2008 Julian Dyke

juliandyke.com

Redundancy

ASM Supports three levels of redundancy

External Redundancy Implemented externally using storage layer Most common configuration in production Normal Redundancy Two copies of each extent maintained in separate failure groups Used with extended clusters Used occasionally in production e.g. CERN Increases CPU overhead on servers High Redundancy Three copies of each extent maintained in separate failure groups Very rare in production

43

2008 Julian Dyke

juliandyke.com

ASM Failure Groups - External Redundancy

Disk Group

Disk 1

Disk 2

Disk 3

44

2008 Julian Dyke

juliandyke.com

ASM Failure Groups - Normal Redundancy

Disk Group

Failure Group 1

FailureGroup 2

Disk 1

Disk 2

Disk 3

Disk 4

Disk 5

Disk 6

45

2008 Julian Dyke

juliandyke.com

ASM Failure Groups - High Redundancy

Disk Group

Failure Group 1

Failure Group 2

Failure Group 3

Disk 1

Disk 2

Disk 1

Disk 2

Disk 1

Disk 2

46

2008 Julian Dyke

juliandyke.com

Normal Redundancy 1 Disk Per Failure Group

Failure Group 1
0 1 2 3 0 1 2 3 4 5 6 7 8 9 10 11

Failure Group 2
0 1 2 3 0 1 2 3 4 5 6 7 8 9 10 11

Disk 0 Metadata Data

47

Disk 1 Primary Primary Secondary Secondary

2008 Julian Dyke

juliandyke.com

Normal Redundancy 2 Disks per Failure Group

Failure Group 1
0 3 4 7 0 3 4 7 8 11 12 15 16 19 20 23 1 2 5 6 1 2 5 6 9 10 13 14 17 18 21 22 0 3 4 7 0 3 4 7 8 11 12 15 16 19 20 23

Failure Group 2
1 2 5 6 1 2 5 6 9 10 13 14 17 18 21 22

Disk 0 Metadata Data

48

Disk 1 Primary Primary

Disk 2 Secondary Secondary

Disk 3

2008 Julian Dyke

juliandyke.com

High Redundancy 1 Disk per Failure Group

Failure Group 1
0 1 2 3 4 5 0 1 2 3 4 5 6 7 8 9

Failure Group 2
0 1 2 3 4 5 0 1 2 3 4 5 6 7 8 9

Failure Group 3
0 1 2 3 4 5 0 1 2 3 4 5 6 7 8 9

Disk 0 Metadata Data

49

Disk 1 Primary Primary Secondary Secondary

Disk 2 Tertiary Tertiary

2008 Julian Dyke

juliandyke.com

References

Oracle Automatic Storage Management (Oracle Press) Nitin Vengurlekar Murali Vallath Rich Long What ASM and ZFS Can Do For You Jason Arneil - Nominet A Closer Look Inside Oracle ASM Luca Canali - CERN Implementing ASM Without HW Raid Luca Canali - CERN

50

2008 Julian Dyke

juliandyke.com

Thank you for listening

info@juliandyke.com
51

2008 Julian Dyke

juliandyke.com