本文共 14396 字,大约阅读时间需要 47 分钟。
ORA-15041 IN A DISKGROUP ALTHOUGH FREE_MB REPORTS SUFFICIENT SPACE (文档 ID 460155.1)
--------------------------------------------------------------------------------修改时间:2013-1-3类型:BULLETIN
In this Document
Purpose Scope Details Capacity of disks within ASM diskgroup are different ASM instance is shutdowned normal/immediate before a rebalance is completed Disk is DROPPING / HUNG state After an add disk command the rebalance is still in place
--------------------------------------------------------------------------------
Applies to: Oracle Server - Enterprise Edition - Version 10.1.0.2 to 11.2.0.4 [Release 10.1 to 11.2] Information in this document applies to any platform.
***Checked for relevance on 02-Jan-2013*** Purpose ORA-15041 is the most common space error reported when disk free space is not sufficient to complete file allocation request. Due to imbalanced space distribution, ORA-15041 can still be encountered although ASM views reports sufficient free space. This article is intended to help the reader to understand the most common reasons for ORA-15041 error although sufficient free space is reported in ASM views. Error: ORA-15041 (ORA-15041) Text: diskgroup space exhausted --------------------------------------------------------------------------- Cause: The diskgroup ran out of space. Action: Add more disks to the diskgroup, or delete some existing files. Scope This article is intended to help the reader to understand the most common reasons for ORA-15041 error in ASM diskgroups.Details ASM spreads file extents evenly accross all the disks disks on a diskgroup accordingly with capacity, free space within diskgroup and its redundancy type. Total free Mega-bytes in an ASM diskgroup is reported in FREE_MB column but the maximum space that can be allocated actually changes with the type of redundancy as summarized below: Redundancy type Max.Space ---------------- ------------------------- External FREE_MB of diskgroup Normal 1/2 FREE_MB of diskgroup High 1/3 FREE_MB of diskgroup v$ASM_DISK.FREE_MB column is simply a sum of free megabyte space reported in each v$asm_disk. On the other hand USABLE_FILE_MB in V$ASM_DISKGROUP also indicates the amount of free space, adjusted for mirroring, that is available for new allocations. Although FREE_MB and USABLE FILE_MB columns reports sufficient free space, an ORA-15041 error can still be encountered due to imbalanced free space between disks. The reason for this is that one disk lacking sufficient free space makes it impossible to do any allocation in a disk group because every file must be evenly allocated across all disks per ASM stripping policy.
Unbalanced disk configuration and certain operations on ASM disks can create this type of
problem. The problem frequently should be resolved after a succesfull rebalance as far as all disks have the same storage capacity and there is no underlying hardware problems. The most common reasons can be classified as follows:1- Capacity of disks within ASM diskgroup are different
2- ASM instance is shutdowned normal/immediate before a rebalance is ompleted. 3- Disk is DROPPING / HUNG state 4- After an add disk command the rebalance is still in place Capacity of disks within ASM diskgroup are different When an asm diskgroup is having the disks in different capacity, one disk lacking free space makes it impossible to do any allocation in the other disks as well. This is expected behaviour because every file must be evenly allocated across all disks. Rebalancing and allocation attempts to make the percentage of allocated space about the same on every disk. File allocation may fail with an ORA-15041 in case of imbalanced space distribution.Extents are allocated evenly accross the disks accordingly with the capacity
(TOTAL_MB) of disks. If all ASM disks are of the same size (e.g. 10 disks, 50GB each), ASM allocator places extents on each disk in a sequence. The first disk allocation is chosen randomly, but all subsequent disks for extent allocation are chosen to evenly spread each file across all disks and to evenly fill all disks. On the other hand, if ASM disks are not of the same size (e.g. disk 1 is 10GB, disk 2 is 50GB and disks 3-10 are 10GB), ASM allocator will place one extent on disk 1, five extents on disk 2, one extent on disk 3 and so on. This is to ensure balanced disk utilization.Extent allocation also differs with the type of redundancy. If redundancy is
NORMAL/HIGH, no allocation is possible when free space in any of the disks is not sufficient for the requested allocation size. On the other hand, in an external redundancy diskgroup, ASM distributes the extents evenly across the disks accordingly with the capacity (TOTAL_MB) of disks and the allocation continue till there exists at least two disks having enough space to complete the allocation.The following test demonstrates the space allocation behaviour according
to redundancy type when there exists disks with different capacity in the diskgroups.DG_EXT: External redundancy diskgroup with 3 disks: 447M, 447M, 70M bytes. DG_NOR: Normal redundancy diskgroup with 3 disks: 447M, 447M, 70M bytes.
ASM/Database instance version is 10.2.0.3.
In order to test the file allocation on different redundancy types (external/normal), files with different sizes are created on diskgroups. In order to ensure the same amount of free space left after each file creation, size of the files created on external redundancy diskgroup is double the amount of the file sizes created on normal redundancy diskgroup. The following table demonstrates the free space after each file creation and the point where ORA-15041 error is encountered on each type of diskgroup.FREE_MB(0): Initial free space at each disk.
FREE_MB(1): After 200M/100M files are created external/normal redundancy diskgroup. FREE_MB(2): After 100/50M files are created external/normal redundancy diskgroup. FREE_MB(3): After 500M/- file is created on external redundancy diskgroup. NAME TOTAL_MB FREE_MB(0) FREE_MB(1) FREE_MB(2) FREE_MB(3) +200M/+100M +100M/+50M +500M/- ------------ ---------- ---------- ------------ ---------- ---------- DG_EXT_0001 447 423 330 283 50 ** DG_EXT_0000 447 421 326 279 48 ** DG_EXT_0002 70 66 51 43 8 ** DG_NOR_0001 447 395 301 ORA-15041 * X DG_NOR_0000 447 395 301 ORA-15041 * X DG_NOR_0002 70 18 1 ORA-15041 * X* File allocation (50MB) on normal redundancy diskgroup fails with ORA-15041 when there is no more than 50M at a single disk. This is mainly because normal redundancy diskgroup can’t allocate primary/secondary extents of file on separate disks due to insufficient space. ** On the other hand, file allocation (100M + 500M) succeeds for external redundancy diskgroup as this type of redundancy only stripes data over available free space in all disks. Further tests show that file creation in external redundancy is possible as long as there is some space in at least two disks in the diskgroup. ASM instance is shutdowned normal/immediate before a rebalance is completed A rebalance can be stopped if ASM instance is shutdowned and it is expected that rebalance should resume after the instance is restarted. However, due to a known issue (Unpublished Bug 5089819) if ASM instance is shutdowned with normal/immediate option, rebalance doesn't kick off again upon a new startup.ASM instance requires to either do a shutdown abort or restart rebalance manually.NAME GROUP_NUMBER TOTAL_MB FREE_MB ------------------------------ ------------ ---------- ---------- DG_NORMAL 2 894 386 NAME GROUP_NUMBER TOTAL_MB FREE_MB ------------------------------ ------------ ---------- ---------- DG_NORMAL_0000 2 447 193 DG_NORMAL_0001 2 447 193A new disk is being added:
alter diskgroup dg_normal add disk '/dev/hdb10';
select * from v$asm_operation;GROUP_NUMBER OPERA STAT POWER SOFAR EST_WORK EST_RATE
------------ ----- ---- ---------- ---------- ---------- ---------- 2 REBAL RUN 1 75 194 181ASM instance is shutdowned:
SQL> shutdown immediate
ASM diskgroups dismounted ASM instance shutdown SQL>Upon startup, there is no relabalance operation going on and free space in asm disks is not balanced.
NAME GROUP_NUMBER TOTAL_MB FREE_MB
------------------------------ ------------ ---------- ---------- DG_NORMAL 2 1341 781NAME GROUP_NUMBER TOTAL_MB FREE_MB
------------------------------ ------------ ---------- ---------- DG_NORMAL_0000 2 447 240 DG_NORMAL_0001 2 447 239 DG_NORMAL_0002 2 447 302Normally, we should be able create a file with approx. 350-400M as
asm diskgroup reports sufficient space.SQL> alter tablespace test1 add datafile '+DG_NORMAL' size 370m;* ERROR at line 1: ORA-01119: error in creating database file '+DG_NORMAL' ORA-17502: ksfdcre:4 Failed to create file +DG_NORMAL ORA-15041: diskgroup space exhaustedSQL> alter diskgroup dg_normal rebalance power 11; SQL> alter tablespace test1 add datafile '+DG_NORMAL' size 370m;Tablespace altered
A new rebalance remedies this situation. Diskgroup has the following free
space figures after 370MB file is created. NAME GROUP_NUMBER TOTAL_MB FREE_MB ------------------------------ ------------ ---------- ---------- DG_NORMAL 2 1341 70NAME GROUP_NUMBER TOTAL_MB FREE_MB ------------------------------ ------------ ---------- ---------- DG_NORMAL_0000 2 447 24 DG_NORMAL_0001 2 447 22 DG_NORMAL_0002 2 447 24 Disk is DROPPING / HUNG state Free space in an asm diskgroup can be imbalanced if a drop disk fails for any reason (lack of space, disk crash, etc.). Disks may be stuck in the DROPPING state in this case.The most common reasons for DROPPING state are that a careless drop disk command
is submitted on a diskgroup runing with full capacity or dropping the disk reduces the amount of available disk space to less than that required for all the existing extents. After a drop disk command, a rebalance is triggered and completed however there exits disks at DROPPING state in this case.It is not possible to allocate space from diskgroup any more as no free space is
also reported in v$asm_diskgroup. To resolve the problem, you can either add more disks to provide extra space or undrop the disk to roll back the drop.- Add more disks
Adding more disks provides starts a rebalance implicity and provides extra space for the rebalance to complete. Once the data is copied out of the dropping disks, they will be expelled out of the diskgroup.alter diskgroup add disk 'path';- Undrop the disk
when an undrop command is issued, it simply rolls back the drop. If the disks dropping has not gone too far, ASM will be able to re-integrate the disks back into the diskgroup. UNDROP DISKS triggers a rebalance implicitly which rolls back the drop and make the space again available to diskgroup. Space should be balanced between disks once the command is completed.alter diskgroup undrop disks; While disks are runing near to capacity, imagine a drop disk brings the disk state to HUNG. Drop disk can't be completed as due to lack of space as current extents can't be fit into the remaining disks.NAME TOTAL_MB FREE_MB STATE
------------------------------ ---------- ---------- -------- DG2_0001 447 97 NORMAL DG2_0000 447 90 NORMAL DG2_0002 447 97 NORMALNAME TOTAL_MB FREE_MB TYPE ------------------------------ ---------- ---------- ------ DG2 1341 284 EXTERNalter diskgroup DG2 drop disk DG2_0002; While rebalance is runing, disk state stays at DROPPING but it changes to HUNG after rebalance is completed.NAME TOTAL_MB FREE_MB STATE ------------------------------ ---------- ---------- -------- DG2_0001 447 7 NORMAL DG2_0000 447 6 NORMAL DG2_0002 447 271 DROPPING
After rebalance is complete, disk state is HUNG as disk can't be expelled out from the diskgroup. NAME TOTAL_MB FREE_MB STATE ------------------------------ ---------- ---------- -------- DG2_0001 447 0 NORMAL DG2_0000 447 0 NORMAL DG2_0002 447 284 HUNG
alter diskgroup DG2 undrop disks; Undrop disk triggers a new rebalance implicitly and resolves the problem. This state can also be resolved with ADD DISK by providing extra space for the rebalance to complete. Once the data is copied out of the dropping disks they will be expelled out of the diskgroup. After an add disk command the rebalance is still in place When a disk is added to a disk group its space is not immediately available for allocation. Since every file must be evenly allocated, extents must be rebalanced off other disks to the new disk to make space evenly available.
Free space will be available in the course of time while rebalance is
progressing. Since rebalance takes a while, users may not be able to allocate files and could get out of space errors (ORA-15041).As a workaround, WAIT option with ADD disk command can be used. If the WAIT option given with add disk, the command doesn't return until rebalance is complete. This may provide more intuitive to users who run disks with near to full capacity.
The following test shows how free space at each disk change while rebalance is going.
NAME GROUP_NUMBER TOTAL_MB FREE_MB
------------------------------ ------------ ---------- ---------- TEST_DG 2 894 48PATH GROUP_NUMBER TOTAL_MB FREE_MB -------------------- ------------ ---------- ---------- /dev/hdb8 2 447 24 /dev/hdb9 2 447 24 Two more 457MB disks are added to diskgroup. alter diskgroup test_dg add disk '/dev/hdb10','/dev/hdb11';Free space is reported immediately in v$asm_diskgroup however it is imbalanced as rebalance is not completed yet.NAME TOTAL_MB FREE_MB(1) FREE_MB(2) ... FREE_MB(3) ------------- ---------- ---------- ---------- ---------- TEST_DG 1788 888 887 885PATH TOTAL_MB FREE_MB(1) FREE_MB(2) ... FREE_MB(3) -------------- ---------- ---------- ---------- ---------- /dev/hdb8 447 36 144 221 /dev/hdb9 447 36 144 220 /dev/hdb10 447 409 300 223 /dev/hdb11 447 407 299 221Free space is getting balanced while the rebalance is progressing. Till the
rebalance is completed large file allocations may fail with ORA-15041 errors although free space is reported in v$asm_diskgroup.As a workaround, WAIT option can be used with add disk command. When the WAIT
option is used, add disk command doesn't return until rebalance is complete. This may provide more intuitive results when running disks with near to full capacity.@However if we tried to do unbalanced disk groups
来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/19423/viewspace-1130911/,如需转载,请注明出处,否则将追究法律责任。
转载于:http://blog.itpub.net/19423/viewspace-1130911/