How to fix poor performance issues on Exadata cellnode flash disks without replacement?

Exadata cellnode flash disks can sometimes have poor performance. If the error numbers are 0 (zero) or very low when you look at the disk details, you can normalize the status of these disks as follows.

We connect to the cell node and run the cellcli command.

[root@oradb1cel01 ~]# cellcli
CellCLI: Release 11.2.3.3.0 - Production on Fri Sep 09 11:34:31 EEST 2016

Copyright (c) 2007, 2013, Oracle. All rights reserved.
Cell Efficiency Ratio: 1.178

We are questioning flash drives with a “poor performance” status.

CellCLI> LIST PHYSICALDISK WHERE DISKTYPE=flashdisk AND STATUS= 'poor performance' DETAIL
name: FLASH_4_0
diskType: FlashDisk
errCmdTimeoutCount: 0
errHardReadCount: 0
errHardWriteCount: 0
errMediaCount: 0
errOtherCount: 0
errSeekCount: 0
luns: 4_0
makeModel: "MARVELL SD88SA02"
physicalFirmware: D20Y
physicalInsertTime: 2011-11-29T10:20:15+02:00
physicalSerial: 1116M08J54
physicalSize: 22.8880615234375G
sectorRemapCount: 0
slotNumber: "PCI Slot: 4; FDOM: 0"
status: poor performance

name: FLASH_4_1
diskType: FlashDisk
errCmdTimeoutCount: 0
errHardReadCount: 0
errHardWriteCount: 0
errMediaCount: 0
errOtherCount: 0
errSeekCount: 0
luns: 4_1
makeModel: "MARVELL SD88SA02"
physicalFirmware: D20Y
physicalInsertTime: 2011-11-29T10:20:15+02:00
physicalSerial: 1116M08J55
physicalSize: 22.8880615234375G
sectorRemapCount: 0
slotNumber: "PCI Slot: 4; FDOM: 1"
status: poor performance

name: FLASH_4_2
diskType: FlashDisk
errCmdTimeoutCount: 0
errHardReadCount: 0
errHardWriteCount: 0
errMediaCount: 0
errOtherCount: 0
errSeekCount: 0
luns: 4_2
makeModel: "MARVELL SD88SA02"
physicalFirmware: D20Y
physicalInsertTime: 2011-11-29T10:20:15+02:00
physicalSerial: 1116M08J59
physicalSize: 22.8880615234375G
sectorRemapCount: 0
slotNumber: "PCI Slot: 4; FDOM: 2"
status: poor performance

name: FLASH_4_3
diskType: FlashDisk
errCmdTimeoutCount: 0
errHardReadCount: 0
errHardWriteCount: 0
errMediaCount: 0
errOtherCount: 0
errSeekCount: 0
luns: 4_3
makeModel: "MARVELL SD88SA02"
physicalFirmware: D20Y
physicalInsertTime: 2011-11-29T10:20:15+02:00
physicalSerial: 1116M08J58
physicalSize: 22.8880615234375G
sectorRemapCount: 0
slotNumber: "PCI Slot: 4; FDOM: 3"
status: poor performance

We look at the details of the flash disks. We prefer the error numbers to be 0 (zero).

CellCLI> list physicaldisk FLASH_4_0 detail;
name: FLASH_4_0
diskType: FlashDisk
errCmdTimeoutCount: 0
errHardReadCount: 0
errHardWriteCount: 0
errMediaCount: 0
errOtherCount: 0
errSeekCount: 0
luns: 4_0
makeModel: "MARVELL SD88SA02"
physicalFirmware: D20Y
physicalInsertTime: 2011-11-29T10:20:15+02:00
physicalSerial: 1116M08J54
physicalSize: 22.8880615234375G
sectorRemapCount: 0
slotNumber: "PCI Slot: 4; FDOM: 0"
status: poor performance

CellCLI> list physicaldisk FLASH_4_1 detail;
name: FLASH_4_1
diskType: FlashDisk
errCmdTimeoutCount: 0
errHardReadCount: 0
errHardWriteCount: 0
errMediaCount: 0
errOtherCount: 0
errSeekCount: 0
luns: 4_1
makeModel: "MARVELL SD88SA02"
physicalFirmware: D20Y
physicalInsertTime: 2011-11-29T10:20:15+02:00
physicalSerial: 1116M08J55
physicalSize: 22.8880615234375G
sectorRemapCount: 0
slotNumber: "PCI Slot: 4; FDOM: 1"
status: poor performance

CellCLI> list physicaldisk FLASH_4_2 detail;
name: FLASH_4_2
diskType: FlashDisk
errCmdTimeoutCount: 0
errHardReadCount: 0
errHardWriteCount: 0
errMediaCount: 0
errOtherCount: 0
errSeekCount: 0
luns: 4_2
makeModel: "MARVELL SD88SA02"
physicalFirmware: D20Y
physicalInsertTime: 2011-11-29T10:20:15+02:00
physicalSerial: 1116M08J59
physicalSize: 22.8880615234375G
sectorRemapCount: 0
slotNumber: "PCI Slot: 4; FDOM: 2"
status: poor performance

CellCLI> list physicaldisk FLASH_4_3 detail;
name: FLASH_4_3
diskType: FlashDisk
errCmdTimeoutCount: 0
errHardReadCount: 0
errHardWriteCount: 0
errMediaCount: 0
errOtherCount: 0
errSeekCount: 0
luns: 4_3
makeModel: "MARVELL SD88SA02"
physicalFirmware: D20Y
physicalInsertTime: 2011-11-29T10:20:15+02:00
physicalSerial: 1116M08J58
physicalSize: 22.8880615234375G
sectorRemapCount: 0
slotNumber: "PCI Slot: 4; FDOM: 3"
status: poor performance

We bring the status of all flash disks back to normal, one by one.

CellCLI> alter lun 4_0 reenable force;
LUN 4_0 on physical disk FLASH_4_0 successfully marked to status normal.
CellCLI> list physicaldisk FLASH_4_0 detail;
name: FLASH_4_0
diskType: FlashDisk
errCmdTimeoutCount: 0
errHardReadCount: 0
errHardWriteCount: 0
errMediaCount: 0
errOtherCount: 0
errSeekCount: 0
luns: 4_0
makeModel: "MARVELL SD88SA02"
physicalFirmware: D20Y
physicalInsertTime: 2011-11-29T10:20:15+02:00
physicalSerial: 1116M08J54
physicalSize: 22.8880615234375G
sectorRemapCount: 0
slotNumber: "PCI Slot: 4; FDOM: 0"
status: normal

CellCLI> alter lun 4_1 reenable force;
LUN 4_1 on physical disk FLASH_4_1 successfully marked to status normal.
CellCLI> list physicaldisk FLASH_4_1 detail;
name: FLASH_4_1
diskType: FlashDisk
errCmdTimeoutCount: 0
errHardReadCount: 0
errHardWriteCount: 0
errMediaCount: 0
errOtherCount: 0
errSeekCount: 0
luns: 4_1
makeModel: "MARVELL SD88SA02"
physicalFirmware: D20Y
physicalInsertTime: 2011-11-29T10:20:15+02:00
physicalSerial: 1116M08J55
physicalSize: 22.8880615234375G
sectorRemapCount: 0
slotNumber: "PCI Slot: 4; FDOM: 1"
status: normal

CellCLI> alter lun 4_2 reenable force;
LUN 4_2 on physical disk FLASH_4_2 successfully marked to status normal.
CellCLI> list physicaldisk FLASH_4_2 detail;
name: FLASH_4_2
diskType: FlashDisk
errCmdTimeoutCount: 0
errHardReadCount: 0
errHardWriteCount: 0
errMediaCount: 0
errOtherCount: 0
errSeekCount: 0
luns: 4_2
makeModel: "MARVELL SD88SA02"
physicalFirmware: D20Y
physicalInsertTime: 2011-11-29T10:20:15+02:00
physicalSerial: 1116M08J59
physicalSize: 22.8880615234375G
sectorRemapCount: 0
slotNumber: "PCI Slot: 4; FDOM: 2"
status: normal

CellCLI> alter lun 4_3 reenable force;
LUN 4_3 on physical disk FLASH_4_3 successfully marked to status normal.
CellCLI> list physicaldisk FLASH_4_3 detail;
name: FLASH_4_3
diskType: FlashDisk
errCmdTimeoutCount: 0
errHardReadCount: 0
errHardWriteCount: 0
errMediaCount: 0
errOtherCount: 0
errSeekCount: 0
luns: 4_3
makeModel: "MARVELL SD88SA02"
physicalFirmware: D20Y
physicalInsertTime: 2011-11-29T10:20:15+02:00
physicalSerial: 1116M08J58
physicalSize: 22.8880615234375G
sectorRemapCount: 0
slotNumber: "PCI Slot: 4; FDOM: 3"
status: normal

When we check the physical disks again, you will see that they are all back to normal.

CellCLI> LIST PHYSICALDISK
20:0 E0HLLP normal
20:1 E13DEC normal
20:2 E0J2DH normal
20:3 E0J7LA normal
20:4 E0L5WK normal
20:5 E0JART normal
20:6 E0K77N normal
20:7 E0L6NR normal
20:8 E0Z0QP normal
20:9 KGDAAL normal
20:10 E0E0KN normal
20:11 E0JATZ normal
FLASH_1_0 1106M06XVW normal
FLASH_1_1 1106M06XWG normal
FLASH_1_2 1106M06XUH normal
FLASH_1_3 1106M06XWE normal
FLASH_2_0 1106M06XDD normal
FLASH_2_1 1106M06XDK normal
FLASH_2_2 1106M06X07 normal
FLASH_2_3 1106M06X5E normal
FLASH_4_0 1116M08J54 normal
FLASH_4_1 1116M08J55 normal
FLASH_4_2 1116M08J59 normal
FLASH_4_3 1116M08J58 normal
FLASH_5_0 1116M08C6W normal
FLASH_5_1 1116M08C6X normal
FLASH_5_2 1116M08C6V normal
FLASH_5_3 1116M08C6U normal

Finally, don’t forget to check the flash cache status.

CellCLI> list flashcache
oradb1cel01_FLASHCACHE normal