Nov. 06, 2014

If you come on this page from google, don’t go further. You are looking for the command *“show storagearray longrunningoperations;”* ðŸ™‚

Yesterday one of ourAï¿½client had an issue during the upgrade from SNFS4 to SNFS5. WhileAï¿½troubleshooting the issue we had to fail and replace one of the metadata drive from the NetApp E-Series disk array.

Usually we just replace the failed drive with the new drive and check everything is good in the Quantum StorNext web interface and don’t bother too much about it.

ThisAï¿½time it’s a bit different since the first issue brought a lot of visibility to the SAN architecture. We wanted to keep everything stable and avoid toAï¿½impact the production system with useless latency, lower performances or downtime visible byAï¿½the users.

So when we decided to replace the metadata drive we wanted to see the progress of the drive replacement to make sure everything goesAï¿½smoothly and to be able to communicate when the SAN will be completely healthy and in an optimal state.

Unfortunately the only management options we have are:

- Quantum StorNext web interface
- NetApp E-Series CLI command (SMcli)

These 2 tools don’tAï¿½provide an intuitive way to check how far we are in the rebuild of the drive. StorNext webui doesn’t expose this detail of information and it took us more than 1 hour, 4 calls to support and few google queries to figure out how to get this information from the CLI (SMcli).

So if you want to know how your rebuild is progressing, you need to type the commandAï¿½*“show storagearray longrunningoperations;”:*

[root@node-1 ~]# SMcli Qarray1a -S -c "show storagearray longrunningoperations;" Long Lived Operations: LOGICAL DEVICES OPERATION STATUS TIME REMAINING 1 Copyback 86% Completed 17 min TRAY_85_VOL_4 Copyback 85% Completed 17 min [root@node-1 ~]#

Before to go there, we poked around trying the obvious commands to get these information with no luck.

*“show storagearray healthstatus”* tells us the array is fixing itself but no information about when it will be done:

[root@node-1 stornext]# SMcliAï¿½ localhostAï¿½ -c "show storagearray healthstatus;" Performing syntax check... Syntax check complete. Executing script... Storage array health status = fixing. The following failures have been found: Volume - Hot Spare In Use Storage array: Qarray1 Volume group: 1 Aï¿½ Status: Optimal Aï¿½ RAID level: 1 Aï¿½ Failed drive at: tray 85, slot 6 Aï¿½Aï¿½Aï¿½ Service action (removal) allowed: No Aï¿½Aï¿½Aï¿½ Service action LED on component: Yes Aï¿½ Replaced by drive at: tray 85, slot 1 Aï¿½ Volumes: TRAY_85_VOL_3, TRAY_85_VOL_4 Script execution complete. SMcli completed successfully. [root@node-1 stornext]#

*“show volumegroup[$number]”* tells usAï¿½who are the members of the volumegroup, but nothing about what is going on:

[root@node-1 stornext]# SMcliAï¿½ localhostAï¿½ -c "show volumegroup[1];" Performing syntax check... Syntax check complete. Executing script... DETAILS Aï¿½Aï¿½ Name:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 1 Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Status:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Optimal Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Capacity:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 558.410 GB Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Current owner:Aï¿½ Controller in slot B Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Quality of Service (QoS) Attributes Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ RAID level:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 1 Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Drive media type:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Hard Disk Drive Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Drive interface type:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Serial Attached SCSI (SAS) Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Tray loss protection:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ No Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Data Assurance (DA) capable:Aï¿½ Yes Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ DA enabled volume present:Aï¿½Aï¿½Aï¿½ No Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Total Volumes:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 2 Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Standard volumes:Aï¿½Aï¿½Aï¿½ 2 Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Repository volumes:Aï¿½ 0 Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Free Capacity:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 0.000 MB Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Associated drives - present (in piece order) Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Total drives present: 3 Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ TrayAï¿½Aï¿½Aï¿½Aï¿½ Slot Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 85Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 5 [mirrored pair with drive at tray 85, slot 1] Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 85Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 6 Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 85Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 1 [hot spare drive is sparing for drive at 85, 6] Script execution complete. SMcli completed successfully. [root@node-1 stornext]#

And if we check the drive itself nothing interesting shows up:

[root@node-1 stornext]# SMcliAï¿½ localhostAï¿½ -c "show drive[85,6];" Performing syntax check... Syntax check complete. Executing script... Drive at Tray 85, Slot 6 Aï¿½Aï¿½ Status:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Replaced Aï¿½Aï¿½ Mode:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Assigned Aï¿½Aï¿½ Raw capacity:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 558.912 GB Aï¿½Aï¿½ Usable capacity:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 558.412 GB Aï¿½Aï¿½ World-wide identifier:Aï¿½Aï¿½Aï¿½ 50:00:cc:a0:43:32:f6:90:00:00:00:00:00:00:00:00 Aï¿½Aï¿½ Associated volume group:Aï¿½ 1 Aï¿½Aï¿½ PortAï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Channel Aï¿½Aï¿½ 0Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 2 Aï¿½Aï¿½ 1Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 1 Aï¿½Aï¿½ Media type:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Hard Disk Drive Aï¿½Aï¿½ Interface type:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Serial Attached SCSI (SAS) Aï¿½Aï¿½ Drive path redundancy:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ OK Aï¿½Aï¿½ Drive capabilities:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Data Assurance (DA), Full Disk Encryption (FDE) Aï¿½Aï¿½ Security capable:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Yes, Full Disk Encryption (FDE) Aï¿½Aï¿½ Secure:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ No Aï¿½Aï¿½ Read/write accessible:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Yes Aï¿½Aï¿½ Drive security key identifier:Aï¿½ None Aï¿½Aï¿½ Data Assurance (DA) capable:Aï¿½Aï¿½Aï¿½ Yes Aï¿½Aï¿½ Speed:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 10,020 RPM Aï¿½Aï¿½ Current data rate:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 6 Gbps Aï¿½Aï¿½ Logical sector size:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 512 bytes Aï¿½Aï¿½ Physical sector size:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 512 bytes Aï¿½Aï¿½ Product ID:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ HUC109060CSS601 Aï¿½Aï¿½ Drive firmware version:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ MS04 Aï¿½Aï¿½ Serial number:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ KSGX0VVR Aï¿½Aï¿½ Manufacturer:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ HITACHI Aï¿½Aï¿½ Date of manufacture:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Not Available Script execution complete. SMcli completed successfully. [root@node-1 stornext]#

But I am happy we discovered the commandAï¿½*“show storagearray longrunningoperations;”Aï¿½*to monitor the copy back from the hot spare to the new drive and to confirm everything completed correctly with everything in healthy and optimal status.

[root@node-1 ~]# SMcli Qarray1a -S -c "show storagearray longrunningoperations;" Long Lived Operations: Aï¿½Aï¿½ No operation is currently in progress. [root@node-1 ~]# SMcli Qarray1a -S -c "show volumegroup[1];" DETAILS Aï¿½Aï¿½ Name:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 1 Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Status:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Optimal Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Capacity:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 558.410 GB Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Current owner:Aï¿½ Controller in slot B Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Quality of Service (QoS) Attributes Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ RAID level:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 1 Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Drive media type:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Hard Disk Drive Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Drive interface type:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Serial Attached SCSI (SAS) Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Tray loss protection:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ No Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Data Assurance (DA) capable:Aï¿½ Yes Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ DA enabled volume present:Aï¿½Aï¿½Aï¿½ No Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Total Volumes:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 2 Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Standard volumes:Aï¿½Aï¿½Aï¿½ 2 Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Repository volumes:Aï¿½ 0 Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Free Capacity:Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 0.000 MB Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Associated drives - present (in piece order) Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ Total drives present: 2 Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ TrayAï¿½Aï¿½Aï¿½Aï¿½ Slot Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 85Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 5 [mirrored pair with drive at tray 85, slot 6] Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 85Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½Aï¿½ 6 [mirrored pair with drive at tray 85, slot 5] [root@node-1 ~]# SMcliAï¿½ localhostAï¿½ -c "show storagearray healthstatus;" Performing syntax check... Syntax check complete. Executing script... Storage array health status = optimal. Script execution complete. SMcli completed successfully. [root@node-1 ~]#

I guess the best would have been to have:

- connected the NetApp E-Series E2700 to the network
- installed SANtricity software GUI on a Windows host
- checked the progress from SANtricity GUI

About disk storage array CLI. I got spoiled early on with DDN S2A (2004) command line, later the SFA CLI (2008) disappointed me. Nexsan didn’t provide any CLI at all (2012). But today I realized NetApp / LSI / Engenio (2014) is definitely not better.Aï¿½Still a lot of progress in the world of block storage appliances to achieve (If it still makes sense for the hardware vendors to work on it).

Leverage TrackIt To Save Up To 40% On Your Cloud Bills!