Wednesday, August 1, 2012

NetApp Usable Space Calculator

NetApp Usable Space Calculator ver 1.0

For people who are looking at a quick way to calculate the usable space of a NetApp Filer, I have developed this tool. It is a stand alone executable (Approx 2 MB) which will run on Windows platforms. No Installation needed. I understand there may be bugs in the tool. I am ready for feedback and would like to improve this tool. For now please download and use and let me know. Please use this tool as a estimate and this tool is supplied without any warranty. It may not report accurate sizes.





Aug 8 2013 - NEW VERSION is 2.0 Refer to my post here
NEW VERSION is 1.2. Refer to my post here
Download NetApp Usable Space Calculator ver 1.0 (2.5MB)


Many people have blogged about the usable calculation formula. I am not going to repeat the steps here. Instead I will give some pointers to those blogs. Below info was used in developing this calculator. Feedback recommended.

STEP1: Calculate the disk size first

Basically if you go to bestbuy and pick a 1TB drive, you actually think that you store 1TB of pictures and music. Thats not the truth. Storage hardware is using the base 10 system and software is using the base 2 system. So no storage is actually lost, it is just a question of how the information is represented.
Hard disk manufacturers assume / Base 10 System -   Kilo = 10 power 3
File system assumes / Base 2 System:   Kilo = 2 power 10 = 1024

KB2101,024 bytes
MB2201,048,576 bytes
GB2301,073,741,824 bytes
TB2401,099,511,627,776 bytes

So a 1 TB hard disk is actually
1000,000,000 bytes/ 1,073,741,824  = .9313 x 1000 = 931.3 GB
In a similar way
750 GB hard disk is actually
750,000,000/1,073,741,824  = .6986 x 1000 = 698.6 GB


Base 10 Capacity
Base 2 Capacity
Right Sized Capacity
72GB FC
69 GB
67.9 GB
144GB FC
137.9GB
135.8 GB
300GB SAS
279.4 GB
272 GB
450GB SAS
419.1 GB
415 GB
600GB SAS
558.8 GB
553.26 GB
500GB SATA
465.7 GB
413.9 GB
750 GB SATA
698.6 GB
620.9 GB
1TB SATA
931.3 GB
827.8 GB
2TB SATA
1,862.6 GB
1,655.7 GB
3TB SATA
2,794 GB
2,483.5 GB

Now from the above table you are clear that how a 1TB hard drive has come finally to 931.3 GB. But how did we get the number Right sized capacity 827.8..This is called the checksum calculation which we will see it next.

STEP 2 : Check Sum Calculation

From 931.3 multiply x 8/9 and you will get 827.8 GB. Simple..Why do we multiply the value by 8/9. For this Taylor Riggan @ Waflhouse.com gives a nice explanation. See here - http://waflhouse.com/?p=47

Reproducing from WAFL House website

The second part of this can begin to get a little confusing.  It has to do with error-correction codes and their use on enterprise-class disk arrays such as NetApp.  NetApp uses two types of disk drives – SAS disks and SATA disks.  The manufacturers of these disks typically create SAS disks for business use and SATA disks for both business and home user (consumer) use.  Given that fact, they include some additional functionality on SAS disks to allow enterprise-class disk subsystems to consistently validate the consistency of data on disks.  They do this through something called a checksum (nothing more than a hash value that they can reference to validate data consistency).  That checksum, on SAS disks, is stored in the same sector as data (over simplifying here, but this is close).  The sector size on SAS disks is large enough to incorporate this (520 bytes per sector) and, as such, you really don’t see a reduction in capacity on the drives after doing the base-10 to base-2 sizing conversion.  SATA disks is where it gets a bit fuzzy.  Home users don’t necessarily need checksums.  So, disk manufacturers don’t include that extra space within the sectors to accomodate checksums (512 bytes per sector).  Well, what do enterprise-class storage vendors do in order to use checksums on SATA disks?  They use up an extra 9th sector for every 8 sectors used!  This eats away SATA usable capacity much faster than SAS capacity.  When using SATA disks, you have to take the size you get from the  base-10 to base-2 sizing conversion and multiply that by 8/9. Back to that 1TB SATA disk that we were using as an example, after base-10 to base-2 conversion we were left with 931.32GB.  After checksum allocation, we’re left with  827.8GB of usable capacity

So a ITB hard disk leaves us with 827.8 GB of usable capacity so far.

STEP3: RAID GROUP DETERMINATION

Again I am not going to dwell on this much and going to give reference to cfheoh article @ http://storagegaga.wordpress.com/2011/10/20/playing-with-netapp-final-usable-capacity/

Reproducing From Storagegaga.com

RAID group is important because it is used to balance a few considerations
 •Performance in recovery if there is a disk reconstruction or resilvering
 •Combined RAID performance and availability through a Mean Time Between Data Loss (MTBDL) formula

Different ONTAP versions (and also different disk types) have different number of HDDs to constitute a RAID group. For ONTAP 8.0.1, the table below are its recommendation.
 

So, given a large pool of HDDs, the NetApp storage administrator has to figure out the best layout and the optimal number of HDDs to get to the capacity he/she wants. And there is also a best practice to set aside 2 HDDs for a RAID-DP configuration with every 30 or so HDDs. Also, it is best practice to take the default recommended RAID group size most of the time.

I would presume that this is all getting very confusing, so let me show that with an example. Let’s use the common 2TB SATA HDD and let’s assume the customer has just bought a 100 HDDs FAS6000. From the table above, the default (and recommended) RAID group size is 14. The customer wants to have maximum usable capacity as well. In a step-by-step guide,
 1.Consider the hot sparing best practice. The customer wants to ensure that there will always be enough spares, so using the rule-of-thumb of 2 HDDs per 30 HDDs, 6 disks are set aside as hot spares. That leaves 94 HDDs from the initial 100 HDDs.
 2.There is a root volume, rootvol, and it is recommended to put this into an aggregate of its own so that it gets maximum performance and availability. To standardize, the storage administrator configures 3 HDDs as 1 RAID group to create the rootvol aggregate, aggr0. Even though the total capacity used by the rootvol is just a few hundred GBs, it is not recommended to place data into rootvol. Of course, this situation cannot be avoided in most of the FAS2000 series, where a smaller HDDs count are sold and implemented. With 3 HDDs used up as rootvol, the customer now has 91 HDDs.
 3.With 91 HDDs, and using the default RAID group size of 14, for the next aggregate of aggr1, the storage administrator can configure 6 x full RAID group of 14 HDDs (6 x 14 = 84) and 1 x partial RAID group of 7. (91/14 = 6 remainder 7). And 84 + 7 = 91 HDDs.
 4.RAID-DP requires 2 disks per RAID group to be used as parity disks. Since there are a total of 7 RAID groups from the 91 HDDs, 14 HDDs are parity disks, leaving 77 HDDs as data disks.

This is where the rightsized capacity comes back into play again. 77 x 2TB HDDs is really 77 x 1.69TB = 130.13TB from an initial of 100 x 2TB = 200TB.

If you intend to create more aggregates (in our example here, we have only 2 aggregates – aggr0 and aggr1), there will be more consideration for RAID group sizing and parity disks, further reducing the usable capacity.


STEP4: SPARE DISKS

Deduct how many spares you plan to allocate.

Below is recommended from Netapp

On NetApp storage, disk failures automatically trigger parity reconstructions of affected data onto a hot standby (spare) disk, assuming that a spare disk is available. If no spare disks are available, self-healing operations are not possible. The system will run in degraded mode (requests for data on the failed disk are satisfied by reconstructing the data using parity information) until a spare is provided or the failed disk is replaced. During this time, your data is at greater risk should an additional failure occur. (With NetApp RAID-DP™, a RAID group operating in degraded mode can undergo one additional disk failure without suffering data loss.)
The number of spares you need varies based on the number of disk drives attached to your storage system. For a lower-end FAS200 or FAS2000 with a single shelf, one spare disk may suffice (configure two if you want to use Maintenance Center). On the FAS6080, with a maximum spindle count of 1,176 disks, more spare disks are needed to ensure maximum storage resiliency, especially with larger SATA disks that have longer reconstruction times.
NetApp recommends using two spares per disk type for up to 100 disk drives, where disk type is determined by a unique interface type (FC, SATA, or SAS), capacity, and rotational speed. For instance, if you have a system with 28 300GB 15K FC disks and 28 144GB 15K FC disks, you should provide four spares: two of the 300GB capacity and two of the 144GB capacity.

STEP5: Root Volume Consideration

Root volume contains all configurations files (ONTAP Files). It is always preferred to separate the root volume from the data volumes on the filer. For optimal management, it is better to restrict the root volume to just 2 disks (minimum required for a RAID-4 parity group). This enables faster reconstruction times in case of a disk failure on the root volume. Deduct how many disks you have allocated for the root volume alone.

STEP6: Volume Snapshot Reserve

Default snapshot space reserve is 20% of the volume size.
This variable is set at the volume level and is set as a percentage of the volume. Data ONTAP removes the defined percentage of volume from being available for configuring LUNs or for file usage with CIFS or NFS. As Snapshot copies need space, they consume space in the snap reserve area. By default after the snap reserve area is filled, the Snapshot copies start to take space from the general volume.

STEP7: Aggregate Reserve
Aggregates are created by default with 5% reserve space. Some people leave this to 0 and set the reserve on the volume level.

STEP8: WAFL Overhead
WAFL reserves approximately 10% of space for performance reasons, and for some part of the metadata. This is available in the new version 1.1


(A ton of thanks to my friend Guru for his help and support in the development of this tool)
Murugappan

6 comments:

  1. any plans to add the 900GB SAS drives?

    ReplyDelete
  2. Actually I did. It is available here for now - https://communities.netapp.com/thread/25020

    ReplyDelete
  3. When you plan to add SSD's? :)

    Nice Calculator

    ReplyDelete
  4. So what is the size of the 900GB you use to calculate?

    ReplyDelete
  5. Thanks for the post - it helped me how to understand the difference between physical and virtual/software view of the same # - and how to convert between the two.

    ReplyDelete
  6. How about updating your application since new SAS and SSD drives are now out.

    ReplyDelete