Saturday, February 25, 2006
SAN woes
Normally I try not to write about the things I am actually doing at work for paying clients. I tend to not want to give away my own or institutional knowledge for which my firm should charge clients. That's why I stick to discussions about collaboration technologies (OneNote, SharePoint, document management) and personal technology (Treo, TabletPC, Firefox, Microsoft Media Center).
But I've been getting frustrated designing for Network Applications storage as part of my current project. And my prior engagement involved in part carving up storage for Windows servers on an EMC Symmetrix. And I've talked with clients about offerings from Compellent, Falconstor, Hitachi, and HP over the years. I guess I would call my market the middle tier of storage; mainframes are not used and only the occasional Unix system is ever connected to these boxes. Mainly they are for Exchange or Notes clusters, Microsoft SQL server clusters, file clusters and/or NAS, VMware ESX servers, and Windows server boot drives. Ideally the storage controller is used as a mechanism to replicate these systems to another site for DR purposes.
Having worked with some of the major vendors in this middle tier I need to speak my frustration. I do not profess to be a storage engineer so I would not mind any corrections, differences of opinion, or even personal attacks by commenters for this post. I am merely expressing my frustration that nothing is perfect is the small slice of the storage world that I play in. Here are my frustrations:
Network Appliance
This is theoretically the new darling of mid-tier storage, but I grow frustrated with their architecture. Their strength is NAS and in many ways NAS is quite appealing. Consolidating some servers? Want to do it without reconfiguring all your apps? Just have the NAS pretend to be FS1, FS2, NYFS18, whatever. Also, you need far fewer NAS heads than file servers to serve the same amount of data. And, in the NetApp world, snapshots of file systems on NAS are far smaller than snapshots of block storage. And you just have the sense that a custom Linux kernel designed for serving files will do so at a faster rate and far more reliably for far longer than a Windows server.
But, you still need some features that require separate servers when you choose NAS. File share virus protection requires a separate server which must hook into reads and/or writes of every file. Yep, the limiting factor on file serving speeds just became Windows! You could argue that if you protect against viruses via desktop scanning, email gateways, and network edge scanning, you're safe enough, but you still have foreign devices that come onto your network. You could also argue that permissions auditing on a NAS system is not nearly as complete as what you would get in Windows (assuming you set that stuff up). And, in the case of my current client, no NAS can run the docsntss.exe service that DOCS Open requires to properly secure documents. It's a law firm, old document management issue, but if you have that issue, you can't use NAS for your documents. If you were also thinking of using something more sophisticated than RoboCopy or XCOPY /D to replicate data out to branch offices, and you didn't want to put a NetApp NAS in every branch office because you were, I don't know, fiscally responsible or something, then you might want to use FRSv2 in the new R2 release of Windows Server 2003 to replicate file data. Again, that ain't happening with NAS. It's not that I don't still believe in NAS (though you'd have to show me more sophisticated things coming from other vendors, such as running real-time scanning on the NAS head itself) but in my current assignment we've decided against it and in favor of clustered file servers.
However, it is the provision of block storage via iSCSI where I find NetApp falling short. I know that NetApp's strength is that you can make a big-ass RAID group and share all the spindles with a variety of applications. So they're ahead of EMC (Clariion) in my opinion in that they actually can do this. But having RAID4 or RAID-DP be your only options here are wreaking havoc in assigning spindles to write-intensive applications like Exchange and SQL. Maybe I'm gullible or an idiot, but I was given, and am sticking with, a write penalty of 6 for RAID-DP. That means that a group of servers that as a whole are performing 3900 reads/sec and 1100 writes/sec on 500GB of data would require over 58 spindles to handle the I/O load, or over 8TB allocated. So, yeah, you have more spindles to use, but you need more because the controller has to perform so many operations to do one write to a RAID-DP array! (The same 3900 reads / 1100 writes would only need 39 spindles in RAID10.)
And, snapshots of iSCSI block storage, because they are just files served off the NAS head, require their same size again for a snapshot. Granted, you have a lot of extra space on the array because you had to use so many spindles, but that just means you are wasting a ton of disk just to provide the performance that far fewer RAID10 disks could perform!
And iSCSI itself, though no fault of NetApp's, is not perfect. Certain legal software vendors recommend against it because its 1Gbit storage path speed is considered too slow. The only card I know of that boots off of it, the QLogic 4010, does not support jumbo frames, which causes consternation among networking types as creating headers for 8K SCSI packets adds overhead in calculation time and a reduction in size of the data portion of the packet. (QLogic will fix this with the introduction of the 4050.) And the otherwise excellent VMware ESX Server doesn't support iSCSI until they finally decide to ship ESX 3.0 some time in the next 6 months.
Finally, a flaw of at least the NetApp 900 family is that it only takes Fibre Channel disks. So if you, in a law firm environment, would like high-capacity storage for infrequently accessed data like disk backups and litigation support images, you need a different unit (a FAS3000 or R200) or you need to pay a fortune for 300GB FC drives. This is true as well of the EMC Symmetrix.
So, for these reasons, the NetApp offering is not perfect. Yes, I may unfairly disregard NAS because of a requirement of nutty old legacy apps. And I may misunderstand the performance of RAID-DP. And I haven't surveyed the iSCSI HBA offerings closely enough, so I may be missing an offering that allows both boot-from-SAN and jumbo packets. And I may not be aware of a newer NetApp FAS model that supports both FC and ATA disks. But all of these taken together cause me some dissatisfaction with the NetApp Filer offering.
Stay tuned for my discussion of the EMC product line. I find that they, too, provide no single product that meets all my needs.
This post was not sanctioned by my company (the name of which I've declined to disclose on this blog anyway); all beliefs are my own and do not reflect those of anyone else.
technorati: NetApp NAS SAN iSCSI QLogic EMC
But I've been getting frustrated designing for Network Applications storage as part of my current project. And my prior engagement involved in part carving up storage for Windows servers on an EMC Symmetrix. And I've talked with clients about offerings from Compellent, Falconstor, Hitachi, and HP over the years. I guess I would call my market the middle tier of storage; mainframes are not used and only the occasional Unix system is ever connected to these boxes. Mainly they are for Exchange or Notes clusters, Microsoft SQL server clusters, file clusters and/or NAS, VMware ESX servers, and Windows server boot drives. Ideally the storage controller is used as a mechanism to replicate these systems to another site for DR purposes.
Having worked with some of the major vendors in this middle tier I need to speak my frustration. I do not profess to be a storage engineer so I would not mind any corrections, differences of opinion, or even personal attacks by commenters for this post. I am merely expressing my frustration that nothing is perfect is the small slice of the storage world that I play in. Here are my frustrations:
Network Appliance
This is theoretically the new darling of mid-tier storage, but I grow frustrated with their architecture. Their strength is NAS and in many ways NAS is quite appealing. Consolidating some servers? Want to do it without reconfiguring all your apps? Just have the NAS pretend to be FS1, FS2, NYFS18, whatever. Also, you need far fewer NAS heads than file servers to serve the same amount of data. And, in the NetApp world, snapshots of file systems on NAS are far smaller than snapshots of block storage. And you just have the sense that a custom Linux kernel designed for serving files will do so at a faster rate and far more reliably for far longer than a Windows server.
But, you still need some features that require separate servers when you choose NAS. File share virus protection requires a separate server which must hook into reads and/or writes of every file. Yep, the limiting factor on file serving speeds just became Windows! You could argue that if you protect against viruses via desktop scanning, email gateways, and network edge scanning, you're safe enough, but you still have foreign devices that come onto your network. You could also argue that permissions auditing on a NAS system is not nearly as complete as what you would get in Windows (assuming you set that stuff up). And, in the case of my current client, no NAS can run the docsntss.exe service that DOCS Open requires to properly secure documents. It's a law firm, old document management issue, but if you have that issue, you can't use NAS for your documents. If you were also thinking of using something more sophisticated than RoboCopy or XCOPY /D to replicate data out to branch offices, and you didn't want to put a NetApp NAS in every branch office because you were, I don't know, fiscally responsible or something, then you might want to use FRSv2 in the new R2 release of Windows Server 2003 to replicate file data. Again, that ain't happening with NAS. It's not that I don't still believe in NAS (though you'd have to show me more sophisticated things coming from other vendors, such as running real-time scanning on the NAS head itself) but in my current assignment we've decided against it and in favor of clustered file servers.
However, it is the provision of block storage via iSCSI where I find NetApp falling short. I know that NetApp's strength is that you can make a big-ass RAID group and share all the spindles with a variety of applications. So they're ahead of EMC (Clariion) in my opinion in that they actually can do this. But having RAID4 or RAID-DP be your only options here are wreaking havoc in assigning spindles to write-intensive applications like Exchange and SQL. Maybe I'm gullible or an idiot, but I was given, and am sticking with, a write penalty of 6 for RAID-DP. That means that a group of servers that as a whole are performing 3900 reads/sec and 1100 writes/sec on 500GB of data would require over 58 spindles to handle the I/O load, or over 8TB allocated. So, yeah, you have more spindles to use, but you need more because the controller has to perform so many operations to do one write to a RAID-DP array! (The same 3900 reads / 1100 writes would only need 39 spindles in RAID10.)
And, snapshots of iSCSI block storage, because they are just files served off the NAS head, require their same size again for a snapshot. Granted, you have a lot of extra space on the array because you had to use so many spindles, but that just means you are wasting a ton of disk just to provide the performance that far fewer RAID10 disks could perform!
And iSCSI itself, though no fault of NetApp's, is not perfect. Certain legal software vendors recommend against it because its 1Gbit storage path speed is considered too slow. The only card I know of that boots off of it, the QLogic 4010, does not support jumbo frames, which causes consternation among networking types as creating headers for 8K SCSI packets adds overhead in calculation time and a reduction in size of the data portion of the packet. (QLogic will fix this with the introduction of the 4050.) And the otherwise excellent VMware ESX Server doesn't support iSCSI until they finally decide to ship ESX 3.0 some time in the next 6 months.
Finally, a flaw of at least the NetApp 900 family is that it only takes Fibre Channel disks. So if you, in a law firm environment, would like high-capacity storage for infrequently accessed data like disk backups and litigation support images, you need a different unit (a FAS3000 or R200) or you need to pay a fortune for 300GB FC drives. This is true as well of the EMC Symmetrix.
So, for these reasons, the NetApp offering is not perfect. Yes, I may unfairly disregard NAS because of a requirement of nutty old legacy apps. And I may misunderstand the performance of RAID-DP. And I haven't surveyed the iSCSI HBA offerings closely enough, so I may be missing an offering that allows both boot-from-SAN and jumbo packets. And I may not be aware of a newer NetApp FAS model that supports both FC and ATA disks. But all of these taken together cause me some dissatisfaction with the NetApp Filer offering.
Stay tuned for my discussion of the EMC product line. I find that they, too, provide no single product that meets all my needs.
This post was not sanctioned by my company (the name of which I've declined to disclose on this blog anyway); all beliefs are my own and do not reflect those of anyone else.
technorati: NetApp NAS SAN iSCSI QLogic EMC
Labels: NetApp
