SOURCE:
Also I would strongly recommend reading the vendor documentation on this carefully.
- START HERE - VMware: iSCSI SAN Configuration Guide
- EMC Celerra: VMware ESX Server Using EMC Celerra Storage Systems Solutions Guide
- EMC CLARiiON: VMware ESX Server Using EMC CLARiiON Storage Systems - Solutions Guide
- EMC DMX: VMware ESX Server Using EMC Symmetrix Storage Systems Solutions Guide
- NetApp: NetApp & VMware Virtual Infrastructure 3 : Storage Best Practices (Vaughn is proud to say this is the most popular NetApp TR)
- HP/LeftHand: LeftHand Networks VI3 field guide for SAN/iQ 8 SANs
- Dell/EqualLogic:
> Network Performance Guidelines
> VMware Virtual Infrastructure 3.x Considerations, Configuration and Operation Using an Equallogic PS Series SAN
ENOUGH WITH THE LEARNING!!! HOW do you get high iSCSI throughput in ESX 3.x?
As discussed earlier, the ESX 3.x software initiator really only works on a single TCP connection for each target so all traffic to a single iSCSI Target will use a single logical interface. Without extra design measures, it does limit the amount of IO available to each iSCSI target to roughly 120 160 MBs of read and write access.
This design does not limit the total amount of I/O bandwidth available to an ESX host configured with multiple GbE links for iSCSI traffic (or more generally VMKernel traffic) connecting to multiple datastores across multiple iSCSI targets, but does for a single iSCSI target without taking extra steps.
Here are the questions that customers usually ask themselves:
Question 1: How do I configure MPIO (in this case, VMware NMP) and my iSCSI targets and LUNs to get the most optimal use of my network infrastructure? How do I scale that up?
Question 2: If I have a single LUN that needs really high bandwidth more than 160MBps and I cant wait for the next major ESX version, how do I do that?
Question 3: Do I use the Software Initiator or the Hardware Initiator?
Question 4: Do I use Link Aggregation and if so, how?
Here are the answers you seek
.
.
.
Question 1: How do I configure MPIO (in this case, VMware NMP) and my iSCSI targets and LUNs to get the most optimal use of my network infrastructure? How do I scale that up?
Answer 1: Keep it simple. Use the ESX iSCSI software initiator. Use multiple iSCSI targets. Use MPIO at the ESX layer. Add Ethernet links and iSCSI targets to increase overall throughput. Ser your expectation for no more than ~160MBps for a single iSCSI target.
Remember an iSCSI session is from initiator to target. If use multiple iSCSI targets, with multiple IP addresses, you will use all the available links in aggregate, the storage traffic in total will load balance relatively well. But any individual one target will be limited to a maximum of single GbE connection's worth of bandwidth.
Remember that this also applies to all the LUNs behind that target. So, consider that as you distribute the LUNs appropriately among those targets.
The ESX initiator uses the same core method to get a list of targets from any iSCSI array (static configuration or dynamic discovery using the iSCSI SendTargets request) and then a list of LUNs behind that target (SCSI REPORT LUNS command).
So, to place your LUNs appropriately to balance the workload:
- On an EMC CLARiiON, each physical interface is seen by an ESX host as a separate target, so balance the LUNs behind your multiple iSCSI targets (physical ports).
- On a Dell/EqualLogic array, since every LUN is a target, balancing is automatic and you dont have to do this.
- On an HP/LeftHand array, since every LUN is a target, balancing is automatic and you dont have to do this.
- On a NetApp array each interface is a seen by an ESX host as a separate target, so balance your LUNs behind the targets.
- On an EMC Celerra array, you can configure as many iSCSI targets as you want, up to 1000 and assign them to any virtual or physical network interface - balance your LUNs behind the targets.
Select your active paths in the VMware ESX multi-pathing dialog to balance the I/O across the paths to your targets and LUNs using the Virtual Center dialog shown below (from the VMWare iSCSI SAN Configuration Guide). Also it can take up to 60 seconds for the standby path to become active as the session needs to be established and the MPIO failover needs to occur, as noted in VMware iSCSI configuration guide. There are some good tips there (and in the Vendor best practice docs) about extending guest timeouts to withstand the delay without a fatal I/O error in the guest.
Question 2: If I have a single LUN that needs really high bandwidth more than 160MBps and I cant wait for the next major ESX version, how do I do that?
Answer 2: Use an iSCSI software initiator in the guest along with either MPIO or MC/S
This model allows the Guest Operating Systems to be directly on the SAN and to manage their own LUNs. Assign multiple vNICs to the VM, and map those to different pNICs. Many of the software initiators in this space are very robust (like the Microsoft iSCSI initiator). They provide their guest-based multipathing and load-balancing via MPIO (or MC/S) based on the number of NICs allocated to the VM.
As we worked on this post, all the vendors involved agreed were surprised that this mechanism isn't more popular. People have been doing it for a long time, and it works, even through VMotion operations where some packets are lost (TCP retransmits them iSCSI is ok with occasional loss, but constant losses slow TCP down something to look at if youre seeing poor iSCSI throughput).
It has a big downside, though you need to manually configure the storage inside each guest, which doesnt scale particularly well from a configuration standpoint so for most customers they stick with the keep it simple method in Answer 1, and selectively use this for LUNs needing high throughput.
There are other bonuses too:
- This also allows host SAN tools to operate seamlessly on both physical or virtual environments integration with databases, email systems, backup systems, etc.
- Also has the ability to use a different vSwitch and physical network ports than VMkernel allowing for more iSCSI load distribution and separation of VM data traffic from VM boot traffic.
- Dynamic and automated LUN (i.e. you dont need to do something in Virtual Center for the guest to use the storage) surfacing to the VM itself (useful in certain database test/dev use cases)
- You can use it for VMs that require a SCSI-3 device (think Windows 2008 cluster quorum disks though those are not officially supported by VMware even as of VI3.5 update 3)
There are, of course, things that negative about this approach.
- I suppose "philosophically" there's something a little dirty of "penetrating the virtualizing abstraction layer", and yeah - I get why that philosophy exists. But hey, we're not really philosophers, right? We're IT professionals, and this works well :-)
- It is notable that this option means that SRM is not supported (which depends on LUNs presented to ESX, not to guests)
Question 3: Do I use the Software Initiator or the Hardware Initiator?
Answer 3: In general, use the Software Initiator except where iSCSI boot is specifically required.
This method bypasses the ESX software initiator entirely. Like the ESX software initiator, hardware iSCSI initiators uses the ESX MPIO storage stack for multipathing but doesnt have the single connection per target limit.
But, since you still have all the normal caveats with static load balancing and using the ESX NMP software (active/passive model, with static, manual loadbalancing), this wont increase the throughput for a single iSCSI target.
In general, across all the contributors from each company, our personal preference is to use the software initiator. Why? In general its simple, and since its used very widely, very tested, very robust. It also has a clear 10GbE support path.
Question 4: Do I use Link Aggregation and if so, how?
Answer 4: There are some reasons to use Link Aggregation, but increasing a throughput to a single iSCSI target isnt one of them in ESX 3.x.
What about Link Aggregation shouldnt that resolve the issue of not being able to drive more than a single GbE for each iSCSI target? In a word NO. A TCP connection will have the same IP addresses and MAC addresses for the duration of the connection, and therefore the same hash result. This means that regardless of your link aggregation setup, in ESX 3.x, the network traffic from an ESX host for a single iSCSI target will always follow a single link.
So, why discuss it here? While this post focuses on iSCSI, in some cases, customers are using both NFS and iSCSI datastores. In the NFS datastore case, MPIO mechanisms are not an option, load-balancing and HA is all about Link Aggregation. So in that case, the iSCSI solution needs to work in with concurrently existing Link Aggregation.
Now, Link Aggregation can be used completely as an alternative to MPIO from the iSCSI initiator to the target. That said, it is notably more complex than the MPIO mechanism, requiring more configuration, and isnt better in any material way.
If youve configured Link Aggregation to support NFS datastores, its easier to leave the existing Link Aggregation from the ESX host to the switch, and then simply layer on top many iSCSI targets and MPIO (i.e. just do answer 1 on top of the Link Aggregation).
To keep this post concise and focused on iSCSI, the multi-vendor team here decided to cut out some of NFS/iSCSI hybrid use case and configuration details, and leave that to a subsequent EMC Celerra/NetApp FAS post.
In closing.....
I would suggest that anyone considering iSCSI with VMware should feel confident that their deployments can provide high performance and high availability. You would be joining many, many customer enjoying the benefits of VMware and advanced storage that leverages Ethernet.
To make your deployment a success, understand the one link max per iSCSI target ESX 3.x iSCSI initiator behavior. Set your expectations accordingly, and if you have to, use the guest iSCSI initiator method for LUNs needing higher bandwidth than a single link can provide.
Most of all ensure that you follow the best practices of your storage vendor and VMware.
Posted at 09:00 AM in EMC Competitors, EMC VMware Tech Stuff, iSCSI | Permalink
Digg This | Save to del.icio.us
1 Visitor Reactions & Comments:
Good dispatch and this post helped me alot in my college assignement. Gratefulness you as your information..
Post a Comment