Home > Big Data, EMC, Hyperbole, Performance, Value > More Records ??

More Records ??


–This has been revised based on some comments I’ve received since the original posting, check the comment thread if you’re interested what/why–

I came in this morning with an unusually clear diary, and took the liberty to check the newsfeeds for NetApp and EMC, this is when I came across an EMC press release entitled  “EMC VNX SETS PERFORMANCE DENSITY RECORD WITH LUSTRE —SHOWCASES “NO COMPROMISE” HPC STORAGE“.

I’ve been doing some research on Lustre and HPC recently, and that claim surprised me more than a little, so I checked it out, maybe there’s a VNX sweetspot for HPC that I wasnt expecting. The one thing that stood out straight away was . “EMC® is announcing that the EMC® VNX7500 has set a performance density record with Lustre—delivering read performance density of 2GB/sec per rack” (highlight mine)

In the first revision of this I had some fun pointing out the lameness of that particular figure, (e.g. “From my perspective, measured on a GB/sec per rack, 2GB/sec/rack is pretty lackluster”) , but EMC aren’t stupid (or at least their engineers aren’t, though I’m not so sure about their PR agency at this point), so it turns out that this was one of those things where it seems that EMC’s PR people didn’t actually listen to what the engineers were saying, and it looks like they’ve missed out a small but important word, and that word is “unit”.  This becomes apparent if you take a look at the other stuff in that press release “8 GB/s read and 5.3 GB/s write sustained performance, as measured by XDD benchmark performed on a 4U dual storage processor”. This gives us 2GB/sec/rack unit which actually sounds kind of impressive. So let’s dig a little deeper, what we’ve got is a 4U dual storage processor that gets some very good raw throughput numbers, about 1.5x, or 150% faster in fact  on a “per controller” basis than the figures used on the E5400 press release I referenced earlier, so on that basis I think EMC has done a good job. But this is where the PR department starts stretching the truth again by leaving out some fairly crucial pieces of information. Notably that crucial information that the 2GB/sec/rack unit is for 4U controller is a 2U VNX7500SPE with 2U standby power supply which is required when the 60 drive dense shelves are used exclusively (which is the case for the VNX Lustre Proof of Concept information shown in their brochures), and this configuration doesn’t include any of the rack units required for the actual storage. Either that, or its a 2U VNX7500SPE with a 2U shelf , and no standby power supply that seems to be mandatory component of a VNX solution, and I cant quite believe that EMC would do that.

If we compare the VNX to the E5400, you’ll notice that controllers and standby power supplies  alone consume 4U of rack space without adding any capacity, whereas the E5400 controllers are much much smaller, and they fit directly into a 2U or 4U disk shelf (or DAE’s in EMC terminology) which means a 4U E4500 based solution is something you can actually use, as the required disk capacity is already there in the 4U enclosure.

Lets go through some worked calculations, to show how this works. In order to add capacity in the densest possible EMC configuration, you’d need to add an additional 4RU shelf with 60 drives in it. Net result 8RU, 60 drives, and up to 8 GB/s read and 5.3 GB/s write (the press release doesn’t make it clear whether a VNX7500 can actually drive that much performance from only 60 drives, my suspicion is that it cannot, otherwise we would have seen something like that in the benchmark). Meausred on a GB/s per RU basis this ends up as only 1 GB/sec per Rack Unit, not the 2 GB/sec per Rack Unit which I believe was the point of the “record setting” configuration. And just for kicks as you add more storage to the solution that number goes down as shown for the “dual VNX7500/single rack solution that can deliver up to 16GB/s sustained read performance” to about 0.4 GB/sec per Rack Unit. Using the configurations mentioned in EMC’s proof of concept configuration  you end up with around 0.666 GB/sec per Rack Unit, all of which are a lot less than the 2 GB/sec/RU claimed in the press release

If you wanted to have the highest performing configurations using a “DenseStak” solution within those 8RU with an E5400 based Lustre solution, you’d put in another e5400 unit with an additional 60 drives Net result 8RU, 120 drives, and 10 GB read and 7 GB/sec write (and yes we can prove that we can get this kind of performance from 120 drives). Meausured on a GB/s per RU basis this ends up as 1.25 GB/sec per Rack Unit. That’s good, but its still not the magic number mentioned in the EMC press release, however if you were to use a “FastStak” solution, those numbers would pretty much double (thanks to using 2RU disk shelves instead of 4RU disk shelves) which would give you controller performance density of around 2.5 GB/sec per Rack Unit.

Bottom line, for actual usable configurations a NetApp solution has much better performance density using the same measurements EMC used for their so called “Record Setting” benchmark result.

In case you think I’m making these numbers up, they are confirmed in the NetApp whitepaper wp-7142 which says

The FastStak reference configuration uses the NetApp E5400 scalable
storage system as a building block. The NetApp E5400 system is designed
to support up to 24 2.5-inch SAS drives, in a 2U form factor.
Up to 20 of these building blocks can be contained in an industry-standard
40U rack. A fully loaded rack delivers performance of up to 100GB/sec
sustained disk read throughput, 70GB/sec sustained disk write throughput,
and 1,500,000 sustained IOPS.
According to IDC, the average supercomputer produces 44GB/sec,
so a single FastStak rack is more than fast enough to meet the I/O
throughput needs of many installations.

While I’ll grant that this result is achieved with more hardware, it should be remembered that the key to good HPC performance is in part about the ability to efficiently throw hardware at a problem. From a storage point of view this means having the ability to scale performance with capacity. In this area the DenseStak and FastStak solutions are brilliantly matched to the requrements of, and the prevailing technology used, in High Performance Computing. Rather than measuring on a GB/sec/rack unit I think a better measure would be “additional sequential performance per additional gigabyte”. Measured on a full rack basis, the NetApp E5400 based solution ends up at around 27MB/sec/GB for the DenseStak, or 54MB/sec/GB for the FastStak. In comparison, the fastest EMC solution as referenced in the “record setting” press release comes in at about 10MB/sec of performance for every GB of provisioned capacity or about 22MB/sec/GB for the configuration in the proof of concept brochure . Any way you slice this, the VNX just doesn’t end up looking like a particularly viable or competetive option.

The key here is that Lustre is designed as a scale out architecture. The E5400 solution is built as a scale out solution by using Lustre to aggregate the performance of the multiple carfully matched E5400 controllers, whereas the VNX7500 used in the press release is relatively poorly matched scale-up configuration which is being shoe-horned into use case it wasn’t designed for.

In terms of performance per rack unit, or performance per GB there simply isn’t much out there that comes close to a E5400 based Lustre solution, certainly not from EMC, as even Isilon, their best Big Data offering, falls way behind. The only other significant questions that remain are how much do they cost to buy, and how much power do they consume ?

I’ve seen the pricing for EMC’s top of the range VNX 7500, and its not cheap, its not even a little bit cheap, and the ultra-dense stuff shown in the proof of concept documents is even less not cheap than their normal stuff. Now I’m not at liberty to discuss our pricing strategy in any detail on this blog, but I can say that in terms of “bang per buck”, the E5400 solutions are very very competetive, and the power impact of the E5400 controller inside of 60 drive dense shelf is pretty much negligible. I don’t have the specs for the power draw on a VNX7500 and its associated external power units , but I’m guessing it adds around as much as a shelf of disks, the power costs of which add up over the three year lifecycle typically seen in these kinds of environments.

From my perspective the VNX7500 is a good general purpose box, and EMC’s engineers have every right to be proud of the work they’ve done on it, but positioning this as a “record setting” controller for performance dense HPC workloads on Lustre, is stretching the truth just a little too far for my liking.  While the 10GB/sec/rack mentioned in the press release might sound like a lot for those of us who’ve spent their lives around transaction processing systems, for HPC, 10GB/sec/rack simply doesnt cut it. I know this, the HPC community knows this, and I suspect most of the reputable HPC focussed engineers in EMC also know this.

It’s a pity though that EMC’s PR department is spinning this for all they’re worth ; I struggle with how they can possibly assert that they’ve set any kind of performance density record for any kind of realistic Lustre implementation, when the truth is that they are so very very far behind. Maybe their PR dept has been reading 1984, because claiming record setting performance in this context requires some of the most bizarre Orwellian doublespeak I’ve seen in some time.

  1. ausstorageguy
    November 15, 2011 at 11:13 am

    “EMC® is announcing that the EMC® VNX7500 has set a performance density record with Lustre—delivering read performance density of 2GB/sec per rack” (highlight mine)

    “From my perspective, measured on a GB/sec per rack, 2GB/sec/rack is pretty lackluster, especially when hen you consider that a Netapp “DenseStak” Lustre”

    Come on John, you’re clutching at straws here – the PR website got it wrong – it’s 2GB/s per rack unit or RU. EMC’s own version is here: http://www.emc.com/about/news/press/2011/20111110-01.htm

    It certainly wouldn’t be the first time a PR agency dropped a crucial part such as a “U” at the end of a statement confusing it for a spelling mistake.

    Now, I note, you did put that little titbit much further down the line, but you’re already spreading FUD upfront and you know it.

    Now read a bit more and you’ll soon see that this solution scales to up to 1.6PB as well as the speed and you’ll start to see where this really shines in the HPC space.

    A coursory glance at the solution paper reveals that this 2GB/s per RU is for the WHOLE solution, not just the array!

    Can you do me a favour, when a competitor releases a new “thing”, instead of pissing over the competitors solution, how about you post instead NetApp’s solution, be it FAS or E-Series?.

    Here’s the challenge: you build a solution using the same components, test methodology and benchmark and achieve 2GB/s PER RACK UNIT, and present that!

    I’m a fan of NetApp, but this constant need to bark at the big dog is getting ridiculous – I’m now getting bored with your blog and as someone who’s multi-vendor trained, I see through your guff and learn nothing new about what NetApp is achieving other than constant looking over the fence and moaning at the neighbours new pool and landscaping.

    • November 15, 2011 at 2:41 pm

      — old version of my comment, now superceded see below for the current version —

      Thanks for the reply, I didn’t look at the EMC version of the press release, though as soon as I’d been informed about the missing U early this morning, I did update my blog, though I reserve the right to poke fun at EMC’s PR organisation in the introduction.
      You claim that “A coursory glance at the solution paper reveals that this 2GB/s per RU is for the WHOLE solution, not just the array!”.
      I’d be interested in looking at the solution paper, as the press releases only point to a fairly lightweight brochure http://www.emc.com/collateral/hardware/solution-overview/h8984-vnx-with-lustre-file-system-so.pdf that doesn’t support a 2GB per sec per RU figure.
      VNX7500SPE – 2RU
      2x 60 drive enclosures – 8RU
      1x 2U standby power supply – 2RU
      12RU for 8GB/sec of performance = 0.66GB/sec per RU (not including the overheads for the metadata storage controller)
      Clearly the reference architecture is not what was demonstrated during the benchmark,
      As I put in my blog, I couldn’t quite see how EMC can claim that they can get 2GB/sec/RU when the VNX Service processors consume 4RU for every 8GB/sec of performance, where do you put the disks ? Based on your feedback I went back and double checked my facts, and it turns out that I incorrectly assumed that a VNX7500 SPE consumed the same number of RU’s as the VNX5xxx DPE’s. If you include a 2RU VNX7500 SPE with a 2RU disk shelf you can in fact get some reasonably good density, though I’m kind of surprised that there isn’t a 1RU standby power system included too. All the collateral I have available suggests that the standby power system is an absolute requirement for all VNX systems. Indeed if you are exclusively using dense shelves it appears that for every 2U VNX7500 SPE, you also need a 2U standby power supply.
      If you do include a 1U power supply with a 2U VNX7500SPE with a 2U shelf (5RU in total), then you’re looking at 8GB per sec / 5RU = 1.6 GB/sec / Rack unit which is still less than the 2GB sec / RU mentioned in the press releases.
      Maybe EMC didnt bother including the standby power supply in their benchmark, so maybe they actually did get 2GB/sec/RU, but from my perspective not including mandatory components for a production configuration seems misleading.
      It is precisely the kind of misleading information that I’ve seen EMC’s marketing organisation put out in the past, and which I believe should be challenged vigorously. You might see it as FUD on my part which is a shame, but if you’d put up with as much nonsense as EMC, has thrown at NetApp over the years, you’d understand why this kind of inaccuracy annoys me sufficiently to write about it. I suspect that innacuracy annoys you too otherwise you wouldnt have bothered posting your comment on my blog.
      So based on your feedback I’m happy to move away from the position that the “2GB/RU” is only for the storage controllers themselves, and I’ll modify the main content of the blog to reflect your concerns and feedback though I still think it’s incomplete without including the overheads for the external power supplies which as far as I’m aware, are a mandatory part of a VNX configuration. You might not care any more, but I do, I make mistakes, and I do what I can to fix them when they’re pointed out. I strongly doubt that EMC will make any changes to their collateral as a result of my, or anyone elses criticism, valid or otherwise.
      As far as pointing out what we can achieve, at least half of the post was indeed what the E-Series is capable of, and it references bench-marked capabilities in our own press releases, and points to a comprehensive whitepaper that describes in detail what we achieve, including configurations that can achieve in excess of 2.5 GB/sec/RU.
      If you want side by side comparable benchmarks, then I can say that NetApp is working on exactly that as referenced in our press release this morning (which came as a surprise to me)
      http://www.netapp.com/us/company/news/news-rel-20111114-469529.html
      I believe that benchmarks should be fair, representative and should contain enough data to make some reasonably valid engineering decisions, and it annoys me when they get “gamed” specifically for marketing purposes, which is why I spend so much time writing about them.
      Regards
      John

    • November 15, 2011 at 2:42 pm

      — Edited to reflect that I now believe the 4U configuration referenced by EMC in their press release is actually a 2U VNX7500 SPE + 2U Standby Powersupply —

      Thanks for the reply, I didn’t look at the EMC version of the press release, though as soon as I’d been informed about the missing U early this morning, I did update my blog, though I reserve the right to poke fun at EMC’s PR organisation in the introduction.

      You claim that “A coursory glance at the solution paper reveals that this 2GB/s per RU is for the WHOLE solution, not just the array!”.

      I’d be interested in looking at the solution paper, as the press releases only point to a fairly lightweight brochure http://www.emc.com/collateral/hardware/solution-overview/h8984-vnx-with-lustre-file-system-so.pdf that doesn’t support a 2GB per sec per RU figure.

      VNX7500SPE – 2RU
      2x 60 drive enclosures – 8RU
      1x 2U standby power supply – 2RU

      12RU for 8GB/sec of performance = 0.66GB/sec per RU (not including the overheads for the metadata storage controller), so clearly the reference architecture is not what was demonstrated during the benchmark,

      As I put in my blog, I couldn’t quite see how EMC can claim that they can get 2GB/sec/RU when the VNX Service processors consume 4RU for every 8GB/sec of performance, where do you put the disks ? Based on your feedback I went back and double checked my facts. It’s possible that if you include a 2RU VNX7500 SPE with a 2RU disk shelf you can in fact get some reasonably good density, but you’d still need a 1RU standby power system included too. Indeed if you are exclusively using dense shelves it appears that for every 2U VNX7500 SPE, you also need a 2U standby power supply. I believe that the configuration referenced in the EMC press release is a 2U VNX7500 SPE with a 2U standby power supply. Even if it’s a 2U VNX7500SPE with a 2U shelf , with a 1U power supply (5RU in total) then you’re looking at 8GB per sec / 5RU = 1.6 GB/sec / Rack unit which is still less than the 2GB sec / RU mentioned in the press releases.

      Maybe EMC did go with a 2U VNX7500SPE with a 2U shelf and didn’t bother including the standby power supply in their benchmark, then maybe they actually did get 2GB/sec/RU in a scalable way, but if so, then from my perspective not including mandatory components for a production configuration seems misleading. I also strongly doubt that you could pull that much I/O from a 24 drive shelf.

      The only conclusion I can draw is that either the 2GB/sec/RU is only for the controller components, or they left out critical components, either way it’s misleading. It is precisely this kind of misleading and confusing information that I’ve seen EMC’s marketing organisation put out in the past, and which I believe should be challenged vigorously. You might see it as FUD on my part which is a shame, but if you’d put up with as much nonsense as EMC, has thrown at NetApp over the years, you’d understand why this kind of inaccuracy annoys me sufficiently to write about it. I suspect that innacuracy annoys you too otherwise you wouldnt have bothered posting your comment on my blog.

      So based on your feedback I’m happy to modify the main content of the blog to reflect your concerns and feedback. You might not care any more, but I do, I make mistakes, and I do what I can to fix them when they’re pointed out. I strongly doubt that EMC will make any changes to their collateral as a result of my, or anyone elses criticism, valid or otherwise.

      As far as pointing out what we can achieve, at least half of the post was indeed what the E-Series is capable of, and it references bench-marked capabilities in our own press releases, and points to a comprehensive whitepaper that describes in detail what we achieve which you can download from this blog, including configurations that can achieve around 2.5 GB/sec/RU.

      If you want side by side comparable benchmarks, then I can say that NetApp is working on exactly that, as referenced in our press release this morning (which came as a surprise to me)

      http://www.netapp.com/us/company/news/news-rel-20111114-469529.html

      I believe that benchmarks should be fair, representative and should contain enough data to make some reasonably valid engineering decisions, and it annoys me when they get “gamed” specifically for marketing purposes, which is why I spend so much time writing about them.

      Regards
      John

  2. ausstorageguy
    November 15, 2011 at 12:32 pm

    Interestingly, as fate would have it, I did just come across NetApps benchmark for Lustre:

    4.4GB/s PER E5400 controller, now that translates to 1.1GB/s PER RACK UNIT, falling short of EMC’s 2GB/s per RU by 900MB/s…. huh… no wonder you’re on the attack.

    • November 15, 2011 at 2:10 pm

      “4U storage system, Whamcloud achieved 5GB/s for read performance and 3.5GB/s for write performance with the E5400”

      5GB/sec read (all the figures I’ve used and EMC has used have been the read figures), = 1.25 GB/sec/RU. If we used the 2U versions of the disk shelves as I said in the blog we’re looking at around 2.5 GB/sec/RU. More than the EMC benchmark result,

  3. ausstorageguy
    November 15, 2011 at 5:41 pm

    RJM > “5GB/sec read (all the figures I’ve used and EMC has used have been the read figures)” – Not True.

    EMC PR > “The EMC VNX7500 achieved 8 GB/s read and 5.3 GB/s write sustained performance, as measured by XDD benchmark” <- not sure what you define as read figures.

    John, You're damn right, I get real cranky at all of them for FUD, and I rip into them at any given opportunity, however, at a glance, almost 1/3 of your blog posts seems to be a montage of anti-emc slants from you. That’s my biggest issue.

    Why?

    Are you so concerned with tall poppy syndrome that you can’t just pitch the merrits of NetApp (and there are many) without writing guff about a competitor?

    The other big issue is this need to change your tune as soon as NetApp “buys it”, such as your thoughts on the engenio range before NetApp bought it, I’ll throw you an example:

    In several comment’s over the last few years, you show a level of distain and doubt for the performance for the IBM DS5000 Series, which, so far as I recall is an LSI Engenio 7900 (Now NetApp E7900) OEM.

    But now NetApp own the Engenio business, it’s all good… right?

    My point is, when any competitor anounces somthing, your first response should not be to… well… not respond.

    • November 16, 2011 at 3:35 pm

      I see your point, and I’ve taken it on board, its maybe a little unfortunate that there are lots of benchmark results coming out which is my favorite topic (just saw something else from HP today that I’m just taking a deep breath and ignoring), and I’m on a bit of a high horse about the way they’re being created published and used. I’m pretty happy with the way NetApp does their bench-marking, and it seems to me that many others abuse the process, which annoys me, so I write about it.

      Re comments on the DS range, I’ve probably taken a few potshots at them over the last few years, especially when they are put up against workloads better suited to FAS, though I’m reasonably sure I said nice things about the SPC-1 performance result IBM got for the 5300, I cant find this now, so maybe it was on someone else’s blog. Even so, I still don’t think I’d put a E-Series or DSxxx up against a FAS array for highly random or virtualised workloads. In the mainstream data-center they’re OK, but they lack the kind of advanced functionality and efficiency I believe people want from a mainstream storage product today. Our OEMs like IBM sell a lot of them into areas where I believe FAS is a better fit, but there are a lot of customers out there who have yet to be convinced of the value of truly unified storage, and those OEM sales will continue for the foreseeable future. That wont however, stop me from trying to convince every one of those customers that they would probably be better off with a FAS, or at least a V-Series solution..

      What the E-Series boxes are really good at though, is screaming raw throughput at a very compelling price point, especially for a bunch of “Big Data” workloads that demand sequential streaming performance. Things like High Def Video streaming, HPC, Siesmic Analytics, and Hadoop Filesystems. Today, these are niche workloads that very few companies care about much, but when people do buy storage for these things, they literally buy truckloads of the stuff, and this kind of extreme engineering fascinates me. While it’s possible to run those kinds of workloads on a FAS array, Big Data customers typically don’t want or need the array to do things like snapshots, or replication, or dedup, or intelligent application aware array based backups, or support for multiple protocols and workloads on the same set of spindles, they would rather just have as much performance as they can get in the smallest and densest package available, and the E-series does exactly that, its everything they want, and nothing they don’t.

      In short the rule of thumb is … if you need something like Lustre or Stornext or HDFS to get the performance or functionality you’re after and you need to buy Petabytes of kit, then E-Series is the way to go. For everything else it’s FAS all the way.

      As a final point of clarification I’m pretty sure that in XX GB/sec/RU used in the Benchmark Results they always use the read figure for the calculation, not an average of the read and write figures, I didnt mean to infer that EMC didnt publish the write performance, just that they didn’t use it or average it out in the final figure as I thought you did with the E-Series number when you said “4.4GB/s PER E5400 controller”

  4. Glenn
    November 29, 2011 at 2:06 pm

    Just ignore the troll… AusStorage claims to be anti FUD, but he wouldn’t comment on a HP/Dell/IBM Post. He’s an EMC fan boy in denial, just hit delete… Your long patient reply’s, don’t help settle anything. They just entice our friend to dig in deeper and reach father. I didn’t even know this guy existed until Chuck tweeted one of his post. After getting caught up, just ignore him… being the bigger man isn’t going to settle anything.

    Too much fire and brimstone from an obvious fan buy…

    ~Glenn

  1. No trackbacks yet.

Leave a Reply - Comments Manually Moderated to Avoid Spammers

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: