Always Question the Metrics

I was talking with one of my raid leaders the other day, and we were looking over logs comparing some of our lower DPSing raid members, and we realized we weren't looking at the same numbers.  So he linked what he was looking at, which happened to be the damage done chart across several Mannoroth wipes from warcraft logs with &cutoff=3 (if you don't know what that is, it's not important). I started to type something like "oh, I didn't realize 3 deaths was when you were calling wipes". Before I even finished typing it, he followed up with "3 is a rough statistic, but a decent cutoff point for wipes". I was fairly astonished that he had read my mind, to which he replied "always question the metrics :-P".

As we approach the legion beta, I've been thinking a lot about how I want to present the data I collect, and how I use it to come to conclusions about huntering.  I think there's been a bit of a shift in the WoW theorycrafting community lately, and a lot of it has revolved around questioning metrics we previously valued.  So I thought it might be fun to go over a few of those and see if I can find any other places where our metrics have fallen short.  

Level 45 Talents: Exhilaration vs. Iron Hawk

A while back, over in the hunter IRC (#beastcleaves), I witnessed a fairly interesting argument over whether to take Iron Hawk or Exhilaration when raiding. What made the argument interesting, from my point of view at least, was that they weren't really arguing over which talent was better (even if that's what they thought they were arguing about), they were actually arguing over what metric was best to determine the best talent.

On the Iron Hawk side of the argument, the metric used was total damage negated over the course of the fight.  It's easy in your log to look up how much damage you took, which tells you how much you would have taken with out Iron Hawk, and compare that number to how much you could heal with Exhil (30% of your HP every 2 minutes).  With this metric, in almost every case, Iron Hawk is the clear winner.

The Exhilaration side of the argument, however, was saying total damage negated isn't a useful metric, but instead we should be interested in which talent is the most likely to prevent a death.  Iron Hawk will only prevent a death in the fairly rare case that the damage of a single ability (or multiple abilities in quick succession) does 1-9% more damage than you have HP.  If it does more damage than that, you'll die anyway.  If it does less damage than that, you'll live through it anyway. Exhilaration, on the other hand, often prevents death to many mechanics, as you can cast it on-demand, when you have less HP than an damage that's about to go out.  For example, if you need to soak a Gaze on Mannoroth, or a Edict of condemnation on on Tyrant Vel, but your HP is too low, you now have an option that could (perhaps) prevent a wipe.  

Of course, both of the arguments were a bit more nuanced than I've presented here.  Though in general, I agree that Exhilaration is the better choice on progression fights, Iron Hawk does have the added benefit of making damage "less spiky" which a lot of healers prefer.  There are several other small benefits to both talents, but the point is, it's a lot more complicated than just saying Talent A does x healing, and Talent B prevents y damage, x > y therefore...

Weighted DPS

An amazing addition to Warcraft Logs recently was the addition of weighted DPS.  I've talked about this on the podcast, but I don't think I've mentioned it on here.  What it does is adds extra value to damage down to priority targets, and reduces the value of damaging trash mobs that will die quickly and easily anyway.  The point is, it rewards you for focusing on what you should be focusing on, and it punishes you (to some extent) for just spamming AoE.  

The reason I bring this up is it's a new metric for viewing the importance of specs that don't necessarily perform as well with the traditional metric of plain DPS.  MM hunters, for instance, aren't do not appear in the top ten parses for any current boss aside from Archimonde. Any of you out there playing MM, you may be frustrated that you just can't out-dps the mages or warlocks on your team (assuming they're close to the same skill level as you).  However, you still see all of the top guilds in the world with at least 2 or 3 MM hunters on their roster. This is because we MM hunters have the ability to do damage that really matters.  Burning down priority targets is our specialty.  So even if our total damage done or DPS is lower than many other classes, the damage we do helps kill bosses, it's never just padding.  

And this is what the Weighted DPS option is there for: to give a metric that shows meaningful damage instead of treating all damage as equally useful.  

Applying this shift to Hunter Math

If you were reading this blog during the WoD beta, you will be familiar with some of the metrics I like to use when assigning priority to our abilities; those being Damage per Cast-time, Damage per Focus, and the somewhat messy Damage per Cast-time per Focus, or "damage per cast-time per focus per cast-time" (it may seem quite silly, but sometimes it helps me to think in those terms).  

None of these metrics tell the whole story on their own.  If you simply look at Damage per Cast-time (DPCT), you'll miss the value of abilities that might not do as much damage, but are incredibly cheap (in focus cost) that make great filler while waiting for CDs of more valuable shots, such as the problem with barrage on Single Target fights, it might do the most damage per second, but it costs so much focus it will costs you Aimed Shots (or potentially Chim Shots), making it actually less valuable than .  If you only look at Damage per Focus (DPF) you'll likely over-value highly efficient abilities that actually cause you to miss out on damage, such as the common mistake of valuing Aimed Shot with Thrill of the Hunt procs over Chim Shot.

My project, then, is to apply this shift in thinking I described above (weighted DPS and the how level 45 talents are compared) to theoretical hunter math.  What I mean is instead of only focusing on metrics that determine the greatest potential DPS, include some that measure, for example, on demand burst potential, or perhaps reliability (I'm sure everyone has had a few miserable openers where Thrill of the Hunt never proc'ed).    

I'm not totally sure how involved some of these metrics would have to be, but I am looking forward to diving into the beta, and seeing what I can come up with.