Monday, June 8, 2015

Ultrarunning National Rankings

 

We are living in the midst of the ultrarunning boom.

Frank Shorter's victory in the Olympic Marathon in Munich in 1972 is often cited as the launching point of the US running boom.  The explosion in popularity of "jogging" and road-racing in general (and marathons in particular) has its roots in that legendary race, as well as the publication of Jim Fixx's seminal The Complete Book of Running which was published in 1977.  Forty years later, ultrarunning is having its moment.  The success of Dean Karnazes' Ultramarathon Man and Chris McDougal's Born to Run have helped to fuel an unprecendented growth in ultramarathon participation.  An estimated 70,000 folks will complete an ultra this year, an increase of over 400% in the past fifteen years, according to Ultrarunning Magazine, the bible of the sport.

While it's still not exactly mainstream--that's about 12% of the 540,000 marathoners this year, not exactly a mainstream sport itself--the interest in our little niche has certainly swelled to unprecedented levels.  Sponsorship money is flowing--well, maybe not flowing per se, but at least trickling in.  Ultrarunnerpodcast and Talk Ultra boast thousands of downloads a month.  Ultrasportslive TV and irunfar.com provide live coverage of major races.  Stupid blogs like this one seem to be multiplying like rabbits.

So I've decided to do my part in servicing the Ultrarunning Boom.  If we're going to be a mainstream sport, let's act like it, dammit!  Let's get in on the sports conversation. Forget fighting about whether LeBron could take MJ one-on-one.  We want people arguing in bars (or, more likely, craft-beer tasting rooms) about who the best ultrarunners are, right?  Well, fear not.  Your prayers have been answered with the Ultrarunning National Rankings.

In truth, I've been kicking this idea around for a few years.  I train with alot of tri-geeks, and they all have their national rankings, both overall and for their age groups, courtesy of USAT.  I thought that was pretty cool, and thought a similar idea--a national ranking for any ultrarunning finisher in the country--would be pretty awesome.  I didn't particularly like the USAT formula, though, which involves comparing performances to estimated times by certain "pacesetters" in the race.  I didn't love the Ultrasignup rankings either, which work similarly, by awarding finishers a percentage of the race winner's time--it penalizes folks for racing in competitive fields, and rewards those who only run races they know they can win.  (Though admittedly the database power of Ultrasignup, allowing them to rank absolutely everybody, is very impressive.)  I wanted a system that acknowledged that not all races were created equal.  And so I found my model: golf.

The World Golf Rankings are very complicated, and may not be perfect, but they provided the basis for what I wanted in my system.  In the WGR, players accumulate points via their performances in tournaments on the various world tours.  The tours are all ranked--the US tour tanks the highest, followed by the European tour, and there are various factors for the Australian tour, the Japanese tour, and all the various mini-tours around the country--and the events within those tours are all ranked, too, based on how many of the top players in the world are playing.  Each tournament is assigned a value of importance--the majors are the highest, of course, with the World Golf Championships a step below.  The combination of what tour is involved, what the event is, and how many top players are there, determines how many ranking points are available, and how many players at that tournament will receive points.  The cumulative points are then divided by the number of events the player finished, and the result gives the player the number used for their ranking.

This was what I wanted.  Just like in golf, some events on the ultra calendar--Western States, Leadville, UTMB--are more important than others.  Placing highly in those events should carry more weight.  After all, which impresses you more: my win in a local 50K, or, say, Dominic Grossman's 19th-place finish at States last year?  (If you have to think about it, the rest of this post probably isn't for you.)  I could simplify things somewhat--ultrarunning doesn't have tours, for one thing--but the basic ideas were there.  Win small events, get a bit of credit.  Win bigger events, get more credit.  Beat the best runners in the country, get even more.

Initially, I had envisioned something akin to what USAT and Ultrasignup provided: a ranking for every finisher of an ultramarathon in the US.  Quickly, though, the impracticality of this idea became apparent.  First of all, there was just too much data to enter manually.  I don't have an automated database like Ultrasignup does, and I can't manually enter, say, 1100 finishers at JFK into a spreadsheet.  Plus, even if I figured out a way to automate it, more problems reared their head.  For one, duplicate names--how would I deal with, say, having five different Matt Smiths in the results?  (I've encountered this problem on Ultrasignup, which sometimes confuses me with another Jason Friedman about my age who happens to live less than 100 miles away.)  Limiting the rankings to top finishers doesn't eliminate this problem, of course, but makes it much more manageable.  Also, golf doesn't award points for every finisher--you have to hit a certain minimum criteria at a tournament for ranking points to become available.   That might mean making the cut, or the top 20, or whatever.  But not everyone gets points.  So not every finisher was going to get a ranking.  I had to make peace with that.

Trying to be as comprehensive as possible, I'm including every domestic race I can, as well as major international ones which might attract the top US talent.  Using the Ultrarunning Magazine race calendar, I started by assigning each race a score on a five-point scale.  Most races were ranked as Level 1: there are over 1000 ultras in the country, and you haven't heard of the overwhelming majority of them.  Level 2 races are slightly more prestigious--they might have some local or regional cachet, or they might have some name recognition by virtue of being associated with a race where a different distance is more important--say, the Ice Age 50K, which gains some prestige by riding the coattails of the Ice Age 50 Mile, but is decidedly the less important of the two events.  Level 3 races have a strong regional importance and maybe some mild prominence on the national stage, but aren't quite attracting the top fields--think Leona Divide or Umstead.  Level 4 races are national-class events--Miwok, Chuckanut, Speedgoat--that are separated from the top only by degrees.  (By default, I assigned all national championship races a Level 4.  They should be a big deal, even though not every US championship is created equal.)  Level 5 is reserved for the true majors: States, Leadville, Sonoma, North Face, and a handful of others that if you win, you just consider retiring right there on the spot since it probably will never get any better.  IAU World Championships and World Cup races are automatically granted a Level 5.

Using the same ratios for points and places that are used by the WGR, I then established how many points were available for each level of race, and how deep the scoring fields go.  A Level 1 race is worth five points to the winner, three for second, and one for third.  Level 2 races score out to the top 5; Level 3 the top 10; Level 4 the top 15; and Level 5 the top 25.  Generally, second place is worth about 60% of the points of first place (again, similar to the WGR), placing a premium on wins, which I like.

Next, we have to calculate a multiplier for strength of field--I want to reward people for racing against the best.  For the WGR, the top 200 in the world are used to calculate field strength; I settled on the top 50.  Each spot in the top 50 is assigned a point value, and those values correspond to a certain factor by which every finishing spot in that race is multiplied.  Of course, this is the first time we've done this, so there is no top 50 to work off of.  Instead, I used the results of the Ultrarunning magazine Ultrarunner of the Year voting from 2014 to set a baseline top 10 for the men and women.  When any of those top 10 run in an event this year, that race is worth correspondingly more points.  Not only does this place a high importance on seeking out top competition, it also acts as a reward for those who have previously achieved a high ranking.  (Once I have enough data for a reliable top 50, this multiplier effect will become a bit more pronounced.)

I decided to base the rankings on the sum of the points earned, not a per-event average, for several reasons that I will get into later.  At this point, my formula was set, and I started the (rather laborious) task of compiling results and manually entering the data into a spreadsheet.  It's a painstaking process.  I use the Ultrarunning website, as well as Ultrasignup, and I wind up doing a lot of Google searches to find results that aren't posted there.  So far, out of the 575 or so races that I have listed through the weekend of May 31, I've found results on about 90-95% of them (several were cancelled or have been removed from consideration due to fat-ass status or other situations that will make getting results unlikely).  In all, over 1400 men and 1300 women have earned at least one ranking point so far this year.

Before I unveil the actual rankings, I know you probably have some questions/concerns/criticisms.  I'm going to try to anticipate some of them now, and address them as best as I can by trying to explain my reasoning. 

Ultra rankings won't work.  You're comparing all different kinds of events: road, trails, track, timed events.  Different runners have different strengths.
True, but that's a feature, not a bug.  That's exactly why I think this is a cool idea.  You can accumulate points in any race, in any discipline.  In theory, the most well-rounded athlete should have the best chance achieving a top ranking.  Think about golf: some courses are long, some short, some with tighter lies or deeper rough.  Doesn't matter.  You have to beat whoever shows up on that day.

You're not accounting for times, or margin of victory.  What about course records?
I want the focus to be on the head-to-head competition, not the times.  Course records are nice, but ultimately meaningless.  You don't get more credit in golf for winning by 10 shots than winning in a playoff.  A win is a win.  Also, comparing times across courses--just like comparing golf scores across courses--doesn't work.

Your race ratings are terrible.
Yeah, well, that's like, your opinion, man.  This is obviously the most subjective part of the system.  I live on the East Coast, so there's probably some local bias involved.  And I certainly can't keep track of 1500 races and know which are necessarily the most important.  What I'd like to do in the future is have a committee of folks spread out around the country, so I can have people responsible for rating races in their home region.  Let me know if you want to volunteer.  When we monetize this thing we'll all be rich.

How come all the timed events/track races are rated so low?
That's not 100% true, but I'll admit, the track events generally are receiving lower ratings than their road and trail counterparts.  My explanation is that, in the current climate, these races are not as highly regarded, and the fields are (usually) much weaker.  This isn't to say that some of the stuff Zach Bitter and Joe Fejes are doing isn't incredibly impressive or historically important.  But the truth is that less people are paying attention to, and talking about, these accomplishments than they are dissecting results from Leadville or Lake Sonoma, and that less of the top runners are showing up at these events.

What about FKTs/solo record attempts?
There's no way to account for these in the system, which I am OK with.  As I've stated before, this is about head-to-head competition deciding who the best runners are.  I don't want to be in the position of deciding whether Zack Bitter's 12-hour record is more impressive than Rob Krar's R2R2R FKT or Mike Wardian's treadmill 50K.

In reality, I think all of these things that we're talking about--FKTs, course records, national bests--are fodder for voting, not rankings.  Which is great.  I find the UROY vote fascinating, and I'm not saying we should replace that vote.  This ranking is meant to be an objective supplement to that subjective process.  Golf has the WGR which is completely objective, and the Player of the Year vote which is completely subjective.  You can use whatever criteria you want to value when you're voting.

You should use an average, not a cumulative score.  Without an average, there's no penalty for running poorly or DNFs.
I struggled with this decision for awhile.  This is where I deviated from the WGR, which uses an average.  In the end, I decided an average for our sport simply didn't work.  First of all, I wasn't comfortable with setting a minimum amount of races to qualify--that seemed way too arbitrary to me.  Secondly, from a logistic standpoint, it was almost impossible.  I'd have to comb through all the results of every single race and record zeros for anyone who didn't earn points for that race, just so I could average that race in later if they happened to record points--just not doable.  Plus, how would I count slower, non-scoring performances vs. DNFs?  You'd like to reward finishing, I guess, so if non-scoring finishes are worth zero points, should DNFs result in losing points?  And then, how would I track DNFs at all?  Most race results don't list them.  If I can't track them, it might encourage people to DNF, rather than record a slower finish.  (The Ultrasignup rankings have run into this issue.)  Finally, some people use low-key races as training runs, or social events.  I don't want to discourage this practice by penalizing people for participating in a race without actually racing it.  Ultimately, cumulative points was the only way to go.

You do a terrible job with international races and international runners.
I'll admit this is the one of the biggest problems I'm having right now.  These are US rankings--God knows I can't do world rankings, though it'd be fun--but there are plenty of top US athletes racing overseas in big races, and I need to account for their results.  And since I'm not keeping track of international runners (except in certain cases when I know they'll run a bunch of US races) the field strength of these races isn't as robust as it probably should be, so the multiplier isn't as significant as you'd want it to be. These races aren't as important on the domestic level as they are overseas, so maybe, it's not as big a deal as I think it is, but I'll readily admit this is a problem that I haven't solved, and I'm open to suggestions.

Level 4 and 5 races are undervalued in points as opposed to levels 1, 2, and 3.
And this is the other problem.  Again, I based this off of the WGR, making level 5 races akin to the majors, level 4 in line with large events like the WGC or the Players, Level 3 some of the medium-sized PGA tour events, Level 2 a small PGA tour event, and Level 1 a mini-tour event. At baseline, winning a level 5 event is worth five times as much as a Level 1 event...and likely more, since the chance for having a field strength multiplier is much higher.  These are approximately in line with the ratios the WGR follows.  But the more I look at it, the less sure of these numbers I am.  Is winning Western States really just five times as important as winning a podunk 50K?  Should it be worth ten times as much?  Or twenty?  I'm a little too far into this now to change for this year, but next year I might need to tweak the relative values.  I could fix this by adding more levels, but I think parsing the different races between levels is going to get maddening.  (I mean, States is a 10, but is Leadville a 9?  Is Vermont a 6, 7, 8, what?  Too complicated.)  Probably the larger races will simply be worth more points.  (Incidentally, I do like the numbers of scoring slots for each level, which are 3, 5, 10, 15, and 25 respectively.  These seem about right and I'm pleased I got that right on the first try.)

ANYWAY...here are the first set of rankings, through May 31.  We're getting into the summer racing season now, when big events start coming more frequently, so I'm going to try to update these maybe every couple of months.  If you like, you can view the entire list here.  Sheet 1 is a list of all the races, with rankings and field strength multipliers.  Sheet 2 is my reference for how points are distributed and how multipliers are calculated.  Sheet 3 is men and Sheet 4 is women.  I have it listed alphabetically by first name, so you can find your name there.  Sorry, I don't have it set up to view the entire list in numerical order.  I'll figure out how to do that at some point I'm sure. Probably. Maybe.

Through May 31


Men
State
Points
Women
State
Points
1
Alex Varner
CA
52.6
Magdalena Boulet
CA
77
2
Ryan Bak
OR
48
Stephanie Howe
OR
65.5
3
Brian Rusiecki
MA
46.2
Nicole Studer
TX
55.6
4
Paul Terranova
TX
45.5
Cassie Scallon
CO
46.6
5
David Laney
OR
43.9
Aliza Lapierre
VT
41
6
Dylan Bowman
CA
42
Jacqueline Palmer
DE
37
7
Chikara Omine
CA
40
Kathleen Cusick
VA
34
8
Mario Mendoza
OR
38
Traci Falbo
IN
33.8
9
Rob Krar
AZ
35.7
Caroline Boller
CA
32.6
10
Jorge Maravilla
CA
34.6
Katalin Nagy
FL
32.5
11
Jorge Pacheco
CA
32.8
Sarah Bard
MA
31.225
12
Benjamin Stern
CA
31.2
Ashley Erba
CO
30.9
13
Jared Hazen
CO
30
Megan Roche
CA
30
14
Jean Pommier
CA
30
Melanie Fryar
TX
29
15
Justin Houck
WA
28
Bree Lambert
CA
27
16
Andrew Miller
OR
26
Amy Sproston
OR
24
17
Ian Sharman
CA
25
Angela Shartel
CA
24
18
Scott Breeden
IN
25
Meghan Arbogast
CA
23.225
19
Alex Nichols
CO
24.8
Rachel Ragona
CA
23
20
Christopher Dannucci
CA
24.8
Laura Richard
CA
22.5
21
Christian Gering
CO
24.25
Nuria Picas
ESP
21
22
Karl Schnaitter
CA
24
Tracie Akerhielm
TX
21
23
Tim Tollefson
CA
22.5
Lindsey Tollefson
CA
20.975
24
Seth Swanson
MT
22.2
Kaci Lickteig
NE
20.275
25
Ben Nephew
MA
22
Amanda Basham
OR
20
26
Daniel Hamilton
TN
21
Camille Herron
OK
20
27
James Blandford
PA
21
Joelle Vaught
ID
20
28
Nate Polaske
AZ
21
Megan Stegemiller
VA
20
29
Jason Leman
OR
20.6
Michelle Yates
CO
20
30
Fernando de Samaniego
CA
20.3
Silke Koester
CO
19.4
31
Jim Walmsley
MT
20.3
Emily Richards
NV
19
32
David Kilgore
FL
20
Kerrie Bruxvoort
CO
19
33
Ford Smith
TX
20
Julie Fingar
CA
18
34
Mike Bialick
MN
20
Ashley Lister
PA
17.8
35
Patrick Smyth
UT
20
Amy Rusiecki
MA
17
36
Zach Ornelas
MI
20
Jennifer Edwards
WA
17
37
Ryan Smith

19.5
Catrin Jones
Can
16.8
38
Zach Bitter
WI
19.5
Robin Watkins
DC
16.5
39
Lon Freeman
CA
19
Gia Madole
OK
16
40
Owen Bradley
AL
19
Suzanna Bon
CA
16
41
Adrian Stanciu
CO
18.5
Emily Harrison
AZ
15.6
42
Jason Lantz
PA
17.35
Kathy D’Onofrio
CA
15.6
43
Bob Shebest
CA
17
Kerrie Wlad
CO
15.2
44
Jared Burdick
NY
16.8
Leslie Howlett
UT
15.2
45
Brandt Ketterer
CO
16
Alicia Woodside
Can
15
46
Catlow Shipek
AZ
16
Amie Blackham
UT
15
47
Charlie Ware
AZ
16
Beth Meadows
TN
15
48
Karl Meltzer
UT
16
Claire Mellein
CA
15
49
Ray Sanchez
CA
16
Jennifer Benna
NV
15
50
Mario Martinez
CA
15.9
Jessica Lemer
WI
15




Joanna Masloski
CO
15




Megan McGrath
NJ
15

All right!  Let the ad hominem attacks on my character begin!

39 comments:

  1. Jay - wouldn't Strength of Competition essentially be redundant with the Race Level? That is to say, Level 4 and 5 races, by their nature, are going to attract better competition than Level 1s and 2s.

    Also, since this system is a participation-based system, it is biased toward the runner who races more often. IMO, this doesn't necessarily make them a better runner, just more prolific. For instance, if a runner enters only three ultras in a given year, but they are all Level 1 races and the runner wins them all, wouldn't they be a better runner than someone who runs in 8 but only wins one of them? Maybe there's a 'Podium Percentage Factor" that can be used in the system, and/or a minimum # of races to qualify?

    But to the point in your narrative, this system certainly wouldn't be perfect. At least not any more perfect than so many other sports/athlete ratings systems.

    ReplyDelete
    Replies
    1. TeeJay,

      For sure, you are much more likely to see tougher comp in the higher-level races. But, the point of the strength-of-field multiplier is twofold. First off, winning a Level 5 race like WS is a huge deal no matter who shows up, but it gets huger the more great names are in the field. Or put it this way: WS and Leadville are both Level 5 races and if you win one of those, that's a career. But, if eight of the top ten show up at States, versus three of the top 10 at Leadville, States should be worth a little more, right? (Making those numbers up on the spot obviously.)

      Second, lower-level races aren't likely to attract multiple top-ranked runners, but not every race the top guys run will be in a stacked field, and there should still be some recognition that the field is tougher than you would normally expect for a level 1 or 2 race. (If Rob Krar shows up at my local 50K, I damn well want some credit for racing him!)

      In terms of your second point: yes, you can accumulate points by cherry-picking small races and running lots of them, but the system is weighted so that it'll be very hard to make up the difference to the top-level races. In your example, you pose a runner who wins three level 1 races (though I'm assuming you mean level 5). That would be tough, but let's say that someone wins three level 4 races in a year and those are their only results. Leaving aside field strength multipliers (which are likely to be substantial in level 4 races, but whatever), those three wins would be worth 45 points. Someone cherry-picking low-level races would need to WIN 9 level 1 races to equal that total. The system is weighted pretty heavily towards winning races--second place is generally worth only 60% of the points of the winner, regardless of the level, so winning just a couple of big races is going to outpace running a whole bunch of small races every time.

      I thought about a minimum number of races, but ultimately i don't like it. I don't want to be the one telling Max King he has to run four ultras a year, or whatever random number I come up with. Plus it's too hard for me to keep track of--I'd have to record every single finisher in every single race, even the "non-scoring" finishers, just to keep tally of everyone's total number of races. Not practical.

      Thanks for your comments. Glad to stimulate some discussion, that's the whole point. Keep it coming!

      Delete
  2. Thanks for all the painstaking thought and effort! You're spot on with cumulative points. Running a race as training is part of what makes this bunch of rock-leapers so unique. Tweaking the weighting of the level 4-5's should smooth out the "prolific bump" mentioned above. Let me know if you need input on evaluating SE US races.

    ReplyDelete
    Replies
    1. Thanks Mac, I may take you up on that. Where are you located?

      Delete
    2. I'm in Alabama. Very familiar with big and small races all over the SE. From the Georgia Death Race on "down". Let me know how I can help. I can at least easily connect you to the big dogs in the region to form a committee.

      Delete
  3. Are you even a real doctor?!

    I think this is awesome - super entertaining.

    ReplyDelete
  4. A system for ranking already exists that takes course distance and elevation into account. It can be sorted by Country. http://www.i-tra.org/page/278/Performance_index.html#tab

    ReplyDelete
    Replies
    1. Jeremy,
      Yes, I've sen that. It's pretty cool and obviously the power of the sheer data is awesome. But I have a couple reasons why I don't like it as much. First--why is a difficult course worth more than a less difficult one? Everyone is running the same course on the same day. Also, it doesn't take into effect head-to-head competition, only time. Personally, I feel like the head-to-head competition is the essence of our sport. Time is obviously the great equalizer but I just tend to gravitate more toward the actual racing. And if it's based off of time, the results are susceptible to undue influence from course conditions or weather, not just between different races but from year to year.

      Delete
  5. Jason, first I really like the idea of rankings because I love statistics and lists so forth. You really put a lot of time and thought into this. A couple things to discuss.

    I think you need to have a higher ranking for US SkyRunner Series races. To this point there have been the Georgia Death Race (you give a 2) and the Quest for the Crest (you only give a 1). They are competitive with money, having a impact on a series standing. This attracts a deeper and more competitive field. I think all US SkyRunner Series races should be given at least a 3. (yes, I have run both of these races but I just wanted to bring up something I am familiar with). Do something similar to what you did with National Championships. In my opinion, the Quest for the Crest 50K was just as competitive and deep (or not far off) as the Cayuga Trails 50 Mile this year.

    Regarding DNF's, I know it is hard to track, but I think they should mean something. Here is what I think - you don't have to penalize DNF's. But, I think if you finish a race in a top position and a higher ranked person DNF's, that should add to your points total. For instance, (another personal example), I finished 2nd at the Georgia Death Race. David Kilgore (ranked 32nd on your list above) dropped out of the race. I don't think that Kilgore should be penalized for negative points, but maybe the top 10 of this race should be rewarded for beating a person who DNF'd who is ranked in the top 50 of your ranking.

    Lastly, I'd like to see a couple scenarios in your original blog post to see how a couple of the runners got their points just so I would understand it a bit more. Like, how did such and such accumulate XX points, etc.

    Thanks,

    Michael

    ReplyDelete
    Replies
    1. Michael, thanks for the thoughtful comments.

      Funny you mention the Skyrunner series. When I first started I had the exact same instinct--all Skyrunner races get a baseline level 3. I was wrestling with it a bit, since I didn't know what the response/interest was going to be for the series, given that it's new this year. And there are so many series out there, I didn't want to paint myself into a corner by committing certain ratings to certain series. But you're probably right about the Skyrunner series. I don't want to start retroactively making changes to every race ranking that people disagree with, but I may go back and change those races to level 3.

      Would you be interested in helping serve on a committee to establish race ratings for next year?

      In terms of the DNFs, they are accounted for much in the way that you suggest: they contribute to the Field strength multiplier, even if people don't finish. For example: right now, Rob Krar counts as the #1 guy for field strength (since I'm using the UROY ranks from last year). He started at Lake Sonoma, so his 25 "rating value" points are used to help calculate the multiplier. Therefore, everyone who finished in the top 25 at Sonoma had their points multiplied because of Rob's presence, even though he didn't finish. Again, next year, once I have some solid data, we'll use everyone in the top 50 to calculate field strength. So, if Dave Kilgore finishes the year ranked 32nd, that's a rating value of 5, which gives you a multiplier of 1.2. So if you were second in the race, instead of getting, say, 9 points for finishing second in a level 3 race, you'd get (9*1.2) = 10.8.

      I'll give you a concrete example as you suggest. We'll use Sonoma as it had the strongest fields so far this year. Let's take the women's race. On sheet 1, you can see I've indicated that runners #3, 4, and 6--that's Stephanie Howe, Kaci Lickteig, and Pam Smith--started the race. From sheet 2, we see that runner #3 gives a rating value (RV) of 17.5, runner 4 an RV of 15, and runner 6 an RV of 11.5, for a total of 44. Below that on sheet 2, we see that an RV of 44 falls in the range of 40.25-50, which corresponds to a multiplier of 1.9. Every finishing point value in that race is now multiplied by 1.9. So, take Stephanie Howe's win. Normally, a win in a level 5 race is worth 25 points. The field strength multiplier of 1.9 means that Stephanie actually gets 25*1.9 = 47.5 points for that race. That would be the case even if Kaci or Pam dropped out--just like the men's multiplier (which, as you can see on sheet 1, was 2.0 for Sonoma) was in effect for all scorers even despite Rob's DNF.

      Here's the problem--I don't know who DNFs a lot of the time. Obviously I knew Rob was running so I counted him. But, most of the online results I find don't list DNFs, so it's tough to pick up everyone. Not a huge problem right now since I'm only using the top 10 for the multiplier, and those are generally people that if they're racing, I hear about it. But next year when the multiplier expands to the top 50 it's going to be really tough. Open to suggestions on how to better capture that data.

      Sorry to be so long-winded! Hopefully that answered your questions.

      Delete
    2. Yeah, there are a lot of series but I think the SkyRunner series is more stable and legitimate. Yes, I would be interested in being on a committee to establish ratings. It is better to have a larger committee if this ends up being something permanent, especially if you could get something like UltraRunning Mag to accept it as something they use.

      Thanks for the detailed description. Makes more sense now.

      Shoot me an email at owenw@mymail.shawnee.edu. I'm also interested in knowing more about Salming - I hadn't heard of them before but would love to try their shoes and/or products.

      Delete
    3. I think you're probably right. Thanks for the feedback. I'll email you soon.

      Delete
  6. Jason, Interesting stuff! If you have an interest, send me an email at corys@ultrarunning.com and we can get you setup with an API to our result data, so you aren't doing this by hand..
    Cory
    UltraRunning Magazine

    ReplyDelete
  7. Man my brain hurts reading the effort you put into this, but thanks! My 9 year old is going to be stoked

    ReplyDelete
    Replies
    1. you have a 9-year-old ultra fan? that's awesome. My 9-year-old could give a shit.

      Delete
  8. Hi Jason, this is Camille Herron. I think you need to somehow weight/add points for things like Course Record, National Championship Record, American Record, World Record, and World Rankings. My performance at the 100K Road Champs, I broke Ann Trason's 26 year old Championship record, ran the fastest 100K ever on American soil, and fastest time in the World since 2008 (so I'm #1 in the World). I ran the #3 fastest American performance and 17th fastest 100K performance ever in the World. Your rankings favorite quantity over quality. I beat the #14 guy on your list by 31 minutes at the 100K... yet he's ranked 14th male and I'm only 26th best female?! I'm normally a very frequent marathoner, but I plan to race ultras more sparingly and go after the American/World Road and Track Records-- I'm chasing the clock, regardless of who shows up or the 'prestige' of the event. Just a thought to somehow calculate into your formula!

    ReplyDelete
    Replies
    1. Camille,

      Thanks for the input. First of all, congrats on an amazing run. Up there right now for performance of the year in my opinion, right alongside Joe Fejes' 6-day record. Your comments are very valid. I'm a little wary of using times/records as a criteria in the formula for a couple of reasons. I went into it in the full body of the post a little bit, but let me address the issue a little more specifically here.

      First off, I'm a bit married to the comparison with the golf world rankings, which also have no way of accounting for historically low scores or large margins of victory. People talk all the time about Tiger winning the US Open by 15 shots as one of the great performances ever--but they also talk about him beating Rocco Mediate in a playoff to win another US Open as one of the great performaces ever, and they both count the same for ranking purposes.

      Records are great and I love the fact that you're making it a priority to chase records and fast times. It's an aspect of the sport that I think gets lost a little in the focus on trail and mountain races and I wish more of the great runners in our sport shared that priority with you. I think by using them as a rankings criteria, though, it opens up too many questions with subjective answers to be a data point. For one thing, how to I compare the relative value of records to each other? Is your world best a more impressive performance than, say, Zach Bitter's 12-hour record? Equivalent? Less? How about Mike Wardian's treadmill WR, which isn't even a race? How about Scott Jurek if/when he sets the AT FKT? Does it matter if you break a record by a minute versus an hour? What if you are going for a record and miss it by a little bit? Do you not get the same credit? What's the value of the fourth-best performance vs, the eighth vs the 20th? Also, what's the value of breaking a record or setting a world best relative to winning a championship or championship-type race? What I mean by that is, if Zach sets the 100-mile record in a solo attempt on the track (or in a small track ultra), should that be worth more, the same, or less, than Rory Bosio winning UTMB? Does your answer change if Rory sets a course record, which is a historic accomplishment but not a world or American best?

      Delete
    2. Let me give you another hypothetical situation. I don't know what the conditions were like at Mad City, but let's hypothetically say they were ideal conditions, you had a great training block, you were ready to break all kinds of records, and you had a great day. Now imagine you come back next year and it's 30 degrees, windy, sleeting. You fight through it, have a really gutsy performance, win another national championship but fall short of your time from this year by, say, 30 minutes. You could easily make an argument that the second performance is more impressive than the first. I might agree with you. I don't want to evaluate the relative worth of those two times.

      I'm not saying these rankings are perfect. But eliminating consideration of times and records is the only way I can eliminate the subjectivity of comparing these relative performances. There is a place in the sport for arguing about these subjective questions--that's a huge part of being fans of a sport. The UROY and Performance of the Year voting is set up to address this very question, and records and world-best performances should unquestionably be factored into deciding who gets those incredible honors. But I'm trying to do something objective and I think assigning a numerical value to these performances (which is by definition going to be arbitrary) limits that objectivity. Unfortunately, your world best, Joe's 6-day, Mike's treadmill WR, these things are going to get left out. You guys have my UROY votes (theoretically; I'm not actually a voter).

      Finally--and this is not a reason that I didn't include records, just an interesting point that I'd argue, at least in theory--I feel like the sports world values championships more than records. Kareem Abdul-Jabbar has the record for the most points, but no one argues he's the greatest basketball player of all time. Barry Bonds has the most home runs, Brett Favre has thrown the most touchdowns; they're not even considered the best at their positions. To bring it back to golf, five players have shot 59, the lowest round ever. Only one of them has one a major championship. I would bet every one of the other four would trade the 59 for a major. Beating the best competition on the biggest stages is what makes the greatest athletes the greatest. If you follow other sports obsessively, as I do, you see that argument get made over and over. Again, that's not a reason to exclude records from the rankings--just an interesting point to make.

      I hope you don't feel like I'm denigrating any of your amazing accomplishments; I'm truly in awe. I just can't come up with a way to incorporate these factors objectively. But as always I'm open to suggestions. I hope you continue your amazing season!

      Delete
    3. Hi Jason, thank you for the kind words! It was my first domestic ultra, so I'm definitely excited to keep trying and pursue the American/World Records. I definitely thought that maybe you could somehow create a point system to account for the race distance-- like 50K and under has a point value, 50 miles and under, 100K, 100 miles and under, 24 hrs and under, and then beyond 24 hours. I feel that you could account for quality of performance at standard "IAAF/IAU World" RANKED distances (10 for 1st, 9 for 2nd, and so on)- 50K road/track/treadmill, 100K, 100 mi, and 24 hrs. Give added points for World (40), American (20), Course (10), and National Championship Records (10). There's a "fastest frequent marathoners" rankings-- they account for the quality of the performance by calculating the time difference between the performance and the median for the race (not sure if this is done with ultras?!). If you run a lot of marathons really far under the median, you accumulate more points than someone who's running a lot of marathons closer to the median for each gender. It's relatively fair! http://www.mathcs.citadel.edu/trautmand/stuff/bestfrequentmarathoners2013.htm

      Delete
    4. Camille,
      When I first started out thinking about this project, I had a similar idea--that point values would vary by distance. 100s were automatically worth more than 50s, 50 milers more than 50Ks, etc. But the quality of field and the relative importance of the race can cut across distances; size of field and who you beat should matter. Winning a small 100-miler with 10 or fewer finishers shouldn't automatically be worth more than winning a big 50K like Chuckanut or a big 50 mile like JFK just because it's longer.

      As I said earlier, I'm wary of getting into awarding more points for records, because I feel like it adds subjectivity into the system. Your suggestions are certainly reasonable and valid, but ultimately open for a lot of debate. For example, in the system you mention, a WR is worth four times as many points as a win. Maybe that's a reasonable estimate. Maybe it's too low, maybe it's too high. I think it's very, very open to interpretation. I don't mind the idea of adding some sort of bonus for records like you mention, though, at least in theory; I just have to figure out an agreeable standard of measurement. We've been talking on here about establishing a committee for next year to better rank the races in determining your value--an admittedly imperfect science that would improve with more people's input. Perhaps such a committee could come to a consensus on equitable point values for records. If such a consensus could be reached, I'd have no problem incorporating it. Though it will make things difficult trying to find out what course records are at various races.

      The time difference thing is always interesting--that's basically how Ultrasignup does their rankings, though theirs is based on time difference to the winner, not the median. For my money though, it's very susceptible to influence from field size and field strength. A very small field, or one with a few massive outliers, as we often see in the ultra world, is going to skew that data very quickly.

      Delete
    5. Hi Jason, it's totally objective to account for Records/quality of the performance. You account for quantity, so why not account for quality too and add points? You tried to develop a system for the prestige/quality of the event and "field quality". Now you need to account for the quality of the time/performance too.

      Btw, I ran/won 2 marathons and placed well at a half marathon in the 6 weeks leading up to the 100K. I wasn't anymore rested than the #14 ranked guy I beat by 31 min at the 100K. My legs feel a lot more beat up after marathons than I did after the 100K. While conditions were good (a bit breezy), Mad City is not a fast road course, which I knew after the first loop (stepping constantly from road, to sidewalk, to bike path- hard to get in a rhythm). I jogged the 2nd 50K just wanting to get the CR and have a good first 100K experience. I'm shocked this was the best time in the World since 2008-- only surpassed by Ann Trason on the all-time American list. I'll definitely go all out at the fast World Champs course (with rested legs)!

      So yes, if you're accounting for the quality of the race/field, you need to account for the quality of the performance/time.

      Delete
    6. There's nothing more objective and legitimate than certified and sanctioned road/track performances, which you can compare collectively amongst performances at other certified races domestically/world-wide. There's far more subjectivity and variability with trails, as you subjectively tried to rank by prestige of the event and quality of the field. This doesn't matter as much on the road/track because a fast performance is a fast performance, regardless of who shows up or the prestige of the event. If I decide to go break the 12-hr and 100 mile Road World Records at a race in San Francisco, my performance shouldn't be given a lower point value because you ranked the event as a Level 4 race (nor should I be given the same points as the "mens winner"- which I could plausibly beat the men's winner while going for the World Record!).

      Delete
    7. Camille,
      You make good points. It doesn't seem fair not to lend special weight to outstanding performances that occur in small or less "prestigious" events. The point bonuses you outlined in your previous comments for various records is an idea I can play with. I'm just afraid of the "slippery slope." As I've mentioned before, where does this leave FKTs and other solo attempts? How about unsuccessful solo records attempts or FKT attempts? If you have a points bonus for a world record, do you need a points bonus for the #4 all-time performance? #8? #17? Wherever you make the cutoff, someone is going to have a complaint.

      But this is not an un-solveable problem. I'll keep working on it. Probably won't change anything right away, but modifications will come in 2016.

      BTW I was not meaning to imply that Mad City was ideal for you or that you were set up for a peak performance--I was using that as just a hypothetical scenario. I guess I wasn't clear on that point, sorry.

      Delete
    8. Great point- the solo attempts are a whole other ball game!

      For all-time performances, ARRS keeps lists for the standard IAAF/IAU World distances. Whether you wanted to award points for ~Top 10 American all-time and Top 30 World, for the IAAF/IAU standard distances, that would work.
      http://www.arrs.net/AllTime/AllTime.htm

      I don't know where this leaves quality of performances for the odd-ball/less recognized distances. There's some lists on Ultrarunning Magazine, but I don't know where to look for world-wide lists?
      http://www.ultrarunning.com/featured/ultrarunning-magazine-all-time-lists/

      Thanks for caring and at least trying to solve it! I'm so new to ultra running, I have no idea where to even start if I try the trail stuff! At least with the road stuff I'm chasing the clock!

      Delete
  9. I come from the XC ski world where each race is scored to a ranking list. Each racer carries a points value (similar to ultra signup). But winning a race doesn't give you 100%. Instead the value of a win is dependent on the strength of the field. Take the top 5 athletes and average the 3 lowest points. Then multiply by a percent (60% perhaps). The winner is assigned this value. All other competitors are assigned a score based on their percent back from the winner. The best XC skiers in the world have very close to 0 points. In this way you can look at an average of the best 4-6 races over 6-24 months and have that as your score. Multiple times per year skiing governing bodies put out new ranking lists. The lists are then used for team selection and funding, among other things.

    ReplyDelete
    Replies
    1. Adam, that's a cool idea. IMO it would be a great way for Ultrasignup to modify their rankings to account for strength of schedule--they have a points value for each runner already, and they use the same sort of % difference to the winner in calculating value. I don't have the ability to rank every single runner pre-race to determine the field strength for every single race, but ultrasignup does. Incidentally how does that system account for new skiers who have not raced before? Our sport grows quickly with new athletes every day. Is there a baseline value that new racers are assigned?

      Delete
  10. Very interesting, in the UK we have a website called RunBritain that does this. They too use a system similar to golf.http://danielsrunningblog.blogspot.co.uk/2014/11/my-guide-on-runbritain-handicap-sss.html?m=1

    ReplyDelete
    Replies
    1. Daniel, thanks, that is an interesting system. An interesting way to account for the course difficulty--as I understand it from your post, the difficulty of the course, including terrain and weather conditions, is basically determined by how well everyone performed relative to their handicaps. As you point out, though, there are a lot of other factors that might skew that data, including field size. (Of course, field size can skew the data in my system as well!)

      Delete
  11. How often will you be updating the list? It would be interesting to see how things change.

    No. 41

    ReplyDelete
    Replies
    1. Adrian,
      You can follow the link to the spreadsheet at any time to see updates, which I do over the course of the week when I have the time. Though for now you won't be able to rearrange the list in point order, it has to stay alphabetical. (Otherwise I won't be able to input results.) I'm anticipating posting every couple of months with an update to the top 50, which should give it enough time to see some movement. Next year maybe we'll aim for quarterly rankings.

      Delete
  12. Nice work, Jason. Picking apart 1000 plus races to value individual performances is downright impossible. Are you familiar with the professional road cycling World Tour rankings? https://en.wikipedia.org/wiki/UCI_World_Tour This ranking system is in the spirit that you have pursued. I've mentioned doing something like this in ultras, but the sanctioning bodies are too disparate in the sport. The commercial interests (TNF, UTWT, Skyrunning) and even the grassroots events (Lake Sonoma) rule over formal sanctioning. Herding ultrarunners is like herding cats.

    ReplyDelete
    Replies
    1. Thanks Gary. The ultra world is a tough nut to crack for sure. Often times I feel like we run into the old problem of the enemy of "good" being "great." If we find like we can't find the ideal solution to a problem we sometimes just say "screw it". This comes up in governance, drug testing, rankings, race logistics, etc. Of course I'd like a perfect system, and I'll strive to make this one as good as I can, but even if it's not ideal, I think making attempts at these sorts of things is a worthwhile pursuit. Eventually maybe outsider efforts such as this can help unite some of the disparate issues you spoke about, if they get enough support behind them. Look forward to catching up with you soon; I may be at UROC if you're going to be there.

      Delete
  13. Wouldnt an ELO rating system work better for this just comparing all the runners as head to heads instead? Then you dont have to worry about field strength, because the ELOs would handle it.
    Though it'd require tracking a lot more people rather than a handful that get points.

    ReplyDelete
    Replies
    1. not sure exactly what you mean--comparing each individual runner to every other runner in the race? would be pretty great but doesn't sound practical for me to handle.

      Delete
  14. Jason - I enjoyed this. Keep writing and contributing to the ultra community.

    ReplyDelete
  15. Hey Jason, I know this is an old post and there's probably a better way to contact you but hopefully you'll see this message. I got here from the link you posted on the women's UROY discussion on irunfar. After spending some time reading your article on Ultrarunning and your post here, and perusing the actual content of your ultra rankings from last year and this year, I'm extremely impressed and intrigued. I mostly just wanted to give you some encouragement on the effort you've put into the task of compiling and constantly updating results into your ranking system. I want to say thank you and make sure you know that your work isn't going unappreciated. For all the criticisms and opinions I'm sure you've received in the last year and a half since publishing this, I want you to know that I think it's brilliant and I like the fact that you've obviously put lots of careful thought into designing the system and into recognizing and being transparent about its limitations. As much as I appreciate the spirit of simplicity in ultrarunning, I also think that a system like what you've put together is in order for use on a wide scale in the sport, and I applaud your effort in taking a thoughtful approach and going for it. I hope your system becomes more recognized or at the very least gets more momentum going behind an objective and comprehensive system of ranking in North American ultrarunning.

    I'd also like to offer to help out in your effort with this system in any way I can. I'm pretty familiar and current with the ultra racing scenes in Colorado and Texas, but I've traveled around and run in a lot of other areas too; mostly western US. I'm also handy with Excel-based data manipulation and programming logic, so I could potentially help automate some of the data input if you're still doing all of this manually.

    Anyway, thanks for all the hard work on this and let me know if I can be of help!

    ReplyDelete
    Replies
    1. Hi Chris, thanks for the input/encouragement! I tried to email you but didn't have your address, so i left a comment on your most recent blog post (which was admittedly some time ago). Check there and shoot me a line. If you can't find it let me know.

      Delete