DRS. MCCAMMON AND HAEGELI'S REASONS FOR REFUSING TO PROVIDE ACCESS TO THE DATA BEHIND THE AVALUATOR'S OBVIOUS CLUES PREVENTION VALUES

According to the Calgary Herald article “Is there a problem with the Avaluator?” (Aprill 19, 2009), Dr. McCammon claims that he “explained to Uttl that the data is available to the public” but “the problem is that the records... are only stored on paper, necessitating a visit to the facility” and we did not yet make this trip.  

However, Dr. McCammon is confusing "accident records" (i.e., written reports of accidents) with "data" (i.e., the list of obvious clues present and absent for each accident record).  We asked for the coded data showing whether Dr. McCammon judged each obvious clue to be present or absent in each accident record (we already have the same accident records as Dr. McCammon does, for example, the Snowy Torrents books describing avalanche accidents in USA).

In contrast to his claim, Drs. Haegeli and McCammon have (a) refused to provide access to the coded data behind the Avaluator's Obvious Clues prevention values, (b) refused to provide even the list of the 252 accidents that remained in their data set after they inappropriately excluded 82% of accidents from their 1,400 strong original sample, and (c) refused to provide the exact methodology (e.g., coding key) specifying how they decided whether each obvious clue was present or absent.

Dr. McCammon's explanation to Elsevier, the publisher of Cold Region Science and Technology

In May and June 2008, Dr. McCammon refused to provide access to the coded data behind the Avaluator's prevention values because, as he claims, they are his intellectual property and have substantial commercial value. Although authors publishing in Elsevier's journals have a “duty” to “be prepared to provide public access to such data”, Elsevier was “not in a position to demand that” and in any case they are merely “encouraging transparency, rather than demanding or requiring it.” (read email from Femke Wallien, Publisher Geology, Elsevier).  Thus, Elsevier's authors have a duty "to be prepared to provide public access to such data" but have no duty to actually provide access. 

"I just want to comment on one specific comment aspect of the presentation before which was that we were very secretive about the method used and we did not allow access to the data we had...

"... There is multiple reasons why we think Ian and myself think that giving access to those 252 data points is not a useful thing to do. In fact the raw data that Bob [Uttl] demands so strongly is publicly available. Everybody can go to into Colorado Avalanche Information Center, get those records in detail, and can basically develop their own database with these accidents. So the raw data is definitely in public domain. By providing this list of 252 accidents, we would automatically bias external validation of the Avaluator and therefore make the project scientifically invalid. And we strongly stand behind that."  (wach the video of Dr. Haegeli's explanation)

Dr. Pascal Haegeli's explanation to the audience at International Snow Science Workshop, Whistler, September 2008, in response to a question from the audience

Question:  "Um, I just have a question about your response earlier. It seemed that, during earlier presentation, as I understand what you said it was that if you made the accident list available it would bias independent evaluations. And I just did not understand what you meant by that. So I wonder if you could say little more about it."

Answer (by Pascal Haegeli):  "Uh, so there is basically this large data set ... in Colorado Avalanche Information Center and Ian coded that in specific way and if we if you were presented with that coding you would basically immediately be biased towards that coding. The better way to do it is to basically start with the clean slate, to do the same work, and than compare at the end these two results together."   (watch the video of Dr. Haegeli's explanation)

According to Dr. Haegeli, Dr. McCammon coded the data "in specific way" and Dr. Haegeli believes that it would be inappropriate for other scientists to know the "specific way" he coded it.

We did that same work and found results inconsistent with the Avaluator

We started from scratch, started with exactly the same accident records as Drs. McCammon and Haegeli (e.g., accidents described in Snowy Torrents), developed a coding method (since Dr. McCammon and Haegeli refused to reveal theirs), coded all accidents, and found a nearly identical distribution of Obvious Clues as Dr. McCammon (2004) had, prior to inappropriately excluding all data with missing values.   

Following in our steps, Dr. James Floyer, commissioned by the CAC repeated the process independently of us.  He found a nearly identical distribution of Obvious Clues as Dr. McCammon originally had and as we also had.  In the next step, Dr. Floyer excluded 71% of the data the same way Drs. Haegeli and McCammon had, but still he was unable to replicate the prevention values published in the Avaluator.  Dr. Floyer found the Avaluator's prevention values to be inflated, even after he also inappropriately excluded all data with missing values.

Obvious Clues Distribution The distribution of Obvious Clues in US accident samples reported by McCammon (2004) for all accidents; CAC Avaluator (Haegeli & McCammon, 2006) for 18% of accidents after 82% were excluded due to missing values; Uttl, Henry, and Uttl (2008) for all accidents; CAC Floyer (2008) for all accidents; and CAC Floyer (2008) for 29% of accidents after 71% of accidents were excluded due to missing values. The distributions of clues reported independently by McCammon (2004); Uttl, Henry, and Uttl (2008); and Floyer (2008) are nearly identical when no accidents are excluded from the samples, even though the samples of accident records are not exactly the same.  The distributions are substantially shifted towards a higher number of clues when accidents with missing values are inappropriately deleted (CAC Avaluator with 82% of accidents excluded and CAC Floyer with 71% of accidents excluded).  Moreover, even the Floyer and Avaluator distributions are dissimilar, that is, Dr. Floyer's study did not confirm the Avaluator's distribution even after excluding the vast majority of the accidents with missing values.

Obvious Clues Prevention Values The Obvious Clues Prevention Values based on the Obvious Clues Distributions reported by McCammon (2004); CAC Avaluator (Haegeli & McCammon, 2006) (after exclusion of 82% of accidents); Uttl, Henry, and Uttl (2008); CAC Floyer (2008); and CAC Floyer (2008) after exclusion of 71% of accidents. The prevention (risk reduction) value of 4 Obvious Clues is 77% according to the Avaluator after exclusion of 82% of accidents; 47% according to CAC Floyer after exclusion of 71% of accidents; and only about 20% according to all data sets that were not selectively biased by eliminating accidents with missing values (i.e., McCammon (2004); Uttl, Henry, & Uttl, 2008; CAC Floyer, 2008). The prevention values calculated from complete accident sets examined by McCammon (2004); Uttl, Henry, and Uttl, 2008; and Floyer (2008) are nearly identical. However, the prevention values calculated from the accident sets after inappropriate exlusion of the vast majority of all accidents due to missing values are vastly different and even inconsistent with each other.

What is Drs. McCammon and Haegeli response to these findings?  

Dr. McCammon now claims, and Dr. Haegeli seconds it, according to Calgary Herald, that the Avaluator's sample of 252 accidents after exclusion of > 1,148 accident is superior to our samples.  This is extremely unlikely since in the studies cited above we worked with accidents from US and we all obtained very similar distributions of Obvious Clues.  

The inflated prevention values of the Avaluator stem from inappropriate exclusion of 82% of accident records due to missing values.

Moreover, if Dr. McCammon and Haegeli want to claim that their sample is supperior they ought to produce their data and the list of the 252 accidents remaining in their analyses.  Until Drs. McCammon and Haegeli produce their data, we cannot establish the alleged "superiority" of Drs. Haegeli and McCammon sample since (a) he has described it in various reports differently (see Uttl, Uttl, & Henry, 2008) and (b) refused to reveal the list of the accidents that are included in his biased sample after excluding 82% of accident records due to missing values (see Uttl, Uttl, & Henry, 2008)

With the exception of Drs. Haegeli and McCammon, everyone agrees: Avaluator's Obvious Clues Prevention Values are inflated and the number of recreational accidents has increased rather than decreased

(1) Original data behind the Avaluator's Obvious Clues are held in secrecy by Dr. McCammon, if they actually exist.  We have no idea which accidents actually were selected by Dr. McCammon for the calculation of the Obvious Clues prevention values published in the Avaluator.  And we have no way to establish the "superiority" of his accident sample.

(2) The Obvious Clues prevention values in the Avaluator are grossly inflated, falsely telling Avaluator users that they reduce their risk of accidents by 77% when in fact they reduce their risk only by a much smaller percentage. We demonstrated this more than a year ago, in March 2008, and Dr. James Floyer (working for the Canadian Avalanche Center) repeated our demonstration in September 2008.

(3) Avalanche Safety Training students are told to disregard the Obvious Clues prevention values even by Mr. Albi Sole, the co-ordinator of the Outdoor Center at the University of Calgary. Why would the largest AST provider use something that is known to be invalid, that is introduced to students with the statement to “disregard it”?  And who will tell Avaluator users who have already purchased the Avaluator to "disregard" the Obvious Clues prevention values? 

Canadian Avalanche Accidents Trends.  In 2005-2006, the season just prior to the introduction of the Avaluator, we had 8 fatalities (recreational and non-recreational), 6 recreational accidents; the 2-year average number of recreational accidents was 6; and the 3-year average was 7.  As of today (April 17, 2009), this season, we have had 26 fatalities, 16 recreational accidents, the 2-year average is 15.0 accidents, and the 3-year average is 11.0 accidents.  Thus, the number of accidents following the Avaluator's introduction on the market has increased sharply rather than decreased.