Contribution, Numbers, Statistics
You all know what I am saying below... I am not at all saying that we abandon citation numbers, etc. I've just seen too many manipulations, gaming the system, misleading claims, so that in the end I only ask: what is the contribution?, what is the quality?, did they do the work? Yes I use Consumer Reports, and I tend to ignore small differences in scores, and go for the higher number even though they tell me that differences of three points or less do not matter--just because it is easy, and the consequences are minor. But here we are talking about something of greater consequence.
Numbers in promotion packets may even be helpful, but in general they are manipulated. I've probably read 1000 dossiers, and so have seen lots of numbers (claimed to be statistics), only some of which are not faked one way or the other. In general, very low or very high numbers are indicative, but they ought be checked against your other evidence, and be treated with suspicion (this is my experience, having been bamboozled at first more than once--if there is a university promotion committee, the best part is that what you miss your colleague will discover). So I have read about 10,000 reference letters, and something like 25%, at best, are really helpful. I know of one distinguished scientist who checks his h-index value each Friday--but he is at the very top of his field, very well recognized and rewarded.
1. In general, what matters in the end is your contribution to scholarship. That is a substantive notion, and letters of reference and your personal statement should indicate that. MIT's economics department, at least in 1980, asked only that question. Numbers of publications, venues, citations, etc are only secondary. I do appreciate the need for numbers and statistics (if they are really statistics rather than numbers misrepresenting themselves as statistics). And playing Moneyball has proved extraordinarily useful, revealing what human judgment misses. Kahneman and Tversky have much to teach us.
As for numbers and rankings, it would be useful to have the most elementary of measures of uncertainty attached to them. When rankings differ by a tenth of a point, it surely matters for bragging, but not for actual information. See # 5 below. The numbers we get from citation sources are claimed to be complete samples, but in fact they are often polluted with junk. What should be the errors assigned to them?
2. If you are using numbers, and almost all citation "statistics" are just numbers
a. comparisons with a relevant cohort are useful
b. be sure they are not stuffed--
1. do most of the citations come from when someone was a postdoc with a famous scholar--so that you compare your candidate with someone elsewhere with terrific numbers, but in fact the high-number scholar's number come from that postdoc period
2. is the source of the numbers reliable--Google overcounts, ISI does not count books but is the most studied by the sociologists of science
3. do they accord with what you know of the contribution?
3. Again I understand the need for these numbers in rankings etc. Just be sure you are getting what you are paying for. Of course, you may be willing to allow the market to use these numbers and rankings to value your goods, but are you so happy when your value goes way down?
4. Universities are fabulous at pumping themselves up--eg. Best in the West, heralding its new rankings in someone's system etc,... Again, I want a university the football team would be proud of (said by the coach at U. Oklahoma in 1922 to the state legislature to get better support for the University)
My point here is that attend to what matters, and use the numbers (pretending to be statistics) to check your intuitions.