In response to the use of citation data in research assessments such as Excellence in Research for Australia, the International Mathematical Union (IMU) and the International Congress on Industrial and Applied Mathematics (ICIAM) are considering producing a mathematics journal rating system to mitigate the exploitation of commercial or national rating methods, see also the 2008 citations report and the 2010 best practice report. They write:
In implementation of Resolution 18 adopted by the IMU General Assembly in 2010: “The General Assembly of the IMU asks the EC to create, in cooperation with ICIAM, a Working Group that is charged with considering whether or not a joint ICIAM/IMU method of ranking mathematical journals should be instituted, and what other possible options there may be for protecting against the inappropriate use of impact factors and similar manipulable indices for evaluating research.”
They tell us further that
The International Mathematical Union and the International Council for Industrial and Applied Mathematics (ICIAM) jointly constituted a Working Group to study the issue of whether and (in the affirmative case) how both organizations should go forward with a Ranking of Mathematical Journals. After discussing the report at ICIAM 2011 in Vancouver IMU and ICIAM decided to start a blog on mathematical journals which became operational on November 18, 2011. Please consider a contribution to the ongoing discussion.
The need for such a listing is increased by the continuing explosion in the number and diversity of mathematical journals. Obviously high-quality, peer-reviewed journals constitute a very important component of the mathematical research enterprise. MathSciNet currently reviews 680 journals cover-to-cover, and another 2000 more selectively. The Zentralblatt Math database indexes papers from more than 3500 journals.
Certainly many of these journals are of high quality and are indispensable to research work. Unfortunately, as the IMU notes there are also some journals which are published with questionable motivations (such a self-advancement) or which do not rigorously peer-review their submissions, and the recent explosion in mathematical journals has unfortunately included quite a few in this category.
Hence, the IMU and ICIAM are leading an effort to develop a workable ratings system. As stated in the recently released working document, various criteria have been proposed for a ratings system:
1. Quantitative bibliometrics, such as citation counts.
2. Reputation, as determined by surveys.
3. Evaluation of the journal’s editorial process.
Bibliometrics by themselves are not a reliable indicator of journal quality. Similarly, a ratings system based only on qualitative reputation would not meet standards of objectivity. And it does not seem feasible to obtain or rely on objective methods to assess the journal’s editorial and/or refereeing process.
Thus, a more nuanced and modest system is being sought. The current proposal is to form a rating committee with appointees from both IMU and ICIAM to provide rankings. The panel would would start by selecting a list of journals that publish papers primarily in mathematics. The panel members would each then be assigned to evaluate a subset of the journals. When these are complete, the panel would then produce a consensus ratings. The panel would then assign each journal to one of these four categories:
Tier 1: A top journal in mathematics or a major subfield of it. Almost all papers published are of very high quality, and it regularly publishes papers that are of great significance. Peer-review is applied consistently and rigorously, and editorial work is carried out by leading mathematicians.
Tier 2: Very strong journal with a carefully run and reliable peer-review process. Papers are generally of high quality, and regularly papers are published which are of significant importance in at least a subfield of mathematics.
Tier 3: Solid journal that generally publishes reputable work and follows accepted practices of peer review, but are generally less selective than journals of Tier 2, and paper quality is more variable. Such journals may play an important role in specific communities, but are usually not considered highly important to mathematics or a subfield globally.
Tier 4: Journals not found to meet the standards of the other three tiers.
We note that this categorization does not obviate the need to assess individual articles rather than journal but it does avoid many of the pitfalls involved in giving each journal a grade ranging from `A+` through `F`. Moreover, as it is clear that such rankings will continue to be be developed and used, it is better they be made by our colleagues rather than by others with other agendas. It is also worth observing that peer review is never compromised by more robust bibliometric data.
Additional information can be obtained from: IMU-ICIAM Journal Working Group Preiminary Report.