Uses and abuses of Scientometrics

The production of original research articles, and the number of citations they generate in international indices, have become the ordinary indicators to measure the contribution of each researcher to the scientific work.

Miguel Ángel Pérez Angón Senior Researcher in the Department of Physics at Cinvestav, he was the founding editor of the Atlas of Mexican Science and the Latin American Catalog of Programs and Human Resources in Physics.

1. Scientometrics
In the last five years the Mexican government apparatus has been consistent in applying the maxim that the social contribution of science and technology is not so clear and much less direct. As a result, federal support for scientific activities has declined significantly. Consistent with this attitude, government delicacy to quantify the impact of its science and technology programs is increasingly evident. In this context, most national institutions have consolidated the culture of evaluation and monitoring of research carried out by their groups of researchers. In particular, the production of original research articles, and the number of citations they generate in international indexes, have become the ordinary indicators to measure the contribution of each researcher at the scientific task.

In this state of affairs, we have become accustomed to using two terms almost as synonyms but it is convenient to define more precisely: bibliometry and Scientometrics. Bibliometry is restricted to the study of research products published in scientific literature, mainly articles in journals, and the citations generated in journals included in the Science Citation Index (SCI), the Social Sciences Citation Index (SSCI) and the Arts & Humanities Citation Index (AHCI). Instead, Scientometrics aims to cover a much broader field than bibliometry and includes the analysis of other factors that may be determining factors for the development of scientific activity: number of researchers, their geographical or specialty distribution, sources of financing, productivity. and repercussion, etc. It is in this last sense that we will use this term in this article.

2. Uses
The use of Scientometric Indicators is increasingly common in developed countries to define their policies to support scientific and technological activities. For us, the analyzes carried out in countries such as Brazil, Spain, China or India are of greater interest (1). In all these cases, the constant increase in the scientific production of its researchers and the interest in reaching the levels of researchers in developed countries stand out. For example, in the case of Spanish science, they were interested in knowing in which areas and specialties their production and impact indicators are comparable or superior to those of the rest of the countries of the European Union.

In our country, CONACyT has systematically published several indicators on scientific activity in Mexico and in some other countries (2). However, the data published by CONACyT are global and are not broken down by area or specialty and even less by institution. With the idea of covering this deficiency, the Mexican Academy of Sciences created in 2002 the Atlas of Mexican Science (ACM). The goal of this project was to integrate statistical databases on all areas of science that are cultivated in our country and make them accessible to all users of scientific activity. This program only had two years of activity but in that period it was possible to generate ample and detailed statistical information on ten areas of Mexican science: biological, physical, chemical, land sciences, agricultural sciences and biotechnology, medicine and health sciences, mathematics, geosciences, social sciences and humanities (3).

The information generated by the ACM is very complete and reliable thanks to the feedback that was received from the scientific community and from the institutions of higher education and research. In particular, there are data on the academic plant (institutional affiliation, academic seniority, productivity and impact, composition by gender, level of studies, nationality and specialty), and on the degree and postgraduate study programs (enrollment, new entry , egress, terminal efficiency), but in the latter case only for some of the ten areas covered by the ACM.

To give an idea of the possible uses of the indicators generated by the ACM, I will mention some of those obtained in the case of physical sciences – one of the areas best covered by this program. Figure 1 includes the evolution of the scientific production (articles in the international indexes) of the ten areas studied in the period 1980-1999. Among them, the increase in production in physical sciences stands out by a factor of six in this period (from 200 to 1200 articles). This increase corresponds to the constant growth of researchers with a doctorate in this area of knowledge (in 1987 only 337 were registered and by 2003 they were 12363). The data collected by the ACM also suggests a possible explanation of why Mexican physicists became the most productive scientific guild in the country in such a short period: it is the most homogeneous group in academic training (there is a minimum of researchers who have only finished his master’s or bachelor’s degree studies) and that he has been subject to a stimulating process of geographical de-concentration: in 1987, 70% of the PhD researchers were assigned to the institutions located in Mexico City, and by 2003 this number already It had been reduced to 35%. It should be noted that this process occurred within the broad growth of the academic plant at the national level and not due to a simple decrease of researchers in the institutions of the metropolitan area of Mexico City. For example, in 1987, UNAM had 51% of doctoral researchers (169, and at that time all were concentrated in CU), but by 2003 they were only 36.5% (451) and those located in CU corresponded to 27.8% (343). Another factor that stands out from the data generated in the ACM is the level of international collaboration that Mexican physicists have achieved and that has allowed them to achieve levels of visibility in the international environment that only biomedical researchers had achieved before: the number of articles published In the decade of the nineties that have generated more than one hundred citations is greater than those generated at this level by the biomedical community. Many of these articles were produced from international collaborations in the large laboratories of particle accelerators and astronomical observation centers (4). A similar situation occurs in the area of geosciences, where some of the most cited articles were published based on the participation of Mexican scientists in international groups that have explored the Chicxulub crater in Yucatan or have conducted studies through perception techniques remotely operated with terrestrial satellites.

There is another very interesting fact that arises from the data included in Figure 1: from the creation of the National System of Researchers (SNI) in 1984, practically all areas initiate a consolidation process and its annual production of scientific articles It increases steadily. This effect has already been detected in other Scientometric analyzes (5) and is a transparent example of the positive uses of Scientometrics.

3. Abuses
We already mentioned some of the uses and applications of the data generated in Scientometrics, all of them located in the macro sphere of science and technology. Scientometric abuses appear precisely when some macro-type results are applied at the micro level, that is, to the evaluation of researchers individually. There is a large literature on the disadvantages of this procedure (6), I will only mention a few of them below:

• The international productivity or citation averages cover very general areas and in most cases do not correspond to the lines of work of individual researchers. These averages vary over time and this possibility is not contemplated in the institutional regulations used to promote or hire researchers.
• The impact factors determined for the journals included in the international indexes vary year by year, and consequently the list of “the best journals in the specialty” is also variable (7).
• There are good quality journals that are not included in the international indexes, particularly in areas such as mathematics, social sciences and humanities. For this type of magazine there is no information on its impact factors, immediacy indexes, etc.

Perhaps the main abuse of Scientometric Indicators occurs in institutions where they are taken strictly in the promotion and contracting processes. In most high-level institutions, academic evaluations are determined by specialists who know the specific topics of interest of the researchers evaluated (the famous peer evaluation) and the data generated by the scientometric analyzes are considered only as support material in the evaluation process. But due to the intensity and frequency with which the evaluation of academic work is carried out in Mexico, the majority of the opinion committees are not familiar with all the issues addressed by the researchers to be evaluated, and then the scientometric indicators become the norm that all researchers who want to be promoted or hired must meet. This, in turn, has generated new evaluation procedures to analyze the appeals of researchers who disagree with the results of their own evaluation. There is also an aberrant use of bibliometric indicators when they are used for exclusively administrative purposes, but I prefer not to dwell on it.

4. The INEGI of science
Like many other developing countries, Mexico does not have clear policies to encourage the systematic compilation of statistical databases on national science. The case of Mexican science is really pathetic and we can illustrate it with two recently published editorials in international circulation magazines that have referred to the state of science in our country. In the first case, published in the American Scientist (8) magazine, Mexican science is well argued because despite how modest our contribution is to the total number of articles

From research published worldwide, it turns out that among the 348 foreign members of the US National Academy of Sciences, seven are Mexican, a much larger number than researchers from Brazil, China or India. In contrast, in the second case, the editorial published in the journal Nature Medicine of September 5 (9) criticizes the performance of Mexican scientists. According to the editors of this magazine, although support for Mexican science is sufficient for the number of Mexican researchers and according to US standards, per capita production and the impact of Mexican science are much lower than those of American science. However, this editorial or the number of Nature Medicine corresponding to that date does not include data that support such a drastic conclusion. Unfortunately, there is no source of statistical information in our country that could refute the argument published in this magazine.

As I conclude this article, I hope to have presented enough arguments about the need to create, and preserve, the INEGI (National Institute of Statistics, Geography and Informatics) of Mexican science, or some instance that would allow us to have updated statistical data on the state that keeps Mexican science at all times. These data could be used by science administrators to carry out real planning exercises of our scientific activity, by researchers interested in studying the dynamics of our scientific activity, or by the higher education and research institutions themselves to provide feedback on their academic evaluation processes. Simply put, to encourage the correct use of Scientometrics.

[References]

1 I. Castro-Moreira, Science 301, 141 (2003); D.A. King, Nature 430, 311 (2004); M. Sánchez-Ayuso, reporte del CSIC, http://www.csic.es/hispano/preside/preside4/infor4.htm; J. Mervis, J. Kinoshita, Science 270, 1131 (1995).
2 CONACyT, Indicadores de actividades científicas y tecnológicas,
http://www.conacyt.mx/indicadores.htm.
3 Atlas de la Ciencia Mexicana, http://www.amc.unam.mx/atlas.htm.
4 F. Collazo-Reyes, M.E. Luna-Morales, J.M. Russell, Scientometrics 60, 131
(2004).
5 C. González-Brambila, F. Veloso, reporte de la U. Carnegie Mellon
(2005).
6 Véase, por ejemplo, P.O. Seglen, BMJ 314, 497 (1997); J.W. Grossman,
Notices AMS, 52, 35 (2005); A.F.J. Van Raan, Scientiometrics 62, 133 (2005); N.C. Lin, Y. Cheng, L. Lin, Scientiometrics 64, 101 (2005); J.A. Tallin, Basic Clinical Pharmacol. Toxicol. 97, 261 (2005); D. Adam, Avance y Perspectiva 21, 181 (2002).
7 M.H. Magri, A. Solari, Scientometrics 37, 35 (1996). 8 F.J. Ayala, Am. Scientist 93, 2 (2005).
9 Nature Medicine 11, 907 (2005).