Main Discussion Thread: Measuring Research Performance

This is the main discussion thread. If you haven't done so already, you need to logon with your GenPORT account (or register if you do not have an account) and join the discussion group.

1. Post your comments. New comments appear at the end of the page.

2. You need to refresh your browser window in order to see most recent posts by others.

3. Main discussion is scheduled for Thursday 7th of June 2018 from 13:00 to 15:00 Central European Time. However, posts are welcome after that date as well.

4. Please introduce yourself shortly in your first post.

Comments

Annecharlott's picture

The central question of the GEDII project is simple: what makes great (research) teams? The literature on what makes (research) teams effective is huge. GEDII focus more specifically at the relationship between gender diversity in research teams and their research performance. How do gender diversity (demographic, functional, and cognitive) in research team affect the quality and quantity of research outputs? In the project we hope to generate new insights how gender diversity helps to build great teams.

As part of the research conducted, the GEDII project carried out a survey across 16 European countries, capturing 1357 individual responses distributed over 159 R&D teams. The overall design of this research project is unique in that survey responses at the team level are combined with established performance measures, i.e. bibliometric publication counts retrieved from Web of Science.

We hope to meet you online at 1 pm for a fruitful discussion following the GEDII results but also around the wider topic of gender and research performance!
Ulf & Anne-Charlott

Ulf Sandström work as a researcher at the Orebro University, Gender Studies and the KTH Royal Institute of Technology, Department of Industrial Engineering and Management (INDEK). His research interests include:
• Research productivity and efficiency
• Structures of financing of research
• Cognitive bias in peer review
• The role of mobility in research
• Gender issues and excellence
• Interdisciplinary research.
• Nanometrics: options for a monitoring system
• Patents and Star Scientists: a Boost to the Company?
• Does excellence Programs Boost Scientific Quality?

Anne-Charlott Callerstig currently work as a researcher at the Centre for feminist social studies (CFS), Örebro University. She has a PhD in gender studies and a background within political science. Her research interests include gender and organisation, policy implementation and in particular in connection to equality policies (in its broadest sense). She has participated in several projects studying gender equality initiatives in academic organisations.

ulfsand's picture

Hi, my name is Ulf Sandström and I am responsible for the bibliometric analysis in the Gedii project. I’d like to start the discussion by pointing at some crucial issues that relates to gender and performance measurement.

Measuring and comparing research performance is quite a controversial issue today. That is especially true in the context of gender equality discussions. Let us for a moment ignore the needs of the Gedii project and raise the question in what way performance measuring is relevant for gender equality?

The answer is the following: For a long time, facts about research performances have not been disclosed in a relevant way. The main difference is size-independent versus size-dependent indicators. Field normalized citation score is one popular indicator but on the downside is the size-independence. Many have been satisfied with field normalized citation scores as they have been equal between men and women.

The resulting figures are very much expected as men and women do not constitute two populations that are on their own and publish articles on different sides. Instead, publications are in general done together with co-authors from both sexes, and as we know collaboration tends to grow quickly.

In my view, being content with this simple picture is unfortunate for the discussion on gender issues in research policy. There are more to add and with that more gender problematic issues comes to the surface. The productivity issue comes with the size-dependence indicators and when these are disclosed the discussion cannot disregard this dimension of performance.

Normalized citation scores (to the mean) are dangerous and might mislead in a situation with large differences in productivity between men and women. Normalisation is based on mean values but rely on skewed data and field normalisation do not give information about the volume of research achievements from the different sides of the gender border.

Therefore, volume is of paramount importance when discussing productivity. The Italians Abramo & D'Angelo (link here) have most clearly pointed at the need for supplementing bibliometric studies with a distinct volume and productivity concept. They should have credit for that.

My claim is that this is exactly what we do in our Survey Analysis and Performance Indicator Research Report (link here). Several innovative approaches and methods allow us to compare groups in different disciplines and we can do that with regard to volume plus citation performance. The basis for this was laid out in a paper by Sandström & Wold (link here). Peter van den Besselaar & Sandström took the analysis further in their paper Vicious circles (link here)

The results of applying the new methods to the teams that Gedii have analysed are interesting by themselves: It appears that groups that have a practice of inclusive gender diversity have higher productivity (measured as Field Adjusted Production). This is important as it might indicate that team science is one way to raise productivity among women.

Why is it important to raise female productivity? This is important because most financiers adhere to regulations or apply informal rules that prioritize those with high productivity. It is an important explanation as to why women continue to be discriminated against by research councils and other financing bodies.

Annecharlott's picture

Thank you Ulf for starting up the discussion! I can see that several persons are now online, and I would like to invite them to introduce themselves and send their comments. Let’s start our e-discussion! Welcome all to share your views and experiences!

alhumbert's picture

Thank you Ulf and Anne-Charlott for very informative summaries.

I am Anne Laure Humbert, and I have also been involved in the GEDII project. Together with my colleague Elisabeth Guenther, I have developed a Gender Diversity Index within research teams in STEM. This provides a measure that goes beyond simply counting heads, i.e. women's proportion within teams (more on this in the next online discussion). When putting this result together with the bibliometric analysis undertaken by Ulf, we see that there is indeed a gap in productivity (as measured by Field Adjusted measures) between women and men... but that this is offset within teams that achieve higher scores in the Gender Diversity Index.

The report is available here: https://www.gedii.eu/wp-content/uploads/D4.3_GenderPerformance_23MAR2018...

How can we make sense of these results? What implications for policy, researchers, Research Performing Organisations and Research Funding Organisations? What further analyses do you see as useful?

Annecharlott's picture

Thank you for joining us! You pose important questions that would be interesting to hear the participants views on - i.e. How can we make sense of these results? What implications for policy, researchers, Research Performing Organisations and Research Funding Organisations? What further analyses do you see as useful?

Annecharlott's picture

I believe myself that the project results indicate that gender equality measures are important to create more equal conditions for women and men to work and publish their research results and that it highlights that this is done within a context i.e. as part of a team (and also the wider organizational context). The results indicates that you cannot only see to the individual. Gender equality measures needed can be around more general equality issues such as working conditions, decision making, making visible implicit gendered norms concerning research and researchers etc but also more specific policies and discussions regarding publishing within teams, such as who are invited to participate, be first author etc

giulia.zacchia@uniroma1.it's picture

Hi, thanks for organising this important event I really think that is time to re-think research evaluation in order to create new responsible metrics that account for diversity.
My name is Giulia Zacchia, I'm a research fellow in economics in Sapienza University of Rome and I did some research about the impact of bibliometrics and research assessments mainly based on the identification of reaserch quality with citation (i.e. IF, H index etc.). Unfortunately in economics there has been an increased use of bibliometrics both for the academic hiring/promotions of the single researcher and for the public resources’ allocation to the universities. This mainly had two consequences:
• produce discrimination within academia against women and minorities;
• boost conformism: the diffusion of a single-minded faculty of academic economists prone to group thinking.
if you are interested in the topic, you can find some papers here - https://www.ineteconomics.org/research/experts/gzacchia.
Also during the G20 Global Solutions Summit hold in Berlin few days ago there has been a meeting for calling for changes to the evaluation of economic research to ensure that economic theory—and policy—is more rigorous, innovative, and in service to society and that academic institutions are more equal and equitarian (http://www.g20-insights.org/policy_briefs/research-evaluation-in-economi...)
I'm really interested in new proposals.

Annecharlott's picture

Thank you for your interesting input and the links! Do you have any specific thoughts on how to "re-think" evaluation in a way that that takes into account what your really interesting article with title: "How Academic Conformity Punishes Women—and Restricts the Diversity of Economic Ideas" suggests?

giulia.zacchia@uniroma1.it's picture

Sorry for the late replay, I tried to attach a picture, but here I can't so I send you a link where you can find a very draft scheme of how it could be designed a research evaluation of researchers that prioritises divisersity and not excellence
https://www.dropbox.com/s/mprn7g72o37csss/Diversity_a_proposal.pdf?dl=0
● In order to prevent corruption and nepotism, it is sufficient to single out the very worst performers with simple thresholds (eligibilty in my scheme)
● than for those who meet the minimum thresholds a second stage of evaluation based on peer review only, in two steps. In the first, all candidates must submit a sample of unpublished work (including not published as a working paper), that will be blindly evaluated. In the second step, evaluation will be based on candidates’ full CV, including teaching experience and the overall match with the needs of the hiring institution(s).

ulfsand's picture

In a project some years ago I took the about 1,000 most productive researchers in Sweden over a period of time and did an analysis of how they produced over different areas. I presented this in Rio ISSI conference 2009. The results was astonishing: the highly productive researchers did publish (per individual) all over the categories. I could restrict the analysis to 22 macro fields and a huge number of the 1,000 were active in several macro categories, not seldom more than 5 categories. So, in my view there is nothing that would be changed from today with what you suggest. This already the case. Maybe Economics is an exception?

joerg's picture

Giulia, your comments on economics reminds me of the work of Heather Sarson, how argues that single or co-authored papers affect with speed with which men or women achieve tenure. For single authored papers, women and men achieve tenure at about the same time. However, co-authored papers slows down women because in case of doubt, their male colleagues get credit for the work done. Would be interesting to see how this relates to our findings where a gender inclusive team actually turns out to be beneficial in terms of productivity.

Sarsons H. Recognition for Group Work: Gender Differences in Academia. American Economic Review: Papers and Proceedings. 2017;107 (5) :141-145.

agodfroy's picture

Quality and reliability of the data are crucial issues when assessing performance. Existing databases are not reliable for all disciplines and all languages. They over represent English speaking literature and material sciences.
As a result HSS and humanities, often published in national languages in many different journals and interdisciplinary works are not always correctly recorded in the existing data. It creates a huge bias which is detrimental for women who represent a large proportion of researchers in HSS and interdisciplinary fields.

Academically maintained data bases, with an EU standard, could be a solution to this challenge.

Projects databases would be very useful too for assessing access to funding and project activities. You could have proposals and funded projects in the database (useful to document success rate). Amount funded would be very interesting too. Same as roles in the project.

Finding some way of collecting data seemlessly is another big issue. Collecting a posteriori is always difficult, slow and not a,ways reliable. The best solution is to collect on the go on a user friendly platform.

Annecharlott's picture

Thank you for joining the discussion! You raise a very important topic - are the current databases reliable at all or to what to extent and what kind of biases do the create? Would be interesting to hear what the other participants have to say about this? Is an EU standard perhaps the solution? Also about project databases - sounds promising

ulfsand's picture

One illustration is that History is probably the largest subject category in the Web of Science, largest in the sense that this discipline has the higest number of journals indexed by the database. The fact that there are many more journals is not an argument here because most of these journals do not fulfil the requirements. Scientific journals are international and publish for an international audience of researchers active in the field. Peers are the only ones that are able to discuss and evaluate the propostions made by researchers in the field. And peers are found all over the world and they follow the international journals. 

agodfroy's picture

Dear Colleague
In what language do you publish? Concerning French speaking research in HSS, journals are not so well represented.
Some journals are very appreciated in national évaluations but are not present in international databases for various reasons. One was the business orientation of databases organized by big British or American companies. Because some top French scholars were very critical they have been reluctant to register some journals.

Elisa Guenther's picture

Hi, I am Elisabeth and I am part of the GEDII team.

I think one important aspect of this discussion is which kind of productivity is looked at. The output and relevance of universities can not only be measured in publications and the ranking of publications. The public tasks of universities and the public good they produce go beyond that - for instance in teaching. Also, writing books might be also important to convey the results of research, and perhaps make it accessible to a wider audience.
So, it is not only a question about collecting more data but at least as important: which kind of data is considered and how the different aspects are weighted.

Annecharlott's picture

Great to have you with us in the discussion! Different types of research outputs and how it is valued is an issue that has been raised in the wider debate. The gender implications of this is important to highlight - what is counted as an important output, why, by whom and to what consequences?

ulfsand's picture

Remember that in the team science report we focus on research and output from teams, education is maybe relevant in some sense but at the end of the day you have to decide what is it that goes on in research teams? I do not say that books shouldn't be counted, they are also in the Web of Science database nowadays, but going there is maybe not really needed as a book is almost always consisting of acutal papers or papers (chapters) that could have been published. If an author refrains from using his collegues as peers to have views and critique from potential readers before publication I would say that this author is very inefficient. Probably, that's not the way to do it. Or, do you have another perspective on this?

Elisa Guenther's picture

I wouldn't say that books are just papers that have not gone through peer reviews and therefore the result of inefficient authors, because this belittles this type of research output and only promotes a specific perspective on how academia could work.

I just wanted to point out that measuring something also means to simplify complex situations. It reproduces a specific perception of what research output could be and hence has its limits. Ther are also other forms of output, which might be important for academia as well.

sandraklatt's picture

My name is Sandra Klatt and I am also involved in the GEDII project. Many thanks for the e-discussion. As for further analyses, a more long-term research of teams seems to be interesting. How does the dynamic in teams change when e.g. the gender balance in teams is changing or the team grows together over time. How does this effect the performance of teams?

agodfroy's picture

I agree teams and collaborations across teams are a crucial issue

Annecharlott's picture

The long terms perspective is really interesting. One of the difficulties we have discussed within the project is how to determine what constitutes a team at all and the way that teams are a "moving target". A long term perspective might give some insights on this bigger issue as well!

Annecharlott's picture

There are now several topics opened up in the discussion, please feel free to continue to comment on these (note you can comment a post directly with using reply at the bottom of the post).

ulfsand's picture

What we know with a number of research data of all kinds is that women produce less papers than men, the relation is like this: when women produce 2 papers men produces 3. This is really problematic. In a research council, to be accontable to the tax payers, the panels have to take production and quality into account. Otherwise they would fail their assignment. Given that men and women have about the same quality a male with 15 papers is better than a female with 10 papers. So, we have to focus on this productivity gap and wonder why this is the case. In my view Besselaar and Sandström (2017) gave a good bit of the answer: there's a vicious circle which basically builds upon gender stereotypes which feeds into the funding process. The process give less money and less prestige, less responsibility and not the best positions to women. In order to change these processes we have to face facts and start a process of change. Too long have we thought that this is something that will change with time. This is not the case! One way would be to change the procedures of research councils and funding bodies, if possible, but also other structural reforms are needed. I'm very optimistic due to our results in the Gedii project. Team science - when teams are inclusive and open to gender diversity in different respects - can make a difference!

Annecharlott's picture

Thank you Ulf, I think this is really a good point! Several Swedish studies shows how implicit gender biases very concrete affect the funding process and that without any measures to control them - they do not change but rather tend to repeat them self and as you write lead to vicious circles! I repeat the link to your article here: https://www.forskningspolitik.se/files/dokument/journal.pone.0183301.pdf

joerg's picture

Thanks Ulf, that makes the problem much clearer to me: there are differences in productivity while quality is similar. To me, this is relatively easy to fix: some recruitment practices should not evaluate the total number of publications but only seek the last 5 (best) publications?!

I recall a paper by Arensbergen, et al 2012. “Gender Differences in Scientific Productivity: A Persisting Phenomenon?” Scientometrics 93 (3): 857–68. https://doi.org/10.1007/s11192-012-0712-y. which states that this gap in producitivty is disappear in younger generations. Is this an overall pattern?

ulfsand's picture

hope he can join the discussion. Myself, I do not believe that this is changing by itself, this is shown quite distinct in the Swedish data, the productivity gap is as large in the youngest generation as for the older ones. 

ulfsand's picture

your suggested practice is probably not applicable to funding processes with a number of applicants that have to be selected based on their total achivements. Or...?

joerg's picture

Hi everybody,

my name is Jörg Müller, part of the GEDII project as Coordinator. I would like to take advantage of this opportunity and pose a few questions based upon Ulf's comment regarding our findings.

The bibliometric data we compiled in our study is based upon the Web of Science database. As Anne-Sophie points out, this does not cover all scientific production. However, knowing its limitations, it still is a very reliable source to evaluate the impact of publications within the science system.

I think an important difference of our study and many existing bibliometric studies is the focus on “organizational” team. Whereas bibliometric studies examine co-authors as collaborators which often span many institutions, our teams are based in the organization where they work. They are mostly “real” research groups located in research performing organizations.

This suggests that the team level might be an important level that resides in-between the individual and organizational level which influences the overall research performance of the team itself but also of its members. I think we should study furthermore how this differs from co-authorship teams. This is one of the interesting lines of analysis we should pursue.

Annecharlott's picture

Great to have you with us in the discussion! You raise a really important point - to move beyond understanding differences in performance, or as I believe also more in general - existing gender inequalities in academia - simply from an individual perspective or, for that matter; to asses team performance not taking into account existing structural inequalities that affect individual performance within a team/organisational context.

Ella Ghosh's picture

First of all, the question of the effect of gender diversity in research teams and their research performance is of great interest to us in the KIF Committee (The NorwegianCommittee for Gender Balance and Diversity in Research). Our mandate is divided - gender balance and the participation of ethnic minorities in research staff, and gender and diversity perspectives in the content of research. Gender balance and gender perspectives are separate strands, but some health studies (from the US) indicate that the inclusion of gender perspectives are affected by the gender of the the leading researcher. We are often confronted with the question of whether science is science irrespective of gender or ethic origin or whether who you are affects what research you produces. Women produce research in fields where gender perspectives are invisible, men can just as well introduce gender perspectives into their research - most migration studies are carried out by non migrants. We assert that gender diversity - and inclusive teams - lead to improved quality. An obvious follow up is - where is the research to follow up this claim? There are many research findings regarding the diversity dividend, but all research findings within the field of academia are a welcome supplement.

I was interested to see the GEDII report enclosed here - the findings are of interest for us in our work. Thank you, Anne Laure Humbert

Some of you may be interested to know that a Norwegian Study is being planned about the impact of the National Publishing Indicator on gender differences in publishing patterns.The Committee for Gender Balance in Research and The Norwegian National Publishing Committee are just in the first stages of planning this study. The purpose of introducing the publishing indicator was to stimulate higher-quality research, where scientific contributions are published in Level 1 or Level 2 channels. Researchers’ publishing activity has implications for the financial transfers to the institutions covered by the system. The effects of the indicator have been evaluated, but whether it has worked differently for women and men has not yet been analyzed.

There is a large body of research literature documenting gender differences in publishing activity. The focus of this project is whether the introduction of the National Publishing Indicator has enhanced or reduced gender differences in publishing patterns. Further, gender differences in publishing behavior have changed during the time that the publishing indicator has been active (from 2006 onwards for the UH sector and 2011 for the institute sector).
In addition to analyzing the main patterns of publishing behavior in a gender perspective, the project will also analyze gender differences in light of differences between disciplines, disciplines, institutional types and sectoral differences (UH and institute sectors). If possible, the study will also take into account publishing patterns in light of gender and age.

Here is a link to an interesting article on this subject from Danish researcher Mathias Wullum Nielsen. Mathias Wullum Nielsen (2017) Gender consequences of a national
performance-based funding model: new pieces in an old puzzle, Studies in Higher Education, 42:6,

To link to this article: https://doi.org/10.1080/03075079.2015.1075197

Annecharlott's picture

Thank you for joining us in the discussion and also sharing the Norwegian experience and the news on the new study which sounds really interesting! You raise a very important question - if diverse team also produce better/more innovative research outputs (qualitatively) than more homogeneous teams?

alhumbert's picture

You stated: "We assert that gender diversity - and inclusive teams - lead to improved quality. An obvious follow up is - where is the research to follow up this claim?"

Very interesting! It is true that so far in the GEDII project, we have only been considering the link to performance e.g. with FAP (Field Adjusted Productivity). It would be very interesting to try and replicate the models we used with another outcome such as quality. Do suitable indicators exist for this? Or how could we calculate them?

ulfsand's picture

one proxy for quality has been tested in the Gedii project and that is what is called the Percentile Model (using citations in percentile groups). What we find is that there's no difference there which can be interpreted in this way: as there's no difference between men and women in overall quality of papers (citation-wise) we shouldn't expect to find that in our investigation. It might be that I miss something here, so please think it over again!

joerg's picture

Hi Ella,

very exciting to hear about this forthcoming project!

As you rightly say, the evidence on the impact of gender diversity on research performance is divided. Some publications show there is a positive impact, others show a null effect or negative. We have compiled some of this evidence in our conceptual framework. In part this non-consistent finding is due to the simplicity of the question: gender diversity in most cases is conceptualized as “counting heads”, i.e. counting the number of men and women co-authors! Research performance of course depends on many other factors.

With the Gender Diversity Index we have tried to make this question a bit more complex and offer to the research community a measure that is more sophisticated for assessing gender diversity in teams, incorporating 7 pillars (tenure, education, care resp., contract, seniority, age, marital status). We are very curious to see how future research can use this tool. Maybe there is some room for considering the GDI in your forthcoming study…

alhumbert's picture

And of course, we will be discussing the Gender Diversity Index in greater detail in our next online discussion on 12 June at 11 am CET...

In the meantime, here is a link to the report: https://www.gedii.eu/wp-content/uploads/D3.2_GenderDiversityIndex_28FEB2...

arroyo_lidia's picture

Dear all,

I am Lidia Arroyo, researcher in the Gender and ICT Group (IN3-UOC) and community manager of GenPORT. It is a pleasure to know how the knowledge is advacing in the measurement of the the impact of gender equality and diversity in research performance. The results of the GEDII project and all the ideas and resources that you are sharing today here are evidence of this advancement. I encourage you to upload these resources on GenPORT,  some of them are already uploaed, such as this interesting article introduced by Ulf  "Vicious circles of gender bias, lower positions, and lower performance: Gender differences in scholarly productivity and impact".   , but all of them are key and very useful for the people who want to know more about the topic.

Annecharlott's picture

Thank you for reminding us of this and also doing such a good job with the GenPORT! It is a fantastic resource and a good way to share and discuss results!

njane's picture

Dear all,

My name is Nuria Jane and I am in HR unit at IBEC with collaboration with Carol Mari in equality and diversity aspects in our institution, IBEC. I am glad to find a lot of resources, results and reviews from all of you. I’ve been check a few data and my main worries is how to monitor all these indicators? How can we begin to involve researchers to this (whit data is perfect of course)? On the other hand, do you have any crucial indicator for researchers in their performance (postdocs or group leaders)?

Again, thanks in advance for all of your useful contributions.

Annecharlott's picture

Thank you for joining our discussion and posing some practical questions on indicators and how to monitor them.

joerg's picture

Hello Núria,

as part of our work during the survey, Ulf has created a bibliographic profile for participating groups. In our deliverable there is an example "group performance" profile in Annex IV while Chapter 2 gives a good overivew of these different indicators. This might be a good starting point. I'm not sure there are any specific indicators at the PhD level or Group Leader level?

njane's picture

Thanks! I will check it

Annecharlott's picture

Our time is up and we would like to thank all of you for an interesting discussion that we hope will continue at different forums! All the best from GEDII and Ulf & Anne-Charlott/Örebro University!

Please visit our website at https://www.gedii.eu for more information about the project and do not miss the two up-coming discussions:

e-Discussion on Gender Diversity Index (June 12th, 2018 11:00-13:00 CET) http://www.genderportal.eu/group/e-discussion-gender-diversity-index
The GEDII team has developed a composite indicator that captures gendered processes
within research teams: the Gender Diversity Index. During this online discussion we aim to
elaborate on its potential to become a useful resource for researchers, equality officers and
policy-makers. What are the next steps?
Online discussion coordinated by Anne Laure Humbert and
Elisabeth Günter

e-Discussion on Sensor-based data for gender research (June 20th, 2018 13:00-15:00) http://www.genderportal.eu/group/e-discussion-sensor-based-data-gender-r...
Research has revealed gender differences in non-verbal behavior within interactions and
conversational styles. Status and gender matter during interactions. Sensor technology and
computational approaches are rapidly evolving, potentially allowing for the detection and
measurement of these behavioral cues. This online discussion brings together researchers
working with sensor-based data in order to discuss current possibilities and challenges for
the social sciences in general and gender research in particular.
Online discussion coordinated by Jörg Müller

Elisa Guenther's picture

Dear all, a brief reminder:
there is another discussion next week about the Gender Diversity Index on Tuesday, 12.06. at 11:00 CET (http://www.genderportal.eu/group/e-discussion-gender-diversity-index). Perhaps some of you would like to join as well.

ulfsand's picture

from Ulf,

and have a good summer vacation!

Add new comment

Active Threads:

2018-Jun-06

4 months 1 week ago
Posted by: joerg
This is the main discussion thread. If you haven't done so already, you need to logon with your GenPORT account (or register if you do not have an account) and join the discussion group.1. Post your comments. New comments appear at the end of the page.2. You need to refresh your browser window in order to see most recent posts by others.3. Main discussion is scheduled for Thursday 7th of June...
Comments: 46