Main discussion thread: using sensors in social science research

This is the main discussion thread. If you haven't done so already, you need to logon with your GenPORT account (or register if you do not have an account) and join the discussion group.

1. Post your comments. New comments appear at the end of the page.

2. You need to refresh your browser window in order to see most recent posts by others.

3. Main discussion is scheduled for Wednesday 20th of June 2018 from 13:00 to 15:00 Central European Time. However, posts are welcome after that date as well.

4. Please introduce yourself shortly in your first post.

Comments

joerg's picture

Hi and welcome to this e-discussion on using sensors in social science research. My name is Jörg Müller and I have been coordinating the GEDII project for the past three years. GEDII is a H2020 project with the objective to examine the impact of gender diversity in R&D teams.

As part of this wider project, we have used Sociometric Badges to study gendered team dynamics. The report of our findings can be accessed here, although we are still busy in analyzing the data and bringing it into publishable form. I guess it comes as no surprise to most of you, that using the badges has not been a simple and straight forward process. Drawing upon our experience, there are two issues that I would like to highlight to get the discussion started.

First, similar to other researchers we have found basic reliability issues with the badges – especially in relation to the turn-taking measures. This is very unfortunate for our research objectives, where turn-taking is of special relevance for gendered interaction dynamics and leadership emergence. We have conducted a controlled experiment to assess the precision and reliability of the badges with very disappointing results. I think it would be interesting to hear how others have mitigated especially microphone based measurement issues with the badges.

Second, I think once we get a better grip on the basic reliability issues, the next stage certainly is to use these type of sensors in theory-related research. We did not manage to get to this level (yet), but it definitely would be interesting to hear others approaches to this. Quite a few studies have been done in relation to “creativity” - but did the badges really contribute to any new insights or understanding related to “creativity”? What are the most proximate questions to explore? We have used badges in relation to gender issues, but I'm sure there are many more topics out there?

JohnParker's picture

Hey!

Reliability was also a major issue in the studies we conducted with the sensors, from them simply not working to them sometimes not capturing interactions that were clearly visible to observing researchers. For this reason we found it invaluable to have ethnographers present and taking careful notes on social interaction in real time to provide comparative data. I'm skeptical of sensors studies that don't take this strategy. We also used relied on relatively straightforward badge measures rather than indicies, wherein issues of data validity might multiply across metrics.

In terms of creativity, the badges of course don't measure the outcome 'creativity,' but rather give us much more precise measures of what past theory and evidence suggests are the key determinants of creative social interactions - intense face-to-face, high network density mixed with diverse forms of cultural capital as measured by unique contributions to a conversation, balanced social interactions that enable contributions from everyone, etc.

JP

joerg's picture

Hi Peter and John, welcome to the discussion!

One the one hand we probably could say that the reliability issues will get resolved since these are technical problems that can be fixed relatively “easy”. However, my concerns is, what does this mean for science and research? How can we make certain that our studies can be reproduced? There is certainly a need for shared protocols in this sense. Some of the publications using sensors go into this direction, but I'm not sure that we can keep up. Peter for example is using a new technology – think about all the different type of smart-watches and sensors out there. Working on shared standards for reporting this type of research is a necessity.

JohnParker's picture

I totally agree. We need standard measures to build a field, and at some point the tech has to solidify, at least around a core set of measures captured in the same way across devices. Also, looked at your report - great work, and great tests of their reliability!

joerg's picture

Thanks John! I think open hardware will be helpful in this respect as well. I hope Oren can join the discussion and introduce the current state of the Rhythm Badges.

Peter Gloor's picture

I am very sorry to hear about the badge reliabilty issues. We have had similar experiences, which is why are are now working with smartwatches instead, see 

www.happimeter.org

and 

https://arxiv.org/abs/1711.06134

alhumbert's picture

Dear Peter

Can you tell us more about the reliability issues you have had with the badges measuring proximity, face to face etc? 

Anne

Peter Gloor's picture

well, we just found that the readings were somewhat inconsistent, so we had to aggregate actors on the day or even week or month level

joerg's picture

The happimeter is very interesting. However, I would be especially interested in new possibilities for turn-taking and speech detection since this is especially important from a gender perspective. I'm pretty much looking forward to the contributions of Jiaxing who has been working on this issue specifically. Anybody else knows other developments in this direction?

frances_unitn's picture

hallo to everybody, i just joined the conversation and I am interested in the topic.

I had a look at the Gedii study and the results are interesting and must have involved a lot of work!

my observations are mostly based on theoretical aspects in the measurement of turn taking. first, i wanted to know if the Geddi group had a reference with research works using measurementes in conversational analysis.

I know there are some applications of social network to standard F2F conversations in the area of social psychology but those are 'old technology research' and studies were performed without sensors measurement. Still they  could be of help for developing new indexes.

have you considered that area of research?

joerg's picture

Hi Francesca,

we haven't been in contact with research groups working on this topic currently, no. But we have consulted (some of) the relevant literature I hope. The truth is that we started with the Badges quite naively, thinking that turn-taking measures would actually correspond to what is going on in "reality". However, the Chaffin paper alerted us that this might not be the case and that's when we started with our own experiments.

I also have to say that I would like to apply and test machine learning classifiers to the data instead of the manufacturer algorithm for identifying turns. John or Peter did you try this? Does this make sense?

frances_unitn's picture

hallo Joerg, i finally joined you.. sorry I was late.

the idea to use machine learning seems good. if you could adjust your analysis and construct a sequencing of the conversations this would allow you to have a contextualized perspective of the results.

in terms of instruments the use of smartwatches - for research purpose - seems also good. they might have a lower intrusivity on everyday interactions

 

 

 

JohnParker's picture

We did get a grad student to use some machine learning techniques to predict certain patterns in the data but it didn't really go anywhere. Likely its possible, but the person we had didn't have the skills or the data were insufficient.

Peter Gloor's picture

yes Joerg, shared protocals are important because the metrics between badges, and between Pebble smartwatches and Andorid watches are not directly comparable. right now our consolidation between Pebble and Android watch happens on the server through machine learning

Peter Gloor's picture

... is very hardware dependent, or instance the sociometic badges have to microphones to see if the wearer or somebody else speaks, Sandy Pentland's team also has new badges where it is done diferently (I do not know how), and we do it very simply by looking who is close, and whose watch reports the loudest speech.

joerg's picture

I didn't realize that the smartwatch works for speech as well? Does it work? I guess this is even harder than using the badges which are closer to the mouth? Does the machine learning make the difference?

Peter Gloor's picture

Yes, Joerg, this is a good idea, it will be a bit cumbersome to collect the training data, but for a controlled environment this should give more accurate results.

frances_unitn's picture

I also find useful from the theoretical perspective research works in the ethnomedology area. they can be rather complex and too detailed but could give nice clues on how to interpret turn taking in conversations, how to construct measurement indexes that might have a social - and in this case a gender specific - relevance.

sorry, I forgot to introduce myself: francesca odella from trento university, I have worked on the issue from theoretical perspective as I am not a technical expert on sensors but they interest me from the point of view of social and interactional impact. 

JohnParker's picture

Hi,

You are totally correct in your thinking. These badges represent the chance to test out in fine detail MANY theories of social interaction that have relied thus far on ethnographic and other forms of observational data collection. It's akin to bringing a microscope into the social sciences, allowing us to see and measure things with precision that we never could before. We address these issues in our paper, as well as cautions about the tech: http://journals.sagepub.com/doi/abs/10.1177/0049124118769091

 

frances_unitn's picture

thanks for pointing me to the new study, John. 

in the ethnographic and ethnometodology perspectives conversations were interpreted as dynamic dimensions of social interactions. in this perspective turn-taking could signify both role reproduction (think about gender dominance) as well as a mean to attach social relavance to otherwise just practical information exchange ( all the experiments with chatting bots are very interesting).

As for the instruments, I wonder if using smartwatches instead of sensors in badges could provide data with more situational information ex. collective discussion. thanks for the references on the use of these watches in emprical research.

 

alhumbert's picture

From current and previous research, the question of the validity of the data seems clearly under question. 

My understanding of reliability is that it is about difficulties of measuring something with precision and/or low replicability over time. 

What about validity? Are sensors telling us something about interactions, even if ever so imperfectly?

And what implications for data analysis? Can we still say something meaningful using these data while waiting for better sensors?

Anne

JohnParker's picture

Hi,

From our studies, which can be found here (http://journals.sagepub.com/doi/abs/10.1177/0049124118769091), we found issues with both reliability and validity in that the sensors would not capture some interactions that were clearly happening, and differed in their measurement capabilites over time and across badges.

Anonymous's picture

yes, by aggregating the data over extended periods of team (as measurement errors might cancel out), which will lead to less granular insights, but they will still be interesting

Anonymous's picture

I am joining the conversation. Any publications to have an idea of the precise methodologies used and the kind of outcomes ? 

joerg's picture

Hello, could you please introduce yourself so we know who you are? Some of the publications have been mentioned already on this page but you can also take a look at our report which cites most of the previous research using Sociometric Badges. The good news is that we have a sort of standard with the Sociometric Badges. Most of the research in this area has used the badges, so there is some sort of basic comparability which is already very good.

Peter Gloor's picture

... and kicked me out :-(

joerg's picture

I like the metaphor of the microscope! What I'm really looking forward to is a different type of analysis. Peter mentioned to aggregate data in time but I want to suggest also to take advantage of the temporal dimensions inherent in this data. We have started to use the interaction data with the Relational Event Model. I really requires a change in mindset to think in time about social phenomena (at least in research). With the sensors there is finally a technology available to monitor interaction in time and start analyzing their temporal signatures. I find this very exciting!

Anonymous's picture

You are right Joerg, the temporal dimension brings in a new level of granularity, the main problem is indeed accuracy. We have been doing it since 2007 with the badges, see Peer-to-Peer Communication Patterns. Proc. AMCIS Americas Conference on Information Systems, Keystone, Colorado, Aug. 9-12, 2007

http://www.ickn.org/documents/gloor_amcis.pdf

Peter Gloor's picture

:-)

joerg's picture

Did you or anybody else try out the Relational Event Model? In our report we started with that and want to pursue this line of analysis. I think this is an active area of development (e.g. there is another approach developed in ETH Zurich called DyNAM) which allows to analyze these temporal patterns also in relation to covariates that might be important.

Especiallly for turn-taking, the REM is very sophisticated in incorporating Gibson's Participation Shifts.

joerg's picture

seems like Peter is not the only person with problems. The following post is by Jiaxing Shen, who can't post:

 

I (Jiaxing Shen) have visited MIT Media Lab for 9 months recently. My main focus is to analyze the badge data. We (with Oren Lederman, a Phd in Media Lab) also come with new voice activity detection method instead of using thresholds. We think the proposed method has latent benefits for improving the reliability of badges. We will also release the code once the work is published. 

Besides, Oren keeps improving the performance of badge. Please keep an eye on the update. You are welcomed to contact him for collaboration. The only concern is that the resources are currently limited. Therefore, it might be difficult to satisfy all collaborations. 

best,

Jiaxing 
 
frances_unitn's picture

hallo jorg, sorry to interrupt with a question not relared to the topic.

I wanted to know if it possible to receive a summary or transcription of this webconversation. many have referred to papers and articles that are very interesting and some issues require more thinking. for example Anna issues are from my point of view very relevant as they question the epistemological relevance of studies using sensors and similar pervasive istruments in social interactions...

 

ps. I am sorry for the technical problems. have you tried to re-enter the group with another account?

joerg's picture

There is not going to be an "official" transcript but what you can do is save this very page to your desktop. In any case, it will remain online and visible for the foreseeable future.

But I agree, it would be nice to continue this discussion between very different researchers, maybe in another format. There are many people that voiced an interest in this e-discussion but could not logon during these two hours. For example there is a very large study going on in Denmark at the Copenhagen Center for Social Data Science using sensors together with ethnographic observations. There are some publications in Big Data & Society coming out of this project you might have seen.

 

Ella Ghosh's picture

Dear colleagues,  I am following the discussion from the sidelines - this is definitely not my field! But it is till deeply interesting, because the use of technology to improve perception of human interaction seems to be evolving fast.

I am a senior adviser in the Norwegian Committee for Gender Balance and Diversity in research, i.e. a practitioner. I have, in the past, worked with methods to document discrimination, and have worked with situation testing and national/European discussions on documenting inequality and discrimination through statistics, qualitative and quantitative methods. In my present job I try to disseminate knowledge on research and available data when discussing equality in higher education.

I recently was asked about what methods exist to measure gender differences in participation in meetings, and referred to the Swedish Research Councils observers who used the app "Time to talk". I understand participants in this discussion are using badges or watches to register speech and interaction - possibly also the proximity of others. I understand your methods of measurement are mainly for research. Do you know of less sophisticated methods available for institutions that want to make sure men and women are given equal speaking time, i.e. monitoring, not research? I can imagine resistance to monitoring on ethical grounds, but some people working with appointment boards are considering such methods. See this link if you are not familiar with this example I referred to from Sweden http://kifinfo.no/en/2017/09/thanks-observers-committees-gave-more-money...

joerg's picture

Hi Ella, thanks for joining! There is one example of a simple web-based application https://github.com/cathydeng/are-men-talking-too-much that I know of. The thing is that this should be automatic. The technology is there, it's called speaker diarization but we don't have a good, simple app we could use (at least I don't know it)...

joerg's picture

Thanks everobody for joining and such a lively dicussion. The thread will remain open for future posts. I hope we can continue the discussion in the future in some other format!

Thanks again for all your valuable contributions :)

frances_unitn's picture

thanks to Jorg for the invitation and thanks to all for the conversation.

all very intersting and updadated topics including the suggestions from Ella (I have seen different strategies to keep turn taking in public discussions but never actually reflect on them).

so even if I do not directly work on sensors if there were another session please do keep me posted about. sorry for those who could not partecipate, it is a pity that the platform had some tecnical problems.

 

 

 

 

 

Add new comment

Active Threads:

2018-Jun-20

3 months 1 day ago
Posted by: joerg
This is the main discussion thread. If you haven't done so already, you need to logon with your GenPORT account (or register if you do not have an account) and join the discussion group.1. Post your comments. New comments appear at the end of the page.2. You need to refresh your browser window in order to see most recent posts by others.3. Main discussion is scheduled for Wednesday 20th of...
Comments: 37