Peer Review


#1

Originally published at: https://www.headshotspodcast.com/peer-review/

#53: Josué and Kelli talk about the academic concept of peer review in games research and gaming media in general. Is a YouTube essay any more or less significant to games research than a journal article? And what would academic peer review of YouTube videos even look like?

Join us on November 3rd in raising money for sick and injured kids. – geektherapy.com/extralife

Questions? Comments? Discuss this episode on the GT Forum.

Notes:

Visit us at www.HeadshotsPodcast.com.

Follow us on Twitter: @HeadshotsCast | @KelliNDunlap | @JosueACardona

Headshots is part of the Geek Therapy Podcast Network.

Geek Therapy Discord server – www.GeekTherapy.com/discord

Geek Therapy on Twitch - www.twitch.tv/geektherapy

Geek Therapy Patreon - https://www.patreon.com/GeekTherapy


#2

I listened to the episode on peer review the other day and felt frustrated through big portions of it because there were some critical points about peer review and research articles not stated. For reference, I’m a PhD candidate at a US university in computer science studying robotics, and a Lecturer in my department as well. I’ve been working in robotics research for the last 6-7 years.

  1. Research articles are written for other researchers, not for the general public.

A researcher’s job is not to explain things to the general public. Their job is to produce new knowledge in their field. They do this by developing new algorithms, ideas, math, etc related to their field, and then writing up the results. Generally academic papers in STEM fields have two goals: 1) inform other people in the field about the new work, and 2) ensure that other researchers can re-produce the results. That second goal means the audience of these articles are other experts in the fields. Communicating technically to other experts in your field is going to be much different than communicating with the general public. In order to ensure other people can re-produce your results, you have to speak technically.

At some point, one host expressed frustration at how they were trained to use jargon during their doctoral training. This is part of doctoral training because jargon is used to communicate with other experts in your field. Learning how to use jargon is essentially learning how to communicate technically about your field. When I write papers and talk to other roboticists, I use jargon all the time. I use jargon so that my message is communicated as clear as possible for other roboticists. However, my lab has visitors all the time that range from high schoolers to business executives. The visitors don’t care about the details of my work - they are only interested in the high-level ideas so there is no need to speak technically. For instance, in robotics the technical term for a robot arm with a hand attached is a ‘manipulator’. So when I write papers, I say manipulator when I’m referring to a manipulator. However, when I talk to visitors I just say arm. For roboticists, the distinction between a robot arm and a robot manipulator is significant. For non-roboticists, the distinction is usually not significant so I just say arm to make things simple.

At another point, the hosts said that professors are rewarded for becoming more inaccessible. While this is technically true, it frames professors and academia is an unfair way. The vast majority of professors are researchers first and foremost. If you go read up on the academic industry, then you will quickly learn that a professor’s job responsibilities are balanced between research, teaching, and service work. Some schools weight each category differently, but nearly all schools still hold research above the other two. Professors gain tenure by doing good research. Good research is done by writing lots of good papers, and good papers are full of jargon. The jargon isn’t there to be inaccessible. The jargon exists to communicate technically with other researchers in their field. A massive portion of a professor’s job is not to maximize how many people can understand their work - it is to maximize how many experts in their field can understand and use their research. Doing that means they have to use jargon. And if they do good research, then they are rewarded. So they are not rewarded for becoming more inaccessible. They are rewarded for being good at their job.

  1. Research papers can be submitted more than once.

A paper being rejected does not mean that paper or work can never be published anywhere. When a paper is reviewed, the output of the review is 1) a decision about acceptance and 2) comments from each reviewer about the paper. The comments state what the reviewer thinks the paper is about (summarizing it), whether or not it is novel research, any shortcomings of the work, and how the paper can be improved. Common reasons to reject a paper are:

a) It’s not actually new.
b) It needs more experiments.
c) The experiments are not convincing.

After a paper is rejected, the author(s) can modify the paper to make it better based on the reviews they received, and then re-submit it to some other conference, journal, or sometimes even the same journal. Then the review process is repeated. After modifying the paper based on previous reviews, the paper is hopefully better and it may be accepted on a second submission. If so, then great. If not, then a new round of reviews are given to the authors and they can continue to revise and improve the paper. It is very rare for a paper to be accepted on the first submission, and it is not uncommon to go through 3 or more reviews before a paper is finally accepted somewhere. When a paper is rejected, journals and conferences don’t say “this is bad and you should feel bad”. They say “this is not good enough right now and here is how to make it better”. Sometimes papers are pretty bad and the reviews can be harsh, but the reviews still point out all the things that need to change about the paper for it to be published.

In theory, peer review doesn’t sound that great due to only a few people reviewing a submission each time and your paper’s publication depends on only a few people. However, in practice it works out pretty well. Each rejection and revision makes a paper better. If you get rejected, then you just revise the paper and submit again. If you’ve done good work, then your paper will eventually get through. Youtube and other social media platforms, on the other hand, sound amazing in theory - they’re platforms where anyone can post information and the audience can decide on the validity of it through comments, likes, etc. However, in practice we’ve seen that it has led to the rise of fake news, echo chambers for divisive rhetoric, and even terrorist recruitment.

I point this out because almost any time I hear this argument about how peer review is too gate-keepery, people only talk about the theory. And in theory, peer review does sound kind of gate-keeperish and social media sounds like a utopian invention. However, no one ever mentions what the practice of these things are. In practice, peer review is a pretty good system (though not perfect) and social media can be pretty scary. The peer review system is a well-oiled machine that publishes thousands of scientific papers each year and the authors are senior researchers, first year graduate students, and everyone in between. It is certainly not perfect, as Katie pointed out numerous times, but I definitely think it is a force for good.

Please keep in mind that I only say all of this because I want the podcast to be the best it can be. I’m actually interested in transitioning into game-based education research after I graduate and I’m broadly interested in any game-based research, such as games for mental health. I really wanted to point all of this out because I felt like the peer review process and research training process were mischaracterized during this episode by both hosts.

The last point I want to make is that formal research (scientific journals, conferences, etc.) and informal publication platforms (Youtube, Twitter, etc.) can co-exist. It doesn’t have to be one or the other, and one doesn’t have to be better than the other. Communicating really technical ideas to the general public is a different skill than being a researcher. They are similar, but different. There are some amazing people on Youtube that run math and science education channels (Welch Labs, Minute Physics, etc) that break down incredibly complex topics for the general public to understand, but that job is quite different than being a researcher. Similarly, there are some amazing people developing new medicine and social theories, but that job is quite different than making informative and digestible Youtube videos. The two types of platforms can collaborate and work together. I think a very interesting conversation can be had about how the types of platforms support one another.


#3

Thanks for posting in the forums, it seems I might have skipped this episode cause I didn’t remember hearing it. Reading your comment really got me to engage with the episode in a different way. There’s a lot in which I agree and disagree with you. But your reply was so extensive and well thought out that I thought It merited being acknowledged and commented on. For reference I’m a 4th year Medical Student, I’m one test away from an MD.First of all, I do agree with you that a researcher’s job is primarily to contribute to a body of science that has some very complex language on the basis of need. What I interpreted from Josue/Kelli in the podcast is that in an ideal world Researchers would also take on the role of educators for the general public, especially because there is a lot of misinformation, from both people outside of the professional field as well as reporters who oversimplify study findings. (i.e. X research shows that protein x is expressed in high amounts in people who have cancer and people who like kittens. Headline: Research shows Kitten cause cancer). In fields like mental health and medicine specifically it’s a lot more important because research implies possible life saving applications and clinicians are taught early on that public education is as much their responsibility as research (From an epidemiologic standpoint). I think it’s an ethical distinction to be made between different fields of study.I agree that the more specialized your field of study becomes the more jargon you have to master. A radiologist has to constantly talk in terms of T1 or T2 weighted MRI Scans. (Something that probably sounds like gibberish to most people) and when applied to research that Jargon is necessary to actually simplify the language of research. Something I worry about (I don’t know if this was what Kelli was thinking about) is that In fields with clinical relevance, a lot of researchers actually double as physicians and they carry that jargon with them as they talk to their patients. In a clinical setting this can be very problematic, and it is in part because institutions do not stress much on educating clinicians on how to distill complex ideas into easily digestible ideas for patients.
As you mentioned, common reasons to reject a paper are:
a) It’s not actually new.
b) It needs more experiments.
c) The experiments are not convincing.

But I think it lacks one of the most vital reasons why some papers are rejected. Because it does not fit well with the journal’s views, perspectives or goals. I could write an article on how Videogame assisted therapy helps kids work through trauma but I would likely be rejected by a lot of publications because they do not view videogames as a profitable factor that would contribute to the journal’s set goals. Perhaps the board members find such research childish or inappropriate. Much like the legendary supervisor that inspired Josué to start Geek Therapy denied the use of videogames as a suitable therapeutic alternative. Similarly many papers on religiously sensitive forms of therapy are rejected on the grounds of the journal being preferably secularized from religious ideology. I think this might also be something where robotics and medicine/psychology diverge because sometimes institutional review boards can have. A lot of political/philosophical biases that are not accounted for. For years transgender medicine was not a researchable subject because review boards did not deem it a respectable subject.

We are taught that one of the most important parts about publishing research is to find a journal where it fits in. Something as fringe as videogame use in therapy is unlikely to find many places where they can be published outside of Videogame centric journals only read by professionals actually interested in that type of research, creating a bubble that doesn’t really extend the conversation throughout the field.

In truth I’m with you 100% on the value of peer reviewed research. Not too long ago I think it was Elon Musk that suggested there be a website where articles could be peer reviewed by the internet. And that idea chilled me to the bone. If you spend one day in Twitter you will find Thousands of people who agree on rather effed up or just blatantly incorrect things and that continue to validate it. For doctors, the anti-Vaxxer movement is a great example. It has been validated by sooooo many people, none of which went through a successfully peer reviewed process.

I think public education is important and I also believe there are instances where peer reviewed science is not appropiate. In Medicine we have Case Studies. These aren’t peer reviewed but they can be REALLY REALLY REALLY useful and they are glorified versions of some things you might see in youtube videos. I think one of the most important things in this discussion is to define validity. Peer-reviewed research is valid in the sense that it is reproducible and standardizeable. Take for example research that finds a medication that helps treat depression. If the peer reviewed research is successful, I can feel safe as a physician to prescribe this medication to my patients because I know when and under which circumstances the studies have shown that such an intervention would result in betterment of depressive symptoms. On the other hand sometimes there are experiences that cannot be easily standardized and vary from a case to case basis. If playing Journey with a kid with social issues helped him better socialize with peers that doesn’t mean that I can consistently prescribe Journey to every kid that presents with social issues. In part I have to be culturally sensitive to the background of the kid and determine subjectively if that sort of intervention could be repeated. In a institutional review board you would likely be rejected because your results are not repeatable or standardize. However that does not mean that publishing that case study would not be a benefit to the clinicians that learn choose to study that experience. I believe this is the sense in which both formats of publishing research are both extremely valid and necessary. Making the clear distinction that peer review is needed to make something reproducible. A youtube like or agree does not count as a peer review because the distinction is that I can like a youtube video because it agrees with my views even if it is not based on evidence. A peer review implies that the professional will return your paper and tell you exactly why your research could be flawed taking into consideration the current knowledge base of the fields. It implies that the review board is always looking at your paper critically and will not readily accept it without a thorough vetting. Because again, peer review means standardizeable and reproducible and so the consequences are likely large scale. Another thing they addressed was how the only way to peer review something is for it to be written in journal format. Perhaps people who dominate a video or spoken format over a written format would have done research with sound science and procedures that would otherwise find it’s way to a journal if the expression of their ideas was done is a written medium. I understand that procedure is important for standardizability, but it is none the less restrictive of possible valid contributions.

Again thanks for posting and looking forward to how this topic evolves in discussion.


#4

Wow thanks a lot for the thorough response :slight_smile:

“For reference I’m a 4th year Medical Student, I’m one test away from an MD”

Good job and good luck!

“in an ideal world Researchers would also take on the role of educators for the general public”

I think it’s fantastic when researchers take on this role, but not necessary. I think what I see from most people is that they expect this to be the case so much that now most people believe and expect it. That said, I can see how the spread of misinformation can make this expectation grow.

“a lot of researchers actually double as physicians”

Ah, this is something that had not crossed my mind. I can definitely see why this would be a problem. I think the podcast was more focused on the topic of general public education, but nonetheless this point is very important.

“But I think it lacks one of the most vital reasons why some papers are rejected. Because it does not fit well with the journal’s views, perspectives or goals.”

I think this may be a difference between fields. In my field we have around 3-4 major journals and around 4 major conferences (conferences are actually a big deal in our field, some have low 30s acceptance rate with like 1k submission). We have other more specialized journals and conferences, but in general you can submit to one of the big ones with nearly topic in the field. For us, the match is more helpful than harmful, so the reaction is more ‘oh you submitted a paper on this topic and that’s exactly the topic we’re interested in publishing!’ than it is ‘this work doesn’t fit this journal’.

“… A lot of political/philosophical biases that are not accounted for. For years transgender medicine was not a researchable subject because review boards did not deem it a respectable subject.”

Yeah you are right, we don’t have any of those problems. Sometimes there can be strong opinions about different approaches (you may have heard of deep learning - this topic is not without controversy). But we definitely don’t have religious or political sway when reviewing work.

“Peer-reviewed research is valid in the sense that it is reproducible and standardizeable.”

Yes, this is the essence of peer view research - something that contributes a reproducible result.

“If playing Journey with a kid with social issues helped him better socialize with peers that doesn’t mean that I can consistently prescribe Journey to every kid that presents with social issues. In part I have to be culturally sensitive to the background of the kid and determine subjectively if that sort of intervention could be repeated. However that does not mean that publishing that case study would not be a benefit to the clinicians that learn choose to study that experience. I believe this is the sense in which both formats of publishing research are both extremely valid and necessary.”

I would agree with you there.

“Perhaps people who dominate a video or spoken format over a written format would have done research with sound science and procedures that would otherwise find it’s way to a journal if the expression of their ideas was done is a written medium.”

I would love to see some other medium be accepted for journals. I know that in robotics we require videos of experiments for nearly all publication venues and usually we also submit 1 minute video summaries for papers. This is mostly to save time for the reviewers, but it has become a pretty big deal for authors. It’s difficult to get published if you can’t show your work too. It’s a long way off from the main medium being something other than text, but it feels like a good direction. One issue I see in moving away from text is simply the density of some papers.

“I understand that procedure is important for standardizability, but it is none the less restrictive of possible valid contributions.”

I don’t think I meant to say that peer review is the only acceptable way of publishing information. I just wanted to say that the way it was covered in this podcast left a lot unsaid. I felt misrepresented listening to it as a researcher so I just wanted to point out a couple important pieces of information that help explain why peer review and academia are the way that they are. I think you have covered a lot of the missing pieces though, and have given fantastic arguments for both peer review and non peer review publishing.

This is my first post here. I actually went looking for some kind of email, contact page, etc for Josue/Kelli to send this too…couldn’t find one so I came here :slight_smile: I will try to start browsing these forums more often.


#5

That’s great GT forums is always open for the discussion of ideas. I guess from my perspective which is clinical by nature, theres a few issues I view differently when listening to the podcast. Its very possible that Josea/Kelli meant to talk about something that neither of us adressed but when I see games research in geek therapy I see it in terms of it’s aplicability in clinical setting where you might see its applicability to more technical fields of videogame research. So I think its a matter of perspective. Ur always welcome to start more discussions and please enjoy all the other awesome podcasts in the network.


#6

Hi @sterlingm! Thanks for taking the time to respond. Posting here is a great way to keep the discussion going! You can also find ways to reach us at headshotspodcast.com.

I think @Gianminni covered some of what I wanted to say about how psychology/behavior peer review is the world I’m most familiar with and since I’m also an engineer, I can understand some of the differences in your peer review experience.

What I was trying to discuss the idea of peer review and how it could be applied in the world of games. I was thinking: Who are the scholars? And why do we only listen to them? In games in particular, does the opinion of the people who experience the games the most not matter? And who decides that? Can a million likes and 500 million views elevate an idea or theory to the level of some person who writes an article that 30 people have read?

I’m using the term peer review liberally and literally. What is the value of having a peer… review your work (or theory/idea) and who are your peers?

I agree that formal and informal platforms can co-exist. The truth is that they do, simultaneously, but I don’t think there is enough collaboration. If “formal” research is siloed and “informal” research is seen by the masses, what effect does that have on our fields? In psychology, we often complain about how research is reported. It’s infuriating how sometimes a research article is completely misreported and then, most people will never read the journal article or white paper or whatever is available.

I guess the last thought I had was that I understand your point about the experts talking to the experts, I’m not saying that’s right or wrong, I’m saying that a lot of people are hungry for information and in some cases could benefit from it, if it were more accessible. My other point is that in some fields (like psychology, behavior, mental health), I think there is value in allowing a diversity of voices, especially those who may not have earned the “right” to join the discussion and yet have insights we’ve never considered.

Please feel free to keep this conversation going, react to more episodes, or start a new thread @sterlingm!

  • Josué