IRR: Agreement Among Coders is Key

For years I have been engaged in a quantitative study at Guatemala’s Historic Archive of the National Police, or AHPN. (See the blogposts below.) In this study coders collect data on sheets of paper according to criteria established and explained in manuals. But when collecting data, there’s always room for human error—this is why the validity of the study hinges on verifying that coders use the correct criteria.

It is important to mention that the mainstay of coding is the use of a controlled vocabulary. A controlled vocabulary gives analysts a framework, or frame of reference, when converting qualitative information into categories without discarding anything, without misrepresenting or misinterpreting all the gathered data. In the case of the quantitative study, applying controlled vocabularies gives analysts tools in the calculation of statistics on locations, actors, events/incidents, or other variables used when trying to answer research questions.

For this task of checking the level of agreement among the coders, we implemented a process called “inter-rater reliability” (IRR), or, in Spanish, CEC (Confiabilidad entre codificadores). IRR is a mathematical measure that gives us insight into the consistency of the classifications assigned to the content of the documents. The calculations are made field by field.

These IRR calculations were based on structured exercises that validate the consistency of the ways that the information from the documents is interpreted and coded; these exercises are done simultaneously to coding.

How we carry out the IRR exercises.
– Working with the data-capture sheet used during sample collection, we create a new document, the Ficha para Confiabilidad entre Codificadores, or the “File for Reliability among Coders,” which contains fields for measuring uniformity in the use of vocabularies.

– To start the process, all coders report to the same room to encode the same document at the same time. Each coder will encode 10 to 15 documents, with a time delay in the exercises varying between 3 and 5 days.

– The process begins by placing the acetate template over the document, placing the left upper edge of the template aligned with the upper left edge of the document. All coders use the same template.

– The template is, essentially, a grid, with numbers along one axis and letters along the other. Wherever on the page the coder finds relevant information about, for example, an event or an actor, the coder notes the coordinates. During this IRR (CEC) process, the coder is looking for information about these fields:

• Document ID
• Identification of actors in the document, actor by actor
• Type of act recorded in the document
• Actors involved in the acts in the documents

– The coder identifies actors involved in the receipt, transmission or authorship of documents, and records the coordinates, using the first letter of the name. If there are no names or surnames, the template is placed on the first letter of the unit or the institution to which the actor belongs. If none of these exist in the document, we use the first word that refers to the existence of the actor. For example, if a document includes the phrase “heavily armed men,” the coordinates are placed at the beginning of the word “men.” It may be that what we observe in the document is: age, sex, position, rank or some synonym for man, woman, person, people, or failing that, the geographical location of the actor.

Consistency is the key word.
By meticulously tracking consistency, we have refined how these exercises are performed. Because of IRR, it has been possible to identify failures in the interpretation, or the need to adjust the criteria or to suggest changes in the way we approach our work.

Therefore, these exercises lead to decisions that improve the controlled vocabularies. By refining boundary conditions, the coders may have more certainty when selecting one of the options in the manual.

Overall, therefore, it is not so terrible when the coders make mistakes, as long as they are consistent in their mistakes. In this way we can identify the discrepancy and correct the interpretation.

THE AHPN SERIES by Carolina López
Quantitative Research at the AHPN Guatemala
The Story of One Document Inside the AHPN
The Art and Science of Coding AHPN Documents
Learning Day by Day: Quantitative Research at the AHPN
IRR: Agreement among Coders is Key
The AHPN: Home of Stories Old and New
The Great Lessons in Research at the Archive
Ten Years and Counting in Guatemala
The Case of Ana Lucrecia Orellana Stormont

[Creative Commons BY-NC-SA, including photos]

All photos: courtesy of AHPN.

---