Systematic Literature Review and Content Analysis in the Area of Communication and Information: the reliability problem and how to fix it

Abstract

This work presents an analysis of the methodological quality of systematic reviews of Portuguese-speaking literature, in the area of Communication and Information by using principles and techniques of this methodology common to content analysis, notably those relating to replicability and reliability. For this, we used 4 indexing bases, namely Web of Science, Scopus, DOAJ and SciELO, under a specific temporal delimitation (2010-2021). By developing a codebook and carrying out reliability tests, we selected and analyzed 49 works regarding the following characteristics: (a) databases; (b) search strings; (c) temporal delimitation (number of years); (d) number of analyzed works; (e) inclusion and/or exclusion criteria; (f) replicability; (g) reliability. The main results indicate a reduced concern of the analyzed works in incorporating or reporting procedures responsive to reliability (n=3). On the other hand, most studies (n=40) proved to be, in general terms, replicable. The main innovation and contribution of this research consisted in proposing and testing a procedure to increase the reliability of systematic reviews by consulting the authors of the papers. We highlight, in addition to the innovative character of such a procedure, its epistemic improvements. Finally, we discuss directions for future research by indicating possible gains that the methodological strategies conceived and tested in this work offer to guide the methodological rigor of systematic literature reviews.