Thu 16th April 2015 - 09:58

Crowdsourcing Quality (Or, Why Openness Matters)

By Javiera Atenas (University College London) & Leo Havemann (Birkbeck, University of London)


Quality in open contexts


In a trajectory that did not simply begin from MIT OpenCourseWare (OCW), via Open Educational Resources (OER), and latterly arrive at a promised land of Massive Open Online Courses (MOOCs), a plethora of institutions, organisations and individuals have attempted through various and numerous interventions to ‘open up’ the education landscape to a wider range of travellers, inhabitants and tourists. And in a way, the question of quality has always dogged open education’s steps. Would open universities attract ‘quality’ students? Would open resources and courses be of good enough quality, and how could we be sure? Open, it seemed, might pose a threat to quality, or at least place a question mark over it. Yet, as openness has gained traction, it has also been suggested that quality might be bolstered and supported through the wisdom of crowds.


One of the problems faced by advocates of open education is a widespread perception of that commercially published materials are quality materials, and that unpublished, open materials may not be. Some may even suspect open resources are simply those which are ‘not good enough to publish’. For an alternative perspective it is worth considering the case of Wikipedia, one of the world’s most used websites, which can indeed be seen as a vast open educational resource. Wikipedia has certainly had its detractors in academia, on the basis that anyone can write and edit Wikipedia articles. This suspicion persists, in the face of academic studies that suggest the overall accuracy of Wikipedia is about as good as that of published encyclopaedias, while at the same time it provides a vastly wider coverage of topics, and in spite of evidence that shows ‘abuse’ (maliciously updating articles with false information) is usually very rapidly corrected[i].


Of course, the success of Wikipedia cannot be said to prove that content that is made available freely will be of high quality. Yet it does suggest that it is pertinent to ask whether and how the adoption of open practices can be leveraged in order to improve the quality of open education.


Open Educational Practices: Collaborating towards quality


Open Educational Practices (OEP) are activities which aim at inclusiveness and gratuity, and support and promote the use, creation and development of open content[ii]. For us, the concept of OEP must embrace, but not be limited to, work done in relation to OER. Instead we envision OEP as consisting of a kaleidoscope of teaching, learning and research related practices, acting within and upon a field of open and closed components, leading towards the opening of access to knowledge and education, while working openly in a spirit of collaboration, transparency and shared endeavour.[iii]


To put it another way, practices occurring within the kaleidoscope certainly include working with various kinds of open components: developing, sharing, reusing, or remixing some form of open content. This might simply be an individual Open Educational Resource (OER), or a collection of them packaged up as OpenCourseWare (OCW), or an Open Textbook (OTB). It might involve developing or delivering a Massive Open Online Course (MOOC), or the use of Open Data (OD) as a learning resource for student research. OEP also includes the question of what you do with resources: OER, OTB or OD can be deposited in platforms known as Repositories of Open Educational Resources (ROER) which (should) support academics searching, retrieving and evaluating these materials.


Beyond these open components, there is realm of collaborative, networked activities such as blogging, tweeting, and presenting about teaching practices, pedagogic questions, learning designs, evaluation of resources, and the like, which are no less key. All of these elements can be interconnected to create instances that support technology-enhanced learning.


So with this notion of OEP in mind, we see parallels with Martin Weller’s concept of digital scholarship, about which he notes, “scholars are being asked to share the teaching materials they produce, to publish in open access journals, to network with others in social media, and to reuse OER.”[iv] With Weller we agree that these attributes represent the beginnings of a possible paradigm shift in the way academics think about scholarly outputs and teaching resources. But while analogue to digital represents a great enabler of such a shift, its key axis might instead be viewed as closed to open. This shift is to be welcomed but in our view, is far from inevitable. While there is evidence to say this is already occurring, academic staff cannot be expected simply to “naturally” develop and harness these abilities.


According to our analysis of academics views on quality development of OER and ROER[v], ensuring that educators can adopt the use and produce of these resources, it is necessary to comply with some basic quality criteria, for example:


●      Use open source software when creating OER to allow modification

●      Provide relevant pedagogical information (aims, objectives, level)

●      Clearly attribute the authorship of the resources

●      Specify the Creative Commons Licenses for each resource

●      Support the development of OER in multiple languages

●      Follow accessibility guidelines for developing inclusive resources


For educators needing to assess the quality of existing OER, JISC[vi] has recommended that the accuracy of the resources, the reputation of the author and author’s institution, the standards of technical production, the resource’s accessibility and its fitness for purpose should be considered. However, it is less clear how and by whom this evaluation process can be performed. It is here that we believe communities, with the help of suitable, purpose-built repositories, can play a vital role.


Repositories and OER quality


While learning resources can be found scattered throughout the web, there appears to be a consensus in the open education community that dedicated platforms (e.g repositories) are needed in order to optimally support the life cycle of OER sharing, retrieval and use. Our work on ROER takes as its point of departure the notion that purpose-built repositories are not simply there to host content: they must create added value for discoverability, for use and repurposing, and ultimately support quality enhancement. For this reason we undertook a literature review[vii] through which we aimed to discover what OER scholars consider the most important features one would wish to see in repositories.


Drawing from our review of the literature, we would argue that the ethos underlying the creation of repositories of OER can be said to comprise four key themes: Search, Share, Reuse, and Collaborate as the purpose of ROER is to support searching for content in a structured way, sharing their own resources, reusing existing materials, creating new ones through adapting or translating, and collaborating with others by interacting, commenting upon, reviewing and promoting resources. However, our analysis found that currently, ROER do not always facilitate access to and/or retrieval of the resources.[viii]


Therefore ROER should include certain technical and social characteristics in their platforms aiming at facilitate the access and reuse of the resources[ix] including:


●      User evaluation tools

●      Multilingual navigation

●      Social media sharing tools

●      Access to the source code or original file

●      Guidelines to understand and apply Creative Commons Licenses

●      Pedagogic description of the resources in their metadata

●      Provision of open source software to facilitate adaptation and modification.


Assuring openness in MOOCs


The openness of MOOCs has at times been open to debate, as many of the commercial MOOC platforms include strict content usage policies that make the content and the data (though often produced by publicly funded universities) restricted under all rights reserved licensing. MOOCs, in order to be fully open, should provide or reuse open content such as open licensed images, OER, open access research papers and Open Datasets, and thereby ensure that the course and materials are open to later reuse and remixing, rather than simply open registration.


We suggest[x] it is desirable to open up MOOCs using three possible strategies


●      Opening individual resources as OER depositing them in ROER, so images, audiovisual materials and assessment carefully describing its aims and objectives, key information and syndicating the authorship.

●      Opening up packages of content that can be downloaded from ROER so resources can be recontextualised in other courses.

●      Transforming entire MOOCs into OpenCourseWare (OCW) so people can make use of them for self paced learning and reuse the contents of the courses.


We consider that when MOOCs have been produced by universities they should include the opening up of the MOOC resources as a necessary part of the process rather than an add-on. This would allow high quality content to be made accessible for members of the general public as well as retrievable for reuse by educators and learners at any time. Universities should therefore, prior to signing contracts, consider whether any restrictive terms and conditions are imposed by their platform provider which actually present a barrier to their open education aims.[xi]


Open Data as OER 


Open Data produced at scientific, research and governmental level has become an invaluable resource not only for other researchers, scholars and for the general public, but for academics and students, as it provides high quality information that can be reproduced, analysed and used for educators across the globe to improve the numeracy and research skills of students alongside deeping in the knowledge and understanding of their subject areas.


The Open Definition includes the concept of universal participation and interoperability in regards OD[xii], so to make participation universal, students need to take advantage of the use of OD by being provided with it in different teaching and learning activities. We consider that as scientific/research data is mostly provided by universities, higher education can now embrace research-based learning models, making use of OD to develop and improve critical thinking and research skills amongst students, as they can learn from researchers based in their own university and country but also internationally[xiii].


With the use of OD, while individual students can benefit, collaborative skills can also be developed. Moreover, students can work across disciplines, improving their literacy, numeracy and professional skills by collaborating with other students, researchers and academics. Collaboration in analysing real research conducted in their universities and elsewhere might strengthen students understanding of good research practices, facilitating independent research, and developing teamwork critical and analysis skills.


We suggest that open datasets used as OER for research or scenario based learning activities are also stored and shared including the contextual information regarding use through ROER. Through these platforms the OD and related activity will be searchable and retrievable and can be assessed and evaluated by the users to ensure quality regarding its teaching and learning value.




Drawing from our research[xiv] academics refer as barriers for adopting OER and ROER their lack of training, language barriers and technological challenges as barriers. These problems are not exclusive to OER and OEP, but also equally affect to MOOCs, OCW and OD, therefore collaboration is key to overcome these challenges and barriers, not only to evaluate the resources or to improve the platforms but to support and train other users in using and creating open content.


Openness, by its collaborative nature, favours crowdsourcing of quality assurance by encouraging users to be not only contributors but also critical reviewers, improving resources and usability.  To ensure that quality is assured in education, is necessary to promote OEP, as these practices are associated with a variety of benefits: Efficiency through reuse or repurposing and widening access to information and knowledge by generating a positive culture of openness, sharing, and collaboration.


However, we consider that significant barriers continue to hinder the takeup of OEP. Although a simple lack of awareness of open resources and practices is often assumed, this could be masking quite tangible regulatory, cultural, skills-based and technical challenges. Institutional policies on openness are rarely actually prohibitive, but might simply be unclear, unnecessarily restrictive, or missing altogether. Similarly, the cultural context in which resource creators work, and the teachers and students skills gaps (literacy, language, numeracy, technical) can difficult adopting these practices, therefore, even with the requisite skills, open educators can be hampered by the available technologies.


Quality assurance needs to be inclusive and accessible, by engaging with the communities of practice as critical partners to evaluate the quality of the resources - from the quality of content, to the quality of the object and its usability - this can be understood as crowdsourcing quality enhancement. Regarding accessibility, the design of OER and ROER needs to consider all the potential users, following accessibility guidelines in order to consciously include people with learning disabilities but also, to include those who have less access to powerful computer systems in the developing world, so including users as evaluators can ensure accessibility. 


In order for the sharing and reuse of openly licensed resources to become adopted as a mainstream educational practice, it is necessary to consider how technical infrastructure underpins such activity and how it can further support and enhance OEP. Collaboration to enable crowd mechanisms for quality assurance are key, as collectively assessing quality might improve the current models enabling trust mechanisms amongst educators and scholars.


Ensuring quality of resources through OEP might lead to efficiency gains for teachers as academics might spend less time a) browsing for resources b) preparing materials c) supporting students with different learning needs and styles, as collaboratively enhancing quality of the resources and facilitating access to high quality content that has been approved or validated by other educators and which can easily adapted to used across cultures and disciplines may allow academics to spend more time preparing their classes and communicating more effectively with their students.


[i] Wikipedia (2015). Reliability of Wikipedia.


[ii] The ICDE’s open educational practices definition can be found at:


[iii] Havemann, L., Stroud, J., & Atenas, J. (2014). Breaking down barriers: Open Educational Practices as an emerging academic literacy. Academic Practice and Technology Conference, University of Greenwich, 5 July.


[iv] Weller, M. (2011). The Digital Scholar: How Technology Is Transforming Scholarly Practice. London: Bloomsbury Academic. Retrieved from


[v] Atenas, J., Havemann, L., & Priego, E. (2014). Opening teaching landscapes: The importance of quality assurance in the delivery of open educational resources. Open Praxis, 6(1), 29–43. doi:


[vii] Atenas, J., & Havemann, L. (2014). Questions of quality in repositories of open educational resources: a literature review. Research in Learning Technology, 22. doi:


[viii] Atenas, J., & Havemann, L. (2013). Quality assurance in the open: an evaluation of OER repositories. INNOQUAL-International Journal for Innovation and Quality in Learning, 1(2), 22–34. Retrieved from


[ix] Atenas, J., & Havemann, L. (2013). A vision of Quality in Repositories of Open Educational Resources. In Y. Punie, Christine Redecker, & J. Castaño (Eds.), OPEN EDUCATION 2030. JRC-IPTS CALL FOR VISION PAPERS. PART III: HIGHER EDUCATION (pp. 54–59). European Commission Joint Research Centre Institute for Prospective Technological Studies. Retrieved from 4_author revised_OK.pdf 


[x] Havemann, L., & Atenas, J. (2014). MOOCs must move beyond open enrolment and demonstrate a true commitment to reuse and long-term redistribution. LSE Impact of Social Sciences Blog. Retrieved from


[xi] Atenas, J. (2015). Model for democratisation of the contents hosted in MOOCs. RUSC. Universities And Knowledge Society Journal, 12(1), 3-14. doi:


[xii] Open Definition


[xiii] Atenas, J., Havemann, L., & Priego, E. (2015). The 21st Century’s raw material: using open data as open educational resources. Open Education Working Group Blog. Retrieved from


[xiv] Atenas, J., & Havemann, L. (2013). A vision of quality in repositories of open educational resources. Open Education 2030. Contribution to the JRC-IPTS Call for Vision Papers. Part III: Higher Education, 54–59.


Language Help