In July 2014, UvA started the implementation of online proctoring for the institution. Read more on their experience with ProctorExam.
A very important question that is being asked within the field of online examination is how safe, reliable and valid is online examination. We believe that an even more important question is what online assessment could mean for the learning process of a student. During this article we will present a case study, which has been conducted at the University of Amsterdam (UvA) in 2014 and 2015. For this pilot the UvA used an online proctoring method, which was developed by the startup ProctorExam.
When learning occurs online; borders, time and location lose their meaning. In this scenario, we cannot expect that assessments have to be connected to a geographical location. Educational institutions offer massive online courses (MOOC’s) and smaller online courses (SPOC’s) through online educational platforms like Coursera and Edx. Students who follow those courses would also like to receive a certificate and get accreditation for participating in those courses (Ho, 2015).
Online assessment is the final link in transforming the complete educational experience from offline to online. Currently, distance learning is being offered all over the world. However, students still have to go to a physical location to take their exam.
In the United States 60% of the students in Higher Education have followed at least one course completely online (Allen & Seaman, 2011). Through this demand the significance of online assessment has increased as well as the demand for online supervision (online proctoring). Currently, online proctoring is mostly used in niche markets where students can follow their complete course online. However, we do expect that the demand for online examination will increase and therefore the market size will grow. Just like any other new development/innovation, the product-market fit within the Netherlands still lies with the first generation of users namely the early adopters. The second generation will be the early majority and will form the demographical core of the potential market and will be quite disruptive.
The early majority users of online proctoring are found in the US. The use of the application varies from business-to-consumer in which a student rather takes his/her exam at home, to universities who obliges their students to use online proctoring. The reason of this is that online examination is more efficient than bringing students to campus to use equipment provided by the university. Also, online examination offers students the opportunity to do the whole process online in which there is no need to come to campus at all.
Online proctoring is not just being used in open online courses and MOOCS. Cima et al. (2014) developed proctoring methods, which are being used at MITx and in a regular full-time course, Introduction to Solid State Chemistry. By offering flexible masters course the course success has increased enormously.
When learning occurs online; borders, time and location lose their meaning. In this scenario, we cannot expect that assessments have to be connected to a geographical location. Educational institutions offer massive online courses (MOOC’s) and smaller online courses (SPOC’s) through online educational platforms like Coursera and Edx. Students who follow those courses would also like to receive a certificate and get accreditation for participating in those courses (Ho, 2015).
Online assessment is the final link in transforming the complete educational experience from offline to online. Currently, distance learning is being offered all over the world. However, students still have to go to a physical location to take their exam.
In the United States 60% of the students in Higher Education have followed at least one course completely online (Allen & Seaman, 2011). Through this demand the significance of online assessment has increased as well as the demand for online supervision (online proctoring). Currently, online proctoring is mostly used in niche markets where students can follow their complete course online. However, we do expect that the demand for online examination will increase and therefore the market size will grow. Just like any other new development/innovation, the product-market fit within the Netherlands still lies with the first generation of users namely the early adopters. The second generation will be the early majority and will form the demographical core of the potential market and will be quite disruptive.
The early majority users of online proctoring are found in the US. The use of the application varies from business-to-consumer in which a student rather takes his/her exam at home, to universities who obliges their students to use online proctoring. The reason of this is that online examination is more efficient than bringing students to campus to use equipment provided by the university. Also, online examination offers students the opportunity to do the whole process online in which there is no need to come to campus at all.
Online proctoring is not just being used in open online courses and MOOCS. Cima et al. (2014) developed proctoring methods, which are being used at MITx and in a regular full-time course, Introduction to Solid State Chemistry. By offering flexible masters course the course success has increased enormously.
Three scenarios tested
At the Faculty of Science from the UvA, three scenarios were tested in 2014 and 2015.
- Students take their exam at home and are supervised by an online proctor who will watch the students through 2 video camera’s. The complete exam is being recorded (ProctorExam Pro).
- Students take their exams at home. This type of exam taking is completely automated and the whole exam is being recorded. The recordings are viewed and analyzed after the exam (ProctorExam Classic).
- Students take their exam at the institute where their computer screens are being monitored during the exam (ProctorExam Light).
It is possible with all these exams types to view the recordings after the exam to examine possible violations by students.
- Students take their exam at home and are supervised by an online proctor who will watch the students through 2 video camera’s. The complete exam is being recorded (ProctorExam Pro).
- Students take their exams at home. This type of exam taking is completely automated and the whole exam is being recorded. The recordings are viewed and analyzed after the exam (ProctorExam Classic).
- Students take their exam at the institute where their computer screens are being monitored during the exam (ProctorExam Light).
It is possible with all these exams types to view the recordings after the exam to examine possible violations by students.
Distance Online Examination at the Pre-master Information Studies as first audience
In 2012 the Faculty of Sciences launched an online pre-masters program in Information Studies. Through this pre-master, students get the opportunity to increase their level of knowledge which is required to start with the master of Information Studies. The expected target group for this pre-master are Dutch students as well as students from outside The Netherlands. Exams that need to be taken in order to complete the pre-master have to be taken at the faculty in Amsterdam. Even during the first edition of the pre-master, there were students that applied from countries all over the world. Because of this, online examination became a necessity, since it would be very expensive to require these students to fly to Holland. Instead, these students could do their exam online through audio and visual contact with a teacher. Although, this process took a lot of time it was still manageable. However, it became clear that the UVA needed professional help for online examination. Although there are American providers for online Proctoring, the UVA decided to look for a European alternative while American companies fall under the Patriot Act, which means that the data stored from online examination would be within the reach of American three-letter agencies.
This is the reason that UVA decided to work with ProctorExam, which led to a successful pilot of ProctorExam Pro with 41 students in July 2014. From that moment on, all students of the pre-master were offered the opportunity to do the exam online provided that their hardware was up to standards. It was possible to take the exam at the UVA if there were any personal objection. However, this only happened a couple of times.
To prevent any surprises during the exam, all students had to do a check requirement of their hardware. If students were not able to successfully complete the check they could not participate in the online exam. Also, in order to decrease the amount of stress during the exam, the students were invited to do a dummy exam, which simulated the online exam and helped the students become familiar with the product.
Students have the opportunity to enroll 4 times a year and currently more than 100 students have taken the exam at home. Only a small number of students come to the university to take the exam.
All the 41 students from the pilot in 2014 had to fill in a survey directly after the online exam. The results stated that 87% of the students were very satisfied with the exam process and the presence of the proctor. Three students (7%) were dissatisfied. The majority (69%) didn’t feel distressed by the presence of the proctor during the exam (image 1). Only 13% of the students did not like the presence of the proctor. Similar results were shown with later exams.
This is the reason that UVA decided to work with ProctorExam, which led to a successful pilot of ProctorExam Pro with 41 students in July 2014. From that moment on, all students of the pre-master were offered the opportunity to do the exam online provided that their hardware was up to standards. It was possible to take the exam at the UVA if there were any personal objection. However, this only happened a couple of times.
To prevent any surprises during the exam, all students had to do a check requirement of their hardware. If students were not able to successfully complete the check they could not participate in the online exam. Also, in order to decrease the amount of stress during the exam, the students were invited to do a dummy exam, which simulated the online exam and helped the students become familiar with the product.
Students have the opportunity to enroll 4 times a year and currently more than 100 students have taken the exam at home. Only a small number of students come to the university to take the exam.
All the 41 students from the pilot in 2014 had to fill in a survey directly after the online exam. The results stated that 87% of the students were very satisfied with the exam process and the presence of the proctor. Three students (7%) were dissatisfied. The majority (69%) didn’t feel distressed by the presence of the proctor during the exam (image 1). Only 13% of the students did not like the presence of the proctor. Similar results were shown with later exams.
While 20% of the students were not satisfied with the privacy statement (figure 4), the UVA and ProctorExam invested a lot of time on improving this statement and securing the privacy of the students. Furthermore, a lot of time was invested by ProctorExam to simplify the exam procedure and improve the look and feel of the product. In 2015, there were 22 students who did an online exam and only 2 students were not satisfied with the privacy regulation. Therefore, they decided to take the exam at the UVA. 11 students (50%) were very satisfied (image 2) with the security procedures. Still, there was a fair amount of students that were neutral concerning privacy issues, however the percentage did decrease compared to 2014. In the upcoming edition of the pre-master the UVA wants improve the communication with the students to create more trust. Whereas the number of students is still relatively low we will keep monitoring the students in order to validate these out-comings.
The UVA developed a customized workflow for the online examination of the students of the pre-master Information Studies. Also, during the upcoming phase the cooperation between the UVA and ProctorExam will be optimized. Still, there are some issues that need to be addressed for example: how long do we store the video images, what are the implications for students who misbehaved during the exam, and what rules need to be upheld when the internet connection is lost. Also safety procedures concerning online examination need to be addressed further.
Meanwhile, the number of foreign applicants for master and pre-master Information Studies has increased. The pre-master program in combination with online exam taking will offer students a great opportunity to prepare themselves for the master.
Meanwhile, the number of foreign applicants for master and pre-master Information Studies has increased. The pre-master program in combination with online exam taking will offer students a great opportunity to prepare themselves for the master.
Digital examination with regular bachelor course Besturingsysstemen
In april 2015 the UVA did an experiment with ProctorExam Light with the regular bachelor course Besturingssystemen. Over 100 students participated in a digital open book exam with the use of ProctorExam technology at the UVA. Students were notified of the rules and the privacy statement of ProctorExam. The students that did not want to participate in the online exam were allowed to do the exam on paper. Due to privacy issues some of the students chose that option.
By adding ProctorExam Light (screensharing extension which monitors the digital workspace) to the digital exam no lockdown software was needed to prevent fraud. According to the students it was a very positive first experience. The teacher would like to use ProctorExam Light for other upcoming digital exams. By piloting these new products it was shown to us that examination could be very flexibly without losing the integrity of the exam standards.
By adding ProctorExam Light (screensharing extension which monitors the digital workspace) to the digital exam no lockdown software was needed to prevent fraud. According to the students it was a very positive first experience. The teacher would like to use ProctorExam Light for other upcoming digital exams. By piloting these new products it was shown to us that examination could be very flexibly without losing the integrity of the exam standards.
Online proctoring with ProctorExam
The Dutch startup ProctorExam offers a web-based solution for online examination so that no program or software needs to be downloaded on the computer. The platform uses 4 components to secure the online exam environment.
- 360° audio & visual supervision;
- ScreenSharing-technology;
- Live proctoring;
- Video recordings which can be viewed an analysed after the exam.
The four components are modular and can be used in a very flexible manner. For example, the institute could decide to use all four elements, or just screensharing if that is preferred.
A week before the exam the students will receive an invitation to do the check requirement test. This test will check the hardware of the students and see if it is sufficient to do an online exam. (Image 3).
- 360° audio & visual supervision;
- ScreenSharing-technology;
- Live proctoring;
- Video recordings which can be viewed an analysed after the exam.
The four components are modular and can be used in a very flexible manner. For example, the institute could decide to use all four elements, or just screensharing if that is preferred.
A week before the exam the students will receive an invitation to do the check requirement test. This test will check the hardware of the students and see if it is sufficient to do an online exam. (Image 3).
The set-up of the exam is conducted with a proctor. In Image 4a one can see the proctor on the right while the student is set on the left. During the exam the proctor will not be visible for the student so that the student can fully focus on the exam. (Image 4b).
Who is Natasa Brouwer, the author of this article?
Dr. Natasa Brouwer is senior consultant at the Education Service Center, Faculty of Science, University of Amsterdam. She is the coordinator of the ICTO-FNWI Program Council that is focused on increasing innovative educational technology at the faculty. One of core themes on which she focuses is making education more flexible with the use of online examination.
