PUBLICATIONS & REPORTS
Key Takeaway
Protocols and security functions for virtual, remotely proctored high-stakes exams can be replicated in a virtual setting in order to achieve the same principles and secure exam conditions as in person exams. Remote virtual exam delivery is an administration method that is a secure alternative to traditional in-person exams and provides advantages that will extend beyond the pandemic.
Considerations for Virtual Written Exam Security
Introduction
The COVID-19 pandemic has forced testing organizations to re-think how they administer exams. When the COVID-19 pandemic forced cancellation of in-person exams, many organizations turned to virtual solutions to ensure that applicants can progress in their pathway to licensure without attending a test centre. A concern, however, has been ensuring that appropriate security measures are in place to protect content, to prevent cheating and to offer applicants a fair and equitable exam experience.
This TIP will summarize our experiences in transitioning exams to remote administration, focusing on security and administrative issues. Traditional in-person exam protocols are easily secured through physical control of the environment. By contrast, virtual testing environments present a variety of exam security risks and administrative challenges that must be addressed to provide examinees with a fair exam experience.
Resource Impact
Moving to a remote virtual exam delivery model requires resources. The greatest cost is not in the technology but in the staff time required to evaluate alternatives and make adjustments to content, processes and procedures. There may be set-up costs, but then costs are usually associated with administration volumes. However, the time and effort devoted to analysis of potential technologies should not be underestimated.
Contrary to some thinking, going digital is not more economical but is in fact more resource intensive. Additional needs include systems checks for examinees and all other participants to ensure stable and reliable connectivity. Furthermore, security requires more vigilant oversight through use of proctors and technology as the exam environment is no longer fully controlled. There may also be added staff to manage props and movement of examinees for exams with several stations, such as OSCEs.
Going virtual shifts some site based resource requirements to technological platforms, but adds additional requirements that must be resourced.
Secure Technology
There are many existing virtual solutions for Multiple Choice Question (MCQ) exams, with some having remote proctoring overlays for added security with home based testing. Proctoring may include an Artificial Intelligence (AI) component that flags behaviours or events for review. When proctoring is not constant, other features such as locked browsers become more critical. It is important to remain focused on the content of the exam when considering technologies. If the solutions offered don’t provide a reasonable match to existing exam requirements, it will be difficult to argue that the virtualized exam is equivalent. It is likely that there is no “out of the box” solution that meets all requirements without some adaptation.
A common consideration with MCQ style exams is to ensure there is a locked browser, but similar security may be afforded by screen captures, keystroke recordings, video monitoring and live proctoring. AI technologies are likely to become increasingly more sophisticated and already provide excellent alert capability for many behaviours. With OSCE style exams, the examiner and standardized clients are present with the examinee, but it is still important to have trained proctors who can focus on the station to record any incidents. Ideally, proctors would not only be trained in behavioural observation, but also on the requirements and expectations of specific exams.
New policies may be required to manage the video data amassed in virtual examinee monitoring. This is personal information that must be protected and safeguarded with retention and destruction policies. Retention periods should align with quality assurance activities and relevant appeals periods.
Secure Testing
With remote exams, virtual security concerns are managed differently, but are fundamentally the same as any other exam modality. Principally, the examining body must validate the examinees identity, prevent cheating, and prevent content exposure.
With in-person testing, the examinee can be checked against official photo identification and also sign in. This provides both a visual identity check and unique signature. The controlled in-person environment also means that there is no risk of examinee substitution. Remote virtual exams must rely on screen based verification and continual video monitoring of the examinee in their environment. This may be considered intrusive, but it is the only way to verify and create a record of identity.
Studies show an unsettling level of cheating in higher education settings. Clearly, the temptation is there for many. Examinees can be reminded of their regulator’s ethical practice standards, and sign off on their personal commitment to honest behaviour and confidentiality. In addition to moral suasion, both AI and direct observation by trained proctors act as deterrents. With longer exams, bathroom breaks present a problem. Psychometric analysis of performance before and after can be used to infer cheating. Regardless, there is a reduction in the control of the environment that affords a greater cheating risk than in-person exams.
There is also an increased concern that examinees may surreptitiously record exam content. This “harvesting” can then be used to help others – sometimes for profit. In-person efforts can be made to prevent recording, but the most likely exposures are from examinees who recall and share content after an exam. Examinees have typically sworn not to discuss any aspect of their exam, however it is widely accepted that this stricture is not universally adhered to. The best ways to mitigate the content exposure risk is to increase the item pool so that future exams are less predictable, or to rely more on performance assessments, like OSCEs, where performance is likely less influenced by any prior content knowledge.
Examinee Impacts
The focus of this TIP is not on the validity and reliability of virtualized exams. Separate processes and considerations, including rigorous psychometric review, are needed to confirm that a virtualized exam is measuring the same competencies. However, it is equally important to consider the change in the experience of the examinee from an administrative perspective. Before going live, testing and piloting will inform how processes and procedures need to be adapted for secure exams. This included tests of registration systems, orientations, security scans, proctoring, and, for performance exams, management between stations, props, scoring systems and incident reporting protocols. Everything needs to be re-thought to support the examinee in the online environment.
As much as possible, examinees should experience exam content as they would in person. The major differences being that examinees are unsupervised in remote settings, and will likely spend longer in the exam environment because of system checks, environment scans, periodic security checks, and managing movement of examinees in the performance exams.
Outcomes
Virtual remote examination is resource intensive and requires considerable adaptation of administrative procedures. Security concerns are heightened, as there is less control of the examinee’s environment. Remote proctoring and other strategies must be implemented to mitigate against the increased security risks. Despite the challenges, there are many benefits, particularly for examinees. Geography is no longer an issue. Exams can be challenged from anywhere, saving time and costs. This advances equity and is arguably more fair and responsive to the needs of examinees.
Touchstone Institute is now committed to offering remote virtual exams having received encouraging feedback on our initial assessments.
Testimonials
"Check-in was smooth and security was scrutinized efficiently."
"The exam went well. It's less hassle for us examinees."
"Having to take this exam alone and at the comfort of my home takes away a little bit of anxiety compared to traveling and being with a number of people around."
"It's easy to take the exam at home because the environment is known to me, and it lessens my anxiety during exam."
"Before the exam I was really concerned and confused about the proctoring, but everything worked really well.
"At first, I had encountered a couple of technical issues but the technical support representatives were so helpful and I was able to proceed with my test as scheduled."
"The support group was accommodating and responsive. I thanked them for assisting me during my exam."
"My exam was interrupted during the last part of the exam but I was easily and seamlessly placed back where I left off. Very impressed with their service."
"Thank you for providing such an alternative."
