Future Proofing
Organization wide commitment to re-imagine our services for the future in response to the pandemic
The Future Proofing Charter
Touchstone Institute’s business is mostly focused on helping to identify and prepare internationally educated health professionals (IEHPs) seeking careers in Canada. As an important source of health human resources, IEHPs deserve the opportunity to show their potential and value and to receive purposeful orientation to the health care environment in Canada. With the onset of COVID-19, all scheduled exams were cancelled and/or deferred – negatively impacting IEHPs on their paths to public service in Canada.
A series of initiatives were identified, and a “Future Proofing” Project Charter was signed by all departments in July 2020. The goals of this project were to create analogous services for all existing clients, enabling access by leveraging information technology while maintaining our psychometric and validity standards. With no other organization sharing our mandate and commitment to IEHP competency assessment, there would be a negative societal impact if we could not achieve our goals.
Goal 1
Deliver our International Medical Graduate (IMG) learning programs entirely online
All IMGs successfully matched to residency programs in Ontario are required to complete the Pre-Residency Program (PRP). This preparatory program employs physician faculty to provide comprehensive orientation to medical practice in Ontario.
An entirely remote learning experience had to be developed in a few weeks and delivered to 221 international medical graduates and visa trainees so they could complete programming in time for the July 1st start of residency training. Learners came from 31 countries and represented 22 different specialties.
Feedback from the learners and instructors indicated that the experience was positive and effective. Learners provided qualitative and quantitative feedback on each session delivered, in addition to their reflections on the program following the first few weeks of their residency. Qualitative feedback indicated that the online modality made it harder for learners to connect with each other and to network; in addition, they wished they could better apply concepts learned through simulation or more case based online activities.
I was pleasantly surprised by how the whole team managed to make remote sessions as engaging as they were. I was very satisfied with the whole experience.
- Learner
Extensive simulation experience has been a hallmark of past programs. The education team worked with our simulation experts throughout the balance of the year to fully re-design IMG programs virtually, with updated simulations and enhanced interactivity for the 2021-22 learners. This enhanced programming was successfully delivered this past summer, after the period covered in this annual report.
Goal 2
Deliver safe in-person OSCEs during the pandemic
Safe-OSCE NAC
Based on the recommendations of provincial and municipal health agencies, Touchstone Institute developed protocols to protect the health and safety of all examinees and staff attending the Objective Structured Clinical Examinations (OSCEs) that test competencies in simulated clinical settings.
Safety protocols include reduced occupancy, screening, temperature checks, physical distancing, guidance on personal protective equipment, and enhanced cleaning. Our physical space was also configured to separate cohorts and minimize close contact, with signage and barriers introduced in high traffic areas.
These safe OSCE preparations made it easy to accommodate the Medical Council of Canada’s requirements for the Fall iteration of the National Assessment Collaboration (NAC) Examination for graduates from international medical schools.
Touchstone Institute successfully administered the first Safe OSCE on September 12 and 13, 2020. More than 80 Candidates were assessed over the course of the weekend, approximately half of the capacity we would normally assess.
Touchstone Staff took the necessary precautions and made the exam safe and I felt comfortable working in that environment.
- Support Staff
Prevailing community infection rates and public health guidance meant that this was the only Safe-OSCE delivered in the period covered by this report.
Goal 3
Transform our Nursing Communicative Competency Test (CELBAN) to be delivered remotely
Digitization of CELBAN
The Canadian English Language Benchmark Assessment for Nurses (CELBAN) is a profession specific language exam for internationally educated nurses (IENs) pursuing registration in Canada. This exam had been administered at eight test sites across Canada to well over 2,000 IENs annually, and is accepted as proof of communicative competence in English by all Canadian nursing regulators. With COVID-19 forcing the closure of test sites, a solution was urgently needed for this paper-based exam.
After a thorough review of digitization options, a solution was identified that would once again provide IENs access to complete the test securely, either through remote proctoring or at a test centre. This prompted the conversion of CELBAN into a digitized form for the reading, writing, and listening components of the test. The speaking component required customization of a different remote administration platform to enable live video interaction with an examiner.
The first virtual delivery of CELBAN was in January 2021. By the end of the period covered in this report, 498 exams have been administered.
CELBAN was developed by the Centre for Canadian Language Benchmarks (CCLB) and has been administered and maintained by Touchstone Institute since 2014. This year, the CCLB and Touchstone Institute have committed to a partnership that strengthens support for CELBAN and recognizes Touchstone Institute’s role in the digitization and maintenance of this test.
It has been a wonderful experience taking exams online for the first time. The most challenge I have is speed especially with writing and time runs so fast. It is a wake up call because even in the work place with this technology most of the times we use computers to document almost everything about our clients.
- CELBAN Examinee
Goal 4
Transform all of our Multiple-Choice Questions (MCQ) exams to be virtual and remotely proctored
Touchstone Institute became a member of the Umbrella Consortium for Assessment Networks (UCAN) in 2014. UCAN-developed tools have been customized to meet our needs, and used for most exams to manage items and publish exams in paper or tablet formats. UCAN also has tools to publish exams online. Converting to an online web-based format allows virtual and remote administration, but proctoring is required for exam security. A proctoring overlay was sourced that included session recording, AI behavioural monitoring, identification verification, and included access for our own live proctors.
By 2021, we were building several MCQ tests on a reliable and robust remotely proctored platform. The first exam listed was delivered in 2020-21. The others were redeveloped in the period covered by this report, but administered in early 2021-22. The administration dates are shown in parentheses.
The first administration of our virtual MCQ was for the Internationally Educated Nurses Competency Assessment Program (IENCAP), an evaluating exam that assesses performance compared against recent Canadian Registered Nurse (RN) graduates. Administrations went perfectly, with all 47 examinees able to successfully complete their exams.
The Alberta Registered Nurses Assessment Program (ARNAP) is a standardized examination that assesses the knowledge, skills and judgement of internationally educated nurses (IENs) seeking nursing registration in Alberta. The MCQ component of this exam was converted from tablet to web-based format.
This exam includes a performance component (OSCE) that was successfully administered in person in Alberta.
The Assisted Reproductive Technology Laboratory (ART Lab) Exam is the certification program for professionals working in Assisted Reproductive Technology Laboratories in Canada. This exam was developed by Touchstone Institute with the ART Laboratory Special Interest Group (ART Lab SIG) of the Canadian Fertility and Andrology Society (CFAS).
The exam is offered in both English and French, and has been adapted for Touchstone Institute’s virtual MCQ platform.
The Internationally Graduated Optometrists Evaluating Examination (IGOEE) tests the knowledge, skills, attitudes and values of internationally graduated optometrists (IGOs), and is recognized by the Federation of Optometric Regulatory Authorities of Canada (FORAC).
In addition to OSCE components, this exam has two tests that were converted from tablet administration to the remote proctored platform: The MCQ and the Therapeutic Prescribing Assessment for Optometry (TPAO), a supplementary written test developed to assess appropriate drug prescribing competencies that was added in 2019.
The Knowledge and Competency Assessment Tool (KCAT) assesses the foundational knowledge and competency of internationally educated dietitians (IEDs) who are applying for registration in Ontario. The KCAT was developed by Touchstone Institute in collaboration with the College of Dietitians of Ontario (CDO).
The KCAT was adapted for administration on Touchstone Institute’s virtual MCQ exam platform and included a proctoring system.
Goal 5
Administer virtual remotely proctored Objective Structured Clinical Exams (OSCEs)
Virtual OSCE
Proof of concept testing for a virtual OSCE platform began in the summer of 2020. A review of available platforms and technologies resulted in a decision to work with a modified Zoom platform featuring a custom-built dashboard. The IENCAP exam for Internationally Educated Nurses had the most cancelled events, so it made sense that we would focus on how this exam could be fully virtualized to address a growing backlog. Stations were modified, and the simulation team had to train standardized clients on-line to modify their interactions appropriately for the Zoom environment.
The first two virtual administrations of the Internationally Educated Nurses Competency Assessment Program (IENCAP) took place in February and March 2021. This marked the first Virtual OSCEs for Touchstone Institute, with the custom-built dashboard displaying examinee instructions, station props, station timing, and announcements for a 12 station exam.
Each exam accommodated 24 examinees. Only one examinee could not complete all stations because of technical challenges at their remote location. Analysis of psychometric indices showed no overall significant difference in the performance characteristics compared to previous in-person IENCAP administrations.
Many thanks for the opportunity to be on the cutting edge of assessment technology....
- Lead Examiner & Nurse Educator
I do not need to travel and worry where to stay in Toronto if it was in person examination.
- Examinee
This exam was very impressive. I proctored both the virtual and the MCQ. Very well done everyone. I was proud to be a part of it.
- Support Staff
This is a great way of testing candidates- Highly recommend and approve of this.
- Examiner
© 2021 Touchstone Institute
