Administration of RELTA

Academic Team

The Academic Team is responsible for the development and validation of RELTA. The Academic Manager oversees the development work undertaken by the Senior Curriculum Writer (Aviation), the Senior Assessment Developer and consulting aviation subject matter experts. The Senior Curriculum Writer (Aviation) is responsible for setting the standards for and maintaining the integrity of test items.

Aviation Operations Team

The Aviation Operations Team is responsible for ensuring the quality control of RELTA and other Aviation English products, supporting customers, and delivering tests and courses in conjunction with our partners worldwide. The Team’s responsibilities are outlined in the agreement between RMIT Training and each RELTA test centre partner.

The Team also oversees protocol and procedure, security, record keeping and finalisation of results, and administers and provides accreditation for all examiners and raters. Any breaches of protocol that may occur during test delivery are immediately reported to the Team by raters via our secure records management database.

The Team oversees regular audits of test centres to ensure compliance. It also organises, administers and maintains documentation on the training, reaccreditation and auditing of RELTA examiners and raters, as applicable.

Obligations of our partners

To obtain and maintain the right to deliver RELTA, partners must comply with the RELTA Test centre administration manual and our quality assurance review.

The RELTA Test centre administration manual outlines the:

• requirements of managing the delivery of RELTA
• operational policies, systems and procedures
• quality assurance review processes and standards.

Partners are required to follow the RELTA Test centre administration manual at all times to ensure the quality of test delivery.

Our quality assurance review process monitors compliance and helps partners identify areas that may need improvement. This process, designed as a checklist, allows RMIT English Worldwide and our partners to assess compliance with the key responsibilities of each party and ensure that standards for maintaining quality in the delivery of RELTA are being met. All aspects of test delivery are included in the standards, including personnel and their responsibilities, test administration, security and facilities. During the review process, objective information is collected to determine the status of systems, areas of responsibility and general areas of quality. Areas that require attention are identified so that preventative and remedial action can be taken.

Test centre personnel

All personnel involved with RELTA must sign the RELTA Code of conduct acknowledging their responsibilities in relation to test integrity and security.

Test centre managers and administrative personnel are responsible for:

• carrying out administrative duties according to the RELTA Test centre administration manual
• liaising between the test centre and RMIT English Worldwide regarding test dates and candidate numbers
• responding to enquiries
• coordinating, registering and processing candidate applications.

Training in the use of the secure records management database is provided to selected test centre personnel prior to the test centre becoming operational.

RELTA examiners

RELTA examiners are responsible for administering the Listening Test and Speaking Test. Examiners may also be accredited raters.

Examiners must be suitably qualified and are required to follow the RELTA Code of conduct, which includes the signing of a confidentiality agreement. All examiners undergo training in RELTA and, if applicable, radiotelephony induction before they can administer RELTA. Examiners are subject to regular reviews and, to maintain their accreditation, are required to participate in recurrent RELTA training. Examiners must be reaccredited every 12 months.

All training for examiners is conducted by fully accredited RELTA trainers. Test centre managers may be trained to deliver training to examiners in order to maintain a sustainable pool of local examiners. Training packs and schedules have been developed to ensure training is delivered thoroughly and consistently in all locations around the world.

RELTA raters

RELTA raters are responsible for rating the performance of candidates after the Speaking Test has been administered. Accredited raters may also be examiners.

Raters must be suitably qualified and are required to follow the RELTA Code of conduct, which includes the signing of a confidentiality agreement. All raters undergo training in RELTA and, if applicable, radiotelephony induction before they can rate RELTA.

Raters are required to undergo a review and be reaccredited in turn every year, i.e. review (2010), reaccreditation (2011), review (2012) and so on. The performance of raters is also subject to regular monitoring and auditing to ensure consistent application of ICAO rating levels as well as intra- and inter-rater reliability. If necessary, coaching and re-training is provided to raters found not to be conforming to required standards.

All training for raters is conducted by fully accredited RELTA trainers. All training of raters is conducted by senior RELTA representatives. Training packs and schedules have been developed to ensure training is delivered thoroughly and consistently in all locations around the world.

Test format

RELTA comprises a Listening Test and a Speaking Test. Both tests must be completed.

• The Listening Test is delivered by a computer and takes 35–50 minutes.
• The Speaking Test is delivered by an examiner and takes 25–30 minutes.

RELTA Listening Test

There are three sections in the Listening Test.

• In Section 1, candidates hear pilots and air traffic controllers talking about routine situations. Candidates answer multiple choice questions based on these conversations.
• In Section 2, candidates hear pilots and air traffic controllers talking about non-routine situations. Candidates type their answers to questions based on these conversations in the spaces provided.
• In Section 3, candidates hear aviation personnel debriefing after an incident has occurred. Candidates answer multiple choice questions based on this conversation.

Accents used in the Listening Test include American, Australian, Brazilian, British, Canadian, Filipino, Indian, Indonesian and Italian.

Candidates can familiarise themselves with the test format by using the free RELTA Listening Practice Test. Support materials include answers and transcripts.

RELTA Speaking Test

There are three sections in the Speaking Test.

• In Section 1 and Section 2, the candidate communicates with the RELTA examiner using phraseology and plain English. The candidate’s ability to communicate is evaluated (not their use of ICAO standardised phraseology).
• In Section 3, the candidate communicates using plain English in a face to face conversation with the RELTA examiner. This is the candidate’s opportunity to talk about the non-routine situation presented in Section 2, and to discuss their experiences, ideas and opinions on aviation related topics. The candidate’s ability to communicate is evaluated (not their aviation knowledge).

Candidates can familiarise themselves with the test format by using the free RELTA Speaking Practice Test. Support materials include sample answers and transcripts.

Sitting RELTA

Scheduling

Test centre managers are responsible for scheduling tests according to the availability of resources and internal requirements, and notify the Aviation Operations Team at RMIT English Worldwide of the test days, type of test and number of candidates per test by creating classes and assigning candidates in the secure records management database. (For more information, please see Records management.)

Test centre managers are also responsible for assigning RELTA examiners to the test and checking that the test rooms are set up and equipment is operating correctly.

Records management

RMIT English Worldwide uses its own Student Results Management System (SRMS) to:

• add candidates to RELTA test sessions
• monitor candidates enrolled in test sessions
• release randomly assigned test versions to registered candidates
• produce results and certificates that can be assesssed by test centre managers and provided to RELTA candidates
• keep a permanent record of all test takers and results.

The SRMS is a web enabled application that has been specifically tailored for RELTA.

Test centre managers or administrators ensure that candidates’ details are correctly entered into the SRMS. Mandatory information required by the system includes each candidate’s given name, family name, date of birth and licence number.

Assessment and scoring

RELTA Listening Test

As the Listening Test is delivered by computer, scores are assigned automatically and uploaded to our secure database. Where there is a query in a response, members of the Academic Team are alerted to determine the result.

RELTA Speaking Test

Candidates’ sound files are recorded and uploaded to our secure database. Two RELTA raters are assigned to assess one test. Each rater listens to the candidate’s recording, rates each speech sample and enters their scores for each section of the test separately. Raters listen to each speech sample as many times as they need to before entering a score. The database then automatically crosschecks the scores given by each of the two raters and detects any differences that would alter the candidate’s overall scores. When this occurs, a third rater is assigned to determine the overall score.

Results

RELTA results are made available once all assessment processes have been completed, and can be provided to the local civil aviation authority, airline or air navigation service provider by the test centre manager as required. RMIT English Worldwide can arrange for these organisations to review results online if required. Candidates can also access their results online by entering their date of birth and licence number.

Certificates (known as RELTA Results Statements) can be obtained by the test centre manager once results are finalised. RMIT English Worldwide does not issue certificates.

Complaints and appeals process

Test centre managers are required to inform all candidates about the complaints and appeals process.

Making a complaint

Candidates who wish to lodge a complaint concerning the delivery of RELTA should complete a RELTA Candidate complaint form and submit it to the test centre manager, who then conducts an investigation of the complaint. The test centre manager must notify the Aviation Operations Team at RMIT English Worldwide about the complaint and of the result of the investigation.

Lodging an appeal

Candidates who wish to request a reassessment of their performance in RELTA may lodge an appeal by completing the RELTA Candidate appeals form. The form must be signed by the test centre manager of the test centre where the candidate sat the test. Once it is completed, the form is then sent to the Aviation Operations Team at RMIT English Worldwide.

All appeals must be received by the Aviation Operations Team within 21 days of the RELTA result being made available to the candidate.

Candidates are required to pay a fee of A$75 to lodge the appeal, which will be refunded if the appeal is upheld by the independent assessor.

Test validity, reliability and security

Validity

The validity of a test is achieved when the test instrument accurately measures what it intends to measure. RELTA intends to measure a range of abilities in speaking and understanding English in an aviation context. These abilities can be described by functionality, such as requesting permission and giving information, and by lexical topics and themes, such as weather and geography. In order to measure these abilities, RELTA tasks are written using language that closely resembles the language used in aviation, such as in radiotelephony exchanges between pilots and air traffic controllers, and in debriefing sessions with a supervisor. Test design ensures that RELTA tasks and content accurately represent the target language use domain; that is, the setting in which this language is used.

Based on the evidence of a number of studies conducted by RMIT English Worldwide, RELTA has been found to be a valid test of Aviation English. In 2014, the RELTA Validity Argument Framework was presented to ICAO. This framework provided evidence- and theory-based investigations of RELTA, including how domains are defined, how tasks perform, methods of scoring and interpreting scores. In 2015, RMIT English Worldwide conducted an analysis of the Speaking Test to determine the extent to which the test elicits language commonly found in aviation contexts. The results showed that the visual prompts used in the test, which guide test takers from one scenario to the next, elicited a sufficient range of both functions and lexis.

Before releasing any RELTA test, RMIT English Worldwide arranges for both native and non-native English speaking pilots to trial the test, and then provide feedback on how relevant and accessible the scenarios and tasks are, and how difficult they found the questions. This feedback is used to revise and improve tests before their release.

Reliability

To achieve a high degree of reliability, RMIT English Worldwide ensures that all RELTA raters are suitably qualified and trained to assess plain English used in aviation contexts. It is compulsory for all raters to be reaccredited biennially. All Speaking tests are rated separately by two raters, with an experienced third rater used in cases where there is disagreement between the first two raters. All Listening tests are statistically analysed and reliability is measured and evaluated. Tests that do not meet minimum levels of reliability are revised accordingly. The performance of raters is regularly monitored and analysed to identify both under- and high-performing raters. Any underperforming raters are required to undergo further training.

Security

The security of RMIT English Worldwide’s intellectual property and RELTA test materials is strenuously upheld at all times. Any person involved in the delivery of RELTA must be registered with the test centre manager and have signed the RELTA Code of conduct undertaking to maintain security and confidentiality in relation to test administration and records management. Overall responsibility for the security of the test centre rests with the test centre manager supported by the examiners and any other personnel involved in the administration of RELTA.

If there is a security breach at a test centre, the test centre manager is required to immediately notify the Aviation Operations Team at RMIT English Worldwide.

Versions of RELTA are randomly assigned from a pool of over 80 tests for each test type and are protected by a software protocol. The online delivery system is fully protected to ensure the security of test versions.

Candidates found cheating on any test will not be allowed to continue the test. Activities considered as cheating include:

• looking at another candidate’s computer screen during the Listening Test
• talking to another candidate during testing
• taking any written notes into or out of the test room
• using any aid.

The test centre manager is required to immediately notify the Aviation Operations Team at RMIT English Worldwide if such an incident occurs. Candidates found cheating are not issued a result and are reported to their organisation’s management.

Test design

RELTA has been designed in response to the ICAO Language Proficiency Requirements as outlined in the Manual on the Implementation of ICAO Language Proficiency Requirements (Document 9835) and Annexes 1, 6 and 11 of the ICAO Standards and Recommended Practices. (For more information, please see the ICAO website.)

According to Document 9835, a test used to assess language for ICAO should:

• be aviation specific (not assess general English proficiency) and be related to the language of radiotelephony which pilots and air traffic controllers use in their work
• assess language based on the six core language skill areas identified by ICAO—pronunciation, structure, vocabulary, fluency, comprehension and interactions
• not assess a knowledge of phraseology, but must assess communicative ability in both voice only and face to face contexts
• assess listening and speaking skills directly (not assess language knowledge or reading and writing ability).

RELTA is practical. It is easy to administer, with simple and efficient pre-test, in test and post-test administration procedures in place to facilitate test delivery, security and reporting efficiency.

The structure of RELTA, the forms of language assessed and the rating processes followed all reflect ICAO’s Language Proficiency Requirements and Standards and Recommended Practices. RELTA is dedicated to assessing communicative proficiency in aeronautical radiotelephony communications.

Test development

The test development team consists of specialists in the following areas:

• subject matter expertise (pilots and air traffic controllers, specialist aviation consultants with dual pilot and ATC experience)
• testing (experts with post graduate qualifications in applied linguistics and psychology)
• test item writing (qualified and experienced curriculum and assessment writers)
• test development (instructional and graphic designers, editors and audio producers).

Test situations are fictitious but based on real life incidents and accidents to elicit more complex plain language. Scripts are drafted by aviation subject matter experts to ensure authenticity and plausibility. Test items are then written by experienced assessment writers who specialise in developing RMIT English Worldwide’s Aviation English testing materials. Item writing checklists have been developed to maintain a consistent approach to test development. Our subject matter experts review all stages of test development to ensure plausibility is maintained throughout the process.

RELTA test impact

The RELTA test construct samples a wide, yet defined, range of communicative tasks, skills and content that are representative of aviation work contexts in its three streams:

• Pilots Heavy (IFR)
• Light Aircraft (VFR)
• ATC.

To study and prepare for RELTA is tantamount to developing relevant and critical communication skills in situations that are familiar to the candidates. Little benefit would be gained from a preparation plan that focused on testing strategies or discrete components of linguistic knowledge. RELTA is expected to provide positive washback in that the test design itself promotes effective and valid learning practices in society, particularly educational environments concerned with the improvement of language proficiency for pilots and air traffic controllers. 

RELTA Code of Ethics

RELTA language testers comply with the Code of Ethics adopted by the International Language Testing Association. The nine fundamental principles of the Code inform the RELTA Code of Conduct and are manifest through rigorous standards of test security, validity, transparency and candidate confidentiality. These standards are supported by our ongoing professional development and insistence on conscionable language testing practice.

RELTA test security is manifold and applied without exception. Validity is pursued through:

• trialling
• analysis
• engagement of SMEs
• responsiveness to feedback
• monitoring activities
• currency with language testing research
• test security.

Transparency for test takers is achieved through:

• practice tests and test information on the RMIT English Worldwide website
• RELTA’s adherence to ICAO Document 9835
• test information printed in trialled, graded language.

Transparency for the participants, whose participation in research activities is always voluntary, is achieved through clear, accessible statements of the purpose and scope of those activities. Candidate confidentiality is ensured through the analysis of all trial and test results in aggregate; uniform blind rating; and strict information management. These standards and their realisation through the Code of Conduct promote equity between all test takers, the protection of their dignity, and the just allocation of RELTA services; proscribe malfeasance among RELTA personnel; and, more broadly, contribute to the integrity of the language testing profession.

Administration of RESET

RESET assessors observe pilots and air traffic controllers at work to determine whether the items contained in the RESET checklist of English language proficiency have been satisfied for ICAO Level 6 (Expert) competency. Supplementary tasks are provided in circumstances where additional evidence of language performance is required to satisfy ICAO’s requirements of proficiency in plain English. These tasks are included in the RESET Assessor handbook.

Assessors are trained to conduct the observation and complete the checklist, and to deliver the supplementary tasks if required.

RMIT English Worldwide accredits all approved organisations and assessors to administer RESET.

Certification

All results are forwarded to RMIT English Worldwide through a secure records management database, then certified by RMIT English Worldwide and officially released to parties concerned. Results can be presented either as individual RESET certificates or a RESET performance assessment report.

Quality assurance

To maintain their accreditation, RMIT English Worldwide conducts regular audits of RESET organisations by agreement to ensure assessors are consistently applying the RESET checklist and delivering the supplementary tasks. RESET assessors are also required to assess speech samples provided by RMIT English Worldwide to maintain assessor reliability.