There are four Principles of Assessment – Reliability, Fairness, Flexibility and Validity. Here we discuss Reliability. This will be followed by additional Blogs which will discuss the remaining Principles of Assessment.
Principle of Reliability
Reliability refers to the consistency of the interpretation of evidence and the consistency of assessment outcomes.
Reliability is one of the four Principles of Assessment. In practice, it means that under the same conditions for the same unit of competency, all assessors should reach the same decision as to whether the candidate is competent, based upon the evidence collected.
Therefore, your assessment tool must provide guidance for the assessor. In reality this means that, for every assessment instrument provided to the candidate, there should be a “sister” instrument for the assessor (the assessor guide). The assessor guide must provide instructions to the assessor to guide their judgement of satisfactory performance or answers to questions. Thus:
- For an oral questioning instrument, you need to provide the questions and the answers expected, together with any necessary guidance on how far the answers can deviate from those provided. There must be a space for the assessor to write the actual answers given and another space for their comments.
- For a written questioning instrument, you also need model answers. Where the answer is likely to vary in wording, this should be stated (Response can vary). Then, instead of a precisely worded answer you can list key points that need to be addressed in the candidate’s response.
Note that for Cert I to Cert III, you are more likely to be able to provide precise written answers. When you are writing assessment guidance for higher AQF levels, the answers are expected to be more complex and key points may be all you can provide.
- For observation of performance, you need an observation form for the assessor to complete during the observation. The form must include the instructions to the candidate and a list of precisely what the assessor should observe during the task. For instance, if it is a pre-start check of an item of plant, what should be checked? Always break the task down into everything the assessor must be able to see.
- Where the instrument relates to performing a task and providing evidence of having done so (e.g. a report) there must still be guidance for the assessor on the key points to look for. It is important to be precise. For instance, if the unit is Cert II and involves typing a letter, are typing, spelling or grammatical errors permitted and if so, what percentage? Alternatively, if the unit is Cert IV and requires the candidate to document a report to the company director, the assessor should be directed to require professional language appropriate to the audience, with correct grammar and spelling.
As trainers and assessors within the Vocational Education and Training (VET) sector, it is important that we not only understand the Principles of Assessment but that we also apply the Principles of Assessment when designing and developing assessment tools and conducting assessment. TAEASS502 Design and Develop Assessment Tools is a core unit in the TAE40116 Certificate IV in Training and Assessment and TAE50116 Diploma of Vocational Education and Training.
Written by Sandy Welton
Please post your thoughts below on the Principle of Reliability.
Thanks for the article Sandy. To summarise, designing assessment requires that questions and answers result in any trainer being able to rely on the instrument to assess the candidate similarly to another trainer including details such as what level of grammar, technical detail etc should be achieved.
Oh wow, maybe everybody gets it but me. I found this piece heavy on jargon and was hoping that for an introductory piece on this task that it could have been put in simpler terms. I read it several times, & it started to clear the fog but still felt there could have been a way to explain it in a more straight forward manner.
Thanks for this article. The clearer the instructions for the candidate and assessor, the easier it will be for everyone. There is fairness in reliability for learners. When designing assessment instruments I need to keep in mind not only the Principles of Assessment but the Standards for RTOs. Having the big picture in mind will help me to express the instructions in a meaningful way. For example, Standard 1 “The RTO’s training and assessment strategies and practices are responsive to industry and learner needs and meet the requirements of training packages and VET accredited courses”. The key idea – responsiveness… Read more »
This article provided valuable information, helping outline important guidelines that help ensure assessment is consistent and fair. Providing clear boundaries that help the assessor and learner understand the requirements may seem obvious but this can easily be forgotten in the process. I appreciate the recommendations in this article and will take them on board as I use and prepare assessment tools.
Thanks for the great article Sandy. Reliability is so important in VET assessment, so it’s good to go over the key points regularly.
The principle of reliability is one of upmost importance in keeping with the integrity and consistency of the unit of competency. A great deal of information these days is ‘open to interpretation’, so having consistent principals like this embedded in the framework is paramount.
In the sector that I work, the trainees that I work with all have there assessments marked by different assessors. So for me, The Principle of Reliability is extremely important, so that it lays a clear benchmark for what the candidate needs to produce – no matter who is assessing their work – making the assessment process fair for all
I must admit that with my ‘shiny ball syndrome’ I went off on my own to find out how many and what type of Assessment tools existed and found 9 different ones including pre-assessments, they have yet more insights for me to learn about, however the two that Sandy has written about here are well worded and nicely explained. I agree that as senior VET folk we must not only know but apply Principles of Assessment as we develop tools for each of our respective work places. Reliability has earnt its place in the workspace. The examples that have been… Read more »
The Principle of Reliability is one of the key features of VET that makes it so valuable and equitable. Given that different assessors may have varying levels of experience and different preferences about “style”, the requirement that there be clear benchmarks and agreement about acceptable answers should ensure that assessment is of sufficiently high standard across the board.
Thanks for the article reiterating the importance of a reliable instrument for assessor’s to use to guide them to assessing competency in a consistent way.
Reliability in an assessment instrument will see that a students answers are repeatable and thus demonstrate appropriate competence.
Guidance for the assessor in the form of benchmark answers is crucial to ensuring that consistency in assessment is maintained not only by different assessors, by also by the same assessor for different candidates.
Hannah this is very true. It is not just a requirement in order for a tool to pass an ASQA audit. It is essential for reliability and consistency of assessment.
Further, reliable and consistent assessment means that we are being fair and without bias. Everyone is assessed to the same benchmark.
Model answers and/or benchmarks are a great guide to determine a candidate’s knowledge.
Thanks for the article, it’s great to be able to read the information in detail and in-depth.
Great article, and shows how important model answers and benchmarks are in reliability.