There are four Principles of Assessment – Reliability, Fairness, Flexibility and Validity. Here we discuss Reliability. This will be followed by additional Blogs which will discuss the remaining Principles of Assessment.
Principle of Reliability
Reliability refers to the consistency of the interpretation of evidence and the consistency of assessment outcomes.
Reliability is one of the four Principles of Assessment. In practice, it means that under the same conditions for the same unit of competency, all assessors should reach the same decision as to whether the candidate is competent, based upon the evidence collected.
Therefore, your assessment tool must provide guidance for the assessor. In reality this means that, for every assessment instrument provided to the candidate, there should be a “sister” instrument for the assessor (the assessor guide). The assessor guide must provide instructions to the assessor to guide their judgement of satisfactory performance or answers to questions. Thus:
- For an oral questioning instrument, you need to provide the questions and the answers expected, together with any necessary guidance on how far the answers can deviate from those provided. There must be a space for the assessor to write the actual answers given and another space for their comments.
- For a written questioning instrument, you also need model answers. Where the answer is likely to vary in wording, this should be stated (Response can vary). Then, instead of a precisely worded answer you can list key points that need to be addressed in the candidate’s response.
Note that for Cert I to Cert III, you are more likely to be able to provide precise written answers. When you are writing assessment guidance for higher AQF levels, the answers are expected to be more complex and key points may be all you can provide.
- For observation of performance, you need an observation form for the assessor to complete during the observation. The form must include the instructions to the candidate and a list of precisely what the assessor should observe during the task. For instance, if it is a pre-start check of an item of plant, what should be checked? Always break the task down into everything the assessor must be able to see.
- Where the instrument relates to performing a task and providing evidence of having done so (e.g. a report) there must still be guidance for the assessor on the key points to look for. It is important to be precise. For instance, if the unit is Cert II and involves typing a letter, are typing, spelling or grammatical errors permitted and if so, what percentage? Alternatively, if the unit is Cert IV and requires the candidate to document a report to the company director, the assessor should be directed to require professional language appropriate to the audience, with correct grammar and spelling.
As trainers and assessors within the Vocational Education and Training (VET) sector, it is important that we not only understand the Principles of Assessment but that we also apply the Principles of Assessment when designing and developing assessment tools and conducting assessment. TAEASS502 Design and Develop Assessment Tools is a core unit in the TAE40116 Certificate IV in Training and Assessment and TAE50116 Diploma of Vocational Education and Training.
Written by Sandy Welton
Please post your thoughts below on the Principle of Reliability.
Reliability is essential to the outcome of competence. Assessment tools that support and guide the assessor to standardise what is required will reduce bias or influence judgment. This leads to consistency and validity of assessment and competence outcomes.
Consistency is important so that all the learners are given equal opportunity and assessed according to the same criteria.
I also agree it is important that assessors come to the same or similar conclusion about a candidate’s competency. Otherwise it is unfair.
Really interesting article. I agree it is vitally important that all assessors are able to come to the same conclusion about a candidate’s competency.
Great article, Sandy. Sets out quite clearly the importance of creating reliable assessment tools and also assessor guides to accompany the tool. I thought the differences in creating the assessment tools/assessor guides depending on the level of the qualification particularly helpful. It can clearly be seen from your points above that the Principle of Reliability is so very important, especially if you are creating an assessment tool that will be used by multiple assessors. By setting out appropriate answers/responses in the assessor guide, this should minimize any interpretation required by assessors and result in the same outcome regardless of who the assessor is.
Thanks Sandy, a great explanation! It just makes sense, hoping it will make sense when I put my pen to the task! I know ‘first time around’ I’ve struggled and got a bit buried in this (as an over thinker!), but reading (& re-reading) reinforces just how ‘airtight’ the 3 principles of Reliability fit together. They should flow to the exact same outcome no matter other ‘influences’ that may be involved. My thoughts go directly to the type of learner the assessor may be and given we are all different in our learning styles, thus are we in our thought processes and approach. As an over thinker I get it! Now to just get over my over thinking!
Certainly highlights the lack of consistency in making assessments that can creep in if clear guidance is not given to the assessor
If the instrument is well designed then this reliability will be possible between assessors in different RTO’s, different geographical locations, different teaching backgrounds and with varying target groups. And the students gain fairness in how they are assessed regardless of variation in background or personality.
I completely agree with this. It is extremely important to have guidance for the assessor to ensure fairness across the board. Where there are multiple assessors in an organisation, it removes personal judgments and teaching styles.
Thank you for the article. A very valuable reminder of the importance of reliability in assessment instruments and for assessor guides, to ensure all assessors are making the same judgements on what is deemed ‘competent’.
The Principle of Reliability ensures that all students have fair and equitable chances of succeeding in their study, without having to worry about differences in teaching styles or approaches, or a teacher’s personality or expectations. The rules are determined and that’s helpful for both teachers and students.
I wish I had read this article prior to submitting my last assignment! (Some points I could have used!)
The Principle of Reliability provides a framework for integrity that supports both the assessor and candidate and further down the line future employers and service recipients. I think it is enormously important that there is reliability in assessment to ensure a uniform baseline for performance, knowledge, skills and wider trust in qualifications.
As the old adage says “consistency is key” and in the field of VET training and assessment, this is absolutely vital. Consistency provides reliability which in turn provides businesses, enterprises, trainers and trainees alike with the confidence that the training and assessment process can be trusted.
This article really highlights the importance in creating instruments for both students and assessors that are detailed and informative. They need to be created with careful and close considerations of all requirements to ensure the reliability of the instruments.
Understanding the Principal of Reliability is extremely important. It gives measure to everyone in understanding what is required and assurances that we should be able to draw on the same outcome.
Thanks for the article Sandy. To summarise, designing assessment requires that questions and answers result in any trainer being able to rely on the instrument to assess the candidate similarly to another trainer including details such as what level of grammar, technical detail etc should be achieved.
Oh wow, maybe everybody gets it but me. I found this piece heavy on jargon and was hoping that for an introductory piece on this task that it could have been put in simpler terms. I read it several times, & it started to clear the fog but still felt there could have been a way to explain it in a more straight forward manner.
Thanks for this article. The clearer the instructions for the candidate and assessor, the easier it will be for everyone. There is fairness in reliability for learners. When designing assessment instruments I need to keep in mind not only the Principles of Assessment but the Standards for RTOs. Having the big picture in mind will help me to express the instructions in a meaningful way. For example, Standard 1 “The RTO’s training and assessment strategies and practices are responsive to industry and learner needs and meet the requirements of training packages and VET accredited courses”. The key idea – responsiveness – should be reflected in learner and assessor guides and the Assessment Plan itself.
The ideas underpinning each Principle and Standard could be woven into the words we choose to explain our assessment instruments and their guide book tools. In this way, reliability becomes an action and ideal not only accountable to the requirements of the training packages and courses, but to a commitment of responsiveness necessary to meet Standard 1. Reliability to me means mapping of a sort to ensure Standards shine through every document we create to collect evidence and assess competency, and that these instruments can reliably be repeated to produce the same assessment results within the same assessment conditions.
This article provided valuable information, helping outline important guidelines that help ensure assessment is consistent and fair. Providing clear boundaries that help the assessor and learner understand the requirements may seem obvious but this can easily be forgotten in the process. I appreciate the recommendations in this article and will take them on board as I use and prepare assessment tools.
Thanks for the great article Sandy. Reliability is so important in VET assessment, so it’s good to go over the key points regularly.
The principle of reliability is one of upmost importance in keeping with the integrity and consistency of the unit of competency. A great deal of information these days is ‘open to interpretation’, so having consistent principals like this embedded in the framework is paramount.
In the sector that I work, the trainees that I work with all have there assessments marked by different assessors. So for me, The Principle of Reliability is extremely important, so that it lays a clear benchmark for what the candidate needs to produce – no matter who is assessing their work – making the assessment process fair for all
I must admit that with my ‘shiny ball syndrome’ I went off on my own to find out how many and what type of Assessment tools existed and found 9 different ones including pre-assessments, they have yet more insights for me to learn about, however the two that Sandy has written about here are well worded and nicely explained. I agree that as senior VET folk we must not only know but apply Principles of Assessment as we develop tools for each of our respective work places. Reliability has earnt its place in the workspace.
The examples that have been given here certainly provided me with a greater understanding of the concept of consistency and therefor useability.
Thank you, Sandy,
p.s this comes from someone one hasn’t studies for some time now.
The Principle of Reliability is one of the key features of VET that makes it so valuable and equitable. Given that different assessors may have varying levels of experience and different preferences about “style”, the requirement that there be clear benchmarks and agreement about acceptable answers should ensure that assessment is of sufficiently high standard across the board.
Thanks for the article reiterating the importance of a reliable instrument for assessor’s to use to guide them to assessing competency in a consistent way.
Reliability in an assessment instrument will see that a students answers are repeatable and thus demonstrate appropriate competence.
Guidance for the assessor in the form of benchmark answers is crucial to ensuring that consistency in assessment is maintained not only by different assessors, by also by the same assessor for different candidates.
Hannah this is very true. It is not just a requirement in order for a tool to pass an ASQA audit. It is essential for reliability and consistency of assessment.
Further, reliable and consistent assessment means that we are being fair and without bias. Everyone is assessed to the same benchmark.
Model answers and/or benchmarks are a great guide to determine a candidate’s knowledge.
Thanks for the article, it’s great to be able to read the information in detail and in-depth.
Great article, and shows how important model answers and benchmarks are in reliability.