TAEDES505A Evaluate a training program

Evaluation varies according to a range of factors. It is the learning itself which is being evaluated, i.e. outcomes are being measured in terms of achieving objectives. Ideally, learning would be being monitored (and perhaps assessed) during the learning process.This allows for changes to be made and errors or deficiencies to be corrected in an incremental way. This method of building on skills and competencies is most effective in learning, as it avoids problems and modifies behaviour in a more beneficial and acceptable way.

If Occupational Health and Safety (OHS) is a strong consideration in the training, monitoring identifies problems and safety issues before they arise. This also enables a more meaningful set of criteria to be developed for the final evaluation of the learning plan.

All organisations vary in their profile or structure. When developing an evaluation model for wider use, the evaluation process should be general enough for use by most, if not all organisations – whether they are for profit, not-for-profit, privately or government-owned. However, in most cases, the evaluation will be to suit one training programme, and as such, it will be much more specific. Alternatively, the evaluation might be used across a large organisation in one functional area (e.g. marketing). The evaluation criteria and process must meet the needs of the individual or organisation that it is designed for.

 

Plan and prepare evaluation

Prior to drawing up a plan for evaluating a training program, the criteria must be determined. The criteria must be accurately identified and the criteria must be prioritized in relation to their power and value i.e. the criteria are evaluated (Bowkett 2007, p.50).

The purposes of the training evaluation must be explored fully

•   What type of training program is it?
•   What is the scale?
•   What are the goals and objectives?
•   Who are the stakeholders?
•   What depth of information is required for an evaluation report?
•   What type of evaluation is sufficient?
•   Who will do the evaluating?
•   What process will be used?

The aim of an evaluation study could involve the following:

•    Evaluating the effectiveness of the training in meeting workforce needs
•    Estimating the cost effectiveness of the training provided
•    Evaluating outcomes of a learning or assessment process or product
•    Determining or trialing new processes and/or products
•    Establishing client satisfaction
•    Ensuring learning/assessment strategies and programs are relevant to client needs and capability requirements
•    Determining resource requirements
•    Providing solutions to organisational performance problems.

 

The scope of the evaluation study may include:

•    Whole of training and/or assessment of the Registered Training Organisation (RTO)
•    Specific training and/or assessment services
•    Specific training and/or assessment products
•    Specific aspects of training and/or assessment systems
•    Partner organisation arrangements
•    Workforce training needs.

Relationships with stakeholders need to be identified in the evaluation plan. The staff or person carrying out the evaluation will have to be selected, briefed and given training in evaluation if necessary

Stakeholders in the evaluation process may include:

•   Employer organisations
•   Training graduates
•   Workplace supervisors and managers
•   Trainers and assessors
•   Training designers
•   Human Resource managers.

 

There are numerous models for evaluation of learning that could be used, or you might adapt one of these models to suit your organisation’s learning programme.

For many organisations the cost-benefit in terms of financial outcomes is most important, but other benefits, such as how much learning took place and how effective that learning was is arguably just as important, in fact, organisational performance is less likely to improve if the learning is not effective.

Collect and collate data and information for the training program which is to be evaluated

To evaluate a training programme, information and data about the training programme itself, including the purpose, must be known, collected and understood. The evaluator will need to:

• Identify the most appropriate techniques for gathering information – will it be quantitative (statistical) or qualitative data and information?
•   Develop the evaluation instruments.
•   Arrange workplace visits and meetings, and get permission to access data and information, where necessary and practical.
•   Collect data and information and store in compliance with the record keeping and privacy policies and procedures of the organisation.

•  Identify and record potentially useful information which is not identified in the evaluation plan.
•   Collate and process data relevant to the evaluation.

Data and information collected may include:

•   Work performance
•   Workplace productivity
•   Employer feedback
•   Trainee feedback.

In collecting information for evaluation, the first decision is how to find out what your need to know. Are there clear Objectives for both the training program and the evaluation (which should be to determine whether the training program objectives were met)? What questions should be asked and of whom? The following are methods or instruments you might use to collect information to address the evaluation criteria (O’Brien 2007):

Surveys: pre-training and post-training (you need to ensure the questions are EXACTLY the same, but the tense will be changed from expectations [before the training] to perceptions [after the training]). Questions will be based on the evaluation criteria, and they must be constructed well to actually produce useful, meaningful, reliable formation upon which decisions (and future changes, including training) can be made.

Questionnaires

Interviews e.g. with course graduates, supervisors or managers, trainers/facilitators and assessors

Mapping (as in using a staff training skills matrix)

Narrative enquiry

Action Research

Case study analysis

Group discussion sessions (e.g. focus groups)

Diaries, journals or logs

Researching organisational performance data e.g.

• Examining organisational documents and systems e.g. Policies and procedures
• Learning and assessment strategies
• Training/assessment records
• Course evaluation questionnaires
• Resources for delivery and assessment including assessment tools.

 

The criteria for evaluation, and the method of evaluation is applied and the results collated. The results (without writers/evaluators comment i.e. findings) are then analysed (i.e. informed and professional comment based on the findings), conclusions drawn and recommendations made.

 

Analyse evaluation data and information, and draw conclusions

An analysis generally means to separate or break up a whole into its parts, to carefully look at each part and to address each of the issues or problems. First, you need to be clear about the situation, then compose a response. Simplify the task by adopting a systematic approach. You should use headings to ensure a clear, coherent and readable analysis.

A case study focuses on one instance or a few instances of a particular phenomenon e.g. an organisation or individual, or aspect or example (or training in this case). It is an in-depth account of events, relationships, experiences or processes occurring in that particular instance (Denscombe 2003).

When drawing final Conclusions, try to leave the reader with your arguments, not just a simple summary of what you have written. For example, what are the most important lessons learned from an analysis?

The process involves:

•  Analysing data and information to identify the outcomes of training and their impact on workforce capability.
• Cross-checking findings where possible by comparing with the results from different evaluation instruments, and/or to organisational or industry benchmarks and best practice.
• Drawing conclusions about the effectiveness (relates to achieving goals/objectives) and efficiency (relates to best use of resources) of the training programme in relation to its aims and scope.
• Documenting areas of training that are satisfactory and those requiring improvement.
• Identifying factors affecting performance and suggest possible enhancements or alternatives to the training programme.

The results of the evaluation are expressed, based on the pre-determined criteria for measurement of success or otherwise, and a judgment or conclusion made.

When monitoring or evaluating learning, do not necessarily avoid the ‘failure’ concept, but use it in a positive way. It is OK to fail, as it is an important part of the learning process. How many really successful people whether in sport or business, have failed before they have become extremely successful?

 

Report on conclusions and make recommendations based on the evaluation

The outcomes of the evaluation must be documented in a report of some kind for management and/or stakeholders. This may be a short report with a summary of findings (learning outcomes, behavioural or innovation change, financial performance etc.) and recommendations, or a much more formal and comprehensive report on a training program. Preliminary findings should be discussed with stakeholders.

Alternatives or complementary activities to the training program which might be suggested include:

•   Analysis of skills requirements and training needs
•   Design of training and assessment strategies
•   Training materials development
•   Training delivery methods and resources
•   Assessment methods and tools
•   Employer and trainee liaison and management.

 

Areas of possible improvement might be:

•   Analysis of skills requirements and training needs
•   Design of training and assessment strategies
•   Training materials development
•   Training delivery methods and resources
•   Assessment methods and tools
•   Employer and trainee liaison and management.