Department of the Army

 TRADOC Pamphlet 350-70-10

Headquarters, United States Army

Training and Doctrine Command

Fort Monroe, Virginia 23651-1047

29 March 2004

Training

SYSTEMS APPROACH TO TRAINING

COURSE AND COURSEWARE VALIDATION

Summary

This pamphlet provides guidance on conducting training course and courseware validations.  It explains the validation role within the Systems Approach to Training (SAT) process and provides detailed procedures for conducting course and courseware validations.

Applicability

This pamphlet applies to U.S. Army Training and Doctrine Command (TRADOC) activities, and The Army School System (TASS) Training Battalions responsible for managing or performing Training Development (TD) or TD-related functions, including evaluation/quality assurance of the training, products, and institutions that present the training.  It also applies to non-TRADOC agencies/organizations having Memorandums of Understanding, Memorandums of Agreement, and contracts for developing training or training products for TRADOC and TASS agencies and organizations.

"R" Forms

The "R" form at the back of this pamphlet is for local reproduction.

Suggested

The proponent for this pamphlet is the Deputy Chief of Staff for

improve-

Operations and Training (DCSOPS&T).  Send comments and suggested

ments

improvements on DA Form 2028 (Recommended Changes to Publications and Blank Forms) through channels to Commander, TRADOC (ATTG-CD), 5 Fenwick Road, Fort Monroe, VA 23651-1049.  Suggested improvements may also be submitted using DA Form 1045 (Army Ideas for Excellence Program (AIEP) Proposal). 

Availability

This publication is distributed solely through the TRADOC Homepage at http://www.tradoc.army.mil/tpubs/pamndx.htm.  It is also available on the Training Development and Analysis Directorate homepage at http://www-dcst.monroe.army.mil/tdaa/.

Purpose

1-1

References

1-2

Explanations of abbreviations and terms

1-3

Systems Approach to Training Overview

1-4

Regulation, pamphlet, and job aids (JAs) relationship

1-5

Validation overview

1-6

Quality control criteria

1-7

Validation management overview

2-1

Validation planning

2-2

Validation team personnel and roles

2-3

Validation team training

2-4

Validation management tasks

2-5

Coordinate validations quality control criteria

2-6

Criticality standards overview

3-1

Learning objective criticality standard description

3-2

Establish criticality standards tasks

3-3

Products of the rating task

3-4

Quality control criteria for criticality standards development

  3-5

Content validation overview

4-1

Content validation description

4-2

Content validation tasks

4-3

Content validation products

4-4

Content validation quality control criteria

4-5

Chapter 5

Individual Trials

Individual trials overview

5-1

Individual trials description

5-2

Individual trials tasks

5-3

Individual trials products

5-4

Individual trials quality control criteria

5-5

Chapter 6

Group Trials

Group trials overview

6-1

Group trials description

6-2

Group trials tasks

6-3

Group trials products

6-4

Group trials quality control criteria

6-5

Chapter 7

Operational Tryouts

Operational trials overview

7-1

Operational tryouts description

7-2

Operational tryouts tasks

7-3

Operational tryouts products

7-4

Operational tryouts quality control criteria

7-5

Chapter 1

Introduction

1-1.

Purpose.  The purpose of this pamphlet is to provide detailed how-to guidance on the following areas of the validation process for U.S. Army Training and Doctrine Command (TRADOC) courses and courseware:

    a.  Establishing validation requirements.

    b.  Planning course and courseware validations.

    c.  Conducting the four different types of course and courseware validations.

    d.  Completing the follow-on actions required for ensuring validation findings result in improved courses and courseware.

1-2.

References.  The references for this pamphlet appear in appendix A.

1-3.

Explanations of abbreviations and terms.  Abbreviations and terms appear in the glossary of this publication.

1-4.

Systems Approach to Training (SAT) overview.

    a.  In accordance with AR 350-1, the Army's training development (TD) process is the SAT process.  The SAT process is a systematic, iterative, spiral approach to making collective, individual, and self-development education/training decisions for the Army.  It determines whether or not training is needed; what is trained; who needs the training; how, how well, and where the training is presented; and the training support/resources required to produce, distribute, implement, and evaluate the required education/training products.

    b.  Training development is a vital component of TRADOC's mission to prepare the Army for war.  As such, it is the responsibility of every civilian and soldier in management and training-related roles in the TRADOC headquarters, schools, field units, and supporting contractor offices.  Management at all levels needs to have a working knowledge of the process, and ensure its efficient implementation.  Doing so will save scarce resources:  personnel, time, process, and unnecessary product development dollars.  The overview in TRADOC Pam 350-70-4 provides the context for producing successful TD projects.

1-5.

Regulation, pamphlet, and job aids (JAs) relationship.  This pamphlet supports and provides how-to guidance for the policy established in TRADOC Reg 350-70, chapter III-3.  The regulation directs the use of this pamphlet in the planning and conduct of course/courseware validations.  Job aids also support this document.  Print the pamphlet and JAs as individual files, or a single document.

Supporting JAs

    a.  Figure 1-1 depicts the relationship of this pamphlet and supporting JAs with TRADOC Reg 350-70.

Pamphlet organization

    b.  Figure 1-2 shows how this pamphlet is organized.  Some chapters are supported by guidance provided in other chapters; refer to each of these to accomplish validation.  The following JAs contain helpful procedures, templates, samples, and other information:

Associated JAs

        (1)  TRADOC Form 350-70-10-1-R-E (Validation Volunteer Background Data Sheet).

        (2)  JA 350-70-10.2, Validation Planning Considerations Worksheet.

        (3)  JA 350-70-10.3, Criticality Standards for Rating Learning Objectives.

        (4)  JA 350-70-10.4, Course/Courseware Content Validation Data Collection Sheet.

        (5)  JA 350-70-10.5a, Validation Volunteer Data Collection sheet for Individual Trials.

        (6)  JA 350-70-10.5b, Instructor/Observer Course Data Collection Sheet for Individual Trials.

        (7)  JA 350-70-10.5c, Validation Volunteer Interactive Multimedia Instruction (IMI) Courseware Data Collection Sheet for Individual Trials.

        (8)  JA 350-70-10.5d, Validation Instructor/Observer IMI Courseware Data Collection Sheet for Individual Trials.

        (9)  JA 350-70-10.6a, Validation Course/Courseware Volunteer Sheets for Group Trials.

        (10)  JA 350-70-10.6b, Validation Course/Courseware Instructor/Observer Course Data Collection Sheet for Group Trials.

        (11)  JA 350-70-10.6c, Validation Volunteer IMI Courseware Data Collection Sheet for Group Trials.

        (12)  JA 350-70-10.6d, Validation Instructor/Observer IMI Courseware Data Collection Sheet for Group Trials.

        (13)  JA 350-70-10.6e, Sequential Validation.

        (14)  JA 350-70-10.7, Validation Courseware Data Collection Sheets for Operational Tryouts.

1-6.

Validation overview.  Table 1-1 describes the validation activities.  Validation is a specialized part of the evaluation function.  It is generally performed during the design and development phases of the training development process.

Quality control/

quality assurance function

    a.  Validation performs a quality control/quality assurance (QA) function.  By eliminating the deficiencies identified by the intense scrutiny that the products are given during validation, effective, efficient training is produced.

        (1)  What the military world calls validation, the academic world normally refers to as formative evaluation, field test, pilot test, or alpha test.

        (2)  While sometimes distinctions are made to indicate how these terms are different, the final goal of all of them is to ensure that the training product actually trains as intended.

        (3)  An instructional unit is validated at the objective level.  Validate that the lesson (instructional unit) teaches the Terminal Learning Objective (TLO), as well as all of the enabling objectives.

Validation definition

    b.   Validation is the process that determines if-

        (1)  Training products and supporting materials perform as intended.

        (2)  The course/courseware complies with all applicable policy/guidance.

        (3)  Personnel receiving the training can perform the objective to standard.

Test validation

    c.  Test validation is a special type of validation.  The test instrument is produced during the design phase of SAT to ensure efficient and effective training.  Design, develop, and validate the test as soon as the objectives are written. 

Note:  A validated test instrument is needed prior to conducting any course/courseware validation trials.

Critical activities

    d. Validation scrutinizes all aspects of the course/courseware.  Since it is impractical and impossible to conduct an intense analysis of all aspects of a course or courseware at one time, focus on specific aspects throughout the development process.  The five critical validation activities include:  establish criticality standards, content validation, individual trials, group trials, and operational tryouts.  The following paragraphs, and chapters 3 through 7, focus on the four types of validation.

        (1)  During content validation, analyze the information in the instructional materials, to determine if it is technically, tactically, doctrinally, and stylistically accurate, appropriate, and applicable for the intent of the training.  Optimally, conduct content validations immediately after the material in each learning activity is drafted.  Therefore, perform content validation incrementally.

              (a)  Correct problems with the content as soon as they are identified, and conduct a new content validation, to verify that no additional problems have been introduced.

              (b)  Eliminate all problems with content, prior to going to individual trials.  Taking products to individual or group trials, that contain content problems, can cause very inefficient training development, which may result in course/courseware that does not train accurate doctrine.

              (c)  Chapter 4 addresses content validation in detail.

        (2)  During individual trials, determine if the product does actually provide the mandated training.  Analyze how effective the individual learning activities are, and how well they fit together to accomplish an objective.

              (a)  Conduct an individual trial of an objective, as soon as it is drafted, to quickly eliminate contradictory, ineffective, and/or inefficient training.  If there are multiple objectives, there are multiple individual trials.

              (b)  If a major problem is found during individual trials, revise and conduct a new trial, to ensure that no additional problems were introduced during the revision process.  Eliminate all problems at the objective level, prior to going to the expense of conducting group trials.

              (c)  Chapter 5 addresses individual trials in detail.

        (3)  Group trials focus on how well the students learn the terminal objective, and how well the various learning activities support the mastery of the TLO.  This validation activity produces the statistical verification that the lesson provides valid training.

              (a)  Conduct a group trial promptly after drafting the lesson, to eliminate contradictory, ineffective, and/or inefficient training.  Correct problems found at the lesson level, and conduct a new group trial.

              (b)  Again, eliminate problems prior to going to the expense of conducting operational tryouts, and exposing actual students to inaccurate or ineffective training. 

              (c)  Chapter 6 addresses group trials in detail.

        (4)  An operational tryout provides a final QA confirmation, prior to reproducing, distributing, and implementing the course.  Sometimes the independent objectives are great independently.  However, when independent objectives are combined with other objectives, it can produce bad training results.

              (a)  Operational tryouts are critical when developing longer courses, since they provide an opportunity to assess how well the entire course works.  This type of validation assesses the quality of the course as a whole. 

              (b)  Conduct an operational tryout as soon as a draft of the course is complete, to eliminate any last minute problems not previously identified.  Eliminate problems prior to going to the expense of reproducing the course/courseware. 

              (c)  Chapter 7 addresses operational tryouts in detail.

Validation records

    e.  Each of the validation activities produces a set of records that documents the results of the validation activity, the decisions made based on those results, and the actions taken to rectify any problems found during the validation.  It is important that all these records are maintained.  The records consist of the worksheets used for the trial; the questionnaires/interview sheets the volunteers and instructors/
facilitators/observers completed; summary sheets; validation reports; and a complete list of the changes made, based on the validation results.  The records provide a starting point for the next revision.  They also provide proof that the training is valid, in case the quality of the training is questioned.  In order to get recognition of the courses from the American Counsel on Education, provide validation records.

1-7.

Quality control criteria.  Each chapter in this pamphlet includes quality control criteria for the validation products and tasks, in the form of standards and performance measures.  If these requirements are met, then all levels of management are assured that the products validated by the validation tasks provide quality training.

Chapter 2

Validation Management

2-1.

Validation management overview.  A smooth running validation is the result of a significant amount of management and advance planning.  This chapter provides how-to guidance on managing the process of validating courses and courseware, to include planning the validation process, establishing a validation team, and identifying resources.

2-2.

Validation planning.  Planning provides the information needed to develop a realistic estimate of the resources required, to accomplish validation activities, establish milestones, and allocate the available resources for the effort.  Do not conduct validation in isolation.  Since this training development process involves both manpower and resources, it involves a number of planning documents.  The first documents that address validation requirements are the Training Development Project Management Plan (TDPMP) for specific course/courseware, and the Training Development Plan (TDP) of the organization.  Additionally, ensure the validation is included in the Master Evaluation Plan (MEP), including specific validation requirements and details.  Also, if a contractor is developing the course/courseware, include the validation requirements in the statement of work (SOW).  The contractor specifies how the validation, identified in the Training Evaluation Document (TED), is accomplished.  Ensuring that validation is addressed, in all the appropriate documentation, also ensures that resources and procedures are in place, to guarantee the training developed is effective and efficient.

Project management plan

    a. When developing the initial course/courseware TDPMP, estimate manpower and workload resource requirements for validation.  Update the TDPMP as the supporting validation plan is firmed up.  Additionally, ensure this information is reflected in the TDP.

Master Evaluation Plan

    b.  Since validation is part of evaluation, the proponent MEP for the year (in which the validation is conducted) should include the validations, or a TDPMP may exist specifically for performing evaluations.  Do not duplicate the validation resource requirements, if they are included within the course/courseware project management plan.  However, include the validation in the MEP timelines.

Validation Plan

    c.  Additionally, develop a detailed Validation Plan for the project.  This document identifies the specific validation activities and resource requirements to support it, including timelines, the products, and the responsibilities for all parties involved.  Use JA 350-70-10.2 for assistance.

Operability testing

    d.  At some point in the validation process, confirm that the materials will operate in a Digital Training Facility (DTF).  Use of a DTF requires scheduling the time to use the DTF, and obtaining user identification and password for use of the computers in the DTF.  Users should register prior to the trials.  Assignment of user name and password takes approximately an hour for each person, if the DTF is not busy.  This user identification and password allows use of any DTF worldwide. 

Note:  Not all training products are designed for use in a DTF, for example, aviation simulators. 

Statement of work

    e.  If the development of the course/courseware is contracted, the SOW identifies how the course/courseware is validated.  Customize the SOW and the Data Item Description (DID) DI-SESS-81524B so the contractor knows and can respond to the validation requirements.  In the TED, the contractor addresses how the validation requirements, specified in the SOW and the DID, are satisfied.  The TED provides the formal feedback on the results of the evaluation process, and specifies the actions the contractor takes to improve the course/courseware materials.

Exceptions

    f.  Before deciding to deviate from the validation model shown in table 1-1, consider the implications of dropping any of the identified requirements.  Often it is very tempting to shortcut the validation process, in an attempt to control costs, or keep a project on schedule.  However, shortcuts jeopardize the success of the entire training development project.  Integrating the validation process, throughout the training development project, builds in the necessary level of quality control, to ensure the final training is valid and reliable.  Validation planning is like planning for any tactical operation-the time and thought invested in this initial planning is returned in the smooth operation of the entire validation process.

Risks of not conducting validations

    g.  Table 2-1 identifies some of the consequences of trying to eliminate any of the validation activities.

Table 2-1

Validation risks

Validation

Activity

Consequences of eliminating a validation activity:

Advantages

Disadvantages

Planning

No advantage is gained if not done.

Having insufficient resources to conduct the validation:

·         Funds.

·         Personnel.

·         Time.

·         Increased costs.

·         Alienation of support personnel.

·         Inability to complete TD project on time.

Content validation

No advantage is gained if not done.

·         Lengthens the development time line if content is not correct.

·         Incurs additional costs to correct late in project.

·         Reduces quality of final training product.

Criticality standards

No advantage is gained if not done.

·         Possibility of increasing cost due to too high a standard.

·         Possibility of inadequate training due to too low a standard.

Individual trials

·         Minimal possible timesaving.

·         Not having to find volunteers with the critical characteristics of the target audience.

·         Lengthen the development time line by not eliminating problems early.

·         Incur additional costs to redo learning activities that do not work; can cause changes to the rest of the course.

·         Reduce quality of final training product.

Group trials

·         Minimal possible timesaving.

·         Not having to find volunteers with the critical characteristics of the target audience.

·         Training one or more entire classes using invalid training materials (which can have legal implications).

·         Giving students credit for completing ineffective training.

·         Sending soldiers to the field, without the training they need to survive and win on the battlefield, or perform the critical tasks of their Military Occupational Specialty (MOS).

·         Incurring proportionally greater development costs, by not identifying the problems until late in the training development effort, and greatly increasing some of the software and media development costs.

·         Extending the entire development timeline, by not identifying problems early.

·         Accepting substandard training and training products just to finish the project.

Operational tryouts

Risks are minimal for very short courses.

·      Conflicts between lessons not identified.

·      Lessons not sequential and progressive.

·      Resources needed to support the training are not adequately identified.

In summary, the quality of the training product is the direct result of a careful and thorough validation.  Quality assurance is a continuous cycle throughout the development process.  As shortcomings are identified, addressed, and resolved, the more cost effective the training development effort becomes.

2-3.

Validation team personnel and roles.  Everyone taking part in the validation process is part of the validation team.  TRADOC Reg 350-70, paragraph II-3-4, provides the basic policy on training development teams.  A variety of different people are needed during the validation process; however, the number and mix of personnel vary, based on the type of validation performed.

Validation team functions

    a.  Table 2-2 shows the functions that the different team members perform.

Table 2-2

Validation team functions

Function

Role

·      Approves.

·      Requirements.

·      Products.

·      Obtains resources.

Senior manager.

·      Management of project.

·      Plans.

·      Schedules.

·      Forms team.

·         Middle manager.

·         Contractor* (in a support role).

Observes.

·         Training developer.

·         Subject matter expert (SME).

·         Contractor*.

Instructs/trains/facilitates.

·         Instructor.

·         Contractor*.

Reviews.

·         Training developer.

·         SME.

Completes learning activities.

Volunteer (nonperformer).

Takes test.

Volunteer (master).

Provide additional support.

Miscellaneous training development, training site, and/or contractor personnel, as required/appropriate.

Analyzes.

·         Training developer.

·         Contractor*.

·         SME.

Revises.

·         Training developer.

·         Contractor*.

·         SME.

*When the development of the course/courseware is performed under contract, the contractor may perform some of the planning, instructor/trainer/facilitator, observer, analyst, and reviser actions.  The final responsibility for ensuring the course validates remains with the proponent.

Validation team structure

    b.  The optimum team structure, and the actions each member should perform, is in table 2-3.

    c.  The validation team is a cooperative effort, guided by the training developer, functioning as the project leader.  Validations are successful when the needed input from SMEs is obtained (soldiers in units, instructors, task performers, and their supervisors).

        (1)  The validation team primarily consists of the training developer, trainer, volunteers, and SMEs in the subject being validated.  Also involve an evaluator with the team.

Table 2-3

Validation team structure

Member

Actions

Middle manager.

·      Plans.

·      Schedules.

·     Manages.

Training developer.

·      Observes.

·      Documents.

·      Analyzes results/recommendations.

·      Conducts statistical analysis (if required).

·      Revises (if appropriate).

·      Approves (if appropriate).

Instructor/trainer/

facilitator.

·      Conducts training activities.

·      Participates in interviews.

·      Provides comments/recommendations for improvement.

SME.

·      Observes.

·      Provides comments/recommendations for improvement.

·      Analyzes results/recommendations (if appropriate).

·      Revises.

Contractor (if applicable).

·      Observes.

·      Documents.

·      Analyzes results/recommendations.

·      Conducts statistical analysis, if required.

·      Revises.

Volunteers (non-performers to act as students).

·      Take the test or training.

·      Complete the questionnaires.

·      Participate in interviews.

·      Provide input.

Volunteers (masters).

Take the test.

Evaluator.

Maintains quality control.

Additional support personnel.

·      Participate in the learning activities, as required.

·      Manage classroom computers (if appropriate).

              (a)  The training developer, a General Schedule (GS)-1750, is normally in charge of the project.  This individual is trained in the conduct of validations, and is the TD SME.

              (b)  The instructor conducts or facilitates the training, as appropriate.  Ensure instructors/facilitators are knowledgeable, competent, and trained on newly developed training materials.  To achieve competency, instructor/facilitator training should take place before they can conduct credible and viable training.

              (c)  The criteria for the volunteers are in paragraph 2-3h, below.

              (d)  The SMEs are master experts in the MOS or Area of Concentration (AOC) on which the developing training is developed.

              (e)  The evaluator is an independent observer, that provides guidance and assistance in ensuring the quality of the process is applied, and products produced.

        (2)  One problem encountered is in selecting the master SMEs.  There are three levels of SMEs:  apprentice, journeyman, and master.  Make sure the training developers and master content area SMEs on this team are adequately trained.

        (3)  The validation team is responsible for ensuring the course/courseware training products-

              (a)  Are thorough and comprehensive.

              (b)  Are technically correct.

              (c)  Result in quality courses/courseware, by applying quality control measures.

              (d)  Comply with TRADOC TD guidance and policy.

              (e)  Meet milestone requirements.

    d.  The training developer, GS-1750, is a key player on the validation team.  This person-

        (1)  Leads and manages the validation effort, to include:

              (a)  Keeping all people involved in the validation process informed of progress, problems encountered during the development phase, changes, and constraints.

              (b)  Being responsive.

              (c)  Providing results in a timely manner.

              (d)  Keeping the TDPMP updated, as appropriate.

        (2)  Provides guidance to the SMEs concerning how they perform their responsibilities.

        (3)  Provides validation training to the assigned SMEs.

        (4)  Provides quality control of the processes performed, and products produced.

SME

    e.  An SME is the content or technical expert.  Ensure the SME used is a master performer of the action/activity being validated.  The SME is responsible for technical content of the validation being conducted.  Specific responsibilities include-

        (1)  Accuracy of the technical content.

        (2)  Comprehensiveness of the validation.

        (3)  Completeness of the technical content.

Evaluator

    f.  An evaluator is responsible for identifying efficiencies and deficiencies in the application process and the production of the validation products.  Specifically, the evaluator-

        (1)  Informs the team of actual and potential problems, and recommends solutions.

        (2)  Identifies efficiencies developed, and informs the appropriate command authorities.

        (3)  Ensures all necessary corrective actions are completed.

Volunteer

    g.  The group of people the training is developed for is the target audience.  During the validation trials and operational tryout, a sample of the target audience is needed to volunteer to take the lesson.  The personnel used for the validation should have the same critical characteristics as the target audience of the training.  One of the training developer's most important tasks is determining the critical characteristics, and how much weight to give to each characteristic when recruiting volunteers.  (For example, if the course is an MOS qualifying course, do not identify the validation target audience as MOS qualified soldiers, since they do not get that designation until after completing the course.)

Volunteers (cont)

        (1)  It is not always easy to gather a good sample of the target audience.  Therefore, begin searching for volunteers very early in the planning process.  Consider all cost-effective means for conducting the validation (e.g., remotely via electronic means).  Assess the impact on results, if not able to conduct direct observations of the lesson or course.

        (2)  Holding the validation trials at a different location, or via electronic means (video teletraining (VTT) or collaboration software) are effective ways of getting sufficient volunteers and obtaining valid results.  Today, collaboration software allows observation of the volunteer's screen as they complete the instructional unit (lesson or course).  Another electronic option is to use VTT to observe a trial conducted at a remote location, to validate face-to-face training.  In either case, videotape the trial and replay it, to better identify problems.

Note:  The instructional unit is conducted exactly in accordance with the lesson plan, and another media is used to observe the conduct of training.  Another option is to use volunteers that possess the critical characteristics of the target audience, but are not members of the target audience.  When choosing this option, be very sensitive to how the sample used varies from the actual target audience.

        (3)  The number of available volunteers obtained determines the validation method used for group trials.

              (a)  With a sample population of at least 30 people, use the fixed sample method of validation for group trials.

              (b)  When it is impossible to obtain at least 30 volunteers, use the sequential method.

        (4)  The instructional unit may require several presentations to obtain data from a sufficient number of volunteers.  A minimum of ten volunteers is needed for group trials, to gather sufficient data to calculate realistic completion times for the courses/courseware.

        (5)  The sample of the target audience population used should include representatives from across the ability spectrum, to ensure the training is appropriate for the entire target audience that it trains.  The size of the sample target population varies based on-

              (a)  Whether a course or courseware is validated.

              (b)  The type of trial or tryout conducted.

              (c)  For group trials, the validation method used.

These factors are addressed in the appropriate chapters for each validation activity.

Instructor

    h.  The instructor instructs, trains, and/or facilitates the unit of instruction being validated, as appropriate.  Ensure the instructor is representative of the average instructor.  While SMEs train some courses or lessons, any available person of the appropriate rank or grade can train other lessons, such as the common core tasks.  The person that performs the training role should have the same critical characteristics as the actual instructors conducting the course, or facilitating the courseware.

Senior manager

    i.  The senior manager provides the resources, in the form of funding and personnel, required for conducting the validation.  It is critical that the senior manager allows the team adequate time to accomplish its mission, and function as a team throughout the validation process.

Middle manager

    j.  The middle manager provides oversight of the validation process, and provides assistance, when needed, to ensure the work at hand flows smoothly and efficiently.

2-4.

Validation team training.  Ensure that all members of the validation team have the training and/or information needed to fulfill their functions in the validation process.  The skills and knowledge requirements vary based on the function that the team member performs.  Requirements for the various team members are shown in table 2-4. 

Project dedication

    a.  The validation team is dedicated to the project for the duration of their participation.  Most of the team is dedicated to the TD project full time, but concentrates only on the validation during the time period when they are planning, conducting, or performing follow-on activities.  Calling them away for outside activities jeopardizes the quality of their work, and could impose cost penalties for the government, if a contract is involved.

Team building

    b.  The project manager needs to promote the team building process throughout the life of the validation team.

Table 2-4

Validation team training requirements

Team Function

Training Requirements

Planner/Manager.

·      Planning process.

·      Scheduling procedures.

·      Resourcing process.

·      Coordinating process.

Observer/Evaluator.

Either:

·      SME*.

·      TD expertise*.

* Generally, do not expect one person to meet both of these requirements.

Instructor/Trainer/ Facilitator.

Certification in the specific type of training being validated.

Reviewer.

Either:

·      SME*.

·      TD expertise*.

·      Statistical analysis process.

* Generally, do not expect one person to meet both of these requirements.

Volunteer (student).

Possess prerequisite skills, knowledge, and abilities for the course/courseware being validated.

Volunteer (master).

Mastered the skills, knowledge, and abilities being trained by the course/courseware being validated.

Additional support personnel.

Training requirements vary based on the situation.  The project manager ensures that all support personnel are qualified to perform, as required.

2-5.

Validation management tasks.  This paragraph identifies the tasks that the training developer performs to plan course and courseware validations, including conditions, standards, performance steps, and performance measures.

Task:  Coordinate validations

    a.  The training developer performs the task shown in table 2-5.

Table 2-5

Task:  Coordinate validations

Task

Coordinate validations.

Conditions

The training developer in charge of validating a course has the analysis data used to design the course, access to the proponency office (that knows where the soldiers in the MOS are assigned), previous validation plans, and a list of all the materials used in the course, with basis of issue.

Standard

Coordinate the plans for validation such that the personnel, materials, and site are all available, when needed.  Ensure personnel conducting the validation are trained in their roles.

Steps: 

Coordinate validations

    b.  The steps to coordinate a validation follow:

        (1)  Identify critical characteristics the volunteers representing the target audience must possess.

        (2)  Determine validation location based on-

              (a)  Type of validation to conduct.

              (b)  Availability of volunteers with the critical characteristics.

              (c)  Coordination requirements.

              (d)  Facilities and equipment availability.

              (e)  Administrative support requirements.

        (3)  Establish a point of contact (POC) at validation site.

        (4)  Coordinate with the POC 60 days before the validation date.

        (5)  Maintain close telephonic/electronic mail (E-mail) contact with all personnel involved, making sure they are ready to take part.

        (6)  Confirm availability of all-

              (a)  Personnel.

              (b)  Target audience volunteers.

              (c)  Site, equipment, and materials.

              (d)  Course/courseware, including all supporting materials, as appropriate.

        (7)  Plan for transport of all materials, resources, and personnel to the validation site.

        (8)  Ensure instructors/trainers/facilitators and other key personnel are trained to conduct the new training (if required).

              (a)  Plan instructor/trainers/facilitators and key personnel training.

              (b)  Schedule the training.

              (c)  Obtain all needed resources to support the training.

              (d)  Conduct/observe instructor/trainers/facilitators and key personnel training.

         (9)  Confirm all validation plans the day before the validation.

Performance measures:  Coordinate validations

    c.  The middle manager completes all performance measures in table 2-6, to successfully perform this task.

Table 2-6

Performance measures:  Coordinate validations

No.

Performance Measures

GO  NO/GO

1.

Ensured a suitable site is available.

c      c

2.

Ensured all personnel needed to conduct the training are available at the site.

c      c

3.

Ensured all personnel needed to conduct the validation are available at the site.

c      c

4.

Ensured all course materials are available at the site.

c      c

5.

Ensured all personnel needed to conduct the validation are trained.

c      c

2-6.

Coordinate validations quality control criteria

Coordinate validations quality control criteria.  Use the task standards and performance measures in paragraph 2-5, above, to ensure that quality control of the products and processes of planning validations are maintained.

Chapter 3

Criticality Standards

3-1.

Criticality standards overview.  This chapter provides guidance on how to develop criticality standards for each learning objective in an instructional unit.  This chapter will aid in determining the criticality of each learning objective selected for training, and includes:

    a.  The factors upon which the criticality standards are based.

    b.  The timeframe in which the criticality standards are developed.

    c.  The process for developing the criticality standards.

    d.  Identification of JAs to assist with the process.

    e.  Description of how the criticality standards are used.

3-2.

Learning objective criticality standard description.  The criticality standard is a measure of effectiveness used to establish validation requirements.  The criticality standard specifies the percentage of students that must pass each objective the first time, upon completion of the instructional materials for that objective.  If this standard is not met, the materials do not validate, and require revision.

Basis

    a.  Do not establish criticality standards arbitrarily.  Criticality standards are based on the rating of a series of factors related to job performance, and the education/training situation.  Each of the factors has a specific score, based on the answer chosen.  The total of all the answers determines the required percentage of students that must pass the objective for it to validate. 

Job related

factors

    b.  The job related factors include the following:

        (1)  Consequences of inadequate job performance.

        (2)  Required speed of response on the job.

        (3)  Availability of master performers, to provide assistance on the job.

Training related factor

    c.  The training related factor is the availability of subsequent practice.

Constraints

    d.  If a student can only perform an objective to the required standard, when one or more of the enabling objectives are mastered, then the standard for that enabling objective(s) is 100 percent, or as close to 100 percent as is practical and cost effective.

JA

    e. The linked JA 350-70-10.3 is used to compute the criticality standard for each learning objective, based on SME input.

3-3.

Establish criticality standards tasks.  This paragraph identifies tasks the training developer and SME perform when establishing criticality standards, including conditions, standards, performance steps, and performance measures.

Task:  Plan for rating the criticality of objectives

    a.  The training developer performs the task shown in table 3-1.

Table 3-1

Task:  Plan for rating the criticality of objectives

Task

Plan for rating the criticality of objectives.

Conditions

Learning objectives for a lesson or course under development are approved; now plan to hold a learning objective criticality standards rating panel.

Standard

Plan to conduct a learning objective criticality rating panel, to obtain the most reliable criteria, without exceeding funding constraints.

Steps:  Plan for rating the criticality of objectives

    b.  The training developer performs the following:

        (1)  Determine if there are minimums of three to five qualified SMEs available locally to form a panel, or perform paragraph (2), below.

        (2)  Select another alternative for rating the learning objectives from the following choices, if SMEs are not available locally:

              (a)  Bring in SMEs on temporary duty (TDY) status (results in increased costs associated with validation).

              (b)  Use electronic staffing with SMEs at other locations (using either video teleconference (VTC) or collaboration tools) to hold synchronous discussions.

              (c)  Use E-mail to staff the objectives and rating sheets with SMEs that independently rate the objectives.

        (3)  Schedule a time and location, if appropriate (or time frame, if paragraph (2)(b), above is selected) to rate the objectives.

        (4)  Ensure there are sufficient copies of the objectives and rating sheets for each rater to have a complete set.

Performance measures:  Plan for rating the criticality of learning objectives

    c.  The SMEs or training developers perform all performance measures in table 3-2, to successfully perform this task.

Table 3-2

Performance measures:  Plan for rating the criticality of learning objectives

Step

Performance Measure

GO   NO/GO

1.

Identify three to five qualified SMEs to make up the rating panel.

c      c

2.

Select the most appropriate alternative when SMEs are not available locally.

c      c

3.

Schedule time and location acceptable to all participants for holding panel.

c      c

4.

Provide all participants copies of all materials.

c      c

Task:  Rate the criticality of each objective

    d.  The SME or training developer rates the criticality of each objective, as shown in table 3-3.

Table 3-3

Task:  Rate the criticality of each objective

Task

Rate the criticality of each objective.

Conditions

As an SME on the tasks trained, the task analysis report for each task and the learning objective(s) for training the task(s) are available, along with JA 350-70-10.3. 

Standard

Rate each objective so the final score for each objective reflects its importance to a student's ability to perform the task under normal job conditions.

Steps:  Rate the criticality of each objective

    e.  The SME performs the following steps for this task.

        (1)  Rates each learning objective on each factor using
JA 350-70-10.3.

        (2)  Compares the results of the rating.

        (3)  Contacts raters to discuss differing scores.

        (4)  Sums the results of the rating factors.  The SME should agree on the rating.

        (5)  Uses the mean of the raters' scores.

Performance measures:  Rate the criticality of each objective

    f.  The SME performs all performance measures in table 3-5 to successfully perform this task.

Table 3-5

Performance measures for rate the criticality of each objective

Step

Performance Measure

GO   NO/GO

1.

Select most appropriate score for each of the factors for each objective.

c        c

2.

Sum the scores correctly.

c        c

3.

Select the percentage on the Critical Standard Conversion Chart that relates to the score for each objective.

c        c

4.

Agree on the final criticality standard for each objective.

c        c

3-4.

Products of the rating task.  The products of the criticality rating for each objective are-

    a.  A percentage for the required level of reliability for each objective in the unit of instruction.

    b.  A list of the learning objectives, based on criticality.

    c.  Data input for the evaluation and validation plans.

3-5.

Quality control criteria for criticality standards development.  The TD/task proponents perform quality control actions as an inherent part of the SAT process.  When developing criticality standards, ensure that:

    a.  SMEs, that are master performers of the task(s) being trained, rate the objectives.

    b.  Personnel complete all the performance measures for the tasks listed in paragraph 3-3, above.

Chapter 4

Content Validation

4-1.

Content validation overview.  This chapter provides how-to guidance on conducting content validations of course/courseware including:

    a.  The tasks performed by the validation team conducting the content validation.

    b.  The data the team needs to collect during the content validation.

    c.  The products produced during the content validation.

4-2.

Content validation description.  Content validation is the process used to verify that the information in the course materials is technically and doctrinally accurate.  During this process, SMEs review all course/courseware materials and supporting documentation, to ensure that all materials:

·        Are doctrinally and technically correct and complete.

·        Include sufficient detail.

·        Are clearly written in the Army Writing Style.

·        Use language that the target audience understands.

·        Use current, appropriate, and applicable references.

Timing

    a.  Conduct content validation in modular increments, while the materials are being developed.  Content validation of all course/courseware and supporting training materials is completed before conducting individual trials.

Personnel

    b.  The training developer plans and conducts the content validation.  The SMEs review the content of the materials.

Personnel requirements

    c.  Ensure SMEs validating the content of course/courseware have sufficient prior training and field experience, for familiarity with the technical details of the course/courseware materials and all reference materials.  While preferred, it is not mandatory to have the training developer and SME in the same room to conduct the content validation.  Conduct the validation over a VTC, or with the use of collaboration software, as long as all participants have the same copy of the materials receiving content validation.

JAs

    d.  The linked JA 350-70-10.4 is used to assist with content validation.

4-3.

Content validation tasks.  This paragraph identifies the tasks the training developer and SME perform when conducting content validations, including conditions, standards, performance steps, and performance measures.

Task:  Prepare for content validation

    a.  The training developer prepares the content validation as shown in table 4-1.

Table 4-1

Task: Prepare for content validation

Task

Prepare for content validation.

Conditions

Given a portion of an instructional unit in initial draft form, verify that it is doctrinally accurate and accessible to several SMEs, has all the materials for review, including supporting resources and references for each reviewer, as well as the developer.  

Standard

Prepare for the conduct of the validation activity so that-

·   The training developer and SMEs can meet either in person, or electronically.

·   The training developer(s) and SMEs all have the same version of the draft materials, all supporting resources and references for the materials, and the JA for reviewing the materials.

·   Team members clearly understand their specific responsibilities for the validation activity.

·   The SMEs have sufficient time to conduct a thorough review of the materials using the JA, before discussing them with the training developer.

·   The validation site accommodates in-depth, live, or electronic discussions, to identify all problems and agree on solutions for correcting the draft materials.

Steps:  Prepare for content validation

    b.  To prepare for content validation, the training developer should follow these steps.  It is not necessary to perform them in order.

        (1)  Identify SMEs to conduct the validation.

        (2)  Reproduce sufficient copies of the draft materials being reviewed, including all supporting resources and references.

        (3)  Ensure the SMEs understand their responsibilities.

        (4)  Distribute draft materials, including all supporting resources and references, to SMEs.

        (5)  Schedule meeting time(s) to compile the results of the content validation.

Performance measures:  Prepare for content validation

    c.  The training developer performs all performance measures in table 4-2 (in any order) to successfully perform this task.

Table 4-2

Performance measures:  Prepare for content validation

No.

Performance Measures

GO  NO/GO

1.

Obtained at least two SMEs to review the materials.

c          c

2.

Reproduced sufficient copies of the draft materials, so each SME has a complete set.

c          c

3.

Briefed SMEs on their responsibilities during the content validation.

c          c

4.

Provided each SME with all draft materials, and supporting resources and references.

c          c

5.

Scheduled sufficient live or electronic meetings, to address all materials being reviewed.

c          c

Task:  Conduct content validation

    d.  The training developer conducts content validation as shown in table 4-3.

Table 4-3

Task:  Conduct content validation

Task

Conduct content validation.

Conditions

Given a portion of an instructional unit in initial draft form and SMEs that are master performers of the tasks being trained, verify the material is doctrinally accurate.  Given all the materials for review, including supporting resources and references for each reviewer, as well as the developer, and sufficient time to conduct a thorough review of the materials.

Standard

Ensure that the training materials-

·      Are doctrinally and technically correct.

·      Include sufficient detail.

·      Are clearly written in the Army Writing Style.

·      Use language that the target audience understands.

·      Use current, appropriate, and applicable references.

Steps:  Conduct content validation

    e.  The training developer performs the following steps to conduct content validation:

        (1)  Have SMEs review all draft materials using JA 350-70-10.4.

        (2)  Conduct live or electronic meeting, to address all materials being reviewed.

        (3)  Discuss the recommendations with the SMEs, to ensure that the reasons for their recommendations are understood.

        (4)  Ensure all involved parties are in agreement with decisions.

        (5)  Document the final decisions agreed upon by all parties.

        (6)  Forward all materials, documenting problems identified and proposed corrective actions, to the development team.

Performance measures:  Conduct content validation

    f.  The training developer performs all performance measures in table 4-4 to successfully perform this task.

Table 4-4

Performance measures:  Prepare for content validation

No.

Performance Measures

GO   NO/GO

1.

Had SMEs review the draft materials, assisted by JA 350-70-10.4.

c        c

2.

Conducted meeting to address all materials being reviewed.

c        c

3.

Discussed with the SMEs their recommendations for all changes to be made.

c        c

4.

Documented all final decisions agreed upon by all parties.

c        c

5.

Ensured all involved parties agreed with decisions.

c        c

6.

Submitted all materials, documenting the problems identified and proposed corrective action, to the development team producing the course/courseware.

c        c

Task:  Review draft instructional materials for content validity

    g.  Each SME reviews draft instructional materials as shown in table 4-5.

Table 4-5

Task:  Review draft instructional materials for content validity

Task

Review draft instructional materials for content validity.

Conditions

Given JA 350-70-10.4, a portion of an instructional unit in initial draft form; and all the materials for review, including supporting resources and references; verify the material is doctrinally accurate.

Standard

Identify and recommend corrective actions for all instances where the materials-

·         Are not doctrinally and technically correct.

·         Do not include sufficient detail.

·         Are not clearly written in the Army Writing Style.

·         Do not use language that the target audience understands.

·         Do not reference current, appropriate, or applicable references.

    h.  Each SME performs the following steps to review draft instructional materials for content validity:

        (1)  Identifies all instances where the materials-

              (a)  Are not doctrinally and technically correct.

              (b)  Do not include sufficient detail.

              (c)  Are not clearly written in the Army Writing Style.

              (d)  Do not use language that the target audience understands.

              (e)  Do not reference current, appropriate, or applicable references.

        (2)  Provides recommendations for correcting errors found during the review.

        (3)  Provides the training developer with copies of the completed JA 350-70-10.4, and any notes taken during the review.

Performance measures:  Review draft instructional materials for content validity

    i.  The SME performs all performance measures in table 4-6 to successfully perform this task.

Table 4-6

Performance measures:  Review draft instructional materials for content validity

No.

Performance Measures

GO  NO/GO

1.

Identified all instances where the draft instructional materials:

c       c

a.  Were not doctrinally and technically correct.

c       c

b.  Did not include sufficient detail.

c       c

c.  Were not written in the Army Writing Style.

c       c

d.  Did not use language that the target audience would understand.

c       c

e.  Used references that are not current, appropriate, or applicable.

c       c

2.

Provided recommendations for correcting the errors found during the review.

c       c

3.

Provided the training developer with copies of the completed JA 350-70-10.4, and any notes taken during the review.

c       c

Completion

    j.  The content validation is complete when all technical and doctrinal inaccuracies and conflicts have been identified, and corrective actions proposed.

4-4.

Content validation products.  The products of the content validation are proposed changes and supporting documentation.

4-5.

Content validation quality control criteria.  The performance measures identified in paragraph 4-3, above, provide the needed level of quality control for individual trials.