Technical b r Report - International Military Testing Association

UNCLASSIFIED,‘UNLIMITEDw “‘$1 

Technical 

r Report 

distributed by 

Defense Technical Information Center 

DEFENSE LOGISTICS AGENCY 

Cameron Station � Alexandria, Virginia 22314 

UNCLASSIFIED/UNLIMITED

. . 

oado2

-* . ‘: .;‘*nn. :: ;.... 1. . . _ _,.“. _ . .- ..~ ~ . . . -. ,, . . . _. - ._.-.. -_- .._. -._..,.. - -- . .- ,-... 

Coordinated By: 

AIR FORCE HUMAN RESOURCES LABORATORY 

AIR FORCE SYSTEMS COMMAND 

Brooks Air Force Base, Texas 

. F a,..,,. - --ovcd for public release; distribution trnlimftc~ 

El Tropicana Motor Hotel 

San Antonio, Texas 

28 October - 2 November 1973 

1 

. . i 

;j. 

: ‘e... 

3 

,J

---. .- 

Ct.! bfi'L ‘110 JvrclrL 

--- _ . . . ..l.- 

i.-. _..-.- 4 

. .2 

J 

L I 

. 

/” _ 

, / /’ 

I d/ -7 _._.___ __---.-.-- - .-..- I 

;~, 1 

I 

PROCEEDINGS 

.) 

45ttrjAnnual Conference 

I 

.__.- - 

of the 

/ Military Testing Association

:: 

FOREWORD 

The 15th Annual Conference of the Mili'.ary Testing Association was 

held at the El Tropicano Motor Hotel, San An:onio, Texas, 28 October - 

2 November 1973. 

The papers presented at the 15th Annual Conference of the MTA reflect 

a diversity of subject ,matter. These presentations from representatives 

of both the m?'litary and civilian community reflect the opinions of the 

authors and are not to be construed as official or any way representative of 

the views and policies of the tini ted States Armed Services or the armed 

services of the foreign countries participat!ng in this meeting, 

._ + 

/ 

iii 

I

- OFFICIAL PROGRAM - 

JJ 15th Annual Conference 

MILITARY TESTING ASSOCIATION 

El Tropicano Lobby 

1300 - 170@ 

El Tropicano Lobby 

Oil00 - 1600 

Hemisfair Room 

Ground Floor 

0900 

0900 - 0215 

0915 - 0945 

1015 - 1030 ( 

28 October through 2 November 1973 

Sunday, October 28 

Registration 

Monday, October 29 

Registration 

Session Chairman 

Co1 Theodore B, Aldrich 

Air Force Human Resources Laboratory 

Conference called to order 

Greetinas by AFHRL Commander 

Co1 Harold E, Fischer 


Keynote Address 1 

Lt Gen John U. Roberts 

'Deputy Chief of Staff Personnel 

Headquarters, United States .lir Force 

Introductory Comments 

Co1 Theodore B. Aldrich 


4dministrative Announcewnts 

Coffee Break

.- . . . _ -._* _.-.,. ..-_. __...,_.... ..^_...I_. . . -_ . . . ,_ _.., . _ .,., _ ,,“.. ;a _ ..,____- 

1030 - 1200 Symposium 

Eiut&-4 cmice 4 C,WCJtiJlg 6olr miettahy pch6ohmance 

and bc.hav.io~~ 

1200 - 1330 


Ground Floor 

1330 - 1430 

Chairman 

Dr. Ralph R. Canter 

Office of the Assistant Secretary of Defense 

(M&w) 

Participants 

Maj James Taylor 

United States Air Force 

Comdr James Murphy 

United States ::avy 

Lt Co1 Todd Graham 

United States Army 

Dr. D. Brian Murphy 

Systems Development Corporation 

Dr. Robert G. Smith 

Human Resources Research Organization 

Discussant 

Dr. Norman 3. Kerr 

Naval Technical Training Command 

Lunch 


Mr. Harald E. Jensen 


Symposium 

Chairman 

Dr. Lonnie D. Valentine 


Participants 

Maj Verna S. Kellogg III 

Armed Forces Vocational Testing Group 

Capt Melvin T. Gambrell, Jr. 


Dr. Harry D. Wilfong 


vi 

?

1430 - 1450 

1450 - 1505 

1505 - 1535 

1535 - 1550 

1550 - 1610 

Monterrey Suite 

Room 909 

1730 - 1830 

Acapulco Room 

Room 902 

DeveCapme& 06 ti:c Az.d Stiviccb Wacati~na.f 

Apt&& Gatietij I 

Dr. M. A, Fischl 

United States Army Pesearch Ins$‘itute for the 

Behavioral and Social Sciences ' 

- 

Coffee Break !! 


Capt ,.lames A. Hoskins 

Air Force Human Resource: Laboratory 

NM devc4ogmca.t~ in dcieus e htguagc pro &ienuj 

te5 .ting 

Drs. C. D. Leather-man and A. Al-Haik 

United States Army, Defense Language Institute 

Presented by: Mr. Sydney Sako 

EnC.i5t~d ~~iccG.crt and chd.5i~i~dic~z fCA tiq in 

t:1 c u. s. Awj 

Dr. Milton H. Maier 

United States Army Research Institute for the 

Behavioral and Social Sciences 

Presented by: Dr. M. A.(Fischl 

The. dcvc?c.pwt t 5 Ei:gLih Ccmp:chca:icn L:.vel 

A camit:g tc.3 t. 404 $2 tei;gn 4 ALde?c,ttA 

Mr. Cortez Parks 

United States Army, Def,ense Languaae Institute 

E%D OF fWi[lAY SESSIONS 

oeoo - 1600 Registration 

’ 

. 

I 

MTA President's welcoming and social hour 

Tuesday, October 30 

vii, ,. 

! 

/I 

I i 

_. 

I 

.., 

.i, : 1. 

: r 

-7

JJ Hemisfair Room 

Ground Floor 

0845 - 0900 

0900 - 0920 

0920 - 0945 

0945 - 1000 

1000 - 1155 

session Chairman 

Capt James A. Hoskins 


:a11 to order 

izn cxptincntaL, muetimedia ccwccfsz dcvcecpmttt 

EOU-AC. 60.2 nw mcrtt& b.tnndahd!, uhmcrt 

Mr. George P. Scharf 

USAF School of Applied Aerospace Sciences 

3p.Gn.l ut.LLizsticn 06 cn- the-job Omiuixg aud 

tccfIrricae ttlaiuriirg OChOO~ 

Capt Alan D. Dunham 

National Security Agency 

- 

Coffee Break 


Mr. Manuel Pina 

Air Force Hlrrran Resources Laboratory 

Symposium 

Ta.m5dnticn 05 tml~~tirlg xe,cmcl1 irtto .tininhzg 

acticl: - a mi,5ing Link 

Chairman 

Dr. G. Douolas Mayo 


Participants 

Fkom tiltltc 1*iclcpcinR: 05 a &taining manngut 

Mr. Walter E. McDowell 

Army Training and Doctrine Command 

Ftom titc vicqcirzt c$ a tiaining 4tica'rcfietr 

Dr. Norman 3. Kerr 


Discussants 

Lt Co1 Donald F. Mead 

Air lraininq Command 

Dr. Earl I. Jones 

Navy Personnel Research ?I Development Center 

viii 

*. 

t , ._ 

‘... . . 

.

1155 - 1200 

1200 - 1330 


Room 909 

1330 - 1500 


Ground Floor 

1330 - 1340 

i340 - 1400 

1400 - 1420 

1420 - 1445 

1445 - 1510 

1510 - 1525 

Reaction Panel 

Comdr Bruce Cormack 

Canadian Armed Forces 

Mr. William B. Lecznar 


Mr. Ralph Canter 

Office of the Assistant Secretary of Defense 

WW) 

Administrative Arxouncements 

Meeting of the Steering Committee 


Mr. William J. Stacy 


Administrative Announcements 

TtiJt @zdba& a,ld i%uXng cbjetivcs 

LtJG Carroll H. Greene 

United States Coast Guard Training Center 

Assessment Systems, Incorporated 

Ptro5pcctivc cfiic$ pcttv ob&m &adchsltip/ 

ma,!cgclw,?~ 5 c.mina: 

Comdr G. C. Hinson 


Tmn5certdwtal lleditfltion (TM) : A t1twa.t otr 

tcchniquc in ;(uAm2 pemonn& eva&&orl 

Dr. R. 0. Waldkoetter 

Consultant, Education and Personnel Systems 

Coffee Break 

ix 

‘7

1525 - 1515 

1545 - 1600 

1600 - 1630 


Ground Floor 

0830 

0830 - 0850 

0850 - 0920 


Capt Nichols: C. Varney 


7tJpticuf~oru 06 camce in.5 t711ctiqn @)I mie.itnry 

t&5 titi,tg 

+ 

Cr. Ronald W. Spangenberq 


Presented by: Dr. Roger J, Pennell 

A5.5c~5mc1t t 05 comp!cx p:,!r&cfno to.2 ccotd~~tio~l 

Cc.nz,~illg ‘Wd yc.zf;C?LnlL7rlCC. ClllAillg 130 d.wqs oh 

ill fcz5ivc .fe5 tir1g 

Dr. Randall M. Chambers 

Georgia Institute of Technology 

Dr. Rayford T. Saucer 

Veterans Adminixtration Hospital 

P.tU ~.iCiCllUj lJJCU5 UWtJWtt i,I di%jl t b&XdntO~~5 

Dr. Edwin Cohen 

The Singer Company, Simulation Products Division 

END OF TUESDAY SESSIONS 

Wednesday, October 31 


Mr. Andrew T. Garza 


Lt Comdr C. L. Idalke'l- 

Naval Guided Missilbs School 

Data Design Laboratories 

X 

i 

,

I/ 

: 0920 - 0935 

0935 - 0950 

0950 - 1OOS 

1005 - 1020 

1020 - i’235 

050 

120 

1120 - 1200 

1203 - 1330 Lunch 

Hemis fair Room 

Ground Floor 

Lt Co1 Quay C. Snyder 

United States Army War College 

SScrdirlct& hhtingb: !a!/ rtot? 

Drs. k!. H. Githens and R. S. Elster 

United States Naval Postgraduate School 

The Kec5~cfc Stlldlj-et.ccttortic techt:ici~lII &wL 

yea3 eva&faliorr 04 Thficc tq;3&5 0s tmkling 

Dr. Virginia Zachert 

Medical College of Georgia 

Coffee Break 


Mr. Kenneth G. Koym 


Mr. klliam C. Osbom 


The Sc.C; Eva4hat~on Tc.d~ui.quc (SET! bhadia 

Dr. John 3. Holden 

llnited States Army Ordnance Center and School 

A mcdda: ap:~zcach to podicicncy tcJ.tA:g 

Dr. Robert W. Stephenson, Mr. Warren P. Davis, 

and Mr. Harry I. Hadley 

American Institutes for Research 

Mrs. Bertha H. Cory 

United States Army Research Institute for the 

Behavioral and Social Sciences 

The ~.tcctcing cicmnnd bar; kiman pekfc-wwcc .totitg 

Mr. J. E. Gerber, Jr. 

United'States Army Infantry School 

Presented by: Capt John P. Otjen 


Lt James W. Abellera 

Air Force Human Kesowces Laboratory 

xi 

3 

V. ._ _- -.C . . - _. -... . 

! ..I . : I-.

1330 - 1350 

1350 - 1420 

1420 - 1435 

1435 - 1450 

1450 - 1505 

1505 - 1520 

1520 - 1540 

1540 - 1600 

1600 - 1615 

Ctiwion-m@certced peh(oe.mancc testing 

in combat M~S ~tS.L& 

Mr. John F. Hayes 

UHS/Matrix Company 

The Ah Fohcc wi(c: Heh brtowZcdge 06, artd 

attitucic,5 .tc.m'ui, titllc Aih Fohcc 

Ors. John A. Belt and Arthur B. Sweney 

Wichita State University 

Cadet m.tcntion at titc ROIJCU? SGtct~~ CoLkgc 06 

Carlada a,!, a &rtctio~ 06 pcti o~x&.bt:~, leadch,lrip 

pc.e6o,tmnncc,and bioq~a~~hic~ vazi&th 

Lt Co1 G. J. Carpenter 

Royal Military Colleqe of Canada 

Lt Knight 0. Cheney 

United States Coast Guard Reserve 

~rtcotuLtcn g4oup5 a3 a mean5 doh e~&ctirlg 

attitctdimt change 

3 Dr. Peter F. Newton 

National Security Agency 

Coffee Break 


Mr. James M. Wilhourn 


A n cvduatin 06 cliagnob t i c voca6uikm~ tca.tbtg 

in Ati Fotcc tcchrticd .tYL~tilg 

Lt William P. Mockovak 


Uni6o?uncrl pllqbicinrts w.Lthout paticrt&: A 

dquandching 06 bc~hce hc.5ouhccJ? 

Dr. Gary B. Brumhack 


An cxpc&~cnthc &@Zzmctztatioh 06 comptctr 

ru,A!&d admiosib& phoDabibL@ Rebtiting 

Mr. W. L. Sibley 

RAND Corporation 

xii

_ 


Room 909 

1730 - 1830 


Ground Floor 

0830 

0830 - 0900 

0900 - 0915 

0915 - 1005 

lC35 - 1020 

1020 - 1045 

I , 

END OF WEDNESDAY SESSIONS 

!’ 

MTA President's social hour 

Thursday, November 1 


Lt Sarry P. McFarland 


Call to order 

CqAK!USEL: k cctn,uutci p,'logtrrnl li*'Itid~ bdkd.5 

qcLLLe,itatiw pzedicto.7.3 $08 qua&ihLii*c cti-tcticn 

p4cdicSvz p4oblem5 I 

Dr. William J. Moonan / 

Navy Personnel Research and Development Center 

I 

Dr. Roger Pennell 


AppI!icntiorzs o{ p4ecfkto olrdcukg acd rc1:ctYicn 

bq Baqesiatt-decinicw tedyque 

'II * 

I 

Mr. S. E. Bowser 

Navy Personnel Research and Development Center 

P 

Coffee Break 

J" 


Mr. William 3. Phalen 


fkw adtn~ndibk pwbab.iLiA~ ~CGLJICJ a$;(ect5 arrd 

ib U~&c&d htj tile LaAgclr Jybtcm 06 irtcextiL*e.~ 

Dr. Thomas A. Brown 

RAND Corporation 

:; . . 

x111 

I 

r 

! 

,

1045 - 1100 

1100 - 1130 

1130 - 1145 

1145 - 1200 

1200 - 1330 


Ground Floor 

1330 - 1355 

1355 - 1420 

1420 - 1440 

Dr. Thomas C, Tuttle 

Westinghouse Behavioral Safety Center 

. . 

Kc~*icrc a;5 Ai.t Fo?cc. jc6 JntiJ@ctio,L culd c6VMC.t 

d;\*c2CpmC,lt ‘tcJc‘lt& 

Mr. R. Bruce Gould 


A mc tizcdo+r $0: dc&zmihg jot snti~ t(ac2i.00 

i,: ti:L! (1 � �� ␛� 

Dr. Lawrence A. Goldman 

Naval Occupational Task Analysis Pr@Jram 

Dr. William J. Figel 

Science Research Associates 

Lunch . 


Mr. C. Amos Johnson 


Job RtiC,cu’ptitw

_A --- -.-a,.- s...-*_-..s.. .-w_- ..__ . 

. 

1440 - 1500 

1500 - 1515 

1515 " 1540 

1540 - 1610 

1610 - 1625 

Dr. Robert G. Smith, Jr. 

Hwan Resources Research Organization 

I 

1625 - 1630 Adnr&nistrativc Announcements 

Terrace Ballroom 

1900 - 2000 

Rcc‘ati.le aptitude .zcquikenrcrr& weah 

Or, Raymond E. Christal and 

Sqn Ldr John W. K. Fugill, RAAF 


Coffee Break 


Mr. Wayne S. Archer 


Mr. Clifford ?. Hahn 


Ccpi tivc ccmpCcxifc(: Iti cust~ciation with 

3 c Zccticv $c 7 mii i.Lxy Cen,fc 23Iri.p &o&h 

Sqn Ldr Brian N. Purry 

Royal Air Force 

END OF ThURSDAY SESSIONS 

I 

I 

Social Hour - "pay-as-you-go" bar 

2000 - 2200 Banquet - Invited S&aker 

Brig Gen Conrad S! Allman 

Commander, USAF Recruiting Service 

Friday, Nove$er 2 

I 


Ground Floor 

0845 

I 


Mr. William E. Alley 


Call to order 

xv 

I 

I 

-. .

Il 

0845 - 0900 

0900 - 0920 

0920 - 0940 

0940 - 1005 

_.~~ 

1005 - 1020 

1020 - 1035 

1035 - 1100 

1100 - 1130 

1130 - 1150 

Administrative Announcements 

khaiktistAation 04 mutt+&-doicc ttitb to 

non-w&c,u via tape r,ccor,da: CNc JtudiU 

Dr. Joseph L. Boyd, Jr, 

Educational Testing Service 

RaciaE did@zctlccs & AFQl, ACE, al:d WATS dco%c~ 

0s A&z Fc.tcc tmic xhninec5 

Capt Stephen B. Knouse, Sgt David F. McGrevy, 

and Sgt Ronnie A. Thompson 


Dcc~ tic USAF CJ~+X.: EXcg~qhicaC InwXcy 

p.*vticrz 0s Zhc AFOQT imdc~ticrtt

1150 - 1210 A pu~po~ cd 4 pa.tiu..t o.ticrtttion/d& otic.&mXon 

#.igl~t ~~ainbtg c0wzp.t 

Mr. Patrick J. Dowd 

USAF School of Aerospace Medicine 

1210 - 1230 I Concluding Comments 

END OF CONFERENCE 

xvii 

, 

.n 

: 

..-: _

1. 

II. 

III. 

IV. 

V. 

VI * 

VII. 

VIII: 

IX. 

x. 

TABLE OF CONTENTS 

I I 

Keynote Address. , . . . . . . . . .Lt General john W. Roberts 

I 

Symposium synopsis: Ccoyc:atiue 

aptihdc sc5cmch p2oyuvM cond1lctc.d 

?h~u~1~ t&c hmcd Fc!‘rcc~ 

VccntiorlaC 7~ tixg Gtroup . . . . .Dr. L.D. Valentine 

'&j&='Verna S. Kellogg I!1 

Capt Melvin T. Gambrell, Jr. 

'Dr. Harry D. Wilfong 

Dcvc,fiopmc.nt 05 the Azncd Scmicti 

Vocdtiorial Aptitude BczXXe~~. . . . . . . . . . Dr. M.A. Fischl 

NW der%~op!?V~~ 13 &I de$m,c 

Zampqc ptc~icimy tot&g . . . . . . . Dr. C.D. Lecthenan 

Dr. A. Al-Z-laik 

Edistcd 5cCcctic11 and cksi.{icatic,t 

tot&g in 3~. I;. S. Auny . . . . . . Dr. Miltor H. Maier 

AN c.tpetim~ hi’, mu.W!hedia c~zxcz 

i 

I 

. . .Mr. Corter'Parks 

dcvticpmeJ1 t ccuuc. gcr WlC’ V!C.azt i 

Jtwdm.h aibnclt . . . . . . . . . . . . . Mr. George P. Sharf 

Optit!t,d utiti:at.icn o; CR - tltc -jab 

.Ct&~ibtiJ:~ - Id tcchcicai fmilt iq 

~clzooe . . . . . . . . . . . . . . . . . . Capt Alan D. Dunham 

Slymposium: T-~~~sPtLtion 06 ttiting 

.tcscat& i,: to .tti&;g act&w - a .I 

i 

n;ti&iJlg till: . . . . . . . . . . . . . . . Dr. G. Douglas Mayo 

Mr. iJalter E. McDowell 

I Dr. Norman 3. Kerr 

/ Lt Co1 Donald F. Mead 

Dr. Earl I. Jones 

Tc3 t &ccdb.tcl: ar!d Ouuhiug cbicc- 

-tivc5 . . . . . . . . . . . . . . . . .Lt 36 Cari*oll H. Greene 

, - 

xix 

Page 

1 

20 

27 

34 

45 

52 

60 

92 

96 

101 

108 

114- 

118

11 

XI. 

XII. 

XIII. 

XIV. 

xv. 

XVI. 

XVII. 

XVIII. 

XIX. 

xx. 

XXI. 

: 

xx 

. 

3 

Page 

129 

141 

147 

154 

170 

178 

2D7 

234 

249

XXII. &tics 06 &?&I .:CAdcfl’dl AtudiCA 

u5im3 .thc Scl.6 &vn2uaticn Tcchiquc 

(SET .$tctdic~l. . . . . . . . . . . . . Dr. John J. Holden 

XXIII. A. mcdu~a~7 npy zcad! 0 ptc

XXXIII. A &cto!~ anot?ytic apphoa& to 2%~ 

cz,itctio~~ pzobkm. . . . . . . . . . . . . .Dr. Ro'ger Pennell 508 

I 

XXXIV. Appticnticn5 o 5 )J WdiCfOh CNkA-blg 

aud 3ct%dior1 by a Garjcniatt-r;‘ccision 

tcchiquc.. . . . . . . . . . . . . . . . . . Mr. S. E. Bowser 525 

xxxv . G?rotthg dmznd5 on honan , * 

kC.5 cu.zcc,5 : A U~CIC ,(~tn i;:.&? tty. . . . . Dr. Thomas C. Tuttle 543 

YYV” ..,\. I . &wCc~ 06 Ait Fctcc job $nti,- 

@!,tion Nld ca’rc.c~‘r dcw?~oymci,. t 

.zc.5~a~l1 . . . . . . . . . . . . . . . . . Mr. R. Bruce Gould 560 

XXX’I I I . A metflodceogy 6o.t dctc/vktkg 

job satis jactioti iti the if. S. 

5lavy . . . . . . . . . . . . . . . . .Dr. Lawrence A. Goldman 573 

XXXVIII. lnttxe3.t Suwcy5 and job p&xxrnti1.t 

41 .I%C A,?rnz.d Sc~v.ico5 . . . . . . . Dr. William J. Fiqel 585 

XXXIX. Job dticGp3iotl it1 tix 

hncd Fotccs ti a btih 

50.1 ~1a&5-i5 atld evaC- I 

llation . . . . . . . . . . . Flottillenadyiral Guenter Fiebig 596 

XL. Pos.~ibitititi cold .kLniXnR.icu 

05 job mla@.ti and job b 

evctecttion i.11 &cd Fo4ce5 . . . . . .Lt!Col H. E. Seuberlich 608 

XLI. Ocaq.~.ztic~la~ am&5 i5 i n .tfic 

'Ro# &othneinjt iLi.t Forrcc . . . . . . .dapt Wayne S. Sellman 

Sqn L'dr John W. K. Fugill 622 

XLII. Col93xticj1 05 peet:{c7~na~ce 

d&x at &'f're jo6 .bk -!kvee . . . . . . ./Mr. Clifford P. Hahn 643 

I 

XLIII. CcgnLtive compk+itf/: 7.t5 a35ocia- 

.tiOlI tt?ltll .54L&tiO,l SOh miediZtrJ 

Ceadc&tip aok% . . . . . . . . . . . Sqn Ldr Brian 14. Purry 659 

XLIV. ,vat* ccrnccpz.3 5ofL the nrcCL5uMm~Lt 

// 

06 attitudc5 attd mo.tivti . . . . . . Dr. Robert G. Smith, Jr. 674 

xxii 

i 

I 

B 

r 

--- .

11 

XLV. 

XLVI. 

XLVI I. 

XLVIII. 

XIX. 

L. 

LI.: 

111. 

LXII. 

LIV. 

L’i. 

LVI. 

Page 

. . . . Dr. Joseph L. EIoyd, Jr. 689 

Racid di~~~.mtccs iu AFQT, ME, 

and WAZS SCO'ICS 06 Ati Fo.:cc 

bmic a%-'~~co . . . . . . . . . . . . .Capt Stephen B. Knousc 

Sgt David F. McGrcvy 

Sgt Ronnie A. Thompson 694 

Ooe5 the USAF C;i(ic:.t Giogtc$?icax 

Invcntoq~ po'~ti011 cj t1:c AFCQT 

imfhwUc)ti~y mea5u~tC aridlo.-C-ta7h.n 

pL.tiooltaei.tq? . . . . . . . . . . . . . Lt Co1 Charles W. Haney 

.Ms. A. M. Kelleher 

1 Melvin S. Majesty 

Wallace F. Veaudry 

rncnt 01 a tni..!.ita.~-/ ~cttotg . . . . . . . . . .Dr. Kay H. Smith 

A pticpo6cd JpatinQ c’Licntatim/ 

diboahttht.ic~t d.fi$l t t~2hb~ing 

co,~cpt. . . . . . . . . . . . . . . . . . Mr. Patrick J. Dowd 

Minutes of the 1373 b?TA Steering 

Committee Pleeting. . . . . . . . . . . . . . . . . . . . . . , 

Citation of the Capt Harry ..i. Greer 

Award for 1973 . . . . . . . . . . . . . . . . . . . . . . . , 

By-Laws of the Nlitary Testing 

Association. . . .,. . . . . . . . . . . . . . . . . . . , . 

List of conferees.' . .I . . . . . . . . . . . . . . . . . . . . 801 

i 

xxi';i 

: 

t 

. . . 

‘... ,_. 

.I ., 

3 

709 

7 2 6 

731 

740 

750 

757 

789 

793 

795

. . 

KEYSOTE ADDRESS 

Lt Gcn John Ir'. Roberts 

Deputy Chief of Staff Personnel 

tfeadquartcrs, Unizcd States Xii- Force 

It's traditional for a speaker, particularly a keynote 

speaker, to open his remarks with the statement that he's 

pleased and gratified to be wherever he is. I can do this 

with a considerable measure of sincerity, sinze being here 

gives me an opportunity to pass on some thoughts to a group 

that holds the key to better use of human resources in these 

days of growing demands. 

In the last few days, I've supplemented my knowledge 

about this audience with, some homework, including a review 

of the minutes of your past meetings. Your achievements, both 

k 

as individuals and as an association, are certainly more than 

impressive. 

Although I do not speak to you as a psychologist or as 

a measurement specialist, I feel a definite kinship with many 

in this group. I have worked very closely with people in the 

Human Resources Laboratory for the past three years. As 

Director of Personnel Plans for the Air Force, and now as 

the Deputy Chief of Staff for Personnel, I know that many of 

the decisions we have made, and will make, are based directly 

on the results of your efforts. Your contributions have had 

a significant impact on the way we manage people: only the most 

naive would deny that we've been able to build a far better 

military establishment with you than we could have without you. 

1 

. . .

!’ 

For these reasons, then, I welcome the opportunity to 

pass on some of my thoughts and, hopefully, to offe,r some 

I 

challenges for the future. I'll start with a couple of 

observations. 

In reading over past conference reports, I find that 

your guest speakers can be broadly categorized into two 

groups. In the first category are the personnel researchers, 

the experts, people like you. They unc'?rstand the behavioral 

sciences. They understand the problems and the lead times 

involved in achieving effective progress. As a result, they 

usually tell an audience like this to ride out the changes 

in administrative personnel, to avoid a preoccupation with 

the operational demands of today's job, to take the long 

range view in the interest of excellence and real progress. 

In the second category, ? find people jike myself, 

managers, or laymen (in the research vernacular1 trying to 

'cope with the day-to-day problems of procuring and managing 

\ 

\ 

. * 

large numbers of people. While WC realize he futility of 

! 

trying to come up with airtight solutions, we tend to ask 

I 

you for formulas and tools (what you might call gimmicks) 

that will enable us to tie up our problemsiin nice, neat 

// 

packages. And, of course, we're always in a hurry. We 

become a little impatient when your answers are not quick 

enough or you try to cualify, perhaps justifiably, the answers 

you do provide. 

, - 

2 

I 

I 

, 

v 

--- 

‘\

From experience, I know that the difference in 

perspective represented in'these two points of view can 

put the manager.and his experts at odds. I would go even 

further. They can, and often do, make effec%ive communication 

difficult, if noz ic?ossible. The tragic thing is that we have 

never needed effective communicatiqn between the researcher 

and the manager more than we do today. The demands on our 

ingenuity to employ, to challenge, to get maximum value from 

those who elect to,join us have never been greater. There is 

too much to kno!

effort, or some other factor, the fact remains ,that today's 

environment has some profound implications for the armed 

forces, especially when it comes to manpower. 

We can talk about this situation in terms of tight 

money. L3e can point to force reductions, or shortfalls in 

military strength. ' What it all means, however, is that the 

military, as an institution, must face up to the challenge 

of having to manage its human assets better -- to get as much 

dedication, as much effectiveness, out of our people as 

possible. In spite of the present environment, we must 

continue to find legitimate'ways to attract the manpower 

we need to keep our forces viable. 

It's been estimated that maintaining an all-volunteer 

force of 2.2 million over the long haul will require one 

out of every three qualified and available men in this 

country to volunteer for active military service. Under 

present policies, one-third will have to be "above average," 

over one-half, "average." This year alone the armed services 

will have to attract better than 400,000 young Americans to 

meet our commitments. But the problem goes beyond numbers. 

They also have to be the right kind of people. Here is 

where we've got some room for improvement. 

4 

_. -. I, I .~ ,__.. .._. 

: 

I 

. . . 1 

. . : 

, . . . - I 

:. . I’ 

I 

: 

i 

i 

I

For example, out of the average 100 enlistees who enter 

the Air Force, 25 leave before they complete 4 y!Lars. Some 

are found unsuitable or unfit for military service; some 

have disciplinary problems; others leave for personal reasons. 

: : 

In adtiition to the personal anguish involved, these people 

represent time, effort, and money, all of which are in shcrt 

supply these days. We know that high school gL-aduates have 

fewer disciplinary problems than non-graduates, by a ratio 

of one to four. (Yet we know tliat the majority of non- 

graduates can do bzell.1 

The question is why? I?;lat arc thc:factors working in 

this area? Can we, can you, identify them? IS it something 

we're doing? Should we try to deal with these people before 

i 

they come on board or after, or both? /These are tough 

questions; but they need to be addressed, and I look to 

you 'for the answers. 

After we get the right kind of pe ple, we also have to 

4 

manage them properly. I sincerely believe young people todhj 

are as eager as ever to do a good job/ 

/ 

I also know they want 

20 ask questions. They want to know why as well as what; and, 

/ 

above all, they want to be challen.@ and provided the 

opportunity to participate in decisions that affect their

This means we have to CJoid arbitrarily poking those 

people into holes. We have to realize and act on the 

premise that what may excite and challenge one man or 

woman may not challenge another. In the long run, 

hsbitrary, Mconvcnient" actions can create a situation 

in rqhich an individual who might havt been it tremendous 

success ends up being bored and indifferent to his work 

or elects to leave us. 

Our need ht.re is obvious. We must have the best tools 

we can get to classify and assign people. WC must have the 

best predictors of success in training and on the job that 

you can give us. WC must be as sure as we arc ablt: that 

the system is doing everything it can to provide each pr:rson 

who corn&s to us the best opportunity possible to grow, and 

learn, and progress to the limits of his or her capabilities. 

I would like to underline the fact that this philosophy 

applies to the minority as well as the majority membera among 

us. 

Like every other ser-lice, the Air Force is tryiny hard 

to satisfy the expectations of its minority wcmbers. NC have 

almost 250 enlisted career fields in the Air Force. A little 

better t!lan a year ago, 91 of them had less. than a 5 portent 

minority population. Today that number is down to about 45, 

and we are heading for 0. Like the other services, we arc 

working the problem and the effort is starting to pay dividends. 

We still can use help, all the help you csn give us. 

6 

. 

. .‘.

What I have been discussinq so far is just a small 

fraction of tho challenge we face. Even thia!incompl+to 

Y 

iist, however, indicates the ireed today for jqqrmsivc, 

decisive action. This is the time for the military to 

question, to Search, to fin& new and better whys to manage 

its human resources, to t.?ke the lead in making its own 

unique contributicns to C:;C growth and development of 

military people as vital members of society. 

We must ask ourselves, What arc the implications for 

testing? Xhat role should tests play in this coming decade? 

Of all the many specific uses to whicjl tests arc ~)*.zt, is there 

any one direction or unifying thcmc? I bclicvc there is. I 

think there is a clear direction in which testing should 

develop, and I think there is an unddrlying theme for that 

i 

b 

,'rection. 

I 

It seems to de that the requirement to

44 Which means what, specifically? 

First, I think it means that tests of the future must 

focus much more on the individual, on his unique talents 

'and desires. 

Second, it means fittir.3 tests into a scheme that 

maximizes the-opportunity for c!loice, both by the individual 

and t,y his employer. 

-- 

). bird, as a function of the first two, it means that tests 

must provide broader profiles of information. Four or five 

aptitude scores do not provide a sufficient basis for the 

kind of cozqlex, comprehensive career programs we have today 

and will need in the future. 

The role cf tests, now and in the future, should be 

to supply a vital piece 0f the information we need in properly 

managing the careers of our people. The quality and breadth 

of this information will be crucial to the quality of the 

decisions he make, but tests must not serve to prejudge the 

individual for a full career. They should clearly be a tool 

in the decision process. By that I mean that they should show 

the implications of various alternative decisions. 

For example, tests are needed, especially with the advent 

of the new instruction systems, that can recommend specific 

kinds of learning experiences that can be applied to an 

individual case. Lee Brokaw, your program chairman, put it 

well recently when he said, 'We must shift emphasis from finding 

the best indfvidual for the job to find the best job for the 

individual." 

t 

8 

3 

. : .: .< 

. -. . . ,. ,. 

_

New systems are also needed to provide a more 

balanced role for tests, roles that clarify where tests 

fit in career development, roles that make sure the 

limitations of testing arc equally clear. 

We also have to remember that testing policies and 

practices must have a relationship to this nation's goals 

in the area of sociai policy. The Civil Rights Act of 1966, 

the Supreme Court decision in the Duke Power Company case, 

and the Equal Employment Opportunity Act of 1972 all really 

make one point. Equal opportunity means giving.all Americans 

a chance, and this chance must be more than just a chance to 

fail. 

As Lyndon Johnson said in the last address he made 

before his death, "History !las not been equal for all 

Americas." We recognize now that tests can be used 

negatively. They can perpetuate a cycle of assignment to 

jobs with limited growth potential. They can confirm the 

"inequality of history," in Lyndon Johnson's terms, by 

making past difficiencies the basis for continued unfairness. 

In short, they can become sentences instead of doors to wider 

growth and development. 

9 

.- .-. 

/ -_. 

. : .

-\ * . . 

Ladies and gentlemen, if we are to have viable armed 

forces in the future, we must come up wiCh reasonable answers 

to the questions and problems ccnfronting us in personnel 

management. You know our requirements; you know how urgent 

they are. 

i! 

Today, as your keynote speaker, I ask you to provide us 

the tools we need to satisfy th?-e requirements. lu'hen you 

bring us the tools, tell us how to se them and how not to 

use them, and make sure we listen. As 

- - - - a manager with a 

primary interest in people, I can tell you we do nerd help, 

- 

,, 

the kind of help only you can provide. I ask that you come 

forward and tell us what should be done. 

For the shcrt term, give us the best answers you can. 

I 

Remember, we can't always wait for the research effort to 

be neatly organized, the r"clght researchek to be found, the 

research effort-to be conducted in a Simon pure environment, 

etc. Very often we need answers now. 

I 

I challenge you to 

i 

respond quickly when the situation deman s. 

P 

For the long term, identify the arc/as where we should 

encourage further research: research that is relevant; 

research designed to tackle the tough &oblems of tomorrow, 

not yesterday's problems; research that will enable us to 

achieve progress that is real and enduring. 

10 

-. 

t 2; F . . 

1. .., 

- 

. 

. i ‘, 

: I’ 

, 

00033

And finally, let the experts and the managers (you 

and I) join forces, with the knowledge that there is 

immediacy and urger.cy in the work we do, that there is 

excitement and opportunity in improving the management 

of the most precious asset entrusted to any organization. 

Xow may not be the best of times, but it is OLX time. 

- 

Let's use it wisely. 

11 

k 

._ ._ 

-. . I . 

'r

. 

Participants: 

Synopsis 

MTA Symposium - Cooperative Aptitude Resedrch 

Programs Conducted through the Armed 

Forces Vocational Testing Group 

Dr. Harry D. Nilfong, AFVTG 

5la.i Verna S. Kellogg II, AFVTG 

Capt Kevin T. Gambreil, Jr., AFVTG 

Dr. Lonnie D. Valentine, AFHRL - Chairman 

Introductory Comments (Dr. Valentine) 

1. The Armed Services High School Test!nq Proqram has gradudlly 

evolved since 1966 into a program of sizable imoortance. 

2. In today's all volunteer era, it will probably assume greater 

importance to the services as a constructive point of contact between 

the services and the educational community with its pool of young people 

entering miiitary age. 

3. For about the past 6 months, the High School Testing Program 

has been under management of the Armed Forces Vocational Testing Group 

at Randolph Air Force Base. Other participants in today's symposium 

are from that organ'zation. 

4. After the participants have each briefly discussed some 

aspect of the program, the floor will be open for audience questions 

and suggestions. It is hop.?d that the discussion period will serve 

to elicit constructive suggestions for the program. 

Program History (Major Kellogg) 

1. A popular stereotype of the recruiter is the "lifer" bodysnatching 

young men from the qhetto and the juvenile courts and 

channeling them to the rehabilitation environment of the military 

services. Nothing could be further from the truth. The services 

need “quality" human resources--military services today demand 

trainable, educable and alert individuals. 

2. The sophistication of today's weapon systems requires the 

military services to expend more time and money training a man to 

an accepttule skill level than they can anticipate realizing from 

his services. Thousar,ds of young people leave the military service 

after their first e,llistn?t. Return on the taxpayers‘ investment in

. - . . 

them is realized mainly in terms of the skills they take back into 

civilian jobs. 1 

3. 

military 

directly 

Because of this requirement for quality manning, the 

services have tied their recruiting and enli'stment programs 

to the measurement of aptitude for vocational training. 

4. The Department of Defense, this year, directed the formation 

of the Armed Forces Vocational Testing Group--to direct, manage 

and guide the Department of Defense High School Testing Program. 

This program is the cornerstone of recruiting the all-volunteer 

force. 

5. The primary tool of the testinq program is the Armed 

Services Vocational Aptitude Battery, c;mnonly referred to as 

ASVAB. 

6. ASVAB is the product of over 30 years research and 

development in the area of military classification--mxt of it 

accomplished by people in this room. Today's en?istee is carefully 

tested for identification of his potential for training. 

7. The genesis of the High School Testing Program goes back 

to 1958, when the Air Force recruiters struck 3n the idea of going 

into high schools with a Vocational Aptitude Battery, enabling 

them to identify specific youngsters with specific aptitudes as 

potential recruits. With the success of the ,Air Force Program, the 

other services initiated sidlar programs. The competition caused 

confusion and projected an unprofessional image of the military . 

services. 

8. In 1966, the Department of Defense directed that the services 

combine their high school efforts. 

I 

9. A group of service behavioral scientists n-et and the 

eventual fruit of their efforts was ASVAB--ASVAB combined the best 

and most desired features of all the militarb classification tests 

and gave the High School Testing Program a product usable by all 

the services. 

10. The concept was good--but differedi parts of the action 

were assigned to the various services--theYArmy was tasked to _ 

develop the test, the Navy was tasked to provide counselor's 

materials and the Air Force supervised the scorinq and return of 

the test to the counselor. Recruiters went back into the schools, 

all using the same test, but still with their parochial interests 

at heart. Thus, ASVAB became an Army test, a Navy test or an 

Air Force test depending upon the representative who got there first. 

13

Both recruiters aid counselors lost sight of the value of the 

battery in helping young people p:an their futures--and got 

Ii hung up on the idea that ASVAB was only a means to get names 

of potential enlistees. 

11. The Armed Forces Vocational Testing Group is the answer 

to the problem. Staffed by personnel of all the military 

services--Army, Navy, Marines and Air Force--the group provides 

a single manager and interpreter of ASYAB for the Department 

of Defense High School Testing Program. Dealing witn t+e local ', 

recruiter through an interservice recruitment committee (organized 

by geographical location and represecting each service) the group 

assures equitable and professional atiinistration of ASVAB. 

12. The mission of thr? Armed Forces Vocational Testing Grouo 

is to place ASVAB ifl the schools, asslrre its currency and validity 

and prov‘.le the student and counselor with assistance and 

information. Our job ends there;

tests for the joint battery from a pool of test items composed of 

those in the interchangeable components of the existing service 

batteries and (C) standardization of the resultant battery. 

4. To c;tablish test interchangzability, all of the service 

classification batteries were administered to a common sayiple of 

3900 Lasic trainees from the Army, havy, Air Force and Marine 

Corps. Testing was arranged for three separate days, 

with only 017e of the service batteries being administered on a given 

day, and battery administration was counterbalanced to eliminate 

effects of test practice. 

5. A stratified subsample, re;lresentative of the full range 

of ability on the Armed Forces Qualification Test, was used to 

establish intercorrelations among all of the tests in the various 

service classification batteries; this resulted in the identification 

of seven interchangeable content areas. Two other areas were 

added to ASVAB to yield an AFQT equivalent. 

6. The nine ASVAB tests yield five aptitude composites which 

are reported to high school counselors through standard format. 

These are: 

. (A) Electronics - describes students in terms of abilities 

relevant to electrical/electronic occupations. The aptitude score 

consists of tests dealing with electrical information and understanding 

mechanical principles. , 

(B) General Maintenance - describes capabilities relevant 

to a variety of mechanica. and trade jobs. The composite consists 

of tests assessing she:, information and spatial ability, 

(C) Motor Maintenance - is concerned with ability for 

engine repair and related jobs and measures automotive information and 

understanding of mechanical principles. 

(D) Clerical - measures ability relative to clerical/ 

administrative occupations. Composite concerned with verbal ability 

and clerical speed and accuracy. 

(E) General Technical - describes students' ability for 

occupations requiring academic ability. This aptitude is taken from 

verbal and mathematical components of battery. 

7. The battery contains 3C0 test items; 100 in the Coding 

Speed and 25 in each of the remainjng eight tests. Actual testing 

time is 112 minutes; generally two and one-half hours are required 

to administer the battery. Speed is not emphasized, although all 

I 

15 

._. _.- 

.s 

‘. : 

..I:. 

. .

.- -- . 

rests are timed. 1) 

e 

8. Te::', items contained in Form 1 of the ASVAB (with the exception of 

those in Coding Speed) were selected from an item pool consisting of 

a'1 ite:ns cortained in the service classification batteries used in 

the '~inte~.~lla.lgeability" study. Criteria for item selecticrl were 

difficulty level (proportion of examinees responding correctly), 

discrimination level (ability of the item to discriminate correctly 

betw:?tin in4jivilluals irho score high and those who score low on the 

relevant abi'lity as reflected in the item's correlation with other 

items of its type), ar,d content validity. 

9. T!c 25 items in each subtest (except coding speed) were 

2r?3;iCJl?d

epresentative sampling of 36 thousand cases producing separate norms 

by geographical region, sex and high school grade, It has been my 

experienc.:, thus far, that these are the three most common comparative 

variables requested by high school counselors. 

4. The other ASVAB research requirement is to expand the 

listing of civilian/military occupations as cataloged in Volume 2 

of the counselor's manual. What is being planned - assuming sufficient 

lead-time to permit publication for the 1974-75 sch,ool year--as a 

two-columinar hierarchical listing of representative civilian and 

service jo5s anchored to aporopriate dictionary of occupational titles 

numbers, 

5. Long-range research needs identified in support of the ASVAB 

program are too-numerous to cite during this brief presentation. 

Howevtir, I would like to touch upon some of the more significant efforts 

that are planned. 

6. First, content revisions are being considered, including 

deleti.on of the tool knowledge subtest, possible shortening of 

some of the other portions of ASVAB and the insertion of an 

occupational interest schedule. Tnese changes should appear in 

Forms 5 and 6 of the ASVAB, hopefully, to be available by the 

1975-76 school year. 

7. It is the general consensus of researchers exposed to the 

AHAB that a highly desirable research undertaking would be development 

of current mobilization base data--representative of modern America 

on as many strata as possihle. What is envisioned is a study broad 

enough to serve many masters and the needs of many agencies while 

simultaneously substituting for the old 1942-44 mobilization data 

base. The many problems involved in planning, securing approval 

for, and crecuting such an ambitious undertaking are well recognized, 

8. Another, equally critical, avenue for research exploration 

i's to establish a mechanism for conduct of a series of longitudinal 

validation studies tracking students at successive staqes along their 

career path, both within and outside of the military service. By 

its very nature, the ASVAB program presents the ideal vehicle for 

performing such research. 

9. A wide scenario of research support requirements has been 

identified for the ASVAB program, including subjects as listed in 

Attachment 1, Armed Services Vocational Aptitude Testing Research 

and Development Support requirements, of the joint services 

regulation covering the Armed Forces Vocational Testing Program. 

17 

? 

*

10. 

Research needs identified by high school counseling personnel 

and by service recruiting, production and classification representatives 

'range across a broad spectrum, covering such'top!cs as: tracing the 

longitudinal career decision process; exploring the feasibility of 

Iremote test scoring and concurrent feedback; establishing a task/ 

aptitude criterion matrix, permitting, in turn, rapid estimation of 

predictive validities for new jobs entering the inventory; reconfigured 

.jobs, or jobs having an insufficient N for traditional validation 

purposes; and a linear programned approach to test administration. 

11. Rather than describe the entire spectrum of our anticipated 

research objectives at this time, I would prefer to return the floor 

to the workshop chariman so that we can explore subjects in which 

you may have a particular interest. 

Concluding Connents (Dr. Valentine) 

1. My.few corrnients represent my personal views--not necessarily 

policy of AFHRL or the Air Force. 

2. AFHRL's role, as well as that of the other service R&D labs,, 

in the program is viewed as one requiring sound R&D to supnort and 

improve the program. Policy with regard to the program is the 

responsibility of service and DOD ffq agencies. 

3. There are three areas in which substantative support R&D 

is needed. These deal with (a) breadth of battery content and 

coveraqe, (b) battery standards (which include both norming and 

composite structure), and (c) va?idity studies. 

4. In its oresent form, ASVAB is limited to subtests either 

"common" to earlier service classification batteries or necessary tc 

provide an AFqT score. Most of these predecessor batteries were 

designed to provide ontimal prediction of training success. Such 

composites tend to intercorrelate substantially. School counselors 

have always been constrained to counsel all students, and the optimal 

test system for them both identifies ability level and provides optimally 

different classification indexes. The services have been able 

to select rather than to simply classify in the true sense. 

IJnder the volunteer force the services are going to be compelled 

to join counselors in a more effective job of differential 

classification. In addition, the ASVAB's present limited content 

coverage short-changes some of the classification needs of some 

of the-services. Thus, the service labs need to give careful 

attention to appropriate added content and to the classification 

models ?lhich will allow decisions about both leve1 

of aptitude 

and thz differential aptitude demands of jobs 

18

5. The mobilization population standards against which the 

services typically calibrate new tests is now 30 years old, and it 

does not take into account special subgroups of particular interest 

for special programs. A broad based national normative;census, 

with provision for identification of standards aopropri,ate to 

particular population subgroups, is needed. Some counfelors have 

expressed dissatisfaction with the service's 194'1 mobilization base 

standird and have,maintained that they want standards based on high 

school age youth, 

, 

:! 

6. Substantative ASVAB validation studies not only against 

service criteria but against civilian career criteria as well, are 

needed. 

7. Careful attpntion to these three &reas, with cooperative 

planning and research by the various service labs, can lead to development 

of a comprehensive and flexible test battery capable of satisfying 

both service and counse?or need->, and can be an invaluable recruiting aid 

to all the services, It’s accomplishment will', however, require that 

the services lay aside parochiai interests, support the appropriate 

IlleD labs with the manpower and other resources needed for its 

accanplishment, and encourage long range cooperative planning and 

research. 

(The symposium was concluded with an audiejce question and 

reaction period.) 

19 

'1 I

DEXEtOPHEX~ OF THE ARNED SERVICES 

VOCATIONAL APTITUDE BATTERY i 

W. A. Fischl, Ph.D. 

U. S. Army Research Institute for the Behavioral and Social Sciences 

. 

The Armed Services Vocational Aptitude Battery (ASVAB) is the test 

battery administered in the joint services high school testing program. 

In this program the services offer to administer a battery of vocational 

aptitude tests in any high school that requests it. Trained test adminis- 

trators visit the school and administer the battery,.the tests are scored 

centrally, and results are returned to the high school guidance counselor 

for transmittal to the student. This program is provided to high schools 

that request it, at no charge either to the school or to the student. 

Identification of Content 

The tests employed in this program were devclop.ed to be a set of 

. short alternate forms of the common elements of classification test batteries 

in use by the services. At the tine of the original development each of the 

servjcss was administering a classification battery of nine or ten aptitude 

tests. To identify- the.cozmon content material of these batteries, the batter- 

its of the Army, Navy, and Air Force were administered to a national sample of 

3900 enlisted men during their first few days of service. Each enlisted man 

took the tests of all three batteries, in a counterbalanced sequence, one 

hattcry a day, over a several day period. The sample was subjected to statisti- 

cal adjustment to equate for diffcrenccs in screening methods and acceptance 

standards among the services, and the test scores were intercorrclatcd 

to determine which test content areas were sufficiently sinilar as 

to comprise the core of a joint test battery. The criterion of similarity 

20

was set as a correlation cocfficicnt of 0.9&‘* and th$s analysis identi- 

Eied the r=ven test content areas of verbal fluency, quantitative 

reasoning ability, spatial visualization, mechanical cbmprchension, shcp 

I 

inftirmation, automotive ‘information, and electronics ,inf ormat ion. An 

eighth content arca, clerical ability, uas added because of its impor- 

tance for counseling and classification, even thougbthc service tests of 

this function did not meet the statisrical criterion. Finally, knowledge 

ahout hand tsols was added as ;I content area so that all four arcas 

assessed in the Armed Forces QualificntL>n Test (AFQT) would be rcprcscntcd 

Construction of the Original Tests 

For all bu: the clerical content area, 2> items, spanning a full 

far.;.= sf diffic-zlty, vcrc sclrctcc! fr,:: t!v2 c?craticc21 tests cf the 

services. These items were assembled so as tb compiise new iorns of the 

/ 

I 

service tests in the respective content xea.sL The clerical ability test 

was assembled by taking. i&t, the most val"d of the service tests in . 

r’ 

this area. It is a 100-item speed tcsti all of the others are power tests 

. 

and are, with the exception of Tool Knowledge; I half-length forms of the 

scrvfce tests in the counterpart areas. 

:'he 001 Sncwladgc test contains 

f 

the same number of items as was contained in the Tool Knowledge pcrtion 

of the then oyerJtiona1 AFQT. 

Exhibit 1 describes the contents of th’!! nire tests of the battery. 

Q . 

To obtain norms, these tests were administered to a r.ationnl sample 

Insert Exhibit 1 about here 

L/ Corrected fer restriction of range and lack of pcrfcct reliability.

I. 

’ 2. 

3. 

4. 

5. 

6. 

7. 

8. 

9 .I 

EXHIBIT I 

TESTS iN THE 

ARteP SERVICES VOCATIONAL APT; IUDE B4TTERY 

CODING SPEED TEST (CS>. IN THIS TEST =ERE IS A KEY AND IO0 ITEMS. 

THE KEY IS A GROUP OF WORDS WI TH A CODE N’UNBER FOR EACH \

of 3050 Selective Service registrants, stratified to represent the 

no'pulation of young men of military age. 

Co,nstruction of Successor Forms 

Coincident with introduction of the original form of the battery in 

the school year of 1968-69, work began to develop two entirely new forms. 

For each of the eight poh-er test-&./ six times the required nunbcr of items 

were prepared. T!~is is customary procedure, preparing a substantial overage 

in anticipation of attrition in the item analysis stage. These items were 

administered to several national samples strarified to represent :he popu- 

lation of military age young men. The total fi was 4000 cases, 18% of 

whom were bJ.acks, and item analysis statistics of difficulty and homo- 

geneity were obtained. Using these statistics, items were assembled into 

two parallel 25-from forms of each of the eight power tests. Each test 

was constructed to be of wife ranging difficulty, with items spanning 

virtually the cntirc range of 59 per cent difficulty (very easy) to 1 

per cent difficulty (very hard) in the reference population'. h'hen coupled 

w!Lth the new forms of the speed test, there were two entire?y new nine- 

test batteries.. 

These batteries were administered to several stratified national 

samples of Selective Service registrants, one forn to a sample. The 

total 2 was 3500 cases, virtually all whom were betxecn 18 and 19 years 

of age, 80% of whom had completed between 10 and 13 years of education, 

and 15.5% of whom vcr'c black. From this administration percentile and 

Army Standard Sccru norms were developed, and test reliability coefficients, 

intercorrelaci7ns, and other characteristics r;ere derived. 

l/The speed test was prepared in Its final forms, because item analysis 

pf a speed test is not meaningful, and item attrition 53 not incurred. 

i 

23 

.A - 

.._. --. 

: .

Characteristics of the Battery 

The entire battery requires approximately 2 l/2 hours to administer, 

! 

The longest test, Arithmetic Reasoning, takes about 25 minutes; the 

shortest, Coding Speed (clerical aptitude), takes 7 minutes; most of the 

t 

others are lo-minute tests. 

Reliability coefficients are presented in Table 1, and test intercorrc- 

lations appear in Table 2. The intercorrelations are of intermediate 

Table 1 

ASVAB Reliability: Coefficients of Equivalence and Internal Consistency 

for Each Test 

‘Test Equivalence 1/ Internal Consistency 21 

Coding Speed .83, .86 h‘ot Applicable 

Ir!ord Knowledge .80, .85 .87 

Arithmetic Reasoning .81, .86 . .87 

I 

Tool Knowledge ;79, .76 .78 

I 

Space Perception .82, .84 . 84 

" 

Nechanicol Comprehension .77, .73* 1 .83 

Shop Icformation‘ .67, .65 .81 

Automotive Information .75, .74 : 

.85 

Electronics Information .80, .79 

1. Correlation of Form 2 with Form 3, in two subsamples, Form 2 

administered first in one (IJ = first in the 

other (2 = 514). 

2. Utilizing Ruder-Richardsonfirmula 20, g = 616. 

(insert Table 2 here) 

, .’ 

:’ 

24 

i 

.83 

_,

t 

. 

‘. 

-.: __ 

_ - .-- 

25 

3 

N 

,

magniture, with the perceptualmotor test (Coding Speedj being generally 

most independent of the rest of the battery. 

The reliability coefficients stem reasonably high for tests of only 

25-item length, and pred.ictive accuracy is enhanced operationally by 

utilizing combinations of sever31 tests. h’hy the Shop Information test 

yielded the lowest equivalence coefficients in the battery is not apparent, 

especially since its internal consistency coefficient is of comparable 

magnitude to the rest of the set. 

Occupational predictions fron ASVAB arc generally made utilizing 

combinations of several tests. These combinations, both with ASVAB tests 

and their counterparts in the services’ operational bdtteries, have undcr- 

gone extensive validation study against criteria of training success in 

large numbers of job specialties. The modal validity cocf ficicnt against 

these criteria has been of the order 0.60, with a range of 0.35 to 0.75. 

In summary, the ASVAB is a nine-test aptitude battery developed to be 

shortened alternate forms of the ccrrron elements of the then service classifica- 

tion bnttcrics. It has been empirically developed, and tried out on large 

representative.samples of the population to which it is intended to bc 

applied. The original form was in USC in the joint service high school 

program for five years, Form 2 has been operational in the schools since 

the beginning of this school year, and testing goals are in excess of a 

million students by the end of the year. 

26 

: . -. 

. -.. 

T..’ .,

“hXW DEVE.LOP?lENTS IN DEFENSE lANGl%E PROFIC IEYCY TESTING” 

PRESEKTED BY DR. C.D. LEATHERHAN ASD DR. A. A&-HAIK OF THE 

DEFEKSC LASCUAGE 1PiSTlTU-l-E !’ 

1973 MILITARY TESTING ASSOCIATION COSFERENCE, SAN !ANTONIO, TEXAS 

28 OCT - 2 KOV 1973 

f 

Dcfrnsc l~ngu.~gc Prnfici.ctxy Tests, commonly referred to as DLFT’s arc tests 

drt;i~;nr-d to l:t;nsurc the language proficiency of all DLI graduates, as well 

as DOD-conncc;ed individuals clniminp, proficiency in a particular forcigu 

I.-tn~t.:a~e. Specifically, the DLPT serves c*10 ;.ljor purposes: 

a . To measure the listening comprehension and reading comprehension 

skills in a foreign lnngu.?gi*. Thcsc two skills range from minimum ability 

whi cl1 rh?y .5r useful in an activity such as directing traffic, to the skill 

nccdrd by a compctcnt interprctor. The test does not discriminate among 

the higher language skills rc.auirrd by a top-level isimultal:rous tranaln tor. 

I 

1~. 

To evaluate and rate the cxnminec’s abi?itd t0 meet the lillglIiStiC 

9 

rcquircmcnts for particular jobs, ‘such as interijre$or, interrogator, and 

translator. ?he test is not intcndcd tu assess other skills, such as 

intrrrogntion techniq~cs, w!:ic!: cay bc rrquircd t:y these jobs. 

I 

The history of military proficiency testing goes back to 1948 when the Army 

I irs,L developed what was then called The irrmy La tr” guage Proficiency Tests. 

Tests in 31 languages wcrc developed and used between 1948 and 1953. Each 

trst consisted of four parts: Parts I and TI covered understanding of the 

spoken langu?gc; Part 111 dealt with readin&; ability: and P:!rt IV mcnsured 

writing ability throug!l I.:uawlcdCe of grarrsr;lr. Due to n.ilitary pressure.+ at 

27 

I 

i 

_ 

. . . . 

,

the time, the development of these tests involved only a niinimum of 

II 

research. During and following tlw Korean War it became apparent that 

the tests, vcrc not ;~Iw;~ys rcl inhlc in discriminnt ing hetwwn individuals 

of varying levels of 13nf;uagc ability. As a result of this recognized defi- 

cicncy, Ar111y USC c agencies rc*qucr;tcd the devcloplwnt of new Arm:’ Langunge 

l’roiicic:,jry Tcs1.s. Accordingly, in 1.955 ci’hc Adjutant Gcncrnl’s Office was 

(the predecessor of The Dcfcnse language Institute) was subsequently directed 

td preparc ln:,~:uagc tests, hutlr to rcplacc esisting i.~~strinnrnts and t o dcvclcrp 

new tests: ‘in langu.il;es for xhich tller(. wcrc no tests at tllnt tinrc. EVCIl 

though t!;e new prototype tests in Russian and Chincsc -Xxnd,7rin were carcfltlly 

clc~~lopec! a n d validated, the ot.hcr new tests, npproxir2ately 35 in number, 

consisted mainly of translations of lhc tx0 prototype tests and wet-c not 

subjected to systematic validation procedures before their ictroduction into 

tlbc Dcfrnse Language Progrnx in 1958. As a rcsul L, Russian proficiency test 

was used as the model ior all European languages and the Chincsc-?bndarin 

test was used for all XsiaLic 1an~;u:ly.c:~. It became apparent following 

extensive USC during the sullscquent ycnrs that these 38 additional tests wcrc 

not as valid or as reliable as had been assumed. Additionally, the USC of a 

skglc test form for each of thcs’c tests over mxc than a decade further 

reduced their cffcctivcnc: :. .i::r to possible test compromise. 

/ 

, 

Recognizing these 

defi.cicncies, The Defense L3njiuage Institute initiated a series of new projects 

in 1366 aimed at developing follow-on DLPT IIs, or second generation DLpTs to 

replace the old ones. The fi.rst group of DLPT LIs, wcrc developed under 

28

contract with the Educational Testing Service, utilizing language exptrts 

from the DLI and from various universities, in addition to resources from 

ETS . Scvcn high-cnrollmrnt languages wcrc included, such as Russian, 

Spanish, Arabic, etc. These initial DLPT 11s are introduced into the Defcnsc 

Language Program at the present time. As these tests are introduced into LIIC 

system and prove to bc operationally sound, other language proficiency tests 

and al ternatc forms wil 1 then tic introduced into the system. Corrznt plans 

call for the dcvclopmcnt of two or three alternate forms as a minimum for 

each high density DLI’T II. DLI’s MD plan includes the construction and 

valic!;llion of a computer bank of tesL items for each language, from which 

multiple test forms can be asscmblcd automatically and actually produced 

in final form, .)oth test bonklcts‘and audio tapes, by co,?lputrr. These wcrc 

the facts and some of the events which led to our current efforts with r?g;trd 

to second gcncrnt ion DLPTs. Additional details concerning specific aspcc!!; 

of the dcvcloyncntnl work program are given ‘belcw. i 

?hcsc early prototype tests were developed by Army Language School pcrsonncl, 

in cooperation wilh rcprcscntativcs from the Personnel Research Branch of The 

Adju.tant General’s Office. Each test has two parts: Listening Comprehension, 

consisting of sixty items and Reading Comprehension also consisting of sixty 

items. All items nrc mulliplc-choice type, each consisting of a lead sentence 

or a short paragraph in the foreign lansuagc and four optional responses 

printed in English in the test booklet. Three of t&se options are plausible 

distracters yhcrc;ts, only one is the correct answer. The options can be words, 

phr.?ses, scntcncc Lrnns la t ions, or paragraph gistings depending on the objcc- 

tivc of the itml. Listening comprrhcnsion item lpads are voiced on a tape, 

29 

I 

/ 

. 

. 

‘. : :-. 

. . . 

-.

ut rcoding comprehension item leads arc printed in the test booklet. After 

completion; the tests wcrc field tcstcd, including a validation !measurc, with 

Army ~~JCS~~~I?IKFI possessing varying c?cgrecs of skill and experience in the two 

r 

languages. The validnt ion instrument consisted of job-oriented F tasks such 

as intcrprcling and translating. 

1 

The median correlation between the Chinese 

test and tltc crilcrion mcnsi~rf VdS -70 and hctwcen the Russian test and tllc 

: : 

criterion c~ensur~ .7s;. The K-R 20 rclinbiliLy of the Chinese test was .96 

a n d the Russia11 t.c:,t .95. 

Language specialists at The Army Language School wcrc furnish4 English 

translatihlls of the two prototype tests mentioned previously and t;crc then 

instructed to use these translations, item by item, i.o’drvLJoping the new 

tests. Since the cordi Lions for dcvinting fro:3 the givrn translations were 

few and rathrr rigid, Lhc resultant new Lcsts WC:rc essrI?nti.nlly parallel in 

i 

many ways: to the prototypes. IJnfortugItcly, not only Get-c the options identi- 

cal, 

I 

i 

but the correct option rcnairicd as the snmc key across all tests. It w3s 

subsequently ‘discovcrcd that this proccdur~ increased the* ChUCcS for test 

c0mpr0niise, much murc than usual. The reason is that 

any DLL graduates would 

cithsr subsequently rrcnroll in a different language Course 

t 

or take a DLPT to 

B 

establislh military competency in a foreign languagr studied clscuhcre. 

Fur thcrmorc, an crroncous assumption was more than 

that since a11 the tests were parallel in content 

therefore all have the same lt;~l of difficulty. For this reason, no attempt 

was made LO standnrdizc raw scores and the snmc skill-level cut-off scores 

were used across al 1 lnnguagcs. This initial assumption was unfortunate 

and was incorrect as later validation data di.sprovcd the theory. 

30 

c 

.., 

.: :! 

‘, ;: 

/ 

.’ 

i 

i

ii DE\~LI.OF>lE:.T OF S):CO:;D CE:XRATIOK DEFEWE LASCllACE PROFICIENCY TESTS (DLFT II) 

It was considcrcd essential to correct the operational situations described 

nhovc and to rcpl3cc the then cl:rrcnl tests which were outdated and 

possibly compromised through the ovcrusc of only one test form per language. 

Accordingly, second Kcocrcttinn DLPTs ilrc being dcvclopcd, printed :tnd 

1 

syst~rx~ticnll~y intrvduccd into Thcb Il(~fcnsc Language Program during o three 

to four y-car puriod. lhc Iii;’ sccontl Rcncration tests have the following 

characteristics: 

il. Although they arc identical in format to the old test, the content 

of cacll is uniqlrc and indcl:a,ndcnt from nny other. Test content is bilsed 

strictly on a description of skill-lcvcl, ObjcctiVFS f o r rach longua{;c. 

h. Current plans specify ztt Icast lhrfc alternate forms for high- 

density languages and tvo forms for low-cnrollcent languages. 

c. All test forms xi11 be subjcctcd to item onnlysis and form-equating 

procedures. 

d. The validarion of inch test wi.11 be based on the collection and 

anslpsis of field dnta conctrninh job-pc’rformance ratings. 

c. Each newly developed test will be independently normcd and new skill- 

lcvcl cut-off scores will be cstablishcd. 

. 

f. Raw scores will be converted into standard scores to achieve mutual 

ccmparabili ty for languagks which have varying degrees of difficulty. 

31 

.

DLL p,m PH~OKITY 1 (DLISDA, p - 0 0 1 5 ) 

(CN) Chincsc 01) 

(‘UC) 

Arabic - - (Kodcrn Standnrcl)k 

T:j;ypL isn 

(A J’> Syr i.an 

I ,*a q i 

Saudi 

(FR) French 

(CA) Crrrzlrl 

(.I,\) .13it311( .‘.l! 

(,LA) Spanir;h (LA) 

(Td) Thcl i 

(VN) Vietnnrncsc ( S ) 

(VS) vicctnnnK?sc (S) 

(RU) Russian 

(KP) Korean 

(CM) Chincsc-Mandarin ” 

(Simpl ificd ch:lr;lctc.rs) 

(SD) Chiwsc-Amuy 

(originally Pric,l-itp 111) 

‘I\..-0

. 

DLI R&D PRIORITY XII (DLISW!, P-0130-1 '2' 

(AB) Albanian (PT) Pro tuguesc (European) 

(BU) Bulgarian (RQ) pomnian 

(CM-N) Chinese-ktiinland (SC) iScrbo-Cro3tian 

(GR) Greek (SR) j Spanish (Castilian) 

( JT) Italian 

(SW) Swahili 

(ML) Ma lay 

(UK)* Ukranian (deleted) 

(PF) Persian (Farsi) 

: ! 

Initial R&D work is underway; completion is expected dur'ing FY 1975. 

DLI R&D PRIORITY IV 

(AA) Africnans (NR) Nor-ncgian 

(PA) Danish 

(PC) Persian (Afghan) r/ 

(DU) DuL ch 

(SL) Slownian 

(FJ) Finnish 

(Cl!) Svcd ish 

(HE) Hcbrcw (Hod) (UR) Urdu r/ 

Prclir;;innry R&D xork is underway; due to the lower priority a schcdulcd cozpie:ion 

date is not,feasible at this time. A/ No DLPTs are,available r,nd structured 

intcr\:icws may bc the rnorc expedient alternative. 

W 

_ 

“’ 

. . 

33 

I 

I 

i 

! \ 

_. r * i ’ 

_. 

I 

d 

/

Introduction 

* Enlisted Selection and Classification Testing 

in the US Army 

Nilton H. Maier 

paper presented 

Nilitary Testing Association 

San Antonio, Texas 

29,oct - 2 Nov 1.973 

For saveral years now testing has been in a state of turmoil. Pre- 

viorsly * tests were quietly given ‘in schools, Industry, and the armed 

St’rviCCs) an,t except for those who wet-c dtrectly involved, such as rc- 

scarchcra and personnel managers , no one really paid too much attention 

to what was happening in the field. In the 1960’s, t’.ough, testing, along 

with so many social practices and customs, was called into question, and 

today tosting practices are regulated by law, the courts, and the Equal 

Enploymcnt Upportunities Cozrxission, %s well zs by the test developers and 

users . Wtcrc3s in Forrim- tines , a new test miG\t be of local interest to 

the Immt*d!nte usgr #, or perhaps to ths ib:ofGssion.?l community if it had some 

unique charxt.:ristics, today a ncv test is subject to scrutiny by a good 

rxlnp parties. In the midst of this concern, the new Amy Classification 

Eattcry, or ACB, was born, and as night be cspt*ctcd, its arrival did not 

go unnotict*d. In my papb:r, I will describe briefly what the new ACB is, how 

it is used operationally, and finally give some comments on how it has 

fared since its intrcdction on 1 Nay of this year. 

What ici the new ACB? 

Tltc new ACB arrived on tho scene at ahout the same time as the all 

� :oluntcor Army, and it is admirably suited to *he current needs for 

sc1cctin.r and classifying enlisted personnel. One advantage is that the 

34

new battery has a broader coveraKe than the old oXe. The two primary 

changes arc that we expanded the number of subtesty in the academic or general 

ability domain, 

1’ 

and that we included more tests of interests and attitudes 

I 

or mca~urcs of the willingness factor. We retained the old standbys of 

Arithmetic Reasoning and Word Knowledge, and we .Idded’ measures of, Scikce 

: ! 

Knowledge and ?fathematics Knowledge to provide more precise measurement in 

this +ri:ical area of general mental ability. By including these additional 

tests, we obtained more accurate prediction of success in the different job 

arcas. (Chart 1) 

The other major improvement is that WC added nore interest measures. The 

c3t licr ACE contained the Classification Inventory, which was a self-description 

instrument to help identify the infantryman. The original inventory grew out 

cf +#.a-n.r.-h hCIn..O A..%--:” 

__-_-_-.. --J--- mu--.. 6 the Korean War. j?e did not, hsmvtr, hz:~ zy ~th:r 

self-description measures ,in the ACB, untii the new’ ACE was dcvcloped. Row, 

i 

we still have a measure of cpmbat interest,, qdxred during the Vict Sam con- 

I 

flict, and three other interest measures in electronics and mechani.:; main- 

tenance, 3nJ an interest in attending to d’Ftai1, used for clericnl-administra- 

tive jobs, field artillery, and for cquipm nt 

b 

operators, such as drivers. 

The intcrcst ncasures add to the prcdicti!n cf success in relevant job 

P 

training, in addition to the measures of general ability and specific measures 

of infomtion in a field, such as electronics information and automotive 

informat ion. 

.I _. 

Th? information tests also tap a willingness factor. They cover 

matcrfal that most anyone with an interest and aptitude in th,c area can 

pick up. Electronics and automobiles are common to our culture, and a 

young persca can readily bccoae expert in these areas if he has the initin- 

, 

a’ 35

u 

tive to engage in relevant activities. hnd as part of our rapid social 

change, young women no longer are as excluded from these areas as in 

former times . 

The Trade Information subtest covers content peculiar to the skilled 

construction trades. 

. 

Finally, we have two perceptual tests that do not involve any reading. 

These arc the familiar pattern analysis.or spatial ability, and a new test, 

called Attention to Detail, which requires the ability to discrfninate the 

letter C embedded in a series of 0’s. . 

The tests are combined to provide nine aptitude area scores, which are 

used as prerequisites for job training. Tbc nine areas cover the full range 

of Army jobs, from combat and field artillery, through rhe a?intenanct areas - 

ejeCtrenir+ -=--?r;J! , 

. Y-.. and mechanical L to clerical administrative end a set of 

jobs called skilled technical, our medics, W’s, miiitary intelligence, and 

t.he like. We have two other job areas that arc more unique. One is called 

surveillance and communications, which includes radio operators and target 

acquisition jobs. The other is called operiltors/food, which includes 

missile crownan, cooks, and drivers. (Chart 2) 

I should say that all of these job groupings are cnpirically based. We 

#did not force the jobs to come out in any special way. We csamined the pro- 

files of validity coefficients both visually and analytically to determine 

which jobs clustered together, and the grouping shown here is what emerged. 

Looking at the subtekts in each aptitude area, it becomes apparent 

that Arithmetic Seasoning, M, is omniprcscnt; it occurs in seven of the 

nine r?rms, plus in CT, which i~ used to determine eligibility for addi- 

tional testing or special programs. Another fact is that the new aptitude 

. . . 

. 

: 

35 

. 

. 

.- 

.< 

-.: 

. I

areas are more complex than the old ones. These contain three to five sub- 

tests each. a.p contrasted to only two in the old aptitude areas. 

The result of putting so nmy tests in each composite is that the 

new aptitude areas arc more accurate predictors of success in job training, 

wtiich is good, but also that the scores arc more hig!lly inter-correlated, 

which at first seems bad. We found, however, that the new area scores do 

a much better job of sorting out the manpower into appropriate job areas 

than did the old ones. 

We analyzed the new scores hy subjecting them to extensive analyses 

througn simulating scores on a coorpul-cr to rcprcsent random samples from the 

mobilization population. WC assigned the simulated “men” to job areas hascd 

on both the new and old aptitude? nrcas scores. We kept the quota restraints that 

ccrrain pcrkenragcs.had to go into each job area which reiiccred rhe real 

Amy quqtas at the tine. The consistent result is that the performance 

estimates were higher under the new set of scores than under the old. The 

increase in predictive accuracy more than compensatcJ for the increased 

intercorrelation. 

Another gain realized from the new aptitude area is that each of them 

contain tests that require the ability to read. In former days, illiterate 

men could qualify for Army scrvIce, and many did. Under the new system, 

some functional illiterates iiro still gettfng through, that is, rend below 

the fifth grade level, but the number has been reduced to an estimated 

five percent. 

Based on the promise of more accurate measurement of job pote,ltial, 

the new ACB was adopted for opcrotional USC, and it was implemented on 

1 Play 1973. Kow let’s see how it is being used. 

37 ,

operational use of the new ACB 

The use of the new ACB has gone through two phases already in its 

: . 

short life. In the two months of May and June, a separate AFQT was 

administered to all Army m~lc applicants. Beginning l,.'July, experimental 

, * 

mental standards were adopted by the Army in which AFQT was dropped as a 

separate test, and an AFQT score is obtained from the ACB. The experimental 

mental standards were also changed:.on 1 July to require one aptitude area score 

of 90 or better for high school graduates and two scores of 90 or better for non- . 

graduates, in addition to a percentile score of 10 or better on the AFQT ob- 

tained from the ACB. 

The new aptitude areas made the experimental standards possible. Since 

each one contains some measure of general mental ability, plus specific 

aptitude measures, a qualifying score of 90 is more indicative of ability to 

succeed. The old aptitude arcas did not nieasure enough general ccntal abiiity 

to use with the same degree of confidence. As of the time of writing no 

I 

decision has been made about which standards to/use as the official ones. 

I 

The hCB is no*r‘ given at tlfe time of application of enlistment, which 

provides the recruiter, counselor and applicant with the best possible 

information about the applicants mental qualifiiations and the job arcas 

in which he is best qualified. 

I 

The tests are adninfstcrcd at AFEES or by Eobile Examining Teams. 

1 

KETs, who take the tests to the applicant. The entire battery requires 

about three hours to administer, which is long!= than the previous testing 

f 

time at AFEES, but the extra tims is well spent. The new recruit actually 

spends less time in testing than before. He used to spend up to eight 

hours between AFEES and reception stations taking tests, but now he 

spends only the three hours at AFEES, plus another half-hour taking the 

38 

I 

\ 

_

$4 

Auditory Perception, or Radio Code test, at AFEES if he is applying for 

training requiring radio code, or at reception stations, if he is not asking 

for a commitment for these jobs. 

The new ACB is also given to women applicants a: AFEES or by the 

Mobile Exanining Teams. An AFWST, i.e. Armed Forces Womens Selection’ 

Test, score is also obtained from the ACB. No additional testing of wouen 

applicants to deternine qualification is required. The new ACB is also 

being readied for-use by reserve components and for retesting at posts, camps, 

and stations throughout the world. Soon the entire dray will be switched 

over to the new ACB. The Ilarines also are using the new At.3, but just as 

wjith the Army, the transition is not yet comple:e. 

Evaluation of the new ACB. 

Gnc consideration in evaluating the new ACE was its effect on the flow 

o:E marginal men, or those with low general ncntal ability. Prior to intro- 

duction of the new battery, our research had shovn that fewer men with low 

A!FQT scores would obtain qualifying nptiturlc area scores of 90 or better. 

There was ‘SO~C concern that the effec:s would be serious enough to affect 

1 

t:he manpower flow. Our argument was that the Amy has little difficulty 

in recruiting men with low mental ability, and the experience since 

1 May has borne us out. In July, when the ceiling on the number of men with 

marginal mental ability was removed, over a third of the accessions were 

in the marginal category,’ In August, the influx subsided, and the number 

was’ more acceptable at just over 20 pcrccnt. Army policy in the past few 

I 

years has been to hold the number of marginal accessions to somewhat under 

20 percent, and that is about the number coming in. 

’ . 

t

The quality of new accessions in the past WTS measured on the basis of 

AFQT scores, which provides a single index of general trainability, All 

men regardless of aptitude in specific job areas, were categorized on the 

, 

bask of their general trainability. Witi: the old aptitude areas, the 

AFQT did provrde the best measure of quality and the procedure was � 

appropriate. With the new aptitude tireas, categorizing men on the 

basis of AFQT is no longer as appropriate: Since the new scores contain 

measures of general ability, the ntimber of qualifying aptitude arca scores 

at or above specified levels provides a more a-curate description of the 

q;Jdity of the accessions. 

We have been working with Army personnel management for some time to 

develop a new basis for categorizing quality. Although at the time of 

wr:Lting there has been no final resolution about the definitions of the 

categories, it is clear that the intent is to broaden the basis by including 

all the aptitude areas in develaping an overall quality index. The basis 

that is finally adopted should make scnsc from all points of view; 

some considerations are that it should have a known relationship to the 

mobbilization population, it should provide useful information to’personnel 

management and it should not stigmatize the individual. One of the complaints 

about the AFQT mental categories is that the Cat IV’s, or men in the marginal 

category, are generally assumed to be the misfits. While there may be no 

way to avoid stigmatizing completing, the new system should be descrfptive 

of trainability in specific areas, which may help prevent a general labelling. 

Another area of concern about tests is that of cffcctiveness of the 

tests for minority group personnel. The tests should be equally effective 

predictors of SUCCESS for all groups, that is, blacks and whites with the 

,,’ . 

40 

I 

-.,: 

._ 

: 

*

same test scores should have the same expected level of performance. The 

‘new ACB is equally valid for blacks and whites, and blacks and whites with 

the same test score have the same predicted level of performance in job 

i 

training courses. 

1 

From another point of view, the question is: are blacks underrepresenz- 

In the Army accessions as compared:!to the population as a whole? And.here 

lthe answer is no. In July, 1973, about one-third of the accessions were 

black, and in August, 20 percent. These numbers compare to about 20 percent 

in fiscal year. 1973, and about 13 percent in population as a whole. The new 

ACB thus is not excluding an undue proportion of blacks. 

The matter of racial bias in employment, in this case joining the Army, 

is of course more complicated than just n mttcr of test scores or even 

prediction of success from test scores. Test scores are only part of the 

jiob training, utilization on the job, and advanced training, would hardly 

I 

be a satisfactory solution fo:,eithcr the individual or the employer. 

1 

Another group that merit; special consideration is women applicants. 

The new .4CB is used to dcterninc mental qualif\cation of Amy women appli- 

c:ants. Thay cust qualify on the combination o Arithlctic Reasoning and 

L!ord Rnowlcdge subtcsts, which provide the AF&T score, and in addition 

3 

obtain at least two aptitude arca scores of 96 or better. The Army has 

had little difficulty in netting the quotas fTbr WAC’s. The ACB, however, 

was standardized on an all malt sample, 

// 

with both blacks and whites 

i.ncludcd, but no wozcn. The question ray nrisc whether the new ACB is fair 

to WOQCll. The answer is yes, to the best of our ability. Based on large 

samples of males and fci;ules in high schools, the population means on 

arithxtic and vocabulary, which constirute the new AFVST, are about the 

smc for both scxcs. As for the aptitude areas, both sexes rcccivc the 

:’ ’ 41

Ii 

same job training, and therefore women require the same aptitude as men. 

If the prerequisite for job training is 100 or average, on a particular. 

aptitude area, wonen as well as men need to meet this prerequisite because 

students of both sexes need to mdster the same knowledge and tasks. The 

ACB scores for women are considered to be appropriate measures of their 

potential to perform satisfactorily in job training. 

In summary, the new Amy Classification Battery has been in use for 

about six months. It is administered at the time of application for en- 

listment to prclride information about the applicant’s qualifications in 

the different job arca. The evidence is that it is not keeping out 

excessive numbers of marginal men, blacks, or women. Reports received from 

the field are generally favorable in that- it is an efficient and effective 

method to assess the potential of applicants. 

. . ; 

:i 

c I 

.

._ 

’ . 

_.- _ _ .I 

- 

, -- ._* 

.: ,_. 

43

I:* . 

Test.' .'. 

Ce~wxal 'Ability Test; 

-. . NE? APTITUDE AkEh COWOSITES ..I - . 

. . . :s 

Arithmetic Reasoning 

-* (AR) 

IGeneral InCornation @I) 

� klathcnatics Kno:Jlcdge (:Il:) 

'k'ord KnowlcdzC _ ow 

-.-. ,, 

Scfcncc Iktovledge 

. 

cm 

&chan$cal Ability Test& _' 

. m 

Trtde Inforclation W-1 

Electronics Infornlatj.on (EI) 

Ncchanical Comprehension wj 

Automotive Info,?;lrtion (AI) 

Ferccp:ual Ability 

. Patter:1 Analysis 

; httcntion to Dstail 

Auditory Percept&o? 

. . . 

. Self DcscrLption 

:: . : 

Conlbnt Scale 

Attcntivencks Scale 

):lcctronics ScalC 

tlaintenancc ScalC 

. . 

. ‘. 

. 

. . . 

. . 

. . - 

c 

. 

. . 

7. - 

AR 

TI 

PA 

AD 

cc 

. . 

. . 

. . . 

I 

. 

Aptitude Arca Cozpositcs 

.: 

Fh EL 

SC 

- 

OF 

- 

XT1 

- - 

CY 

- 

AR 

Gi 

$lK 

. . 

EI 

CA 

. . . J / 

I1R 

. . 

I.7 

El 

NC 

I 

I 

I 

I 

CE 

cx 

is 

AR 

, 

I 

WK 

NC 

PA 

hk 

tit: 

Tl 

El 

Al 

Ct 

I 

AU 

SK 

t:c 

AL 

:. 

. 

ST 

- 

AK 

t1Kt 

SK 

. 

*. .- 

*.. 

GT- 

- 

A!: 

. 

. . : 

. . . . -. . 

. ‘.!’ 

:! 

: .’

THE DEVELOPMEXT OF THE ENGLISH COMI'BEHENSION 

LEVEL SCBEENLNG TESTS FOE FOREIGN STUDENTS 

BY: Cortez Parks 

The English Comprehension Level (ECL) testing system is the ! 

\ 

primary quality control used in the English Language Training 

Program (ELTP) conducted by the Defense Language Irstitute, English 

Language Branch (DLIEL) at Lackland Air Force Base and at numerous 

overseas locations. 

It is a unified testing system in that all test forms are 

origi.lated and controlled by one agency. The construction, 

validation, distribution, administration., and control of the various 

types of tests prescribed by DLI are the responsibility of the Tests 

and Measurement Branch, Development Division, English 1 rnguage 

Branch. 

Among the tests produced by the Tests and Measurement Branch 

are the English Comprehension Level Screening Tests, or "EC%" tests 

as we refer to them, which are the subject of this paper. T'ne ECL 

Screening Tests consist of general English tests, usually called ECL 

tests and the Specialized English Terminology (SET) tests. 

The ECL test is designed to measure an individual's English 

listening and reading comprehension and to ascertain his capability 

to acquire knowledge of and to function effectively in English 

language training and work situations. 

The ECL test does 'not measure speaking or writing ability. 

i 

These skills are measured subjectively by the students' instructors. 

; 

L /; i: . 

45 

:: L’. 

. 

’ . 

--- --. . . . . ..-I.. i . I . . .._ .. . . . c 

* 

A : 

: 

. I/i ; 

. 

'.

This is done by raring the speech or writing production by the use 

of a speaking and writing proficiency .?evel code key. Correlation 

studies made at DLIEL of the instructors evaluation compared to the 

E:CL indicate that the subjective evaluation of thece skills has a 

high correlation with the ECL scores. 

ECL tests, as well as the SET tests, have been correlated with 

language performance of native speakers of English in various 

specialized fields (Electronics, Fiaintenance, - - - - etc.). 

The minimum ECL scores and cutoff scores of the SET tests have 

been established for different levels of training in the various 

specialties of the Army, Navy, and Air Force. These scores represent 

,the minimum English comprehension level necessary for foreign students 

to successfully cope with English instruction in tasks involving 

,varying degrees of difficulty and/or danger. 

An individual cannot normaily be expected to perform satisfactorily 

and efficiently when exposed to training in those fields with an ECL 

score below the minimum requirements. . 

A 100 ECL score indicates that the foreign students can use 

standard training materials without any difficulty. 

A O-25 ECL includes beginning language students with no preViOlJs 

- 

English language background or those with a weak English background. 

A O-39 ECL indicates that the student is at the elementary level. 

Language instruction given at -this stage consists principally of 

carefully selected basic vocabulary, sentence pattern and related 

pronunciation drills. No formal specialized instructions is attempted. 

* .’ 

46 

.,A---- ._ 

.’ 

\ 

, 

h 

.:. 

. -‘I 

\ ,‘;, ‘-, 

t

ILevel. 

I I 

A 40-59 ECL indicates that the student is at the intermediate 

A 60-69 ECL is the qualification level for direct entry into 

'CONUS or 3rd country apprentice specialized training. 1 

A 70-79 ECL is the required qualification level for direct entry 

into C0hT.S or 3rd country in most basic courses (e.g. primary pilot 

training, etc.). 

80 and above is the qualification level for professional career 

and advanced courses (C & GS schools, sur?eons and advanced medical 

courses, pilot transition, advanced flying safety officers. demolitions, 

control tower operators, etc.). 

ECL levels are not percentage scores but are scores converted to 

established qualification levels; i.e. an individual with a in ECL 

knows much more than twice the "amount" of English t!lan one with a 35 

ECL, and the progressive proportions between 45 and 90 ECL are not the 

I 

same as those between 35 and?O. The ECL spreid "value" is nc: linear, 

! 

but is more like a modified logarithmic curve. 

The language proficiency levels required -or entry into the 

various training courses are set by the servic Is 

/ 

responsible for 

operating the training facilities. Applicant; failing to meet the 

I 

language proficjency level requirements for direct entry are programmed 

through the English Language Branch for additional language training. 

// 

In early 1962, a number of the service'schools reported that a 

signcficant number of students were entering trai..inp courses who were 

meeting the ECL requirements but who were deficient in the specialized 

terminology peculiar to their specialties. This was especially true in 

.A- __-. 

47 

_. 

. 

: . .. 

. .J 

$. 

b k‘2 

p ” 

. . 

.I

the areas of electronics, mintenance, and weather. The service 

schools were finding it necessary to give these students extra- 

curricular instruction in technical terminology. At a Tri-Service ECL 

Test Conference at Air Training Comand (ATC) in lS64, a decision was 

made to add SET tests to the ECL screening tests. 

SET trsts were developed in five specialized ares; electronics, 

maintenance, supply, weather, and wdical. Thus, Series 6500 was the 

first series of the ECL screening tests that consisted of ECL and SET 

tests. 

By use of the SET tests, Xilitary Assistance and Advisory Groups 

(ELUCs) were alerted to the necessity of prograuming students through 

the DLIEL Specialized phase prior to their enrollment in COWS training 

courses. 

As the result of a ianguage training sumey consucted by DLI upon 

its conception in 1966, it was decided to add General Amy and General 

Na,cy terlnfnolo,y to the screening tests. lkese txo additional tests 

were incorporated into t3e 6700 Series screening tests. Also, several 

of the technical training schwls were requested to submit lists of 

technical terms along with basic course materials to DLTEL for use in 

the development of SET tests. These materials are updated periodically. 

The 7200 Series SET tests now in use and the 7400 Series which 

will replace the 7200 Series in July 1974, consist of two forms each of 

Electronics, Maintenance, Supply, Weather, Xedical Professional, Xedical 

Service, General Amy, and General Sary tests that mssure the 

comprehension of technical terminology considered essential by the 

respective technical training'schwls. 

48

The ECL test consists of 120 questions, 6Oi of which are on tape 

and test the aural comprehension of ti,e student. 40% of the questions 

i 

test reading comprehension. 40% of all objectives tested are taken 

1 

from the ALC Elementary and 60% from the lnternediate texts. Items 

are presented in the form of questions, s:atements, and dialog in the 

, 

:! 

listening part of the test and in the form of incomplete statements, 

underlined objectives, vord order, and complicated sen:enccs in the 

written part. 75% of the questions consist of vocabulary and idioms 

and 25% test the students’ abilities to select the proper structure. 

All items are of the multiple choice type with four choices of answers, 

only one of vhich is the correct ansbvr. The standards for. the 

construction of items are extremely r&d. 

The resulting product is the service trial edition of the 

1 

screening test consisting of 12 forms of ECL tests a-f two form5 cf 

each of the SET tests* 

Validation of the ECL and SET tests are accomplished by pretesting 

the service trial cditian k-fth the forrign students at the various 

technical school::. Since the students at the technical schools are 

all in the upper quadrant of language Iproficiency, 

the ECL tests are 

( 

pretested also among students in the cower quadrant at overseas schools. 

The number of cases used for experimental testinS depends .u,>on 

‘I 

the type of statistical analysis re&ircd. For conuuL:ing item _ 

anal,s&, at least 100 cases were used. Fcr computing correlation or 

for norming, at least 300 cases were obtained. 

49 

i

A great amount of effort is being expended in the Javelopment 

of the ECL screening tests in order to obtain the highest reliability 

and validity for each form of the tests. For test reliability, both 

the internal item consistency and the alternate-form reliability are 

carefully checked. For test validity, we check the content validity, 

construct validity, and predictive validity. The reliability index 

and the construct validity coefficient of each of the different forms 

have been kept above .90. The coefficients of the predictive validity 

have shown above .65. X11 of these cutoffs are considered high 

st .mdards in test validation. For an example of statistical data, 

please refer to handout, Table 1. 

We have also correlated the ECL tests against the Test of English 

As X Foreign Language (TOEFL) tests, the University of Michigan 

. 

language proficiency tests, and the American University language tests. 

Pretesting has also been conducted at San Francisco State College, the 

University of Texas, Tesas AfJl, and the University of Minnesota. The 

,results were around .80, indicating a high correlation with their 

criterion tests of English proficiency. 

The SET tests do net use converted standard scores as do the ECL 

tests. They arc designed to use percentage scores with pass/fail 

cutoff scores. For the 6500, 6700, 6900, 7200, and 7400 SET tests, we 

have used minus one standard deviation as the cutoff point since these ’ 

are proficiency type tests.

- -... 

!krxfrm::i pn~sihlc score 

TABLE I 

hEL-69 ^. ?!\-69 - - SU-69 - - WX-69 - - ME-69 AR-69 N&69 

- - B- A B A B A B - A - B - A BAB 

100 lfJ0 1 N 120 100 100 100 100 120 120 100 100 100 100 

?kran Score 70.4 70.9 81.2 77.0 54.1 53.6 62.8 63.3 87.3 87.8 66.5 67.0 63.5 63.3 

1.‘ Standard Deviation 15.8 15.3 25.6 24.2 18.1 17.8 20.3 20.1 24.1 24.9 15.0 13.7 15.5 15.3 

I 

i ..,, 

, . 

Ncnn Difficulty Index .65 .67 

Leon Discrimination Index .43 '.37 

Reliability Index for 

lnternal Consistency 

(Kudcr-Richardson 

Formula 21) 

.93 .92 

Alternate-form Reliability AxB 

Cocfficfept (Pearson 

Product-Xomcnt Correla- .87 

t ion Formula) 

,’ ’ 

1.’ Validity Coefficient ,79 .79 

of Correlation between 

Specialized Terminology 

Tests and ECL Tests. 

.62 .59 .57 .57 .65 .63 .64 065 .66 .66 .62 .63 

.49 .48 .47 .47 .49 .50 .43 $44 .37 .32 .37 .36 

.97 .96 .93 .93 .95 .95 .97 .97 .91 .89 .91 .90 

AXB ixB 

.96 .89 

AXB 

.95 

.88 .86 .77'. .75 .86 .86 .76 .76 ,70 .72 .61 .69 

Axi3 

.94 

AXB 

.88 

AXB 

.92 

_ ” 

,_,- ’ 

,.-:-

;Lv EXPERIMEXTAL, iWLTIMEDIr\ CAREER DE\‘ELOPMXT 

CoutiSE FOR SEW WNTAL STANDARDS AIRMN 

George P. Scharf 

Air Trnining Commnnd - USAFSMS, Chanutb 

This paper is a report on an experiment to determine if 

modifying the Fire Protection Specialist Career Development 

Course (CK) could improve the CDC as a t:aining device for 

low ment31 aptitude airmen. ?louL-*cations to the regular 

CDC included simplifying the k-ritten materials and adding 

1s hours of nudio supplementation. The experiment used 

three groups of Sew Yenta1 St3ndards (WS) Project 100,000 

3irmcn enrolled in three versions of the CDC. The control 

group took the regular fire protection specislist CDC. T!IC 

first csperiment31 group took the modified CPC with nudio 

suppicmentntion. The second expc ‘imental group took the 

modifieci CDC without any audio supplement3tion. Criterion 

for effectiveness was the final grade performance of the 

personnel in the three grouns. Results indicated both 

experimental groups had mean final gr3Je scores significantly 

brttcr than the control group. flok’ever, there ~3s no 

significsrit difference between the mean fincti grade scores 

of the tuo experimental groups. Mean course completion t imcs 

for the three groups were within 3 week of one another. In 

this csperiment, the audio-supplemented CDC ~3s not 3 costeffective 

method of course presentation. 

. In recent years the federnl government /has been quite conccrncd with 

improving the utilization gf the nation’s manpoouer resources. In the 

late 60s and until 1971 there was considerable pressure on the milit:lry 

services to accept and train personnel with’ lox mental abilities. OIlC 

such progmm, Project 100,000, begnn in 1966. This program cnllcd for 

the milirary scrviccs to accept up to 100,OpO cnlistcd pcrsonncl scoring 

between the 10th and 30ti; ptr:cntilcs on the ArmAt Forces Qunlific3tion 

Tcs t (XFQT) . These 3rc pcrso;ls in the Cate’ory I\’ ment31 group, 

f 

As A p3rt of Project lOO,OO!? zithln the Air Force, ;I number of cnreer 

fields were sclccted for impiemcntin g the &se of large numbers of nu~rginal 

Iptitudc, Sew Hental Standsrds (SYS) personnel. hbilc US airmen were a 

psrt of Project 100,000, not all Project 100,000 rrirmcn wore X\IS people. 

The definition of Sew Mental Stnndnrds is, Vhosc who have scores 

in the 1016~ h3lf of the ?lo,.:al Croup I\ One of the career fields 

sclcctcd to utilize A>15 3irmen ~3s thnt Protection Specinlist, _ 

XFSC 571so. 

.\?lS personnel were assigned to the Fire Protection Spcciaiist Course 

nt Chnnutr’ directly from Lachland. However, to merit this assignment, the 

52

student had to have (or achieve before leaving Lackland) a reacting level 

of at least the sixth grade. hhen the ?;?!S airmen entered training,. it 

was found that most of these personnel with low mental abilities and lou 

reading lcvcls could complete the Fire Protection Course if given adequate 

rcccdial instruction, counseling, and personal guiJance. On the job, however, 

where structured remedial and personal help were not as readily available, 

lox reading ability caused great difficulty for thcsc airmen in studying 

and completing their Fire Pro,cction Specialist Cnrtcr Development Course 

(‘SC). As a result , AK Headquarters dircstcd that an experiment be set 

up to detcrminc if simplifying the CDC and adding nuJio sus?lcmcntation 

would ensblc studcn:s to complete their CDC fnstcr and with better grades. 

The cspcrinental design finally selected call4 far comparing three 

couivalent groups of 915 airmen who had successfully graduated from t?e 

h:,Sic Fire Protection Specialist Coursr at the School of Applied Aerospace 

Sciences, Chsnutc. The AFQT scores for the SVS cspcrimental subjects 

ranged from 20 down to 10. Upon reaching the t’icii and applying for their 

CDCs, these airmen were enrolled in one of three different versions of the 

Fire Protection Specialist CD,. The students in all three groups took the 

same prc-test nnJ final or p>st-test. The three UK versions covered the 

same basic information, but their modes of prcscntxtion differ4 as foIlok’s: 

CDC 57lSlA. Airmen 61 took this course were the Control group. This 

is the convcrtionnl Fire Protection S?rcislist CDC. 

cnc 5711-V. This was 331 experimental, JJlUltiJ2tdiil CDC whi:h USC~ short 

sentences, large, single-~olumn-pcr-~3gc print, contnincd more pictorial 

natcrials th:ln the regular CDC, and used cassette tape rccorctings to 

summarize, cnphasi zc , and express in different terms the information read 

by the stulicnt . So new or different information xas included in the 

recordings. Khilc dctailcit instructions on usrng the tape player vith the 

b-ritten material arc a part of the CL’C test, the procciiurc was sorlewhat 

as fOllOh’S: 

11) After reading an instruction sheet on the tape player and 

directions at the start of the first CIK \‘olumc, the stuJcn: studied the 

first part of his CDC. After a few paragraphs, he was instructcJ to put 

a particular cassette into the tape plnycr and turn it on. 

(2) The tape recording rcfercnced the pnswgcs :hc student had 

just wad. Speaking in turn, a group of up to four male voices then 

di scusscd , summnriz~:d, restated, or reviewed the int’ormatinn in the 

referenced paragraphs. r The student skis then direct& hy a rccorJed voice 

to turn the Player off, i!r then returned to the b-rittcn text until hc 

reached the printed instruction to again turn on the tnpr player. This 

procedure was t‘olloKed throughout the entire five voiumcs of the CDC for 

a total of approximately 1s hours of nuctio rcvicx of whnt the student 

h.-tJ rend. 

\ 

. 

_. 

t 

53 

. ,_..

(3) \~olumc introductions, chapter introductions, and chapter 

sunnaJries were recorded with a female voice to give more variety to the 

tapes. 

CK 5710u)B. This was the same, less verbal, more pictorial 

CDC S?%iout the tape recordin@ and with all references to using 

a tnpc player removed from the written text. 

Procedures 

Starting in August 1971, a graduation roster of each class in the 

F:ire Protection Specialist Course \r’as furnished :hc Training Research 

Applications Branch. The first entries into the CDC experimental program 

g:l:3Jwtcd from the Chanutc course on 7 Septczber 1971. Khen an WS airman 

graduated, his name and social security account number were furnished to 

thz Estcnsion Course Institute, Edwational Systems Branch (ECI/EDS\‘), 

at Gunt er AFB in Alabama. These names were then put on an identifier list 

a’t EC1 as eligible for the CDC experimental program, 

a. The number and flow of‘ MIS personnel available for this 

experiment never became as great as the originally projected 16 per month. 

One rc,rson for this was that Project 100,000 unofiicially decreased to 

Project 50,000 some months before the experiment began. As a result, the 

number of SYS students entering into the Fir-c Protection Specialist Ccurse 

Jccrcnsctl by more than half. Thus, by the tine the CDC course materials 

wrc\ prcP:ircJ, there was a shortage of eligible participants for the 

cspcrimcnt. 

b. Effective 1 April 1972, Change 4 to AI:N 33-3, Enlistment in the 

Regular Air Force, cut off the MIS student flak- entirely. Under this change, 

Xlr Force enlistecs must have an AFQT of at icast 21, and if the AFQT is 

bctwecn 21 and 30 the enlistce must be at least a high school graduate. 

This change in enlistment requirements stopped entry into the Air Force 

(and thus eventually into this csperimentl of AX airmen. As a result, 

the number of casts analyzed in the study is as follows: regular CDC -- 

24, experimental group 1 using the tape recorder -- 25, experimental group 2 

u.sing only the i-written course materials -- 19. 

:\s applications for the Fire Protection Spc

The measures of 

an analysis of their 

control CDC group. 

, 

effectiveness ci the two experimental CDC groups was 

final course grade performance in comparison to the 

Xl1 caiculations were performed using statistical packages available 

in the FIAT0 I\’ computer-based teaching system. There are presently four 

PLATO IV terminals at Chanute connected to a central computer located at 

the Urbana campus of the University of Illinois. The mathematical formulas 

nrccssary for any analysis, including those involving unequal n’s, are 

included in the analysis programs used. The person raking the analysis is 

required only to select the type of analysis desired and input the raw data. 

:\nsxers appear on the screen immediately. Thus, it was possible to examine 

a great amount of raw data concerning the subjects of the study and receive 

back an almost instantaneous readout of the analysis selected. 

Overall results obtained from this study are summari-zed in Table 1. 

SOT . mean 

AFQT standard deviation 

w 

Resident course grade ’ 

mean 

Resident course grade 

star&r2 deviation 

Prc-test grade mean 

Pre-test grade standard 

deviation 

Post-test grade mean 

Post-test standard 

deviation 

Set -mean gr3de score 

increase 

Set-mean grndc score 

increase standard 

deviation 

ComPletion time mean 

CornPlction time standard 

deviation 

Table 1 

CDC 

5715lA 

n=24 

15.33 

3.41 

’ S3.75 

3.53 

56.56 

10.5s 

66.75 

7.7i 

10.17 

9.59 

I 

/ 

j 

1 

I 

/ 

! 

i 

/ 

i 

CDC 

37100 

n=25 

15.0s 

3.55 

s3.ss 

3.69 

57.40 

9.57 

72.80 

10.73 

15.40 

8.72 

205.00 days 211.50 days 

, .’ 

76.35 days 50.26 days 

55 

I 

\ 

CDC 

571008 

n=19 

15.05 

5.10 

64.61 

s.b,s 

61.95 

10.7s 

76.00 

11.07 

14.05 - 

s.9s 

205.3’ days 

31. b.1 days 

. . . 

‘< 

: ! 

‘8 :-

*ro establish that the three groups were equivalent in ability, a 

one-way analysis of variance ~3s conducted on the means of the students’ 

XFQTs, final grades from the resident Fire Protection Specialist Course, 

and their CDC pre-test scores. Table 2 summarizes the one-say analysis 

of variance on each factor among the three groups. 

\ 

Table 2 

- - 

Source of Variation F - d.f. E 

AFQ’P 0.045 2,65 = 0.9558 

Resident Course 

Final Grade 0.341 2,65 = 0.7122 

CDC fre- test 

Cradc 1.628 2,65 = 0.2042 

As cnr. be seen, the F ratios for each of these three student variables 

were all non-significant, thus indicating comparable groups. 

In order to test the cffcctivcncss of the instructional methods, a 

one-k:3y annlysis of variance ws conducrcd on,thc zeans of the post-test 

CLN: (final grxic) scores achieved by the three groups. Table 3 presents 

this analysis of variance. 

‘fable 9 

Souric i) i VdriJtion F - d.f. e 

Post-test score 5.966 -1,65 = 0.0098 

Inspection of ‘I’ablc 3 shoxs that the post-test score F ratio was 4.966; 

this is a significan? vnluc indicating the post-test performance of the 

?hree groups was not comparable. 

To compare the differences of the three groups on the post-test 

pcrformanse factof, a Series of t tests were run. Their results are 

sumarized in Table 4. 

56 

._ 

..- _.

. 

TL 

Mean post-test score 

of Es Gp 1 (revised 

text, audio supplement) 

I 

72.80 


of Lx Cp 2 (revised 

text only) 

76.00 


of Es Gp 2 (revised 

text only) 

Table 4 

T3 


of control group 

t d.f. E 

66.75 2.258 47.0 .06 

Yean post-test score 

of control group 

66.75 3.227 41.0 .05 


of Ex Gp 1 (revised 

text, audio supplcrwnt) 

76.00 72.80 .967 42.0 .44 

The inferences from Table J are that both the audio supplemented CDC 

and the revised text only CDC yielded significantly higher grades than the 

regular CDC. However, there was no significant difference in the final 

course grades of the two experimental grwps. 

To establish the equivalency of the compietion times, 'a one-way 

analysis of variance was made on their means. This analysis is shohn in 

Table 5. 

Table 5 

Source of Vciriat ion F d.f. IT 

COl:pietion time of 

CDC in days 0.064 2,65 = .9,383 

The indication of this analysis is that the CDC completion times for 

the three groups were not significantly different. 

57

Discussion 

In this study, 20th experiment31 CDC groups had sta;istically significant 

better learning scores thnn the regular CDC students. Hcwever, there was 

no sisnificnnt difference in the learning scores of the Go experimental 

groups. Khilt not statistically sigri ficnnt , the experimental group without 

tlw 3udiO supplementation hnd a mean final grade that ~3,s 3.1 points 

higher thnn the cxperimentnl group using audio supplementation. At the 

s3me t imc, the net-mean sr3dc score increase of thcsc tuo groups was 1.78 

Iiighcr ior those using 3uJio supplementation. Both experimental groups 

.iiad greater net-ncan gr3Jc score inc,l:eases than the contrcl group. 

l\%ilc costs of 3 program such 3s this :irc a rather elusive thinE and 

hnrd to pinpoint , it xould nppear fhnt the product ion, impleinent3tion, and 

ndininistrn:;on of t!lc 3uJio-supplezentcd CIK ~3s bctvecn three 3nd four 

tines the cost of the rcSuI3r CDC. This factor rcanins about the same 

wfilen proj cctcd over a much 13rger number ‘01‘ students thnn participntcd 

in this experiment. For 311 three CDCs the production Of the hlittCl1 

n3tcrinls is cquivnlcnt. In addition, the larger the printing of 3 CDC, 

the less cost per student. ilorseuer the premise that the larger the audio 

grcsup the Jcss the cost per studcnt’does not appc3r val id, :ilii:c tlicrc 

ni:ght bc some slight savings in buying 3 lnrge number of c3ssettcs :InJ 

t3:pt pl:l\~crs ( the loss rntc plus wintcnancc nnd rcplaccmcnt COL‘:~ of 

these items wculd nullify nny projected rcductioirs in cast per student 

lwscd on a lnrge number of users. Consequently, it does not 3ppcar cost 

effective I‘or widespread :40t;tion of an audio-supplemented CDC program 

\

However, the mean time for those using the audio tapes was a week longer 

than those using the other versions of the CDC. 

times for the three groups was not significant. 

The difference in completion 

Crnc lusions 

As a result of the analysis uf the data evolved from this study, the 

following conclusions are hereby stated. 

The the experimental groups had better final CDC grades than did the 

control group. 

The differences in grades of the two experimental groups xas not 

statistically significant. The audio-supplemented experimental group 

had a net-mean increase of their final .grade over their pre-test grade 

1.78 pr,ints more than the non-audio supplemented group; but the non-audiosupplemented 

group had a mean final grade 3.2 points higher than the 

audio-supplemented group. 

Any higher grades of the experimental grouns can be attributed to 

t:he revised text as much as to any 

groups were shah-n as equivalent in 

Yean completion tines for the 

another. 

&dio supplementation was not 

in this experiment. 

Ref crenccs 

audio supplementation as all three 

mental ability and pre-course knowledge. 

three groups were within a week of one 

a cost-cffcctive means of instruction 

Effectiveness of Exrerimcntal Training Naterials for LCW Ability Airmen, 

Kaynce S. Sellnan, Captain, USAF, AFHRL-‘LX-70-10, June 1970. 

59 

.-

Optimal Utilization of On-the-Job-Training 

and Technical Training School 

Captain A1q.n D. Dunham 

This paper discusses work planned and completed during the AF 

Human Resources Laboratory’s effort to develop techniques for 

optimizing the use of On-the-*Job Training (OJT) and Technical 

‘l’rai.ning School (TTS) as methods of initial upgrading for non-prior’ 

service personnel. 

1. INTRODUCTION 

The author’s previous MTA paper (1972) discussed criteria relevant 

to selecting optimal OJT/‘TTS mises and then described the results of 

wor:k which established the feasibility of obtaining OJT cost data. 

Significant improvements in the O.JT costing methodolog have been 

obtained in the year since the first report and progress has been made 

in the application of a measure of one other relevant criterion, the 

quality of the training methods. This progress is discussed in the 

following text. 

II. ‘REFTSEMENT OF CJT CCSTTSG METlIODOLOGY 

The original OJT costing methodologl identified the following cost 

factors: 

. ” 

: 

Table 1 

@JT Cost Factors 

Student Time 

Instructor Time 

Delayed Entry into Training 

Records Managcmcnt 

Remedial Trainins 

Equipment and Materiais 

60

Data concerning these cost factors were collected by strvey for the 

Communications Center Operations specialty for OJT to the 3 (semiskilled) 

level for non-prior service airmen. These surveys asked 

supervisors in this job specialty to provide detailed ?aea on several 

items, including time spent learning a number of skiIlk specific to 

Communications Center Operations. The Supervisor2 had a wide range 

of experience with OJT. The resulting OJT estimate was considerably 

less than the cost of TTS for the corresl,onding training co*-lrse. 

The most important aspect of this research was that it established 

the feasibility of obtaining cost estimates fQr USAF O.JT through the 

use of a survey technique. 

One undesirable result was the high response variance encountered 

for many of the crucial questions. This resulted in relatively wide 

confidence limits for tht! WT cost estimate? but not wide enough to 

cast doubt on the cost comparison. 

Another undesirable characteristic 

was that a large section of the survey instrument was specific to 

Communications Ccrlter Operations. A survey instrument independent 

of the job performed would reqllire less resources for survey administration 

and data file maintenance. 

Kf. THE DEVELOPhlENT AND EVALUATI@N OF ALTERNATIVE 

METHODOLOGIES FOR ESTIMATING THE COST OF AIR E’ORCE OJT 

In June of 1972 Manpower 2nd Personnel Sy&.tems Division of the AF 

Human Resources Laboratory let contract PF41609-72-C-0048, designed 

to improve upon the original OJT costing methodology. Dcring the first 

phase of this contract the contractor used the original survey approach 

and two alternative survey techniques to simullaneously collect cost data 

concerning OJT to the 3-level for USAF Admi istrative Specialists. 

One of the two new alternative survey techgiques 

‘i 

asked supervisors 

to estimate information for e!even variables

in the instrument, but most of the redundant questions were asked only 

once to minimize the length of the survey. 

Of the 295 surveys sent to supervisors at 88 Air Force bases, 20’7 

were returned. All responses were manually e.&ted and keypunched 

for transferal of the survey data to magnetic tape. 

The questionnaire’s length might have been responsible for a reduced 

response rate toward the end of.the instrument, but response rates for 

all three survey approaches were sufficient to enable co,nparison of the 

alternative txchniques. 

A detailed report of the survey data and an in-depth discussion of 

their implications will be forthcoming soon in an AF Human Resources 

Laboratory technical report. For brevity and to minimize duplication 

Table 2 describes item content and mean responses without discussion 

of their implications. The reader is cautioned against ixferring strong 

conclusions from this limited display of data. 

.4 cost estimate for each OJT cost factor is exhibited in Table 3. The 

cost models used to compute these estimates were selected from several 

cost models based on various combinations of the three survey approaches. 

Again, a more rigorous discussion will soon appear in AFHRL publication. 

The OTT cost per trainee of S1545 can be %ompa.red to the marginal cost 

per trainee o course 3Af3R 70230, S2271. There are criteria other 

than costs which are equally inrportant, viz. , training capacities ar.d 

quality of the training. Thus, conclusions regarding optimal OJT/‘TTS 

mists for the Administrative Specialty based only on data in this report 

would be reached without considering other important criteria. These 

criteria are discussed l~~ a Technical Report in review at AFHRL. ’ 

Z The cost of Tech schoorwas obtained from FUSD Corporation data. 

- ! 

. -._. 

62 

. _ 

, _- .- _.. 

. 

._. .~. _. .*. - -. A

L .._. 

Table 2 

Mean Responses for Admin 5pecialist OJT Survey 

Responses Based on Supervisors’ Total .Expericnce 

F 

Item Content f Mean Response 

Persons in Section 

Days Delay in start of &level O.JT 

Days Delay in start of &level OJT 

Weeks to 3-level Proficiency 

Weeks from Proficiency to ,Ictual 

Aw.a.rd of 3-level 

I 

7.27 

8.60 

8.44 

13. ?3 

Sumber of S-level Trainees 

/ 

3.31 

1.34 

Sumber of 5-level Trainees 

1.06 

Aci&tional Training Capacity 

1.113 

Operate with fewer personnel if no OJT 

297 

Are Trainees Productive? 

82% ; 

0,JT Superior to TTS Grad 

22: 

TTS Superior to O.JT 

32°C 3 

So Differ ence between OJT and TTS Grads. 

465. 3 

Percent Fai!ing Apprentice Knowledge Test 

25.64 

Weeks Remedial Training 

Trainee Remedial Hours/Week 

/ 

3.60 

9.07 

Instructor Remedial Hours/‘Wcek I 

5.28 

Training Accord Keeping Hours/Week 1 

1.94 

Ski 11 Area4 

.Career 

Security 

Supervision 

� 

Wks. to Prof. 

2.86 

2.G5 

4.80 

Instr?ctor 

1 

Hrs/Wk. Trainee Hrs/Wk. 

- 

7.78 

3.89 

2.94 3. 3.31 3.59 1. 78 22 

6.90 

Equipment 3.78 

9.14 

Publications 4.51 

7 . 8 3 

Forms 3.64 

I 2.65 

5.43 

Communications 4.99 

3.27 

8.28 

Documenta!ion 5.02 

6 . 9 2 

Library 2.70 

1.16 

4.02 

Postal 3.10 

2.07 

5.04-. 

3 Per cent ot total valid responses that responded affirmatively. 

4 Corresponds to the Job Proficiency Guide skill breakdown for this 

speciality. 

63 

, 

. . , - _. . ._ 

’ 

-- 

. 

-- 

. .: ___.. _ -.... __ ._, _.-. _) -.-- * . ..A-. _- ..- 

v

Table 2 (Cont’d) 

;I Week Productive Trainee Hours (Mean Response) 

, 

1 14. a7 

4 17.92 

a 23.44 

12 27.25 

16 30.78 

20 33.21 

Responses Based on Supervisor’s Experience Previous Week: 

Item Content h’lran Response 

Week of Training for Trainee 

Percent of Training Completed 

Percent of Skills known prior to OJT 

Days of delay in start of 3-level O.JT 

Trainee Hours OJT 

Instructor Hours 

Record Keeping Hours 

Days of delay in start of 3-Icvcl OJT 

Percent of skills known at time of arrival at unit 

(TTS Grads) 

Additional weeks to 3-ievel Proficiency 

, 

lo.88 

56.21 

10.93 

9. 16 

9.07 

ifi. 88 

1. 88 

6.71 

29.57 

5.05 

Responses Based VII Supervisor’s Record -Keeping for one Week: 

Item Content 

L 

Mean Response 

Number of Trainees 1.3 

Trainee hours instruction 10.61 

Trainee Hours productive 24.73 

Instructor hours/trainee a. a9 

64

Cost Factor 

Table 3 

Sunmary of OJT ‘Costs 

Co&/Trainee 

Trainee Time . $ 579.38 

Instructor Time 

591.35 

Delayed Entry 

196.05 

Records Mawgem? nt 137.70 

Remedial Training 40.03 

Equipment and Material 19.18 

65 

s1545.49

Tables 1 through 6 summarize the more obvious conclusions concerning 

the three survey approaches used. When sample size is no problem and 

irhen information specific to the skill groups is not nqeded, the survey 

approach used should be that of Table 6. When sample size is a problem, 

the ‘methodology of Table 5 would suffer from the same problem as that 

of Table 6, so that a form of the original survey approach appears more 

advantageous. Most limitations on sample size can be overcome hy 

extending the time span of the survey. / 

Selection of a survey approach thus involves trade-offs. The Phase I 

report of this contract recommends use of the general survey approach 

of Table 6 while retainins some of’the questions used in the original 

methodology. This synthesis of the survey approaches was used in 

Phase II of the contract and appears as Appcndis A of this gaper, 

66 

;: 

: :-

Table 4 

Original Survey Approach - Responses Based on Supervisor’s Total 

Experience 

Advantages (compared to alternative’methodologies): 

a. Larger sample at any point in time 

b. Bl’eaks out information by skill groups specific to speciality 

or job performed 

c. Can be completed in one sitting - minimizing turnaround 

and survey control workload 

Disadvantages: 

a. Speciality - dependent 

1. makes aggregation of OJT cost data across specialities 

more complex 

2. requires that a different survey form be developed for 

each speciality 

3. rcquircs multiple formats for mair??enance of OJT 

cost data. 

b. High response variation, resulting in relatively hil:h variance 

around OJT cost estimate. ; 

67

Table 5 

- - 

-Alternative Survey Approach - Responses Based on Supervisor’s 

Esperience During Previous Week 

- 

Advantages: 

a. Can be completed in one sitting; 

IL Speciality-Independent; 

c. Variance reduced somewhat compared to original methodology; 

d. Recall required is for more recent experience. 


a. Provides no data on costs related to specific skill groups; 

b. Requires presence of CJT trainee during previous week, 

which may result’ in a reduced sample size for any point in time. 

i 

68 

. 

‘.. -. 

-.. 

’ . 

_ .-- ~ . . ; ., I . . . .(. 

pd 

. . 

-. 

-.._. 

. - 

. ~ 

, 

.

Table 6 

blternative Survey Approach - Responses Based on iecord-Keeping 

for One Week 

F 

Advantages: i 

a. Generally reduced response variance compared to other survey 

approaches; 

:: 

b. Speciality - independent; 

c. Recall involved limited to one day of work. 


a. May have reduced sample size due to need for currently 

enrolled trainee; 

b. Turnaround time is longer; 

1. may result in reduced sample size due to increase in 

time in tne field; 

I 

2. will require more record keepfng by survey control. 

� � 

69 

‘. 

. 

, 

. - 

. . -,. 

� 

I’ 

.

. 

Phase II of the OJT costing contract is nearly completed. During 

this phase the contractor utilized the survey techn’iquc developed in 

Phase I to collect cost data for OJT to the 3-level for the following 

specialities: Fire Protection, Pavements Maintenance, Fuels, 

XIatcriel Facilities, and look. 

The results of this phase will soon be ready for publication. Persons 

interested in these results should be able to obtain information from 

Nanpower and Personnel Systems Division of the XFHRL in the near 

f u t u r e . 

IV. Qlr:lLITY GF .THE TRAISIXG METHODS - 

The OJT costing work described above shows that the Air Force 

can and is obtaini-ng OJT cost estimates compatible for comparison with 

TTS co. t estimates. One assumption required for a straightforward 

comparison is that these two methods of training provide trained 

personnel of equal quality in terms of their productivity on the job. 

Cost estimates which included ‘quality’ in some manner would be 

tenuous figures based on unsupported assumptions because measures 

of quality useful for this purpose d:, not exist. Hence, cost estimation 

and quality comparisons are treated separately. 

The following discussion of methods of quality comparison is a 

summary of some work completed at Manpower and Personnel Systems 

Division, AFHQL. A thorr\ugti analysis of the ‘quality’ problem is in 

preparation and should appear as an AFHRL technical report within a 

year. 

The problem of comparin g the quality of training metllods in the 

military context deserves a thorough theoretical discussion of the 

impediments faced by researchers. It ~7s thought expedient to publish 

some interesting data in this article with only a short discussion of 

problem traits to allow interested persons an earlier glimose of the 

results. 

previous research concerning this subject is limited. One exercise 

was the aut_hor’s first simple comparison of mecan Speciality Knowledge ’ 

Test (SKT)~ scores, which essentially showed no differences between 

the performance of OJT a.id TTS graduates on the SKT for the Communications 

Center Operations specialty. This may have been indicative of future 

5-D-n, 1972 

i 

-- 

70 

3

esults but lacked depth in that it did not account for related variables 

that may have some concomitant effect on SKT scores, such as the 

pre-training abilities of the trainees. 

Another AF’HRL study, The Road to Work: Technical School 

Training or Directed Duty Assignment?, by Mr. W. B. Lccznar, 

used several criteria which are thought to be related to performance, 

including: Job Difficulty CompositeG, Number of Tasks Performed 

(range l-372)7, Average Task Difficulty per Unit Time (range l-7)7, 

and operationally prepared Airman Performance Report (range l-9) 

overall evaluations. Several variables thought to cause variation in 

these criteria were used in addition to OJT/TTS as grouping in an 

analysis of covariance or? ” . . . a two treatment-one concomitant 

variable multiple regression model. ‘*8 Analyzing these results, 

Mr. Lecznar concluded that, “The inclination is to say that in these 

specialties formal resident technical training provides little or no 

advan!age over on-the-job training. ” 9 

Airman Proficiency Ratings (APR’s) would appear to be a 

‘natural’ ;Oi. rleasuring some aspects of performance, but they 

currently suffer from such a high degree of inflation that their 

lack of variation severely limits their potential for this purpose. 

The Navy has qlso done some interesting work in developing 

productivity indices. lo This concept may eventually be a fruitful 

avenue for future work but a few shortcomings must be overcome. 

0x-W prfJbk?Il? is that it will be difficult to account for changes in the 

quality of output or in the type of output. In addition, output level 

in many work centers in the .4ir Force is strongly related to the 

demand for the output, which could be a troublesome variable to 

account for in a productivity index. 

A logical attack for this problem would be to select the best 

measure of productivity, performance, or proficiency, a priori, 

and then accept the results obtained with observations on that 

measurement. State-of-the-art for this type of measurement is 

6 hleade kChrista1, 19r 

7 

8 

LcczIlciI’, 1972, 

Ibid, 11. 3 

11.2 

9 

10 

Op. Cit., p. 

BUNAVPERS, 

9 

196~1 

71

still primitive, so measurements used are those available, which -‘.. :! 

ii-. 

are usually designed for another purpose. This is true of 

c 

Specialty 

Knowledge Tests (SKT’s). 

f 

SKT’s are used operationally by the Air Force as promotion 

criteria. They are specialty-specific paper and pencil tc6ts updated 

frequently by Air Force supervisors with current experience ir, the 

specialty. An SKT score for an individual may be said to indicate 

that individual’s potential for performance or productivity within 

the given specialty. SKT’s are taken for promotion purposes, thus 

all with a score for a particular sKT..had a common motive for 

excelling on the test. 

If either OJT or TTS were ‘superior’ to the other, one would 

expect this to be manifested in the graduates’ performance. Studies 

discussed above found no evidence to support the hypothesis that 

OJT and TTS differ in the quality of their braduates. Because SKT’s 

are specialty-specific, used by the Air Force for promotion, and are 

taken by both OJT and TTS graduates with common motives for 

excelling, SKT’s may be useful for examining this hypothesis. 

Thus, SKT scores were used to compare the specialty !cnowlcdge 

of OJT and TTS graduates who received initial upgrade training in the 

same specialties for which O.JT cost data has been or is being collected. 

The research hypothesis tested was that OJT and TTS graduates are 

of equal quality. Operational testing of this hypothesis was conducted by 

comparing the individual SKT scores of OJT graduates with those of 

TTS graduates, holding constant other factors thought to be related 

tc SKT scores. The rationale i%r including or excluding ‘related’ 

variables will not be developed here. Analysis ‘of covariance was 

used to test the operational hypothesis and also to test whether the 

‘related’ variables explained some of the variance of the SKT scores. 

Table 7 lists the variables included for each test specialty. Separate 

analyses were conducted for the specialties lis’ed in Table 8. 

f 

If 

72 

. . 

. ,.,

Variables 

- 

SKT Percent Right 

TESTIM - 

Table 7 

I\ Variables Used in the Analysis of Covariancc 

w - p - - - - - - - - - - - - - - 

Description 

Individual’s first score for Test Specialty 

Days between dat.e of enlistment and date 

of test administration 

NEGRO Categorical: 1 if Negro, D if other 

FE,.iALE Categorical: 1 if Female? C if other 

ADMISISTR:ITIVE AI 

GENERAL AI 

MECHANICAL AI 

Aptitude indices as computed from the 

Airman Qualifying Esams tnkcn prior 

to enlistment. Values arc in raw 

score form. 

TEST REVISIOS NUMBER SKT’s are systematically updated. This 

is a set of categorical iarinbles indicating 

which test form an individual 

received a score for. 

2 

OJT 

O.JT X ADMIN AI 

TTS X ADMIS AI 

OJT X GES AI 

, 

TTS X GEN AI I 

OJT X MECH AI 

TTS X MECH AI 

Categorical variable indicating method of 

initial trairing. 

1 of OJT, 0 if TTS 

’ Interaction between OJT variable and 

Admin AI score 

Interaction between TTS variable (1 if TTS, 

0 if other) and Admin AI SCOW 

Interaction as above 




73

Air Force Specialty CodelJ 

551X6 

571x6 

622X0 

Table 8 

Air Force Snecialties Analvzed 

‘Description 

Pavement Maintenance 

Fire Protection 

Cook 

63 1x6 Fuels, petroleum 

647X9 Materiel facilities 

702x9 

Null Hypothesis 

- - 

1 

Table 9 

Administrative Specialist 

Summary of Hypotheses 

Variables Tested 

- - 

2 01-r 

OJT x (appropriate) AI 

TTS x(appropriate) AI 

3 OJT x (appropriate) AI 

TTS x (appropriate) AI 

O J T 

ciiAlso,TeSSPeC~’ lhe ‘X’ in each code is reulaced bv a 4 or 5 

depending on-whether the ?KT was for promotion ‘tc E4 or’ E5 in that 

specialty. Both of the 4 and the S-level SKT’s were used for all 

specialties in the Table. 

74

. 

For each test specialty three regressions were computed in the 

following order and format: I; 

i 

.I. SKT = BI(OJT) .‘ B2(NEGRO) + B3(FEMALE) A Bq(OJTxAI) i 

BS(TTSxA1) * BS(TESTIM) t B7(TEST FGlX$ 

2. SKT = BI(OJT) + B2(NFGRO) + B3(FEMALE) 1 Rf(TESTIM) + 

B7(TEST FORM) i BB(AI) ,! 

3. SKT = B2(SEGRO) 4 B3(FEMALE) r. BS(TESTIh!l + B-;(TEST FORM) i 

Bg (AI) 

Three F tests were comlxted based on .k? increase in error sumsof 

squares between, respectively, regressions 1 and 2, 2 and 3, and 1 

and 3. 

Null hypothesis numbe.r 1 concerned whcthcr inclusion of the two 

interaction variables, ‘OJT s AI’ and ‘TTE s AI’, resulted in significant 

red :stion in the error sum of squares. These variables provide 

for the nossibility that ability, as measured by an appopriate Aptitude 

Index, has a -differential - impact on SKT scores obtained by OJT and 

TTS graduates. Thisis also sometimes calldd a ‘slope’ test. l2 

I 

The second hypothesis concerned whether the O.JT dummy 

variable significantly ~mpro?ed the error suni of squares. The regression 

did include a!1 appropriate Aptitude kdes score. but not in 

interaction form. This is the ‘intercept test. Rejection of the null 

hypothesis would support a contention that OJT and TTS graduates 

have different average SKT scores, holding other variabies constant. 

The final null hypothesis provided an ‘ov~;tlI’ 

I 

test. It tested 

whether inclusion of both - the - interaction var$b!cs and the OJT variable 

resulted in a signif:cant reduction in the error s;1m ot squares. 

f2 The tcchniquiused% ??%putationa:ly si/kilar to that discussed in 

Lecznar, i372. Wonnacott & Wonnacott, pp 77-79 and -Johnston, .. -. 

pp 192-207, contain discussions of analysis of covariance using 

multiple linear regression as a computational approach, 

, ..’ 

75

Table 19 

F Ratios for Test Specialties for Which the Rebuctiox 

in Error Sums of Squares Due to OJT/TTS Variables were not 

Statistically gynificantl3 1) 

Test AFSC 

55 140 

55i3z) 

57140 

5715$3 

62240 

6225z114 

6315014 

6475D2 

792502 

H 0 

1 

2 

3 

1 

2 

3 

1 

2 

3 

1 

2 

3 

1 

2 

3 

1 

2 

3 

1 

2 

3 

1 

2 

3 

1 

2 

3 

DFNUM 

1 1 

2 

1 

1 

2 

1 

1 

2 

1 

1 

2 

p 1 

1 

2 

1 

1 

2 

1 

1 

2 

1 

1 

2 

1 

1 

2 

I 

I 

121 2.42 

122 3.30 

121 2.88 

1494 0.07 

1495 -0.65 

1494 0.38 

2710 P. 27 

.2711 a.89 

2719 II. 57 

i1914 

‘1915 

i 1914 

I 

621 

.I 622 

621 

2983 2982 

I 1428 

1429 

9.97 

4.37 

2.22 

2.46 

1.32 

1.89 

4.19 

5.77 

4.98 

6458 9.61 

6459 P. 28 

6458 P. 44 

1.39 

4 . 0 3 

2.67 

5.55 - 

2.38 

3.97 

13 At them%%r--- 

14 TELTIM was included in these regressions. This variable was excluded 

from other F tests because it was not statistically significant at the -01 

level. 

- 76 c

,P 

Table 11 

Test Specialties for which OJT/TTS Variables 

Proved to be Statistically Significant15 

63140 -1,2716 -2.29 4.40 - .20.' -6.36 - - 39.42 .179 1 2509 26.66 

64740 -2 . 29v -1.71 - .39 - 4.66 - - 31.41 .246 1 2411 30.56 

=: 70240 2.63 -1.27 -0.72 - - -5.54 .1916 .2416 28.47 .203 1 10716 14.80 

NOTE: Numbers in columns under variable names are regression coefficients. 

VQ 

:. . 1Sat the .Ol level 

16variables 

. 

for 

. 

which the F Ratio was computed. Data used to compute these rcgrcssions and 

test statistics were obtained from files maintained at Personnel Research Division, AFI!RL. 

Records, or observations, used were for personnel who, up to the time they took on SKT, 

served only in the specialty for which that SKT was written. Final computations were 

completed under Project 6323, Task 0205, Study 4903.

_-. 

. . 

._A 

.-. 

Table 9 summarizes these hypotheses. Table 10 lists the test 

specialties for which the F ratios would not ‘allow rejection of the 

null hypotheses. For these test specialt~~no differences were found 

between the SKT scores of OJT graduates and those of TTS graduates. 

Table 11 displays regression and correlation statistrcs for the test 

specialties for which one or more of the null hypothesis was technically 

rejected. Figures 1, 2, and 3 show the data of Table 11 

graphically, holding constant the dummy variables NEGRO, FEMALE, 

and TEST FORM. 

Figure 1 illustrates the results for SKT test specialty 63140, 

Fuels Specialist. Acceptance of the second hypothesis means that 

there was a statistically significant difference in the performance 

of QJT and TTS graduates on the 63140 SKT. Ability as measured 

by the Mechanical AI apparently has the same relationship with SKT 

scores for both OJT and TTS graduates in this specialty. 

The same results obtain for test specialty 64740, Materiel 

Facilities Specialist, escept the difference in mean scores as 

evidenced by the “OJT” coefficient is slightly larger. The diq>lay 

would be similar to Figure 1. 

Test specialty 79240, Administrative Specialist, is interesting 

in comparison because not only is the mean SKT score higher for 

V’OJT*’ graduates, but also ability may have a differentiZl%ii&t on 

OJT and TTS graduates. Figure 2 illustrates these results.

. 

Figure 1 

FUELS SPECIALIST ‘! 

SKT - 

A 

39.42 _ 

37.15 _ 

I 

b 

TTS 

1’ 

> I - a 

- - - - - - 

Lr--‘; -.. 

OJT 

I 

- --_-. --_ 

Figure 2 

ADMIXISTRATIVF 

- - - 

SPECIALIST 

SKT 

TTS(Slope = .24) 

b 

.- -_ - - - -. . 

fz sY’-‘- I 

Figure 3 l- 

ALL OTHER SPECIALTIES A&LYZED 

-SRT.------- 

OJT 

M e a n 4 

SKT _ 

Score 

I 

I 

Dilference = 1.27 

r 

MECH Af 

GES AI 3 

b AI %=

Figure 3 shows how the other test specialties would have appeared 

on the graphs - no difference in intercept or slope, The OJT and TTS 

lines are coincident. 

The differences in SKT scores found for 0,JT and TTS graduates 

arc statistically significant, but small. Larger differences might 

!UVC justified a strong statement conctbrnmg the quality of OJT relative 

lo TTS. XT scores ;11 tjcst I~C~IS~I~C only performance potential 

wilhu2t considering otht>r (.a *i!lI)otlents 0i performance, so snipe 

fcrenccs in SKT scores tlh UX~U~C~VCS have limited implications. 

Thcsc results provide no evidence to support a contention that OJT 

and TTS graduates differ in quality. 

V. CONCLUSIONS 

- - 

Work completed and underway at AFHRL is providing the Air Force 

with 0,JT cost estimates compatible for comparison with TTS cost estimates. 

This is a valuable first step hut these cost data have limited use 

by lhcn~selves. They must be used in conjunction with measures of 

other criteria. 

Quality of training received is one of the criteria not incorporated 

into the cost estimates. Individual performances on Specialty Knowledge 

Tests were used in this paper to compare the quality of OJT and TTS 

graduates for a limited number of specialties. The differences in SKT 

scores found using analysis of covariance, were too small to allow 

rejcctiorl of the research hypothesis that OJT and TTS graduates are 

of cclual quality. There is clearly no alarming difference in quaJit$ 

as measured 11s SKT scores for the specialties analyzed. One may 

not generalize either the cost results on the SKT results to other 

ZiEcinltics. 

Future work concerning OJT/TTS tradeoffs should include improved 

OJT cost estimation, repetitions of the SKT analyses of this paper for 

other specialties, development of a computerized algorithm for assigning 

non-prior service persomel to OJT and Technical Training Schools, and 

dcvelopnlent of metliodologier. for determining the capacity of units to 

conduct OJT. This last item deserves special emphasis because it is 

the key to efficient utilization of !Air Force training resources. 

AN Force operational units have manpower standards against 

which OJT trainees are drawn. Each additional trainee replaces’ a 

more qualificcl individual and requires the time of other personnel 

to conduct 0,J’l’. Assignment of an excessive number of trainees 

80 

?

would result in unacceptable degradation of unit productivity. Even 

if OJT were less %ostlyV1 and produced equally well “qualified” 

graduates, there is a limit beyond which either TTS would have to 

be used or else manni=tandards would have to be changed. 

Training costs form a large par? of the DoD budget. Since the 

near future holds only budget cuts, one may anticipate increased 

interest in establishing procedures and data bases for selecting 

optimal OJT/TTS mixes. AFRRL research in this area provides a 

sound basis for continuing development of these methodologies. 

i 

81 

._ .-. . ,

_ 

I I 

/ 

REFERENCES 

f 

i 

i 

Dunham, Alan D. - The Estimated --T-.-e C;st of On-the-Job Training to the 

S-Skill Level intheommumcatronsCenter 

Operations-q 

r&IRL TR 72-56, Lackland AmTexas: Personnel Research 

Division, Air Force Human Resources Laboratory, June, 1972. 

Bureau of Naval Personnel - Manpower Allocation - and -Productivity 

Measurement Models, AD?i)l 3OJ, 1969 . 

Johnston, J. - Econometric Methods, 211d Editron, New York 

McGraw-Hmm.- 

, Lecznar, William B. - The Road to Work: Technical School Training 

or Directed - Duty - Assignmenn-KI%mm-72-29, 

Texas: Personi%%mDivision, Air Force 

Lackland=, 

Human Resources 

Laboratory, April, 1972. 

Meade, D. F., & Christol, R..E. - Development of a Constant Standard 

Weight Equation for Evaluating-J%%Difficulty, AFHX-TR-7P-444, 

m-5. Lackland AFB, Texas:?sonnelDivision, 

Air 

Force Human Resources Laboratory, November 1976. 

I 

Wonnacott, R. J. & Wonnacott, T. H. - Econometrics, New York: 

,John Wiley & Sons, Inc., 1979. - - 

I 

- - 

__ , - 

: 

82 

r 

..i 

‘ -4, 

: ,. 

: I’ 

.

APPENDIX A 

OJT Cost Survey as sent to USAF 

supervisors with trainees upgrading to 

the 3(serniskilled)-level in the first five 

specialties listed in Table 8. The surveys 

were administered during the third quarter 

of 1973. 

83 

3

FOR OFFICIAL USE ONLY 

OJT COST SURVEY 

AFPT 80-5x6x-109 

This survey is part of AF Contract #F41609-72-C-0048, 

for which Personnel Research Division of the AF Human 

Resources Lab is the contract monitor. 

AIR FORCE SYSTEMS COMMAND 

BROOKS AIR FORCE BASE, TEXAS 

FOR QFF!ClAL :SE ONLY

- 

. . . 

, _..- - 

z%",'p PESE (Capt Dunham,, 

SUIJECl. OJT COSt Survey 

10: OJT Supervisors 

_. 

DEPARTMENT OF THE AIR FORCE 

AFHRL PERSONNEL RESEARCH DIVISION 1 AFSC 1 

4106) 

LACKLAND AIR FORCE OASE. TEXAS 78236 

I! 

/ . . 

. ‘. 

i I 

/ 

i’ 

I 

,I 

I' , 

. 

MAY 3 1 I??3 

1. The 

~.- 

purpose of the attached survey(s) is to collect data concerning 

On-the-Job Training t0 the 3-skill level. T!lfs survey data, along with 

information from other sources, will be used in decisions concerning OJT 

and Technical Training School. 

2. Answering the survey questions with some thought and effort wfll aid 

Air Force decision makers in the management of your AFSC. 

3. Permission to conduct this survey was granted by i!q L'SAF/DPXOS, 

reference Air Force Pa ..onnel Test (AFPT) Sumbcr 60-5X6X-109. 

FOR THE COXXAXDER 

Chief, Personnel Research Pivision 

. . , - 

.-- 

85 

.. . 

:

. 

. . . . . *. ’ . -. . . *’ . 

. 

� - *. 

z. ,- � � 

INSlRUCTlONS TO OJT SUPERVISORS 

The accompanying survey is port of a research effort directed toward 

evaluating the costs end benefits of “On the Job Training.” Your cooperation in 

completing the survey is requested. While it will probably toke less than on hour 

your time, the information you provide will be vx valuable to the research and 

help to improve Air Force policies concerning OJT and Technical Training School. 

If you do not quite understand a question, give the best answer you can and 

feel free to write in an explanatory comment next to the question or on the buck of 

the form. If you are completely uncertain about what a question means, enter a ‘I?“. 

If a question, for some reasOn, does not apply to your unit, enter “N.A.” 

The survey is divided into two parts: A and B. Part A asks you to try to 

-. 

make the best estimates you can about your average experience. 

Part 6 asks you to keep a record of activities, each day for a week. It is 

important that you do this d&, so that what was actually done is fresh in everyone’s 

mind. If you alsa feel that the week you reported on is not representative of your 

I 

normal operotions, so indicate by writing in an appropriate comment; and if you con, 

indicate what the average value ought to be in your judgment. 

If you have any questions, contact Capt Dunham, Autovon 473-4106. 

*. 

: 

86 ‘r 

. 

. 

. , : :* 

f i 

2, 

of 

will 

.

SPECIAL INSTRUCTIONS 

Q Number[ml 

T 

Upgrade Training in AFSC 1. l-m/ 

1. The trainee’s supervisor should complete this survey. Approximately one half (l/2) 

hour will be required to complete Part A, and 

necessary for Part B. 

2. When answering the questions, be sure to 

- 

five minutes per day for a week will be 

have a Job Proficiency Guide (STS), and 

the Consolidated Training Record AF-623 for each person undergoing training, handy to 

refer to. 

3. The person who fills out this survey is encouraged to ask for the help of others, such 

as the OJT Monitor or an instructor when uncertain about the answer to a question. 

4. Part A which should be completed immediately, is to be returned together with Part 

B within 2 days. Do not start Part B before completing Part A. 

5. If there is difficulty in deciding what information is being asked for in any question, 

contact Capt Dunham, Autovon 473-4106. 

BACKGROUND INFORMATION 

NAME. I I 1 I I I I 1 i i I r-1 I- 

-i 

Last 

Firstitial 

Middle Initial 

GRADE If Air Force NC0 enter “4” for E4, “5” for E5, etc. 

If Air Force Officer enter “0.” 

If Civilian enter last digit of GS GRADE, e.g., “1” for GS.ll. 

_ ..-__ 

SOCIAL SECURITY NUMBER 1 II n-1 

] ]--ii 

PAS CODE I-] 

/ 

I 

, 

. . 

87 

, 

r--j 

, . . 

i 

‘.: _. 

f 

. J? 

, 

I 

i

Ji 

PART A 

1. How many trainees do you have upgrading to the 3 and 5 level in your 

section? 

2. V,%en a man(or woman) first reports directly from Basic Military 

Training, it may take some time before he actuaily begins training and 

work, even though-his “date of entry“ to training may be the same os his 

reporting date. This delay may bc due to personnel procossiog, the need to 

wait for security clearance, or some other co;rse. Approximately how mony 

days dors it take before the newly arrived “helper” actually begins OJT? 

3. There is also delay in entering training associated with the arrival of 

a 3 level from Technical School. In addition to personnel processing, 

familiarization with procedures specific to your situation may be necessary 

before he/she actually begins 5 level training. On the avcroge, this delay 

is: 

4. On-the average, how many weeks elapse between ochicvement of 3 

level proficiency and actual award of the 3 skill level AFSC? 

5. What week of training is your most average i lcvci \hclper) in? 

6. What ?b of the 3 !evel proficiency training do you cstirnczte he has 

completed? 

7. When he arrived what 9; of the d-uties of o 3 level could he complete? 

8. If you stopped doing OJT training would you be able to reduce the 

number of NCO’s in your work area without significantly reducing 

effectiveness? (Insert a “1” for Yes, or a “0” for No). 

9. During the training period for 3 level OJT, the instructor (troiner) 

must spend some time keeping training records up to dotc. On the 

average over the who!e train:ng period, how many hours (or fractions of 

one trainee? 

does the instructor (trainer) spend in record keeping for 

I 

SO ON TO NEXT PAGE 

: 

III u 

3 level 5 level 

2 

trainees \ trainees 

! 

work days 

Iwork days 

L2 I--.1 - 

LQ 

-- 

UJ 

:‘0 

-. 

El 

96 

r I 

I I 1 

Hrs. 

.

Port A (co&d) 

I -. . 

10. The newly arrived Tech School-trained 3 level is not as productive 

at first as the OJT-trained 3 level is, although he may soon close the gap. 

a. In your estimate, what percentage of the worklocci of an OJTtrained 

3 level can the Tech School graduate handle immediately 

after his arrival? 

b. How many weeks does it take bcforc the Tech School-trained 

3 level works with as little supervision as an OJT-trained 3 level? 

c. After both types of 3 levels are awarded their 5 level, on the 

overage do you consider either to have superior performance? 

(Inset-t a “1” for Yes, or a “0” for No.) 

d. If your answer was “yes, ” which type of 3 level do you 

consider to have better performance? (lnscrt a “1” for OJT, or a 

“0” for Tech School .) 

‘1. if extra (remedial) training is conduct4 in your office for trainees 

who fail the End of Course Exam (Aoprcnticc Knowlcdgc Test), answer 

the following questions: 

a. In your experience, what pcrccnt of the 3 lc.~el troiqees foil 

the End of Course Test the first time they tokc it? 

b. On the average, how many wccLs of additional troiring are 

given to airmen who fail the End of Course &an before they take 

the test again? 

c. How many hours per week, during the normal work week, does 

the trainee spend engaged in tfris remedial training? 

d. How many hours per week, .Iu:ins the normal work week, does 

the instrejctor specc! engaged in this remedial troining? 

12. If you stopped doing OJT training and hod no replacements for the 

troinecs could your section continue to perform its c;issio:i without 

significantly reducing effectiveness? (Insert a “1” fcr Yes, or a “0” fcr 

No.) 

i 

89 

GO ON TO NEXT PAGE 

. -.. 

I . 

r 

. .‘/ 

3 

7 

I Percent 

cl.2 

tieeks 

El 

-. 

L -I 

I-l-1 - .- 

Percent 

- - i 

13 . .-! 

Weeks 

cl- 1 

’ HrT.- 

-- - 

Ill _- --- 

Hrs . 

cl

Part A (cont’d) 

i 

/ 

Q Number \Jl--ill 

r 

13. Based on your past experience, and, if you feel you need help, the experience of 

other qua1 ified personnel in your section, list the average number of productive and 

non-productive hours of work for the trainee upgrading to the 3 level for each week 

between start of training and award of skill level. For instance, in the fourth week of 

training your trainee spent approximately 30 hours receiving instruction and reading ond 

10 hours doing productive work. Your secdnd entry would look like this: 

4 

Note that the hours for each week must sum to 40, and you must have an entry in every 

week. If, on the average, trainees complete tra’.?ing between the 12th and 16th weeh, 

then the entry for th4 sixteenth week should show a “40” under productive and a “0” 

under instruction. 

Weeks of Training Trainee Productive 

(to the S-level) Hrs Per Week 

1 

4 I22 I 

8 

12 

, 

\ 

Instruction 8, Reading 

Hrs Per Week 

L- ‘--I 

-; r-m 

-- --_ 

.1I 

I 

1 

1 

i 

I i 

iii 

16 I ---~-I 

. . . _ ._ EJEI 

20 c l --i I 

Li. 

14. What is the total number of personnel in your secti/on (officer, enlisted, 

and civilian)? I 

15. In addition to the trainees you now have responsibility for, how many 

more 1 level trainees could your section train right niw without 

significantly reducing the effectiveness of section o&rations? 

(ignoring the limit on aurhorizcd number of personnel) 

16. If you had to lose a qualified 5 level for each new 1 level trainee 

(helper), how many 1 level trainees could your unit train right now 

without significantly reducing the effectiveness of section operations? 

90 

GO ON TO NEXT PAGE 

il 

no. gf-@s. 

n- 1 level 

trainees 

rl 

1 Clel 

trainees

PART B 

FOR OFFJCIAL USE ONLY 

FOR ONE WEEK PLEASE KEEP A RKORD AT THE END OF EACH DAY OF THE 

AMOUNT OF TIME SPENT IN EACH CATEGORY. 

17. How many DDA airmen do you currently have enrolled in 

OJT to the 3 level? 

18. Record dail.y the total hours your 1 level 

trai..ses spend on reading and receiving 

instruction each day. (Be sure and ask your 

trainees for their assistance in completing 

this question). 

Man- 

T”es 

m 1 r-y-; 

- ..- 

I I 

AZen 

19. Record dairy the total hcurs your 1 level - - - - p 1-1 -7 y--J- I ” i 

trainees spend in activities contributing to 1 -.--2 ’ ’ 1 __I _ _’ . _.__ i 

office productivity. 

20. Record daily the total hours of 

instruction provided by each grade of 

instructor. 

L 

Q Number I-l-TT--ll ._ 

E4 :’ 1 : i 

I---! 

-.-p 

---I 

I 

-__I 

-77 I i iTj 

AFTER YOU HAVE COUPLETED THE E9 p _. i--’ 1 i.. j:T1..1 [J-y .--’ \- t---J 1 .I[_1 

ENTRIES FOR FIVE DAYS RETURN 

THE SURVEY TO YOUR BASE GS-5 i ,.-!.--l i::~i *c--n 1371 n--l 

CBPO 

FOR OFFICIAL USE ONLY 

GS-6 n-j ::izj cl-: I -1 .I 1 .1-i 

5

SYMPOSILW 

TRi.“cLATION OF TRAINING RESEARCII INTO TRAINING ACTION - A MlSSING LINK 

In t rcduc t ory Remarks 

C. Douglas Mavo 


I accepted the chairman’s job in this symposium with the understanding 

that I would rot have to be the kind of chairman who is neutral at fmpartial, 

but instead that I could expose my biases on ~!le Translation 

of Training Research into Training Action, the same as anyore else. So, 

in introducing t*:e topic it should not be surprising if i express some 

of the thoughts that I may have been repressing during the two decades 

that I have been involved in training research and in translating it 

into training action. 

During this period, we have won a few. We are all familiar with instances 

in which the results of training research and development have 

been implemented into the en-going training operation and have contributed 

materially to it. Two examples that readily come to mind in Naval 

Technical Training (the area of opcrntion with which I am most familiar) 

are programmed instruction during the decade of the 68’s and the implementation 

of computer based instruction which is underway on a substantial 

scale at the present time. 

But for every R&D project that has had an impact upon training there are 

numerous ones that have not. Xov it-can be argued that this is toe nature 

of the beast, that risk taking is an inherent and necessary part of the 

R6D process. No doubt this is true, hut I submit that the normal risks 

associated with RCD do not even begin to account for the unused, and in 

some instances unusable, volume of trafning research. 

Much of the systematic civilian work in the area of translating educa- 

Lional research into educational change has been accomplished under the 

rubric of “linear change models in education.” Xost of these models 

describe a linear sequence of functions that include: research (the function 

in which new knowledge is produced), development (in whfch a product 

or procedure based on the new knowledge is engineered and evaluated), diffusion 

(in which the generality and extent of applicability of the prod.lct 

or procedure is explored), and adoption (the function of implementing an 

appropriate form of the product or procedure in an on-going educational 

situation). It is generally conceded that in order for such a model to 

work, at least one of two conditions must exist; either the people involved 

in the linear sequence of functions (i.e. research, development, 

, t:’ 

/ 

I 

92 

,;- 

‘I 

. 

: _.. 

. I 

\ 

‘. .

. 

‘\ 

diffusion, and adoption) must be intrinsically motivated toward the 

common goal of improving education by means of research, or they must 

be extrinsicallv motivated by accountability with resjject to accoaplishing 

the common goal. ?lore often than not, a combination of both 

intrinsic and extrinsic factors are at work in varying degrees in the 

instances in which the Iinear change model functions’ properly. 

Moving to a more specific level, and with special emphasis Upon the 

mili:ary situation, there are a number of conditions that contribute 

to the “rrissing link” in translating training research into training 

action. For convenience they may be divided into three groups, namely, 

those associated with research personnel, those associated with training 

personnel, and the interface between research personnel and training 

personnel. 

First, research personnel. Not infrequently research personnel conceive 

of a “good” study and then look for a training situation in which to do 

it, without seriously considering whether the findings of the study, if 

successful, could be implemented in that or any other school. In my experience, 

it is not as unusual as one would wish for a researcher to 

complete a training research project in which his hypotheses arc sustainrd. 

and yet be unable to state in what way his f'inding can be used to improve 

training or. training cf ficiency . 

A simple solution to the problem would seem ;to be to state in advance for 

each applied research project what action should be taken in the event 

that any one of several possible outcomes of the study should result. 

Carried to its logical conclusion this.should prevent the researcher’s 

wondering what to do with I%.s results, upon !conplction of his research. 

Sometimes the researcher does not wonder whit to do with his results 

because he does not feel that it is his responsibility to point 3ut 

practical implications of his research, that his responsibility is linited 

to properly conceiving, Jesigning, conducting, and reporting the 

study. 

What about training personnel then? Are th, 1 y not eagerly awaiting any 

research results that will improve their tr/aining operation? Well, yes 

and no, but mostly no. They are busy with/the day to day pressures of 

conducting training. Both having the research conducted in their school 

and implementing the results tend to disrupt the training operation and 

Pil? additional work on a staff that alrcidy feels that it is overloaded. 

Often training personnel feel that the R$,D was not their project, that 

any credit that might accrue from it is likely to go to the researchers ” 

or to a higher echelon in the training organization, and it is just hard 

for them to get very enthusiastic about it. Besides, several months often 

elapse between the time the data were collected and the time the more 

or less unintelligible report arrives. 

93 

. . . - 1 

1 

.* i I, 

ir

This lra~s ;IS to our third set 01’ conditions L tla; ten.!:; to affect the 

production of usable research and the proper USC’ of ir, tikc interface 

between research personnel and training personnel. This may well bc 

the area in which the greatest source of our t.rouble lies. Too often 

research personnel do not accompany the research report when it is 

submitted to training people, do not make a prcse:ltatitin to training 

people concerning it , and do not a.ssi,t in its ir,?lcme2tdtir~n. 

Clearly, it takes much more initiative on the p.irL of L!le trainfng 

people ta deciJe to make changes in their training operation on the 

basis of a report than it does 1.0 reach a c!ccisic:l to t3ke :~c~ion 

when the ii;iLiior of tile repot-t Jcscribes thr sLudy, cl:::-i!‘ics potnts 

that are n,?t understood, and makes logical rc~oxxnciatitins in ;I Cacc 

to face situation to the people who are rcsponsibit? for efficient 

training i:: the course conceri:ctl. Tht prob,i!~i 1 i :;; of successful 

inplementat i,>n is iurtlier enh:~:~cct~i ii the rcst~,tr25er culitinuc‘s t o 

work wil.1 cognizant training personnel througll~.. the implementation 

of the research results. Normally this does not require d great de.31 

of time and in my view, at least, is time better spent than in utilizing 

this same time on another research study, with t?le prnbabil ity 

that the results of neither of the two studies will be implemented. 

We have long known that in implementing any concept, responsibility 

for its oc~xrring must be fixed. I ;im tliereii~rk~ suggasting that each 

organization engaged in applied K&D sl~ould have ;I sma!i group, rcporting 

directly to the Techr.ical Director ol: tO ;ltt! Colrma~din~ Cf I iccr, 

whose primaq responsibility is two-fr,ld, first, to Ic7nk externally 

to fmplementation 0i RSD products and, second, to look i.ntern.llly to 

the design and product ion of products that at-c capable uf being implemented. 

Thus , on ~ha one hand this grolup wtr\:ld er.sxrc that RSD 

products were properly presented to nppropri.ate Lraining personnel, 

normally by the research personnel who did the work, and that assistance 

in inplementing the products was continuck! aIs lcng as npproprfate. 

On the other hand, prior to approval of an nppl ii-d research project the 

group woul\! ;\ssess the prr>bablc s~ileabil ily oc t i.c product Lllat the R&D 

project was likely to produce and so advise tl~z- Technic-al Director or 

Commanding Officer. If the purpose oi tl~r projcci was to produce a 

usable product and the proposed project W;IS juJl;eJ not to bc rapable 

ot producing such a product, it would be refcrrt\L? ilack to the orlginator 

for revision. 

At this point I will give the other members UC tli? pace1 a c!inncc to 

express their thoughts, repressetl or othcrwis~:. f hope tlzt they iirc 

going to tell us that I a$ the :)nly cne wlicl has c5served t!lc problems 

Z have rrilated. that cver;rthing runs smoothly in the training world 

with which they are familiar. But we will hnvc to wait and see if 

that is in fact the case. The plan for the syr,!:.-siua is to have two 

prepared papers, one by a trainer (Ilr. Wal tcr ?kIL~wei 1) .ind one by a 

researcher (Dr. %rman Kerr) . Thr papers will r?:en !:e discusscc’ by 

Lieutenant Colonel Donald Yead and Dr . . Will inn >:>onan. 

94

In addition, if I have said anything in my introductory remarks that 

is worth disagreeing with or otherwise commenting on, I hope that 

Colonel Mead and Dr. Moonan will include this in their discussion also. 

Following the papers by the two discussants, we will afford an opportunity 

to the four members oE our Reaction Panel to make a few impromptu 

remarks. Upon completion of this structured portion, WC will be 

pleased to open up the symposium for comments or questions from the 

floor.. Our first speaker is Mr. Walter E. XcDowell frcm the Army Training 

and Doctrine Command at Fort Monroe. He will be speaking on the 

basic topic of Translation of Training Research into Training Action, 

from the Viewpoint of a Training Manager. 

. 

. , . . 

/ 

x ,’ 

. 

,.I _ 

,; . . 

95 

.., I: . 

.

SYMPOSIUM 

I! 

"TRANSIXCION OF TRAINING RESEARCH IhTO TRAINING ACTICN - A MISSING LINK?" 

/ 

From the VLewpoint of a Training Manager 

* 

/ I 

! 

Mr. Walter E. Mcbcmell 

Supervisory Educatio{al Specialist 

Headquarters US Amy Training And Doctrine Comand 

Fort Monroe, Virginia 

96 

I

II 

When given the requirement to propose Human Factors-Behavioral Science 

research for the coming year, we are faced with a soul-searching situation 

in which we must ask ourselves two questions: one, are we really getting 

the taxpayers' dollar benefit from the work that has transpired in the past; 

and, two, will the research we are asking for be likely to yield productive 

results in the future. Another question which also must be scrutinized when 

it surfaces is, if we aren't getting a masimum return on our investment in 

research, how can KC, as trainers and managers, translate the results of 

future research into better returns on our investment dollars? First of all, 

d it is not purely 3 translation problem. The process of bridging the gap 

between the ccnduct of rcscarch and the implementation of rcscarch, by the 

Army trainer, is a continuing one. It begins while the problem is being 

defined to both the researchers and the trainers; it continues during the 

conduct of the research; it continues while the final report is being 

drafted, and it- still continues long after the final report has been published. 

Let's b:Rin at the beginning. In avery real sense, utilization begins at 

-. 

the time tl~c probla to be researched is presented. The training problem must 

be carefully defined and documented to insure that both the researcher and the 

trainer arc speaking the same language and that the researcher fully understands 

the problem that the trainer believes he has. After the trainer submits his 

request for rcscnrch, the researcher must then submit a statement to the 

trainer on what he views the problem to he and how he intends to seek out the 

solution. A common understanding and agreement of the problem to be researched 

is essential if a usable end-product is to be the result. The research approach 

may have to be modified in the light of continuing support requirements. somctimes 

it may have to be recast if the research proposed is not understood by 

the trainer to be appropriately responsive to the problem as hc perceives it. 

Constant inter&tiodbetween researcher and trainer takes place during the 

conduc: of the work. Cor;lmunication is achieved informally on a continuing 

$asis through -he USC of interim progress reports, periodic reviews, briefings, 

and discussions. The, trainer requesting the research must be kept ccjnstantly 

aware of the work taking place_, and the researcher must be kept abreast of 

significant military changes within the parameters of training problems. Hopefully, 

the interaction and open ccxzmunication will keep the evolving solution 

directly aligned with the trainer's problem. 

As relevant data becomes available, that is, data which bears upon the 

problem, it is provided to the trainer. Such coarmunication involves both 

those management functions which can implement cl,ange and those responsible 

for actually carrying out the drcision to make the change. Management, simply 

dccrccing that change ie made, will not, alone, get the job done; the “notinvented 

here” syndrome must be overcome. The "doer" --t!re guy at the working 

level--must be convinced t.hat the change is good and that it will be of benefit, 

not only to himself, but to the prospective students and to the system, othcrwise 

it will be doomed to failure. 

97 

, : ._ 

.: . 

2

When the report is subnitted, it is reviewed and star'fed with ap~r

Scientific and Technical Information (CFSTI). All iriiiortant are the informal 

and conti.r*,ing contacts between researchers and potential users. 

A report by Dr. William A. McClelland, at the Cor!fercnce on Social 

Research and ?Iilitary Euqyzment at the University ofi Chicago, June 1967, 

gives an insight into possible characteristics of unTuccessfu1 and successful 

research iinplemcntation: UNSUCCESSFUL RESEARCH EFFORTS : 

(1) Poor communication. Neither briefings nor reports effective 

communicated the validity and opccntional value of the research. 

(2) Lack of timeliness. The product of the research effort did not meet 

a valid, contcnporary rcquiremcnt. It was available too lntc or too cariy, 

or it was too tangential in nature. 

(3) Degree of change. Too many c’.nngcs in operating procedures were 

required. For cxamp le , training was shortcncd (or lengthened) too much, or 

the csisting Army structure was incompatible with the indicated change. Existing 

or tradititinal practice may have hccn too strongly threatened. 

(6) L?ck of strong command support. Not enough people at high enough 

cchc ions wanted to change. 

, ’ 

(5) costs. 

not fc obtained. 

Funds and personnel required had not been programed and could 

I 

(6) Lack of cngintcring capability. Thd training experts required to 

translate the research rindings into more ~\phrationsll; usable form did not 

exist or were not: available+ 

3 

I 

(7) Policy problem. There was a lack o G doctrine under which to fit a 

new or improved training or operatik,nal capability. 

:,3) Insufficient “snlcsmanship”. f’rojc L t people did not devote enough 

effort to “selling” the product. At one tim , for esaapic, we believed this 

was not the job of the research agency. 

f 

Possible reasons for successful implcmcntation I are largely the ob-zcrse oi 

this list. SUCCESSFUL RESEARCH EFFORTS : I 

(1) Timeliness. A rccoznizcd instructional rap was filled. The.work was 

obviously relevant to a planned or on-goin revision in Army practice. 

(2) Command in terts t . 

P 

There was n strong operational command interest, 

including that of a subordi.natc command. 

and working levels. 

Interest existed at both management 

(3) Engineered product. The end-product was concrete. It was a material, 

plup,-Ln item, specifically cnginecrcd for a given situation requiring little 

99 

.Y

. additional Army effort to adapt it to the operational setting and requiring 

ri no doctrinal changes. 

(4) Earlier acceptance by others. Some other service or civilian 

institution had accepted and successfully used the product or a ver; similar 

one. 

(5) Personal-interest. An individual Army officer or g,:ruup oE officers 

or key civilians associated with the work were convinced of its WOL-';II ,!r~ti WL-I-C~ 

willing ~0 serve as forceful prapc'nents. 

: 

. *. 

100 

. . _ 

: --_ 

- . ,. . .

SWPOSIL?! 

"TRA!EXATION OF TRUIU'ING !tESEARCH INTO TRAINING ACTION - A MISSING LINK?" 

From the Viewpoint of a Training Researcher 

Dr. Soman J. Kerr 

Director of Research 

Naval Technical Training C0mar.d 

Naval Air Station Yemphis 

?Iillington, Tennessee 

;- 

101 

., 

-_. . . . I ’

Perh3ps those of vou i:> trnining management thlr:k I &I you so-:C i::j~s:ice* 

I cJnn3 t a.grci!. ‘~Iw tralni:lrZ manager, in zy percept (on of him, .~lw.i~s Sf?tZkS 

answer:’ yesterd:~v to tomorrirw’s problems which he is causing to‘!.\\.. i’.t? tradition.~llv 

accepts 3 truixn such 3s “individualization is ‘@“’ o\:’ “trainine 

tine must br> rtlducc-d hv 10%” snd, vith unquestioning blindnc*ss, blunders 

forwdrJ wit!; 3 GrXFZKor.:p,1 innce actions. L’nqucsLian3blv, hl: ::;Is!z instfz:,-t 

ix-elk- know that ;mv cltlfust7.cnts .to the system will rl>sulz in sx:~ x,arvini: 

degree of turmoil. But it zust b> th3: he hopes the* t!w :urmof; will be‘ 

temporary rather th;ln permanent--and he does not willin!:lv accept rhr fact 

that s.xh actions may he IIct ive curriculum, the training zanagct is also 

knokm to be cspab!c of putting top.ether a curriculum on a complex 

102

weapons system (for example) over night. 

The typical training manager will fight the instructor billet battle 

shemingly to the wire by a continual wailing over inadequate numbers of 

instructors; yet kc is verv villing to accept neu curriculum administration 

responsibilities for his overworked school staffs. Those xho question this 

should ask themselves bov rzny or in what proportions were new instructor 

billets provided Khen additional curriculum burdens such as human relations, 

defensive driving, and remedial reading programs were directed for inclusion 

into existing training programs. Yet, these good men seem obsessed tqith 

management and its precise controls, and often decline well-intentioned 

advice and assistance. That is putting it mildly, for it is onlv with reluctance 

and suspicion that zany training managers will accept a rationale 

or the experience of others who ore not only directly in but senior within 

his line organizational structure. 

It night be that the training manager has the utmost confidence in him-. 

se1 f--not necessarily in his abilities to improve the training, but most 

ascLredly in his zosie to be able to handle the many and varied resoonses 

he x.-ill derive fro:! both judEnentall\- sound and highlv questionable decisions 

which he has malit? in the nare of training manafzezent. After all, is this not 

what trail,*ng man3zezent is all about? 

, 

Have I left iut planning--or the lack thereof--by the training manager? 

I have not meant to. As any manager knows, planning is an essential part of 

his daily routine. ?!y Lord, does he pian! He has training plans, personnel 

pipeline plans, SER plans, CIKX?3 plans. . . Heaven knows how manv other 

schemata he considers in the laying out of training requirerpents and in the 

execution of training activities. But ask the training nanagenent community 

for research plans and stand by for a high fog count in the responses. Better 

pet, ask for definitions of training research and you’re likely to get respocses 

In the order of “soneti,ine not needed”, “something received after the fact”, and 

“sonething that is needed to verifv ;?v actions.” The training nanager, often 

will inp to spend hours gruzbl ing, perhaps, but unquestionablv counting the nmber 

of sessions in a score of curricula to develop data to satisfy an obsolete 

report on a hands-or. vs thecry ratio, is generally not prone to review the 

research conducted oc highly pertinent areas of his responsibility. Yet that 

research which might well enable him to better conduct the business of training 

may be in evidence all about him, gathering the patina of GSA dust. 

Early in pre?aratlJn for this s:=nposium I decided that my assigned topic 

“From the Viewpoint of a Training Qesearcher” p(ave me wide latitude in pinpointing 

--at least to the satisfaction of my own perceptions--where the problem 

exists in the translation of training researchnto training actions. I think 

that I have now zade that ooint several times over and I supposed that, since I 

probably have a goodly percentage of my listeners in a state of spirited agitation, 

I should smile broadly and back ‘slowly out of t!le nearest exit. I promise that 

I will do just that very soon, but first let me say that I tried to locate strong 

support for my conte-tion from the literature. 

103

- 

..\ . . 

i I 

Co ny dismay I found th.%t zany of the articlts on thy subject in research 

and development professional journals are rcallv self-deprecating. Ke in 

research are reJl 1y rather ;)r~ni~ to take on the problems of the world: rr) 

blame ourselves for another’s inability or unwillingness to comprehend. The 

chest beating is often, in re.s1itv, a hushed humble breast-thumping which marks 

the contrite, and thrre are xnu in research who continually trv to fin‘! the 

missing link of translating cr,tining research into traic,ing actio:, alone. Lt 

was amazing how spars2 v.Is t!x :n.zterial in support of thee rather parocLia1 

position I have presented. .‘as .L latter of fact, I found the strongest proponent 

of the view thsc canagcix;:t r1hlx5r:s to take seriously well done research in 

tha management field. I uim;C of criticism. Thus, if researchers :zant Co 

fcrl badly about their rolz il: :hri missing link bit, thev mtrclv have co turn to 

sources suck 3s tilt? . Arcricx3 . !‘.ius.ltion !n of the problem faced and the solutions reouired. 

Let me quotr Beman again: 

i1 

“If we don’t do this (providing the careful definition and 

_ 

the solutions required) , if we continue to allow a situation 

to zsist where vn’l’ .Ire constantly wondering how much support 

WC are givirig RSD, then we are openly admitting that we as 

manapers ha.

.I’ , 

So, you see, the solutlon to the problem is clearly in the hands of 

management. 

Now, being in research I cannot help but come forward with not one but 

several approaches for solution. 

The problem is not easy to solve: there are cany levels of competencies 

as well as many specialties in the research function just as, in production 

facility, one will find a variety of competencies and specialties within the 

work force. In the research function, for example, there are personnel who 

work well in fundamental research problems but who, perhaps, are Impractical 

In the broad field of applied research. The converse is axiomatic. The 

Department of Defense RDTLE structure speaks to such specialties in the functional 

terms of: - 

6.1 Fundamental research 

6.2 Exploratory development 

6.3 Advanced development 

6.4 Engineering development 

6.5 ?lanagement and support, and 

6.6 Operational Systems development, 

recognizing the unique facets of research and development. It seems to me 

that management should recognize, if not the categories, &en certainly the 

functions of .the several types of research programs. 

Glenn Bryan, in a published article entitled “The Role of Basic Research 

in the Total RLD Process”, in the January.1973 issue of Research Yanagement, 

conceptualizes a series of models which address the enigma of placement of the 

various types of research into an organization. 

ALlong the node18 arc several organizational ones with which we are 

familiar. First, he depicts.3 “Linear Production Xodel” which, for my 

purposes today, adapts the DOD research functions into a production scheme. 

! 

. ’ 

105 

3 

. 

- _--- - 

_* --. 

-_. 

: 

.

You can see, the block to the right draws upon and feeds responses to the 

several research functions to the left. This situation is somewhat idealized. 

It appears in the straightforward approach that flow to and from the blocks 

is continuous, unimpeded, and simple. It would be ‘nice were that so. 

Another model of Bryan’s is the “Departmental llode1.” 

' I* 

7--- 

I .-- 

2; 

; .? 7 

I 

, ;---I 

, ‘L .r- -* 

:._ . . . . . ,*.s.. ' _. .* '-~.-,J 

. 

* _ 

. . . . . 

: __. 

This model has decentralized the research function and superior authoritv 

which Bryan calls “?fanagement” in the model takes total responsibility for 

coordinating the several research functions and moderating communication among 

them. Bryan states that such a system as depicted in the model requires 

superior technical competence at the management level to function effectively. 

I might also add that it will take superior managerial conpetence to keep such 

a conpetitive system functioning toward the overall goals of management. 

Another model is the “Project Yodel” which has had muc‘n usage throughout 

the Defense establishment. 

The project manager is supreme in this model. He buys. what he needs 

and what he needs is largely a function of his perception in achieving project 

goals. Bryan points out that such a structure has both good and bad 

-.. 

I 

/ ’ 

106 

-. I 

.* 

, . 

,a’ 

_. . . 

, 

- -. 

- 

. .

merits. Inasmuch as the highly focused research efforts, that are in line 

with project goals, are often well supported since, they are usually an adjunct 

to a high-visibility, large project, that is good: The drawback, of course, 

is that there is little general support for research not directly in evidence 

of project goals. Research runs the constant riSk of being a necessary evil 

in this model. 

I won't shot: you Bryan's "Organic Yodel." There is little need to, for 

consider a tree, if you willi'laden with lush fruits. Then consider its root 

structure, seemingly dis0rganiz.d. ugly, and dirty. Then consider the roots 

as what they are, the source of the hcalthv fruit: the roots that are often 

misunderstood as to their function and their problems. That the roots can 

isolate the proper nutrients that are necessarv for effective fruit production 

is without question. Also without question is the fact that indiscriminate 

root pruning will almost certainly lave adverse effect upon the fruit. 

Bryan's tree is more an analogy than a model. It is too simple to portray 

in any real sense of the world what the interaction between management and 

research is: it merely portrays what it should be. 

And this, perhaps, is the reason that we live In a world of linears, 

departmental, and project models--variations, perhaps, of those described. 

X reasoned approach by management is necessary to bring about precisely that 

fine-grained relationship of mutual respect withfn organizations among its 

major functionaries. The cartoon she-zing the manager in his plush office 

speaking to a person in a laboratory coat, and bearing the caption: "You 

researchers--make some kind of a breakthrough!" has got to cease being funny. 

, 

It would seem to rnz that the underslanding of the RhD function, the 

placement of that function within the dy; amics of the organization, and the 

understanding by management that reasoned direction is reouired will bring 

the closure we all seek and elim!nate thy need to worry about the translation 

problems between training research and training actions. 

I 

. , . 

107

108

Prr 

-. . Kerr sOg;er,ts what I perceive t3 be the !*cnl ke.v to 

ixvine renen 1-c t: PI-oduct? from tf!e 1-L~~-3tc:*y.~ -. .J_ to the 

classraoa, an-l thrtt i s concrehensive trainI 7-16: !aesearch 

plans and requi-ements. I beliel/e t3e mzn?l:Crs’ susnicion, 

reluctance, and resistance xay freq::entlv be 

traced to research conceived in the isolatT,:w of a 

laboratory 3n.i aubxesxentl:,, offere% to solve :I training 

p:-obiem. Typically, t?lis -e&ire? tI1? Oril-in:\l research 

y:Boduct. to he ::;@3ified, a-fjuse,e3, ?.n? exn?? !ei:, with 

p:*cJictable results. It is !i>t sur?risinc th?t hook 

tkelves a r e loaded with researc!? rzgorts neuc:* npnlied. 

I :-I many instances, I suspect it is hecause they were not 

. 3”ci:7ncJ _ ._I in re::ponse to st,:te,i oper*ationrl? I’rvh le.ms . 

Alone the same lines bLit sxewhat rfeeger T f+t>l that 

r.sjor zodifications must be zade to zain’cr7?T*.linated 

aznagexent of the multiple 3X rese??ch nr-t?ncles. 

Current ly 6.1, 6.2, 3rd 5.3 efforts zre ot’ttln lnit iated 

a’ 

, 

109 

- 

-: 

. 

I- 

.I

. 

i I 

/ 

Without cross-validation or coordination k:it?l t$aj.ning 

managers. It is not unheard of to ha.re a contractor, 

whose unsolicited proposal is funded with 6.1 monies, 

arrive at a military instaliaticn without announcement 

or coordination. Similarly, there are examolesiof 

duplicated efforts while topics neediny: exylora,,torv 

research remain untouched. .4t this time the nl'litary 

services are open to criticism in thzsarea. ?er:;nn:lly 

I believe that no research, re.gar1less of tunctlonzl 

category 6.1, 6.2, or 6.3 shm$,ci be undertaken unless 

it supports stated user requirements. Recently an 

Interservice Training Review Fcard has been established 

which could make si::nifican: contributions in this area. 

!5?3ile I agree with Dr. Kerr that a trainin?: research 

requirement document is an essential Item, I don't believe 

thst It is solely a riansqenent resnonsibility. Ideally 

th.is document would be develope:? in coordination with 

training researchers. Carried a step further, it could 

be a matrix consisting: of a taxonomy of sirnlficant 

training research areas on one axin ?nd researc?] a~::encZes 

on t!le other. \!hen 0ny;oinr research efforts are Flatted, 

those areas not being addressed beccne rendlly arnarent. 

At this point, I fully endorse the close coordination 

between researcher, nanaqer, and trainerl??resente? !‘.‘I 

Mr . McDowell. Sever-n!. points of his presentation vrtrrant 

reemphasis: A meet ink; between researcher and tr.aininir 

manager shouid be held prior to initiation of the **~cp3nch; 

-...--r- 

the statement of work shou?d he soor3inatei an2 ztnprcved; 

proposed research products should be revhewed and'approved 

as developed; and close interaction should !le maintained 

as the project progresses. l)is -.I coordinated spesrhesd 

will open a pathway through nanaqement's anti-research 

field of sharpened fence posts ,fescribed by Dr. Kerr. 

Another area of general arreenent a!?.on~ he presenters 

is the problem of placement of R&D acti .ities in the 

organizational structure. Dr. Xayo suiT.ests 1 the RhD 

group should report directl:: to the technical di:lector 

or commanding officer. I ccncur in this suecestion. 

flacement in the upper echelons of ti;e,oxan~zation~l 

structure is critical since it dictate, 

d the interface 

between the traininrzr nanaqer an,l researcher. In aJdition, 

it eases access to top decision makers which is essential 

f-0 r support of research projects requiring major t.xDenAitures 

of resources or hnvinc significant impact cn the 

training activity. 

. 

110 . 

;

4. 

- 

, 

i3ecause of several unique aspects a brief description of. 

Air Training Command’s RS3 xrgsnizational structure might 

be of interest to this audience. Tt should be understood 

that the responsfbility for perronne? research in the 

Air Porte has been assJgr:sd to the Iiuman Resources 

Laboratory, Air Force .?.vstens Co.zmand. Research requirements 

from each InZIJC!q command are fork!arded to them for 

accoa~lis?.ment . 

S;s:nek::hat znu;u:il 1 :: the fact that tke 43 ;,TC R&D orsanizstic,n 

is locs’;e.-l under r!:e 2cput;z C::ief .>:” Staff fj:- 

Plans rather !!;?+-I , ur. ,? er t:?e .u Y$ v. -“or OCCh:!: eal Training 

3,” DCS far Operntio!:s, xI;ic!. int-,,lves f1 A 5 *-s.,, .-:-t and navigator 

training. ‘ft:r, placemen: re:mitZ ClOSf? lntcrscticn between 

fu*.Ire ccr;.mand programs and R&C requirements, permits more 

conorehensive scnnni:ir; of state-cf-the-krt technology than 

riigj-.t be rassi5lc under 22 agcnc*; responsi!yle f o r specific 

train;nc, 3rd provides 3ccesI: tc the commkncier. 

Anctt7er u::ique ?spcct 1s T!le pcrz~cr~zel rm;ce-UD o f t!le Hq 

RS3 staff, xi::ch cona:sts of ‘i ccmputer sreciallnt , two 

r** ‘70t-fli~!;t s:mula:.c:* sreci?lists, a nn:‘l,-stor simulator 

specialist, 82 n.uJio-:ri31;a1 s?cci2list, an? two bc!:avioral 

scientist::. ?3Ch specialist ?s. c!-Larged to interact wit:h 

3CldeZiC, clvilinn, and cilitsry 2.Gencies 3evelopir.g 

products or con~ucti r.? research in 95s arc? of resronsi- 

5i:it.v. Infc:rxt ion concerninc ne::’ tcc!:nc.lnpie.s W+h 

potential use in the cornman? !: C>CrAiRat~3 iiith thiappropriate 

DC,” and/or fcr::lardeci >o the arcrooriate 

trsining 32t tvity for cccrdination. Khere interest is 

s!~oxn srranqements are na5e for *:Isitations to the research 

s:;ency or !-rizfinc of ?ccroDriete staff rcxonnel by the 

researcher. ~2ch spez iaxs t is ;t 1 s 0 respcxible for 

in:tiatin[; Requests for F’ersonncl Research in his area of 

interest to be conducted 5:: the i-iunan Resowces Laboratorx. 

Trainin< Research Applications Rrs!:chcs or ‘?RARs have 

been established at esch Technicail Trainin? Center, 

monitored by the 112 R&D group. The TRAR works for the 

sckcol opext iox office? and has iccess to the SC~CO~ 

cczmncln?er. E 3 c t; CR,A9 has a clvil:an CS-11 educational 

spec4alist snd txo r3iLita:y: behavlorsl scie::tists. 

These organiac?t !onn’ nerrora ql.iick reaction studies for 

tra.ln1r.q mzi!191c:‘:: Cbtl+UCt CI~pliCSTiOn stulies xith 

‘,rsinirq re:eiirchirs 23 ,?irectc,d 5:: t h e I!? R%D group, 

an;t Znitiatc resc:1rc!; requests to be acconclished by !!RL. 

111 

. --.-. ----

, , 

. ., 

\: ; -* 

All TRAB projects and research requests are coJlt??'lled 

by the HQ R&D group to prevent duplicated effort. 

A Research Review Board made up of the technical directors 

from Technical Training, Flleht Training, and Plans 

has been established to validate command research requests, 

In addition, they monitcr the action taken on recommendations 

resulting frcn completed research studies. The 

Research Review Ro,zr,1 responds directly to the Chief of 

Staff and the Vice Commander. 

The final area of arreement mentione\d by our presentors 

deals Fith t:he :'cqui:'ecent fOI> cc7zunicatlnns between 

reseclrcher an-1 ti*2i::in^: mzr.?g:er. If a research plan ir; 

developed an2 t?ere In close cocrdination dulling: the 

initiation, coWfAct, and re?ortinF; of research, communications 

proS1em.s are rarely: experienced. 

I hsve come to sevt~~1 ccnclusicns regardin< communications 

during-: the past three years of coordinating the 

interaction het3een researchers 2nd high ec!lelon decision 

makers. Some of m.v \--onclu sions concur and ot?lcr:; differ 

from the positions taken tcrinv bv our speakers. MY 

remar'ks reflect ny Fersonol opinions only and are specifically 

directel tox?rd t!:e interaction between the 

researcher and highest level decision makers. 

Researchers are their own :.:orst enemnies in the transition 

of research Froducts from 'vhe laboratory to the training 

environment. Speci flcEilly : 

a. As a group t?ley are particularly poor briefcrs. 

They generally arr:ve preysr ed to deliver a three-hour 

briefing to a person !*J??o has limited time for discussion. 

Their briefing format ususlly follo?Js that of an AFA 

journal article an.3 i:; usually stronc;ly statistically 

oriented. Althouc!l experts on their studies, they seldom 

address the critical aspects affecting: the decision, i.e., 

the total resource impact of implementing their tJor!

. The publishing of selected bibliographies and 

reviews of literature may look great on t,,he vita but it 

tarnishes ths researcher's image among mdnagers. 7: 

shudder when these docur;r;lnts reach the headquarters 

because they invariably precipitate two qy,estions: “IS 

this what we are spending our money on?" an3 "What the 

hell an I supposed to do with this pile of junk?" This 

is particularly damaninq when expectations are high concerning 

a new project and this tvpe document arrives 

representin the initial work expended. Fly recommendation 

would be to ho12 the document and submit it at the conclusion 

of the project or incorrorate the entries with 

your interim reports. 

C. The tine-honored format of the tvoical research 

report may be justified for puciishinn ease and communications 

between professional scientists, but it is ill- 

.designed tc convey meaninL,- -'ul information to training 

managers. Arrain I ?:ould encourage you to consider your 

audience and write to meet their needs. Xinimize the 

discussion of your stat istical design and consider 

presenting this information as an appendix to the renort. 

I rcall:: reccmcend Arorrinr t!le Al??, rerzrt format and 

k;oing to esecuti;re s.umzary type reportin;T. 

d : An:! ?inall::, V;hen a fast reaction situation 

requires, stick your neck c*ut. If a decisicn has to be 

mde , don't su,z.qest you can provide a sclution in 

4 18 months. Use your knowledge of research data, and 

coce up wit:? viable options. Validate 2s vcu go and 

modif'y ,the approach+s required. 

i 

I 

.’ 

I 

113

SYMPOSIUM 

$RANSLATIOII OF TRAINIXG RESEAXC3 IPiT TRA.I?IING ACTION - A HISSING LIMK 

EARL I. JOIIES. 

NAVY PERSOX1CL P.ES!XRCH .ZID GZVCLO?X::T CEtITZR 

SAX DIFGC, CXLIFOP.XIA 92152 

Having rc-ziexed Kith zrcat intezst the pazers of Hayc, KcDoxell and 

Kerr I shall attempt to summarize their main points, conrent upon them, 

and then as is a discusZant's prlvllege, * '* provide some comments of my own. 

Dr. Nayo, with whcm I have worked closely for the better part of 

two decades makes several major points. His first major point seems to 

me to be the rationale of this symposium. It is that a great amount of 

training research and development goes unused. he stipulates that 

failure to use R&D cannot be accounte? for by the simple assumption that 

risk taking is a fundamental part cf P. &D nor by the fact that some of 

the unused.training R6D is indeed unusable. He leaves out an important 

(but fortunately small) class, namely, training R&D that has been used 

but should not have been. 

The next major point in Dr. Mayo's paper is his reference to 

"linear change molels in education" with the stipulation that conditions 

of intrinsic or extrixic cot ivation must be functioning for the model 

to work. 

From here Dr. Mayo moves to the position that most of the conditions 

functioning to obviate use or transla:ion of training R&D are associated 

with (1) research personnel, (2) training personnel, or (3) the interface 

or interaction between research perscnnel and training personnel. With 

respect to research personnel Dr. Nayo perceives the major sin to be 

conceiving studies in which researchers look for training situations in 

which to execute but fail to consider (in the planning stage) the utility 

or implementability of the R&D results. As a solution to this unwholesome 

state of affairs Dr. :!ayo has the pragmatic notion that researchers 

should pre-state and pre-judge the training actions to be taken contingent 

upon the possible R&D outcomes and accept responsibility for such 

planning. I 

! 

. 

With reference to training perscnncl Dr. Kayo describes their work 

overload, their dislike of waiting forever for utilizable R&D results 

or recommendations, their commonly net identifying with the R&D task or 

project, and their feeling rhat whoever might get credit for research 

based action, they (the training pcrscnnel)'wiil not. 

. 

114 -- 

.-.. . . ,_ _ 

: 

._ 

-:.* . -.f.. 

.’ 

’ , : a. . ./ \ . * *, . ’ ‘1, , ; q 

~z-“J

Dr. Mayo sees the interface or interaction between researchers and 

trainers to be the most critical problem. here he recognizes that the 

continuity from X&D to translation to change in training can be accomplished 

only by appropriate interaction in which researchers and their 

managers take the responsibility for the translation process and assist 

the training personnel in the a:tion process to the point where a;;rcpriate 

inplcncntation,has been accomplished. 

Mr. McDowell's presentation is a natural follow-on to Dr. b!ayo's. 

He perceives the requirement for proposing resea..-. m-5 as a soul searching 

responsibility for use of the taxpayer's dollar invokring the questlor.s 

of benefits gained from Fast !?'.tD and projected yield of future ?.i-3. he 

, perceives ttle t~'a::,ers' and maxt~cr~ :-ales as not limited to ;r-!nslatlon, 

but a process 3f bridging the ~au j . t;etween cznCi:c +i of research and im?lementation. 

This process: is continuous according io Xr. NcDowell and 

2equires constant interacrion between researcher and trainer free problem 

identification, 

implementation. 

XE.? planning, conduct cf X&D through translation and 

Mr. McDowell is not so prone as Dr. Xayo to perceive a vast graveyard 

of unused i%D. He qualifies his remarks in a way which implies 

that the Army may not suffer the "missing link" biimna to the extent 

that the iiavy does. 

Where Dr. ?-!ayo exhorts the need for trainer---csearchar interaction, 

. . 

Mr. McDowell talks of a system 11: beIn&. He refers to interacticn that 

"takes place;" data whic!i are "provided to the trainer;" reports which 

are' "reviewed and staffed with appropriate elements of the organizstion;" 

and specific examples of frcq.uent and essential in-olvenent of the rcsearch 

team in "active participation." He percc2vzs iise of F.&D byproducts 

as increasing, and dissemination of FLD findings among FotentiLl 

users as essential. 

In closing, Rr. McDowell excerpts from Dr. ?!cClelland's 1967 report 

eight characteristics of unsuccessful research efforts and five characteristics 

of successful research efforts. 

The final presentation by Dr. Norman Kerr is a shocker. in some 

communltics it might even seen blasphemous. However, in the long course 

of the history of education, the history of training and the history of 

tralnlng 

. .a 

OCD, A\v Dr. Kerr's remarks spark a flame of truth--albeit a truth 

so mixed wit!1 exception, complication and dark shadow that its exnlication 

is no slnplc'matter. 

Dr. Kerr's paper rtatas what the other pa?ers only imply. The 

problem may not be a missing link but a faulty chain anchored insecurely 

and subject to vectors of power and stress that make the task 

of translating and a??lying SD results in a positive manner unduly 

dil'ficult . cm. 

even ic~ossible. 

!15

Dr. Kerr's major poi::::: are that training, managers obvf,lte rather 

than facilitate RGD, t!la: :!;cy ~CSSCSS Dove r wherc3s rcsearc!s-s do nc:, 

and that the power disp.lrI: \' dt all 1evCls of in:c:raction betuaen 

training managers and traZ::i:>c researchers are tht* central f3cu.s for 

consideration in this s;?.Fo::ium. 

. 

Dr. Kerr carries this ar~uzcnt with 

conviction, verve, eloq::.~:::~ ‘inil numc?rJus cxampl~s. Althou;> hc provides 

a scorching indict-+::: of training nanaGcrs, Dr. Kerr does not 

back slowly out of the n*~.i:‘~t exit. iie finds to his disma;: that his 

review of research literii::::'c' does not provide the confirma:l>a he 

seeks Cut that canagemcnr lZ:iraturc dJes. x'ith sur:)ort : r 3.: rcscarcti 

nanagczent +litcratcre an,! ?;i:: Own kn~3.2led):e. of the KkD comm*:nZfv'zj 

nar.st;em~ent models, Dr. +er:' ; roviiies Some ?ositivc proposals :zr 

rapprochement. 

To my ear at least, tf:~ three pa,-ers xe have heard are es:remely 

use:ul accounts of the suz:~~-rs or failure of ail::ary orr,an::;i:zons to 

exploit RLD as a means c: u:t:rCIJing militar>- trai::ing. As XC might 

expect the ;apeFs strongly :~:l~t both the nature ar,d nurture of the 

problem and the histories .‘: the presenters. I, too, will ;:‘c to 

overccx tfiat dilerza Of ,.I::. OhS~t~c:'s--hOx tc F,et outside o:‘:y own 

skin h%ile 5eir.g securely 1*3:::;d within it. 

The does not per-it :z'-plotc coxent cn "li::t'~r change r.e>els;" 

suffice it to say that th.c.:: .kFe usc!‘ui devices whlc!l like :nr'st models 

have variozs constraints. Y?:c primary one is perhaps the r;~iti ior 

intrinsic or extrinsic mot1v.k:tion. tie:-c :*1r . :.:($p n'e 1 - ' ' cJ 

cnuticz.3.ry 

observation tht a decree t-y -~nagc:ncnt (one type c! extrinsic zot 1vc1- ' 

tion) vi11 not get the job :!:::c is fair war-zinc a::J is echo& zn 

differer:t ways in both Dr. X;Clcllan.i's l&t and In Zr. Kerr's 

adnoni:ldns. Estrinsx t.0..Ivation is then sometimes necessary but 

seldm sufficient. ?!owcver , riven that one of the mctivaticn&l 

conditions is necessary, t!.~ ?roblcm becomes one of ccntrolliq: zhc 

direction of its outcome. 

Within the conter't cf T!:~s symposium, Dr. Kavo's point ',t;.rt tran::L.rtion 

of R&D into action dcg-i!t;.l% upon researc5 pcrscnncl, trarnlni; pcrsonnel 

and their interaction 2:: SO 

! 

strongly sc;?ortcd by WcDcwcll, Kerr 

116 

3

and logic that it neck little cX:Zcntary except to wonder why, when the 

problem c;\n be cast i::to SUCK sin;le terms, the solution to the problem 

has no: 2~3 tigo beer. routinely applied. 

Mr. IIcDowell’s y..lper 5s SO canv~ncinglp positive that it would 

seem, fcr the A:T.Y .\I Icast, the solution is at ban-’4. 

Shifting now tc .?r. t'.err prsviies a somewhat stark contrast. !ie 

ignites SC:;?: d con~1~~r~tizn that we are hea>ed with ashes and wonder 

as in the “Perils cf i‘zulizc” i: there can ever be a ha;py c::ding. fiy 

followi::< .':.. i&r:' i:::Y!:Cr, the :ncst obvious answer uculd appear to be 

a qualif ic,'. ";:+a'; ." :'..,;--.r.:.. 

. . . . . - LC-&^-. Lik;tyit\-te p: to &\-axpcsltcve 

point, the need for rcrearc!: and 

nanagemcnt 3y.ir.L1:ft I:?.s tc hccctc an lcttgx-21 force for prcduct mXl- 

. 

provenent a;:3 3: .:-1z;xt I 'CT. . effic:cr.f.,*. Had Dr. Kerr’s search of research 

literature ttTe:i ;GZP*~~-.AT mere extcsive he xould have found a good deal 

of SuF~cr:. :t;c ~.-y-7.:‘:--~ a. A..k.“ cf yin? &, Ixmsdainc, Glsser, Rigncy, !4ackie, and 

even Earl Lz;.*s x::L,! not onlv he ionsoiing but might dispel his notions 

of contrite. 

Tf my IrxLq-ctation 05 these papers hss been ccrrect, I heartily 

endorse not only :&I:. ~.afc:- ;ti::l hut their eqlication of factors which 

obviate i~~3,e,ent~.tI0\;: of 2.53 into training action and their recorzendatiOX 

fo:* ;asitiLc s:c:,,s f;:- cat31~,zing, facilitating and improving the 

translation ‘and implementation process. Yet I think they hnvc not gone 

far enoqh. \Shat seerii to me to bc missing arc n set of models for training 

mansgcricnt , K&D management and a mcta model, superordinate to training and 

K&D, cshic!i c0uI.i gnidc both the training community and the H&D community in 

the spirit of intcgrckt ion, establish a11 the ncccssa~ intcrfnccs at all 

le\pels of function In both training and training CF,D and proi’idc such mechanisms 

as a Training Rcsenrch A,!risory Board and 1aSnrntory schools--in short, a model 

which cou1J faci:it;itc the kin& of interactions all three of our speakers have 

proposed in their o::n unique styles. 

117 

,

TEST FEEDBACK AND TRdlNINC 0EiJECT:)‘ES 

BY 

CARROLL H. FREE&E, LTJC USCGR 

UNITED STfiTES COAST GUARD TRAINING CEHTER 

b 

CAPE HAY, N. J. 

/ 

Note : The views expressed in this paper are those of the writer - 

and do not necessarily reflect those of the USCG Trainicg 

Center, Cape !tay, N.J. 

118

INTRODUCT IO11 

The public seems to be generally willing to accept a decrease in 

the manpower of the armd services, As a result of this acceptance the 

active draft system has been deleted as a method of acquiring service- 

men. Whether this acceptance is warranted remains to be decided however, 

and in the meantime this decrease in manpader is something with which 

the mi I itary forces must deal. Ihy must we deal with it? Decausc, in 

the event that military or naval forces must be called to action on 

short not ice, the American people wi II expect and demand the same 

readiness and response uhich has always beer. cha.-acteristic of the 

armed forces. An uneffectivs response at such a time could cause a 

breakdown in one oc all of the systems which oefend nut N.st ional 

Security. It is apparent then that,irr the face of major sutbscks in 

the quality and quantity e>F individuals joining the .~nnc~I forces, WC 

myst continue to maintain 3n effective state of military readiness 

which is responsive to situation derunds. 

Hany tires we lu:u track of the overall importance *IF our jobs 

when WC become involved ii) the ,,-ts of everyday bt~sItx5s. tioweve r 

such large problems as th.)t mentioned .bb?\.- are not r-cnudicd by one or 

even several decisions or changes. The overall situation, in this 

case is Jffcc;ted by many ~1al1 varidblcs Jnd ‘utiun to it will 

be dependent upon many SKI I 1 ad jus cmcnt s or changes .hich will in 

turn have an overall efiee:t. 

The situaticn with which this study ~!c;~ls is one of those small 

‘4 

. . - 

119 

__. ____ . . . . . . - 

3 

. 

,. ’ -. 

-. . .. . . .

areas in which changes can be made-which will affect the ability and 

readiness of the men presently being trained by our armed forces. 

To increase the abilities of our servicemen in their various 

special ties, as well as in the general knowledge of those things which 

pertain to the mi 1 i tary forces,, is an objective which will help 

accomplis? the readiness which the public expects. This can be 

achieved even in the face of lower en1 istment rates and manpower 

cutbacks. In increasing retention and usage of information given to 

trainees we can project that this ability will enable them to operate 

at an increased level of awareness, and efficency. 

At the Coast Guard Training Center in Cape May New Jersey we often 

encourage instructors to give information to recruits in regard to per- 

formance both on the’ positive as well as negative side. This feedback 

is what most people base their evaluations of their abilities in any 

particular area on. 

In an early experiment on feedback Hut-lock (1325) emphasized 

encouragement or incentives and their effect on learning. He also 

encoura,ged the use of verbal praise or criticism and added power to the 

preference for verbal as opposed to other types of feedback. Plowman 

and Stroud (1942) showed that seeing their errors corrected on an 

examination allowed high school students to eliminate approximately 50% 

of their errors on a retest one week later. This result was probably 

due a great deal to the practice effect but does point tm :he value of 

feedback in enhancing ability. More recently Page (1958) reported 

c 

120 

-2 -.-. . ._ 

. ‘. - ‘-“ .-. - 

.,-. __ 

.

’ 

I 

‘* ; ,. 

‘. 

I I 

results which bear out Hurlock’s earlier emphasis on praise and blame. 

He found that students who received comments on examination papers did 

better on the next examination then did students 90 received no comnents. 

/j 

Ii 

Huch of the previous research in this area has been conducted 

comparing feedback with no feedback situations. 

: : 

Curtis and Wood .(1929) 

hwever’, tested four methods of scoring examinations and found that 

those methods which provided for discussion of questions were the most 

effective. Stone (1955) found that the effect of feedback was correlated 

positively with the amount of information contained in the feedback. 

Students who received full explanat ion,as to why one answer was wrong 

and another was correct, did much better subsequently than did those who 

received less, or no, information. Sassenrath and Carverick (1965) 

compared four treatment groups for retention and transfer of learning 

and concluded that the type of feedback i!s not nearly as important as 

the fact that the group%ets it. In the ‘present experiment it was 

I 

hypothesized that the introduction of weekly quizzes prior to the first 

progress test in recruit training would &crease scores on the progress 

test and this increase would vary positi Jely in accordance with the 

I 

amount of feedback given on these quiz&. 

P 

APPARATUS Quizzes consisting of twenty five questions each, base; 31‘1 

the first and second week subject matter were constructed. Questions 

covered the same subject areas as those cn the third week progress test 

but were changed to appear in a slightly different form. 

‘. 

121

All quizzes were given in well lighted classrooms under 

comfortable conditions. 

SUBJECTS and DESIGN Five hundred and sixty eight Coast Guard rrcruits 

who entered the service between 15 April and 15 August 1373 served 

as subjects in this experiment. They were assigned to companies, 

apprcxiaately fifty men each, as they reported for duty. The varying 

numbers in each co,:gany caused the number of subjects in each treatment 

group to fluctuate. As each company of men began training they were 

assigned a treatment category, 1, 2, 3, or 4. This continued unti 1 four 

companies had gone through treatment 1, four companies had treatment 

2, five companies had treatment 3, and four companies had treatment 4. 

The total number of subjects in each group was as follows. 

Treatment #1 - IL5 

Treatment $2 - 134 

Treatment k’3 - 170 

Treatment 14 - llg 

568 = N 

Subjects were therefore assigned as randomly as possible to treatment 

groups . Four instructors took turns administering the three treatments 

to help control for the effect of any particular instructor personality. 

PROCEDURE Prior to this time no quizzes were given to recruits other 

than the third and sixth w&k progress tests which consist of 50 

I 

mu!tiple choice questions on material covered in lecture, course material 

and text. Quizzes were introduced at the end of weeks one and two prior 

to the first progress test. Companies received their assigned feedback 

/ 

. 

.* 

. 

,, 

: 

’ , : I I : 

, , 

122

condit 

“first 

ion over both qu izzes and were afterwards tested using the standard 

progress test”. The treatments for each group are specified below. 

Recruits were told that quiz grades would not be a part of their record 

and were for their information only. 

TREATMENT #l 

After the quiz each man kept his paper and the answers were reviewed. 

When no questions came up in regard to an item the instructor was told to 

explain the answer anyway. Recruits were encouraged to ask questions and 

discuss each one fully. 

TREATMENT 4’2 

The companies which took part in this treatment were told which 

choice was correct in each case, but were not given any further information 

or allowed to question it. Thus they received some feedback but not the 

amount available to treatment group one. 

TREATNENT .$3 

Since we were introducing two independent variables in the feedback 

and the quiz itself, it was necessary to be able to say wether change 

was due to the feedback or merely the introduction of the quiz. It was 

for this reason that group three received only the quiz and no information 

as to how well they did on it. Instructors required that answer sheets 

be turned in upon completion and gave no information on correct answers. 

Subjects received no feedback other than how they felt they had done on 

the quit. 

TREATMENT $4 

The control groups proceeded through training as other companies had 

until this time, with no quizzes until the progress Lest at the end of 

the third week of training. 

123

ESULTS 

A Kruskal - Wallis one way analysis of variance ylelded an Hc value 

0 

significant at above the .OS level. Group comparison’s, again using the 

i’ 

Kruskal-Wallis formula, were made which indicated significant differences 

between several of the specific groups. 

!! 

INSERT FIGURE 1 HERE 

These results imply that a definite difference existed between the 

group which received maximum feedback from the quizzes and both the 

. control and the quiz - no feedback group. This difference bears out 

the hypothesis and was presumably due to the effects of feedback. The 

max feedback group ‘did not however, differ significantly from the 

mlnimim, feedback group (group I I). 

I 

In this respect the increase in 

I 

feedback did not cause a siTnificant difference from the lesser feed- 

I 

back level and.fails to bear out the finer implications of the 

hypothesis. The trend was toward an increase in feedback predicting 

I 

increase in performance, but in this case it,did not approach 

significance. 

OlSCUSSlON 

The results of this experiment bear out the hypothesis in regard 

// 

to feedback versus no feedback and seem to confirm a statement made - 

previously (Sassenrath and Garveri ck, 1965) that the type or amount of 

feedback Is not of extreme importance in these situations. The 

important thing is that there is some feedback. The data shows a 

trend, although not always at significant levels, that as the amount 

of feedback increases so does performance. 

124 

.- 

I 

1 

I 

b. 

..I 

.i, :I 

ii

Instructors noted that throughout the experiment the groups which 

received max feedback showed a good deal of interest in other areas of 

training and asked questions which concerned more than just the quiz. 

One Important factor in this case seems to be that of human interaction; 

the chance to speak with, or question, someone higher up in the chain 

of corrmand. This interaction was seen to have an effect in terms of 

hefghtened interest in many areas of military life. These companies 

received the treatments twice over a two week period and the question 

rrops up wether the difference in group one’s performance over that of 

three and four was due to the educational aspects of feedback or the 

psychological aspect of interaction with someone of higher rank on a 

semi informal basis. Both factors probably affected the results but it 

would be interesting to see wether the feedback WJS valuable because of 

its information or because-of the human interaction. 

A certain type of practice effect probably occurred also in that a 

majority of these men indicate a fear or apprehension concerning test 

taking. Allowing them to take two quizzes prior to the progress test 

prcbably helped el iminate some of the anxiety normal ly surrounding 

this test and present in group four. 

Due to the nature of the environment subjects in this experiment 

could not be assigned to treatment groups on a completely random basis. 

There is a fluctuation *n recruit education level that occurs at various 

times throughout the year,and it is possible that this fluctuation may 

have had some effect dn resui ts. However companies were assigned a 

treatment group in sequence as they entered training and this fact should 

have ramdomized the effects of the seasonal fluctuations. 

Whatever the reasons for the effects. of feedback in various situations, 

’ , 

.- _- 

. ----. . ‘. 

.-.. . 

., ” 

. -. . -.

we know that it does enhance performance and increases retention of 

information to a recognizable extent. This realization should be of 

prime importance in developing any training program and should incoroorate 

the use of human interaction in increasing a pe‘rsons interest and 

knmledge in their chosen area of service. Feedback allows individuals 

to assume their Identity as humans with individual scores, performances, 

abflities and characteristics. Feedback is the basis of how we feel 

about ourselves and we rely on it to continue shaping our ideas. The 

more of it we get, the better we are able to evaluate our abilities. 

in a society which deals in numbers, feedback is a chance to ccnserve 

our identity and enjoy our work. As such it should be at the heart of 

any training program. 

126

RESULTS 

OVERALL SIGNIFICANCE AT BETTER THAN .05 LEVEL. 

REQUIRED 7.82 OBTAINED H = 8.19 

AFTER THE CORRECT ION ” Hc = 8.21 

GROUP COHPARI SONS 

H 

I vs II LESS THAN .05 LEVEL - 2.51 

I vs Ill HIGHER THAN .Ol LEVEL- 10.15 

I vs IV HIGHER THAN .Ol LEVEL- 23.34 

* 

, 

I 

/ 

FIGURE I 

i I 

/ 

I 

I

Ii 

1. 

2. 

3. 

4. 

5. 

6. 

BIBLIOGRAPHY 

CURTIS, F.D. and WOOD, G.G., A study of the relative teaching value 

Of four common practices in correcting examination papers, School 

Review 1929, 37, 615-23 

Hurlock, E.B., An evaluation of certain incentives used in school 

work, Journal of Educational Psychology. 1925, 16, 145-49 

Page, E.B., Teacher conments and student performance; Journal of 

Educational Psychology. 1958, 49, 173-81 

Plowman, L. and Stroud, J.B:, Effects of informing pupiis of the 

consequences of their responses to objective test questions. Journal 

of Educational Research, 1942, 36, 16-20 

Sassenrath, J.M, and Garverick, C.M., Effects of differental feedback 

C*om examinations on retention and transfer. Journal of 

Educational Psychology, 1965, 56, 259-63 

Stone, G.R. The training function of examinations. Research report 

No, AFPT RC-TN-55-8, 1955. USAF Personnel training research center, 

Lack1 and Ai r Force Base. 

128

*'CwaE**: An autonated testind svstem for the evaluation 

of abilitv and/or wi I linaness to orofit fron a 

trainino situation 

David B. Yinson, Ph.?. 

1719 “‘edical Towers RIdrl, 

Ifouston Texas 770?5 

I n 1?7Q, CO?,a conouter software nrooram cnoable of 

assessi na the effect of cerebral concussion on connitive 

function uas reported. Usino renression analvsis. Coo 

coTbared baseline data with post- iniurv cerforpance data 

to calculate the arohahilitv of inpaired coanitive functicn 

associated with cerebral concussion. 

Pui ,di:.- 3r: exr?erience I?;nnd w i t h coo, a coflbutpr 

sc!tware proflr8m ?dq?F WAS de\‘elpPed to evaluate cert.?in 

cocnitive ?nd bersonaiitv functions, and from the interaction 

of these functions as specified ht; al~,orithms, tr, 

credict t h e cubiect’s- a b i l i f v ahd/or wiliinnness t o profit 

frcm a trainin bra,cral7. 

I n t h e DoO?i svsten. nerfnrmanta d a t a a r e acouired hv 

the administration C! tests to assess selective attention, 

short-term nemorv, rate nf inductive reasoninn, sneed snd 

accuracv of continuous addition, level o f asniration. abilitv 

to read, and mental attitudes. Certain bioaraohical data are 

inotit t o t h e svster:. but sex and culture are not comnuted. 

!dritten i n Fortran I V , the rWnnF software uses arnroximately 

27 K characters with two over lavs of I I I/ and eK 

resoectivelv. Ifsing a n XDS-ndn i n tine-share, nerformance 

data are innut and conoared with stored information to 

generate a renort which auantitativelv nrades a subioct on 

five factors nreviouslv identi ficd hv factor nnalvsls as 

predictive of sunarvisor ratir2s of ioh nerfornance. 

The five factors as defined in the DWflnF nrint-out 

reports are mental craso. self-confidence, cooneretiveness. 

emotional stabilitv, and leadershin. nQoF?F nradinns a r e 

from I.0 t o 5.0, reDorted to one decimal ncint. A. arndinn 

of 3.0 represents the mean of hinh nerforminr! normal sub;ects. 

PCT?F has been on-line since ln7l. ‘ran receipt of data, +o 

user’s receipt of the renort. the turn around time is less 

than ten minutes ner sub iect, nnvwhere in ‘I !rth 4merica. 

. I \ 

. 2 -- 

. 

_..*.-:7-~,/,. : * 

i 

8 . . , ! 

,. ..-. 

-. . 

. ./ 

_ . 

1 

,

-L - - 

t. -. .‘.. .) 

-- 

.’ -a- - ..:_ . 

-. 

; i 

130 

) 

* .

Althouch suoervisor’s ratinos pf work parfornancc are 

aenera I I y acknow I edged to be tr i a::*‘,! toward t5e mean. nrsdinds 

of profess iona I fcJrba I I csaches .?r-x not. \?>aches’ nrnd inns 

cf certain c?serva? I e behav i.-r aqs,-: i ated (I i t h on-the- ieb 

oerfcrmance are as shown in iilbIP I . 

T ,A I: :. i‘ I 

-c A’.’ 

- LEJ.3.i 1 >I*? ;nyr 1 pF’!rE rf- ?a,,‘.::. :: 1 L IT:’ .?cccrss 1 \‘rqr$$ 

. 

the agreement is as shown in Table ?. 

Conf i- 

dcnce 

. . 

, 

.i \ 

t; \->ness i i itv shin 

: - -

I 

Knew I edce of -31 

Product i s i ty .Jl 

P r i d e ir: X’ork -5 

Leerni r.c ?ate -7s 

Over3 I 

. ! ,? 

‘.‘e n t A 1 COnii- conncr.t- c t?! - !.cader- 

, 

. 

i 

. ,- --.. 

: ,: 

’ 

._ 

.--I- _. -*. __. . . . . . 

_. _ _ _ ._ .._ - - ._ . . . __j. .._ . ,- ^_. , 

. . . . i; 

h 

Q’,’ 

\ 

\ 

,’ ,/. 

. - 

.’ \ 

‘.. .). 

‘guyJ’J 

I 

: , 

‘S. 

: ‘. 

: I

. 

STUD\’ ?: Performance dat,l were acouirct from 855 emnlcrvee 

candidates of U.S.Steel (Texas \\lt)rks). nn each s u b iect, a n 

earlv model af PROF?F: w a s nroces~:cd. These r?* and co-nuted data 

.n.?re forwarccd .to t h e !‘enartrcnt f S t a t i s t its, Texac- 4 P. ‘1 

L’niversitv. nn t h e same suhiects. t’.S.Stcel provide;. io5 a n d 

training data; :l.S.SfPtxl asset-e(! Texas A R ” “niversitv &hat 

e~lriovce can: i dates were ne i t her hired ncr not-hired cn t h e 

-asis o f =?;\?:E renor:s. T h e :WC +ets o f data (PCPPF a n d 

1 .S.Steel’) itere kenf entlrelv t.c‘n.jra+c u n t i I ~bev reached Texas 

-4 ;, ” IJni\ersitv f o r ,tn.?l\,si-,. II.S.SteP1 da’? consisted of 

f C’ V t’ F .3 1 rat I ncs nerf?r!-e.! b,J qurt\r-v i sors, ‘? r .- certain data-nfrcc,-rd 

o n birs?/not Cir;r, ?t.bl? 6. 

r ,7 i: I. F 6 

. c:‘P.I f,TS c-i: 19F r:YF~: “TC II.S.CTF~L n4?-l?!r.S Or: 

: I I 

;I I 

“enta I Cf. I!:? 

:,‘onfcrmi tv 

133 

b!?rL ?erform,?nce. 

C.cnr c s?+ctv rules. 

Fs70r-?ra? i vcness P. confnrmi tv. 

fnotl-nai stabi I i f v . 

Auth-r itarinn leadcrshir. 

I ndustr i.a I enainecrina ratinn. 

Tines chanced denartmenir. 

Clus*er nlacement. 

ncce-~ L ancc overt ire. 

.I\ccrntanrc assinnnents. 

11nre31isticatIv hirlh, oonr c l u s t e r 

-lacemcnt. 

Ilnre?l istical Iv hirlh. m o r n 

Ihscntceisn. 

tlnrP?!i5?ical Iv hioh, n(Yrcr wof c 

>erfnrnance. 

!1nr0a ! isticnllv hioh, poerer 

?ccentance assianments. 

IearninG. 

Hirt*:/L’r)t 1’ i red. 

Fmo+ihnal sta5ilitv. 

t .clor er.3 .+ivrness i conformitv. 

4bsenteeism.

Clerical-technical Al I narsmeters 

er 

er 

et 

cat-technical 

Cal-technical 

Cal-technical 

Cl,~rical-technical 

Production, Latin 

Pr-ducticn, Latin 




Production, White 

Production, !chite 

Production, b!hite 

Production, !“hite 

Production, :Jeoro 

Product ion, t!eOro 

\lental *Irasn 

\ 

Conforri tv 

Confidence 

COnneti*ivcness 

Coonerat i vcness 

A I I naramnpter-s 

?tenta I ?-rasn 

CoODera+i veness 

COnf idcnce 

“enta I rrasn 

1 3 4 

Hired/tJot Hired. 

Non-authoritarian leader-shin. 

Doorer PgCIRE. mOre likelv hired. 

(men Onlv). 

Safetv violations 

Safetv violations. 

Authoritarian leadershin. 

boorpr “=‘QnF. better sunervisor 

ratinns Of emotional stabi I itv’. 

r’nreo I is+ icallv hinh, nonrer 

sunervic-Or ratinns O f 

emotional stabi I itv. 

Poorer ‘3AnF, f-et ter superv i sor 

ratinns Of anOtiOnal stabi I i+v. 

Fmotinnal str;?i I itv. 

Authoritarian Ieadershir. 

T i m e s ck?nOec! deoartnents. 

fmotional sta5i I itv. 

fInreal istical I v hinh, rnor clu+ter 

nlacement. 

L’nroalisticallv hinh, nOre 

absenteeism. 

L!nreal istical Iv ninh. noor work 

rerfornance. 

C!nreal istical Iv hioh. more safntv 

violetinns. 

i’nreal is+icai I v hind, ?oOr 

acceptance assianments. 

Emotional stani I itv. 

Eetter c=?nnF , more acts of 

rhvs i ca I annrnss ion. 

Accontance overtime. 

,

_ 

-StJ9J - ECTS F’QqPF 

QQFI?ICT’ I1.q.S. QATIWS Or: 

Production, Neqro Venta I Grasp, .A5srnteei{m. 

Setter QQAQF. 

1 

i 

I’ 

I 

m o r e nrievances filed. 

Production, Negro Cooperativeness 3etter o?PRF, ooorer cluster 

blacemen*. 

product ion, Yeor Conform i tvl! Cluster nlacelent. 

Froduc? ion, S!eoro Connetitiveness tafetv viOliz+iCns. 

Drcduction, ‘dear0 Confidence t’nresl istical I v hioh, moorer 

cluster rlacement. 

(In Table 6, all nredictions are in the exoectec direction unless 

otherwise noted.) 

$ u ‘.: ‘.! A Q y -. 9lthough oerformance d a t a f o r oor3E nav b e acnuired 

by non-brofessionals, bv the use of telecommunication nets, 

computer pardware a n d commuter softwares, nrnfessione! iudnment 

and experience are used in evaluatina those data. 

PP3F’E is independent of the 

avai labi I itv and cost. geoaraohv, rerson3l 

and Q?QEE is designed to conserve tine. 

the examiner, 

I 

There is no oractical linit to the nur!Per of net-sons who 

can be evaluated Sv QQ?SE in a fast. reliable, valid and costeffective 

wav, and t h i s evaluaticn nredic+. =hilitv and/or 

willingness to erofit fron a trairinrl situ-s,t;on . It is felt 

DSCSE cou Id Se of sianificant usefulness ir the evaluation of 

annlicants for militarv dutv anc’?or for 

on militarv service. 

t+. eva I uat,ion of those 

E!v u s e o f cregression analvsis. 

1 

base-line 

i 

berfornance data 

of QQO?E can be connared with anv subseacen? retest nerfarmance 

data, and the orcbabilities of chanae can ?e calculated. “QOPF 

nav also be used in serial studies to det rrine rever5iSilitv 

c f batholoaic chsnaes associated with arli/?c, P disease or trauma. -’ 

* - 

. 

135 

‘. . --- 

, .__ 

. . -. 

. -_. 

. . 

‘.

I . 

2. 

3. 

0. 

1335 

1943 

1948 

1963 

9. I96 I 

19. 1963 

I ’. . 

I?64 

., i 

t: 

Stroop. J.R.: Interference in serial verbal 

reactions. .I. FX”. Psvcho I . 

Lewis:, K., Dcnbo, T . , ‘e5.t inq?r, L., t 

Sears. P.: Level Of ?sn;ration, I ?I: 

Thurstone. I. . !. . 9 ihurctc?e. T.C.! T2.y 

Prinarv ‘.+r>nta! Ahi i it;ec. In+orf?rdi.?tP 

Form A’!. (:hicaoo: ‘cience '?eSe?rch 

Associates. 

Vinson, “‘.F.: nL ‘tpct iv i tv in t!lr? 25SPSSmerit 

O f the A”vrO+nxic patiPn+. .I. 

Fsvc~05oratic 3ese2rch 

Vinscn. r.a. 5 Do>hins, I..?.: ?biecTivP 

psvc!to lo= ic 3ssescment c.f the tkrotoxic 

natient 3nd tbc res0cnse lo trc.?tmPnt. 

J . Cl. Fndocrinolqqv rind “etcbcl i5f7 

2 

V i n s o n , P.D. D. cai i7, C.“. : n5 iective 

neasz.remt~nt O f nsvc?c?iololic decline. 

rr-0:ced i nOs. 5 t h Int. Canqrpss ilerontc Io\ov 

V i nson. ?.D. : flbiectivitv i n t h e ~T~PGSment 

Of psvc+ohioloai: decline. V i tn Llumana 

Vinfon, :J.f\. . Ecnrnes. T.7.‘. ;J*~s5el I , U.F.: 

The: ef fct? o! 31 t--r-.< cnnsc iousni?Cs nn 

inf’orm?tiOn rrocessicn. Sc5eI lenner ces. 

Labs. 8 

! 

. . . 

. _ 2. .-. -. 

t 

. . .: 

./’ 

. . 

136 

.” . . _. c 

. : ,. . - 

- : 

..__ ‘._ % . ,..., . . . . .: >* ..^..... -.. . . - 

/

12. I967 

13. 1967 

1:. I 06-i 

17. ic):i 

19. 

I”. 

3n. 

,‘I . 

In72 

1072 

IQ73 

I”73 

Vinson. P.B.: Information nrocessina in 

the human suhsvste~. Instrument Sot. of 

America (Aoo: lo stlct ion) 

Vinson, ft.R.: lnfcrmation nrocessino in 

man-machine 5vstems. J. Pssn. Adv. L’edical 

Instrumentation 

Vinscn, !?.P.: L studv @J the effect of 

time o n mental eff iciencv and emotional 

stabi I it\:. .-I c. . ~sL.i’ko I . Assn. Annua I 

::ee :tn,y, * Piv. 2?, ‘:‘3sh i nctcn DC 

?‘inson, r-e.: @erfornance testinn a s i t 

relates to iniurv. ?.!at’ I . Onsearch Sounc i I, 

Division on ‘-‘edicaI Sciences 

Vinson, 2-l.: Comnuter evaluation 

rel ative to iniuries. l?th Uat’ I . Conf., 

‘.‘edical P.soect; oc cnorts, Poston 

Vi nson, D = Constant, :.A., ouackenbush, 

J . ? . 8 -CA;;, .I.“.: C:utomated ncnta I 

Status examination: bre a n d post drua 

effects. Crus lnformaticn kssn., Computer 

Ycrkshob, Ch i taco 

Vinson, 0.6.: Automated assessment of 

chances i n CICman nerformance. r,rand 

“cucds. Savior I’niversitv Cal leac o f 

*t etiicine, Januarv ?I. 

Vinson, I1.P.: Telecommunications and the 

automated men +a1 status exjmination. 

r!a:’ I . Telecommunications Conference. 

LJouston. 

Vi nson. 9.D. : Con conntiter oronrammi na 

nredict nest-operative return to function? 

Facultv, “?/hat’s ?‘PW i n orthonaedics,” 

hn . Acad. orthonnedic Surneons, S a n A.ntonio 

V i n s a n , ?.P. P. !-Jarner, f-9.: DfJlJ,TFe a 

multi- lannuace automated osvchiatric 

screeninn tool. Civi I Aviation “cdical 

4ssn.. cuada’l a iara. ‘.Tex ice 

137

ASSESSMENT SYSTEMS, I CCORPCSATE2 

“PRO?? 

OCTOFER 12 1973 

ASI- 

I I 

FlGlJRF I /’ . 

nEsTaL GRASP: 3.1 TP;E Pr ILITY TC LE.ARN F?cl?: 4 SYWFCLIC 

f!F A P9OCESS 01 PROCEZUQE. ‘,;F PTILITY T C UE:3C?ST4N3 

PEPRESEb’TCTION 

l+C!‘d P SUP-T;VSTFM 

SELATES T O F TCTGL SYSTE,“,. .- A’::ITY TC THIKX C F bLTE?fdATE SOLUTICNS 

T@ ??C”LEXs. 

SELF CONFIDECCZ: 3.4 :t’, IKSIV:ZUCL’S FEELIKG OR EXPECTaTIOb TPFT t’E 

‘*ILL ?l4STER A PRCCE2URE &KS/c77 ?LnFP.?h’ PS SlJCCE:?Fl!LLY I N FLiTUFE 

AS HE llbS IN TPE PPST. 

COOPEPPTIVENESS: 7.0 1ZEt:TIFIE:: ‘n’lTp AUTt’JCEITY. T4KES DISECTION FROf? 

yy0S~ 1% AUJ’P3RITY. -IDECTIFIE? ‘,‘::!’ CC?PC!?PTE GOPLS. SUtOFDI RATES 

“-TS$::AL * t 

S(?ALS TO CCRPCSCTE ;;C.tL’;. 

, 

_. , .’ 

133 

- . 

_ _.__ -. - _. “. 

1 

I 

I 

-._ . . -. 

. . 

: _ : 

.. 

I

Flr,LJRE ? 

Assessment Systems, Inc. - 

LEADERSHIP 

EMOTIONAL 

STABILITY 

‘\ \ . . 

MENTAL GRASP 

(c) H.C. VINSON. 1970 

139 

2 

2...*’ 

. . -.- 

CNO: 

DATE: 

ASI-1734567RQ 

FORMAT: “PROBE” 

f-)ctoher 17 to 

SELF CONFIDENCE 

COOPERATIVENESS 

. . 

. 

. 

. . 

‘... 

..-

ASI -1?34567E(3 

,’ 

. 

I 

140 

.- 

.: . 

.-... 

-, I _. . 

. 

. . __ .., . . . 

.A

‘.. ‘, ) :’ 

,/.,‘.’ : * 

,’ i / ’ ; 

INTRODUCTION 

Prospective Chief Petty Officer' 

Leadership/Management Seminar 

Commander'G. C. Hinson 


The need for leadership in the military services is axiomatic. At all levels, 

managers--those with a responsibility to get a job done--must be able to direct 

other men to want to do a job well. Recently, within the Coast Guard, there 

has been increased emphasis placed on the leadership abilities required of 

E-7 through E-9 Chief Petty Officers. Late in 1972, the Commandant tasked 

the Training Center at Governors Island, N. Y. with defining the elements of 

leadership a Chief Petty Officer must possess, and preparing a "nuts and 

bolts" course to teach these prospect':e leaders this managerial skill. This 

paper outlines the procedures that were used in development of such a course. 

BACKGROUND 

Whit is the essence of leadership? Can it be taught in a classroom? Can 

this be taught at all? Can it be taught to.everyone? Can men be transformed 

into leaders? A Questionnaire sent to E-9 Chief Petty Officers revealed a 

leader to be intelligent, a specialist in .his rate, imaginative, dependable, 

capable of making quick decisions, espe'.iilly under pressure, and additionally 

one who inspires his men. Could we develop a course to produce such a 

person? Is this ttle man who could get the job done? Research in this field 

showed that many leadership courses are available. Most courses are designed 

to shape men into what sorfeone thinks leaders ought to be. In general many 

efforts have been disappointing and a waste;of time and monqv. Even extensive 

training at military academies has failed on many occasions to transform 

cadets into effective leaders. The reasons for our apparent failures should 

be explored, are our methods of instruction? the students or a combination of 

these two factors to blame; or is somethingjelsc at fault? What should be 

changed to improve leadership training? rla'y studies show that the most 

important dimension of a leader is his rela 1 ionship to his subordinates. He 

will be most effective when his men respect! and trust'him. But respect and 

trust are built upon something deeper than/personality and friendship. This 

made a lot of sense when we re-read the chd racteristics cited on the CPO 

questionnaire: intelligent, specialist in rate, imaginative, dependable, 

capable of making a quick decision, espec'ally under pressure. It is clear 

that respect and trust must be resultants/of i performance oriented character- 

istics. Fred E. Fiedler has conducted extensive research on leadership at 

the University of Illinois. His studies show that it makes little sense to 

speak of a g,od or poor leader. There are only leaders who perform well in 

one situation but not in another. In these situations, he differentiates 

between task-motivated and relationship-motivated leaders. Those primarily 

141

Ii 

interested in building personal relationships with their fellow workers are 

said to be relationship oriented; those interested primarily in getting the 

job done are considered task motivated. Fiedler also asserts that a taskmotivated 

leader will have a greater probability of success than a relationship 

motivated leader. Based upon our analysis of the literature available, 

a course in leadership should stress (1) the psychology of motivation and 

the satisfaction of doing a job well, (2) analysis and control of leadership 

situations, and (3) building a relationship of mutual respect between the 

l.eader and members of the group. This was the basis for the Coast Guard 

Leadership Course and our challenge was to develop realistic rather than 

idealistic leaders. ' 

HISTORY OF DEVELOPIi;S THE COI;RSE 

E-7 through E-9.Chief Petty Officers are Coast Guard front-line managers. 

They most frequently fill middle management roles serving to clear the way 

for command actions and letting the command know how the cre\/ feels. But 

on the other hand, Chiefs are assigned smaller units as Officer-in-Charge 

and ir'. Executive Officer. These possible assignments require having both 

a narrow specialized knowledge in a technical field and broad knowledge of 

the service in general. They also must be able to communicate vertically 

and horizontally within the command. Our leadership course was tailored then 

to meet these needs. We sought also to provide prospective Chiefs with a 

basis for decision making. A leader has three methods of exercising his skill: 

he can be authoritarian, democratic or depend upon his men to exercise responsibility 

and good judgment to get things done. Although the good leader 

must use all three techniques to effectively operate, current literature 

promotes democratic leadership or group dynamics as the preferable choice. 

This conflict results in confusion. To quote the Harvard Business ReVitW: 

"often he [the leader] is not quite sure how to behave; there are 

times when he is tom between exerting 'strong' leadership and 

'permissive' leadership. Sometimes nc:$ knowledge pushes him in 

one direction ("I should really get the group to help make this 

decision"), but at the same time his experience pushes him in another 

direction ("I really understand the problem better than the 

group and therefore I should make the decision"). He is net sure 

when a group decision is really appropriate or when holding a staff 

meeting serves merely as a device for avoiding his own decision 

making responsibility." 

We felt a Chief's decision making abilitv wculd be proportional to the information 

he had to draw upon. Consequently, we stress the forces operating 

in the decision making process. 

ii] fiscal restraint.. 

responsibilit/ ana .* +.hority 

(3) legai boundries i' 

132

(4) military.- considerations 

(a) chain of comnand 

(b) personnel evaluation 

(c) discipline 

The successful leader is aware of all relevar ,t forces and can put them into 

perspective when making a decision. He is flexible enough to not fear change, 

when change may be indicated. Therefore, the Coast Guard Leadership Course 

emphasizes analysing situations, knowing available options, and selection 

of efficient strategy. Our course: 

SUMMARY 

1. Supplies information (or indicates where information can be found) 

concerning relevant f?t-ces. 

2. Teaches analysis techniques‘to enable one to know self, subordinates, 

superiors, and peers. 

3. Stresses the selection of the most appropriate 

ship techniques, and 

4. Provides the opportunity to develop communicat 

and producti .e leader- 

ion skills. 

Our goal at the Training Center is to produce a realistic leader; one who 

is well grounded in Coast Guard programs, precedures, policies, and problems; 

one who knows his importance in the management of resources; one who is 

sensitive to the needs of his subordinates and his superiors. We feel that 

our four week Leadership Course can achieve these goals. 

APPENDIX 

Outline of Curriculum 

143

II 

Enclosure (3.) to CO TRACRN Ltr 1510 dtd 

PROPOSED PBIJSPECTNE GRIEF PETTY OFFICER LRADERSRIP/.WAGEMRNT SEMINAR 

c 

Monday 

a. IntrJductfon to the training center and check in. 

b. erecting by Commanding Officer, Executive Officer, Senior Chaplain, 

and the ?rcsidcnt of Governors Island Chief Petty Officers Association. 

c. Rxpl;lnation of the seminar's czcthods and goals - emphasizing skill, 

not theory. 

d. Admix?ster the Leadership Opinion Queztionaire 

e. Prest%jt historical leadership studies. 

Tuesday 

a. ln dere-.h course orientation; How to accoaplish the job 

b. The puychalogy of motivation including perception, role identification, 

communicatfons and role playing. 

c. Achievement motivation including security vs. challenge quiz and 

review (quizzes are personality preference Ueaeures). 

. Uedncsday 

a. Contir\tat.ion of achievement motivation including .Xa~hisvellianis~ 

and surmnaty. 

b. Leadcr.ship styles; Fiedler Development. 

c. Leadet.ship Matrix and review Leadership Opinion Questionaire questions. 

Thursday 

a. 

to 

b. 

C. 

Friday 

a. 

b. 

C. 

Situs,ion evaluation and solution choosing - obtaining facts prior 

decision making. 

Retur. LOQ and discuss the results. 

Leadership games, including %urvivaJ on the Moon”. 

1 

The impnrtance of communicating, encouraging suggestions. 

Retake and review of LOQ. 

Review and discuss pain research. 

-SECOND UEER- 

Monday 

a. The Coast Gusrd,'including the movie "The Eighth Mission", Coast 

Guard Organization alId the Chain of Command. 

b. Career Counselling which includes methods of counsellfng, self 

understanding, human understanding, evaluation of.people, basic sales- 

Eanship* effective listening, counselling problems, interviewing and 

specific areas of CC personnel policies (rotation, etc), P.A. assistance, 

social security, insurance survivors benefits and annuities, personal 

financial rxlnagement, 

current job market. 

civilian careers both after retirentnt and the 

144

Tuesday 

a. Drug abusr including types of drug6 in use and the reasons 

people turn to drugs. 

b. Continuation of career counselling. 

Wednesday 

a. Drug abuse continued; alcohol, and the Drug. Exemption Pxogrcn. 

. b- Dfscipline and grievances. 

c. The Uniform Code of Military Justice. 

Thursday 

a. Human Relations 

b. Body Language 

c. Pundxnentals of recruiting 

Friday 

a. Hunan Relations 

b. Public speaking 

c. Public informatioa 

THIRD WEEK 

- - 

Nonday a. Human relations 

b. Budget process including Sub-Head 30, planned projects and procuremeat, 

cost consciousness 

c. Allowance lists : 

d. Procurement, including advance planning, work lists vs. procurement 

lead tFue. 

Tuesday 

a. Human relations 

b. The Fersonnel Xauual (CG-207) 

Wednesday 

a. Human relations 

b. Tftlc "B" property 

c. Personnel records 

d. Health records 

Thursday 

a. Pay md Allowances - pay offLce proceedures 

b. Tr3vel - household effects 

c. Leave, leave rations, computation, types 

d. CMXPUS benefits, elegibility, orthodontics 

e. Efedfcal 

f. Rctixment 

145 

.

Friday 

a. Educational opportunities, including se F ice Ischools, ' 021, USAF1 

and off duty education. 

/ 

b. Advan&ments, including career patterns, servicewide exams.' 

C. ' uniform regulations. 

* ..' : ,, / .1 

d. training procedures. 

-FOURTH -WEEK 

' 

, 

r 

Monday I 

a. Organization, delegation and planning of relatkvely complex 

tasks, including long range goals and intermediaty objectives and 

the need for follow up. 

b. Clerical, including effective writing, correspondence manual. 

Tuesday 

a. Forms. reports and publicitions. 

,b . Classified material, handling stowage, access and the need to know. 

Wednesday 

a. Records keeping 

b. Rules of leadership. 

Thursday 

a. CPO responsibility up and down the line: a general summation 

of what a CPO is, ranging from accident prevention through experience 

to rumor control by not participating in speculation; a chief's 

presence in the work area not the CPO mess; supporting the service 

and superiors and not adding to gripe sessions with junior enlisteds. 

Friday 

a. check out 

b. Graduation 

and Ethics". 

- CD or SO address class - theme "Personal Integrity 

I 

ADDITIO&U TR%;\fS& ASDESIREEb(NTCHTS) 

1. Self Taught Reading Xmmrovement 

I 

I 

2. Self Taught Xathematics Improvencnt 

3. S&f Taught Spelling Improvement 

I 

TEXTATIVE SEXIS.AR TEXTS 

1. 'What Every Supervisor Should Know", Lester R. Bittle, Second 

Edition, 1969, S&raw-Hill Inc. 

2. 

3. 

Naval Leadership, US Naval Institute i 

"Right Down the Line", Charles A. Pearce, Ed, 1955, Arrow Books, Inc. 

4. Various articles on motivation, achievement and leadership. 

Q 

146

. 

9. r‘. Zaldkoetter 

Sxsc1t~n:. 3ducntizsal p1 Persame System 

Indiazspclis, Indiaca 

Zte aide exmmc tkait is zrceived whez we beg+= to tfrl,r.k of ways to 

>rcr2fe 5f.e best UBPS Jf f.mn= resources lzxludes 63 ruch for attentim, we 

zay te disturbe:! by tbc c.zcle::ity of this challenr;e. T'ho tuxan subject, 

rrker sr friend is nzt easily directed ir, :hc zest Fredictable charnels 

l;itLxt occasi:r.a1~:' irterferin:: with our pz-:gaaed @OHS by tice, territsrial 

x value diffcrczces. In bchaviljr31 ..A. --ii*icatix, -_- trninicg, and 

zther c:c.:iti~cizr ., csrc-i . - -cc-- -_-I ti ( a rath is s:c=ht which CBE ~ivc desirable 

sthi231 ar.d er-ducti*:c >osu!.tr; zi.tb.xt zhc zaj3rity :f pors:ns involved 

resi,stir,g the .A.rc;ti~r. ir. zhici't rtzay;ed 3r coopelled to f.:Llxl. 

;‘vcr the pars i:; tke CE~C~V~X addressed under the quest for adequate 

Dcrsxncl evaluatim tcc:;A.quss, tkere has been such cznccrn nbxt evaluatisr. 

~?.icfi car. reasin.7bl:~ 3Gsxrc tht a perssn cith acceptable ski.11 Is 

selected :zr r?.ntcvcr CT?: rtunitcs fxnd a-ailnble. ?niS &JOn~ ha13 n3t 

ditilnis!te~ as frc.7ter an2 pester effort bas been sade tc, cot at skill, 

ability, interest an5 ~c2smality factors-n:hich are ‘celieved to be fundaynta1 

sources SI xtrintxe a.C!'ccti3,C; beha-rizr. As a pssiti0n guiding neasurer-ect 

~esi..7z., a crcerint d-.l‘:t is i**equeztly dimissad ahcnever the 

pr3tlan >,I rt:tirati:r. is cxed or tttc “sense of tinicg” evarencss shorn, 

rhic:? 5ztt see= t.3 i~~lae~ce 5rw s’killcJ betavior Is brought t:, bear in ar.7 

nrctlezt situ3tix. : 

Xth scze ncEunrti?s abxtt ti%t hve !xen teneratin~ forces for 

-ers-v-e1 ~ -...* oval-2tl2n - zasuris~ icdiuidna1s prinaril;' with paper-porkcil 

tests ir. aci:icsia; s.xc txe 7f criterix be:?avior -- there ramins also 

a prizici* qwr:isn rc&l2C; ttc deslro and y?rsistence nodded to zaintain 

beha-:i.or ZCtc? tk? recxd ?C ackievezezt and aptltudc is indo:ccd. 

Yhettter ~2 view pcrs:ns zeasurc5, ?;ettlnl; ready to be noasurod, or at a 

specified ir.terval after zeasarcxnt, a constraint is still Dresont that 

zakes one feel xxctsy t3 tte de:.-ee that a strmg level 31 continuing 

nstivntizn io as! f:rcsccablc mlcss very M.gh qualifying exwdards are 

603c?:17W mplic 5. ‘.?lat ~~~icc Is ttore 2x preparing ladcncndont bohavisr 

:hat till di?c:t’ly Cr’ntribUtC t> zcater efCiciency aad eventual proixctitit-.’ 

7ite17*Jt . - 

C 3!2:13?td? 

e -m’-eiw- - -: ..I’ u;:‘n y~ssivc and coercive nettsds and authcritdmn 

147

partlcularl:I eco-centered ~Jals. Under favorable conditions individuals 

will espend thzir Lntclloctual and ecotisnai r‘cs5urccs to attain specifled 

gDal6 lrithout being &wed by harsh instructi and built In wnlshzent 

should the:: not coz??:; to set Schcduks. Again, if :rm will’susynd 

critical anal::sis fnr a zmcnt, I as.4 that you give your attention to t!!o 

concept that the wst CJnVinCinG teaching and experience results from your 

havi2,r pereitted :-xrself to be 63 enmossed in a practice task, :.-our sense 

of anareness is nl5r;t nhJ:l:; mided 0:; the.culnisatisn 3f the action 

bein,-’ ncrf Jrx:i . Y.Ju s2ck -ml; to deterzinc the depee of cxccuti2n by 

vhlch & zi;:-.t schievc n finer s:?23c ?f ncrf:mance and in site m:f 

aILsi :y.xrself tz eva:~t~te the rcsu?t.s in term .;r' s?.ze L‘thar instructive 

cspericnce. ‘ihen tf;c activity d-CS 7t SatiSf; ;-J,u :?ith the anticipated ' 

re6ults 5or.e c:rrCcti';e fccdtnck is USUa:::’ ??D?icr! t.3 cbtctin adjusted 

future res~:r.scs t.7iar5 the pn3. Eel+-cted. Selaz [lid critical ir.rir;ht 

xcur where .ac c':u?d osccrtnin tkzt ra:: cmrgy cat-3 r.3t tte nrir-ary 

33tivntin,: fC7rc?, but :-a',':lcr lith :te prnmr . nr^7uzt 3f er.er.blce-nolvinu

shift the level of consciousness to let go of value-laden thought. 

Although not qulto In the 6838 way as hypnosis does, ?H produces with the 

nantra applied, an almost Immediate change in the mental and physiologIcal 

functionIn& (?fallaco, 1970). , 

! 

This short dofinitlm Is practical1 y all that need be stated about 

the Identity Jf ‘l?! as a technique. The agreetent on Its potential effect 

in personnel avaluation is yet t3 be aclcnorledged. 30 let us exercise 

the collective IanglnatIon to,the point that we may odtit thero at-o many 

contingencies which vi11 become evident in future developments in personnel 

evaPJatIon, testing and personnel system analysis. Tilding the crest of the 

wave with statl6tlcal applicatisns and computerized innovations, testable 

b&avIor has been nainly explaloed as s dependent variable In response to 

tasks set before tho hunan subject. Rchavlor not directly percoptlble 

through the nearly always observable channels of peychonotor evaluation Is 

routinely dlsmlssed as not in the te-rltorial interest of the practicing 

behavioral sclontlct. Few mdels oi hmaa behavior should be propxted 

to account for altored states of consciousness and connected behavior. 

How nay these Stat06 tend to produce systematic varlatlons in behavior when 

all pOsSibh characteristics In selected individuals appear to bo Blnllar? 

Xhat Related Ideas and finctions of ‘37 Can 3e Sug,ce.sted? 

To a certain oxtent I believe NC Clelland (1973) psas speaking of 

something bordorinl: on a change In a theory of testing when he 6pecIfIed 

"tests should involvo operant as ee2.l as respondent behavior," and "te6ts 

should sample operant thought pattern6 to get naxlnua generalIzabLlity to 

various action outcome6."w Re goes on to say that “the. tester of tho future 

Is likely to Get farther in finding genera,l.Izable competencles of character 

lstlcs across llfo sutcones If he starts by focusing on thought patterns 

rather than by tryin,: to Infer what thoughts must be behind the clusters of 

action that come out In various factors in\ the traditional trait analysis.” 

Yet tic Clelland has only seen fit to apF?O'obCh the edges of the testing 

dolaIn, and bala.&cas precarl~us1:~ to not ihply further that ne rzl;ht well 

.Study those p8ychdc variables mhIc:1 IrngactjLn the levels of cm6cIou6ness 

and thus affect the thought patterns. 

To get behind thought patterns 

/ 

for testing pUrpo608 Will avo Interested 

sclentlsts no end s? culzzlcal moments -&en the:: attenpt to learn about 

the uconditlons 05 knowInS” that many of ,us were taught to be forbidden 

territory. ?oday obscrvln; human enerC::$research trends In Sidphgoics 

and such lnter-llscipllnary flefds, If thk ps::choph::sIclst does net emerge 

to work nith the innwotlve experimental clinlcan the future of huzan 

understanding and rseaouzenent could well be recove4 froa the hands of 

conservative theorists and Rractlticncrs. Row could this ever happen 

experienced pspch~logists will say attempting to think c.f flaws in the 

1sgi.c of this pawr. Very ueli., then, I ~111 hazard several predictive 

views under uhlch, the TtT influence cf defined level6 0: csnscieusnoss 

might g-lvc dlffcring lnsl5hts in relatisn to how certain varletloe of measured 

behavix- czuld be interpreted. The-, II these predictive vicns do 

149

- 

not suggest t3 you ieasibic stud:: probleas, I m nistakenly fo2loHng 

heretical research dcvel.:yzeats, and ?crsonnel evaluation should bo 

declared oni:: a field for refinlzg proven p s*.chnetrl.c.- nethodoloc:. 

ViewIn,- the 1?.: t.ccki;uc 2s a suppwtive treatcent, let us visuailze 

a situnticn rkere the measured out:ut 3f an i r.Y.vidual halo becorc erratic 

bqd this ~erson.scens to be chrzicaliy fatigued. L7nleGG there is some 

p!ysiolo&ical da:al;o, h e sh.-.uld rc?i;n%a :.y:: 136s af applied sk 1: whcro 

zcr=al 1~8s is nst eqectcd to ncconyan:: a hea-.--- dcgqdatlzn cf shill. 3; 

i:Sl.Cg TiI ,:;‘aIlace (1372) :?as ‘:rith aWaxeS zedfzators, aLtin& as their owfi 

cxltr2ls, EkX7ed c!xnps ir. ‘T:n (zJre fr3nta: aZ.Sm ar,d theta zavcfz), a n d 

decreased i.:

his own hypnotic condition and complement hls further awareness by the 

practice of TM. A heightened sense oi huslaa understanding l.6 hypothetically 

in re:ch and the behavioral scientiet would be ready to analyze aspects 

of con&clousness all but left to loagiinatlon and theological epeculatlon. 

This exploration of inner space, a6 rith outer space, will not appeal to 

everyone and to some conjures up threats to sanity and human integrity. 

'TO motivate selected individuals to undergo experiences which to date have 

generally proven enrlchlng should add excitement to learning while 

nexlmlzlng performance and creativity. 

'i'he person xho risks his own +ell being and those material thing6 

acquired in the fully consclou6 realo of “getting and doing” will likely 

reconsider goals and values when exposed to dlf?erent levels of c~neclousne68 

in which spontanecus trial and error behavior are minimized. No 

tendency to coerce learning and task behavior should be tolerated, but #hen 

Selected lndlviduals master the experience o? altering their conscious 

levels o? awareness, they will COEtribUte to greater creativity and COmposure 

in their living and aorklng environcents. ?3oth political and rellglous 

mcvenents have long profited fPC= having dedicated IndlvlduaIs and A 

structured soth3d f>r preparing thegselven to face challenges as potential 

opportunities. In effect to know ahat ConSCiOUS variables In alterod states 

of awareness oren a su'bject to greater insight for accelerating learned 

behavior, I believe, till reveal human resources only seldom tapped and 

hardly ever articulated a6 hopeful attributes for personnel evaluation. 

Jill you accept the existence of such a tochnlque as 3-f and does It 

threaten or serve as an optlnlstlc note in searching for well-integrated 

neasures to predict the ?Urther cxplorstlon of promising human resources? 

Can We A??Trd to Deny Consclouszcss In ?ersonnel 3aluatlon? 

‘ilhenever a prajacted diiscusslon of something like "states of consclousness" 

Is put lnt3 written words, the fear that much of what is said till 

seem beyond ratlonal csnprehension beglns to be very constraining. Cn the 

other hand the scientific endeavor to plow a new section of thought is even 

nore prOvoking. Xhether ny discourse on 'I?! will pique your interest or 

dlsdaLn, really should not matter. The bnslc purpose is to Ict a newer 

theoretical and experimental alternative be expressed 60 that personnel 

evaluation does n3t become toa self satisfied with fundamental principles 

Of human performance and the OriEln Of measurable qualities. 

Surely, operant behavior can be interpreted as re6ultlng from the type 

of environment the individual gains through sensation and genetic inheritancd. 

!?owever, that cnvironmenta I awareness Is able to be altered by 

variables which are not immediately perceptible. Zverythlng from the electromagnetic 

fields and atc-spheric changes to psychic interference by adverse 

energy patterns developed betKeen individuals seem to give us pause to 

wonder what does cause behavior we tr:: to evaluate. In a recent publication 

Roger6 (1973) made an cnllghtenlng review of sO%e new challenges in psychology 

and pointedly questioned “Is tMs the only reality.” You need not 

151

. 

agree with his own brand of cUn.lcal nethodology, yet I ieel great 

significance must be put in his serious query about needing to risk 

the investigation of parazornal phenorzena. 1 

I 

An approach as seen through the channe 1 of 34 can help open the 

dark door so prohibited by conservative investigator's rhon inquiring 

about the other reality. ?lhere hunan belhge in all walks of life have 

ca.relessly sought A finer awareness with druEs and other Illusory substances, 

that behavior has led tJ zany uzlxtuate incidents. At least 

, two Army general officers have a+~ xnted voluntnrll~: lctrsduced presentation 

of 'I?!.? for interested personnel, but oz'::: after being initiated into 

'I?i practice thecselves and ex?cricnclng pors.Jr,nll:; favorable results 

(Xastnan, 1973,). 

E:e;? hcrizms are ss)ing to be atta'ncd nrj uses ,>f T!4 and other statos 

of cohscio*Jsncos are delved ir.t:l t.: discover the mxr‘ceo and functlorJng 

of our thought patterhs. If TP chcmntcr new e~perionces that are eelf 

actualizing fx us and lead t-7 firmer !xwn~~v vit.b.ln and be::md our om 

de.goes of awareness, we can be z~crc cor.fldcr.t in understanding Individual 

behavior me a.re nssessinc and hxr tuzxm rctxurccs should be properly used 

in diversiried x-pnizati:nal c:i.zztes. ?!:cre wl?l be a constant urge to 

study the cxzplete huzar. s::ster in the iz?erncti:n with cnvirsnnent and 

productive results being obtained. ?he or.rtnsis on nethodologj in ovaluatian 

cannot bc i~~.dred, honcver the finest nxl:r616 procedures are nisgulding 

if we ‘are not evaluating the &ills or tFalts wo think should be undor 

study. 

i a 

In scarc:lln!; fu'r vat-'i~-36 ways to Deaf&e and apply our findings 

about human behavior, the t$oa!- to cor~crve e,zd develop a full range of 

hunan resources nust not be &cited b,.v lack isf newer concepts and subsequently 

nethods in personnel eva'uatim. !hrinr a recent lecture by a ~011 

known astronaut (!Xtchel!., 1973). hc stated that he was cosvlnced of tho 

teed to study the spcctruz of c~~~sciousnesslllspla~rcd by the hunan organlan, 

2nd !?.as rrcnfized sxh a grou-3 f3r this iury;ose, Yiith this kind of forceful 

inquiry t&in{; place in ntmerous internqtional study canters, we nust 

a,reo such tochniquos a~ IF: 6hould bc sf 6+ intcrcst in xur pr3fessiocn? 

franc-of-reference. 'There are hu.zan abilit,,lcs which apparcntl:r transcend 

the traditiUnal cxpln?,ntioz ~,f h.-‘w sirilis 'f'c norzaiI; acquired and used. 

;‘le should see .;'h?t effects these abilities hn*~c 38 031‘ thexics and evaluation 

practices eve:1 if the y-zqxxt seezzb unsettling. In the ho,ne that 

my discussion ;f Tra~,sccrzcr~aZ . .- ." ::cditnti>r ins r.:t n?lcnatcd :-3ur 3cIn 

creative urgesi;, I 7n5 ivith t&n %?ditativ/ rczark attributed to Pinstein: 

11 . . . Imaginatix is r.3rc byrtant than :*&33ledpz.” _. 

_. - 

, 

.’ 

152

Abram, A. I., Paired associate learning and recall: a pilot study 

comparing trnnscendestal meditators al.th non-meditators. Unpublished 

progress report. Serkeley: University of California, Education 

Departnent, February, 1972. 

Doucette, L. C., Anxiety and transcendental naditation as an anxiety 

reducing agent. Ucpublished paper. Yar,l.lton, Canada: l!c PIaster 

Universit::, January, 1972. 

%stnan ) M . , The military neditators. Ilrny/?:avy/tir Force Tines. July 4, 

1973. 

Fiske, 5. 3., Thousands finding neditation eases stress, The !:ea York Time, 

Bxenber 11, 1972. 

Kanellakos, D. P. and ?lel?in, iv., The practice of meditation as a rceans to. 

the fulfilfzent of the ideals oI hunnnistlc and transpersonal peychology. 

Paper presented at the 10th Annual Meeting of the AaeoclatIon 

o f ~uzmnistic Psycholsg:~, IIonolulu, Hawaii, Axguot, 1972. 

>!c Clellaxl, D. C ., Testing for corpetence rather than for @8intelligence18. 

tierican ?sycho?ogist, 1973, 3, 1-14. 

Mitchell, 5. D., cuter space and TSP. Lecture presented at the 3rd Annual 

PSI, Inc. Conference, Indianapolis, Indiana, October, 1973. 

Xogers, C. il., Soce new challenges. 

379-337. 

,‘J-,orican Psychologist, 1973, 2, 

Vallace, 3. K., ?hyslolo&ical effects of transcendental nedltatlm: a 

proposed fourth state of consciousnesn. Unpublished doctoral dlseertation. 

Los Angeles: Unlversit:; of California, 1970. 

.’ 

i 

i 

/i 

-. 

. 

: 

: 

___- ’ . -. - 

-e- ,- .-.~4-~--..~~-A-~~.& 

.\ 

. . 

‘\\ . * : ,; 

’ 

. . . 

‘\ \‘ 

*. 1 

‘, ( _ 

\ : -. . ; 

I 

/,, 

:’ 

_. __, . . 

-..;c.- _ 

__. -c-,: 

.i 

. . ., 

. 

d,t s t: i,@ 

., 

153 

. _ 

3 

. 

-.:

'. 

"IMPLICATIONS OF CARREL INSTRUCTIO!I ON MILITARY TESTING" A PRENARRATEO SLIDE PRESEhTATION 

BY OR RONALD W. SPANGENBERG OF THE MEDIA DESIGfi & EVALUATION BRANCH, TECHNICAL TRAINING 

DIVISION, AIR FORCE HUMAN RESOURCES LABORATORY 

AFHRL TECHNICAL TRAINING DIV. This presentation is entitled Implications of 

MEDIA DESIGN & EVALUATIOJ BRANCH Carrel Instruction.on Military Testinq, Ron 

Spangenberg speaking. For proper synchronization 

you should now be seeing the specially designed 

focus frame of the Media Design & Evaluation Branch 

1 

. 

of the Technical Training Division, Ai.r Force 

Human Resources Laboratory. 

Focus frame 

1 

I 

CARREL INSTR!!CTIC!N 

Student working at standard 

carrel 

I 

2 

Basic Electrical and Electronicsf 

School, U.S. :Javal Training 

rCenter, San Diego, California 

Students using workbooks and 

programmed texts at "dry" carrel 

Some test equipment on shelf. 

4 :c 

. 

._. 

What does carrel instruction mean to you? 

For the Navy student at San Diego it means a 

programmed instruction booklet with some 

items of electronic test equipment. 

For the planners at Ft Benning, carrel 

instruction means media. 

::. :.: HRL Media D/E Lab -2 

. . 

..\. 

I'. 

. 

_ 

i' 

-. . 

'. 

,' 

__-. e-..-- 

. '. 

..__ .'., 

-- . . . 

-- . . 

..-.. 

* .,*

,., ,. . . . . . . 

:. . . . ,. 

,. ,: . ,: ,. ,. 

., 

. : :,:. :. . . :, .,I. 

. . ., . 

,:,. . : .: :, :. :, :; 

.:.: ,. . . 

, : ,:, 

~~--‘,::z...:...~ : . . . .;; ‘..:..~:.i’-r~.....:.~.~.~.~.~....,..::?~..~.:r’.. :y... ‘..& ..i 

.:,.:,, . . ‘:’ . ..y... ..:.. ‘.::%‘:.:.:.: :. _,,,. ,._. ,, ._ :, 

..i 

..~...::..r..,,., 

. . . . 1. ,.. I..:: .:.:.: 

$.s:’ ,.‘.‘. _> ‘:_ :x:y: 

.:..,:::,“?. 

,,.,._ :.>. . 

:‘:, 

. . ..i.‘..._ _ 2,. . . . :‘.: ,._, (,_ > ‘.:‘:.: .;c,,.: ‘,~.‘;~~ 

....: .y:... c :::.: :.: :fi:.‘,‘;. ..:..:.,.:.:-..: . . . :.>...,1 ; .:::‘: ..:: (,,., 

.._,, . . . . :::.: :.>:: ,,.., .“’ ._ _, 

.o:... :. ..>, ::.::~;:..:..: . . :,::: 

CARREL INsTRJCTIoN _ S+ANGENBERG 

:, .; ‘,: 

.: ._.. 

.. 

,,;‘,:y. :.:... ::....>:.>::. /’ : .,i ,...: i:,..::iT.‘,;.‘,’ :c:::,. .‘. . . :,,:,.:... 

,>:. ._,... :,:.., ..:.:.:..A:; y.::‘..,., _ 

. . . 

:I “:: .:. :.... :’ ,, ” “‘_ 

‘.. :::.. 

,., 

.: .: 

.,., 

;:. .:.:::j::. :;.; 

; L ,> 

.,_ :,., ‘.: 

: . . . . . . .:,.,; . ': :::. . .,. ...'.:., ., 'k, ..:.:p ,_: ...: 1. .,.( f_. j:.: '.. .5' ;::. I.. _,:j: ;:$ 

..:.:. .', " 

:...:k':.:.::. .,. .,...:, ,~ ..:;.:.; .2 .-: :'.:'.:.:'~I.~ .'._ ,(,, ,:: 

.,:. ..: .:., :.:: . . ..j.... ,.___. _" . . . . . ; ,;..:.. .I:'.':' 

. 

..'. .:.,i ,, 

: ,,. f. .)... : ._..: ._ :..::: :: : . . . . :: "i' ,:,, . . . _.,.,.: . .:::. . 

: ..:%.j:: 

:,. '.,Y ;y:;;;. ,, _, :...:::;: 

.. -j;?.; 

AL!IO-&lJAL SroKY BOARD -/-l-r 

; i 

('. College Library .$$ The carrels atfkt San Jacinto College have 

Mt San Jacinto College 

: ':i:i prenarrated filmstrips. 

: ._ Gilman Hot Springs, California 

,,; 

. '. 

Study carrel containing: 35mm 

filmstrip projector and reel to 

reel tape playback unit. 

5 

UPT Learning Center 

!Jilliams AFB, Arizona 

: 

.: .".' 

Two-man study carrel designed 

to represent T37 cockpit. Audio 

selection directs the student 

through procedural lessons. 

** 

I 

2 

II 

Undergraduate pilot trainees at Williams work 

'4th A/C control panels 

6 ) 

I 

. '.: . . 

I 

Student watching Audiscan w4th 

Security Police Training 

Program 

while Security Police at Lackland can view 

c. audiscan presentations at a table. 

.: .: 

8 

7 

: :'.;; ;' 

._ . / . .I':.::: 

: ::.. 

EDIA DESIGli & EVALUATION LAB 

FHRL 

:artoon figure with media 

equipment 

‘.., 

.' 

.. .",': .y ..:,., ., ,,;: 

:A 

. . . . 

4 

. 

I 

: I 

I 

.: 

-.:i 

: .: 

:_:. . . 

.I _’ .:,2 

. . 

. ..‘_.. .:.. 

-:: 

., : :.. .,,; ,;, ‘I:‘. j:;- -::.~~,~:::~;j~::~~~-- ‘.;y;jj 

:; ., 

,: ,....., ( ‘. I.. 

..,. :. .’ .: ,_ ‘.“::;$ 

i 

I 

-.;; 

..” j..: 1 

;I,: In the Air Force Human Resources Lab the Media 

. . ..$ Design and Evaluation Branch of the Technical 

'> Training Division has been involved in 

. .-..... . . ._... ,..: .:_, .:.: -i::.. y _.-, ‘: -:.; HRL Media D/E Lab -2 

,.:: 

155 

.

,.‘j :,;. ‘.: :. .:‘. 

,_:1...... 

.'.::.I :.'. :- . . :;.,: 

':., .: 7.; . . ....'y CARREL INSTKUC llUN - M’HNbtlYUtKb 

,, ,. 'i'.' : . . ..z : ., . . . . 

:,,: y. 

. . .:. . . . . . . 

. . . . . . '. 

AUDIO-VISUAL 

., ,;. ,/. ,.,. "..: '. ..,, _;._. 

SIDRYBOARD mmL/lT 

:. 

:. . . ,, :..::. 

'.. ., 

.,_.$ 

.". . 1 

.' ,'IcARREL DEsIGti carrel design and instructi'on for several years. 

I 

Carrel with calculator and tape 

recorder visible. Hay to 

operate calculator orogram is on 

screen. 

9 

. 

, 

kleapons kchanics Course 

Compulsory Pezedial Trainin? 

Lowry AFB, Colorado _ 

Students viewing prenarr-dted 

slide presentations at standard 

carrels. 

Mechanics Course 

Compulsory Remedial Training 

Lowry AFB, Colorado 

Student viewing PSI&6 Program 

at Performance Aiding Carrel. 

ar;el inset for Performance Aidi g 

arrel with wiring for makinq 

ontinuity checks. 

I 

I 

5!e have set up carrels in a learning center 

oresenting prenarrated slides most frequently. 

Ue have designed and built a performance aidinq 

carrel in which an inset panel in the center of 

the working surface can be r-enlaced with 

specific oerformance boards, 

Such as this board designed to refresh the 

students in making continuity cheeks with the 

PSM G Hulti-mzter. 

156 

. HEL Xedie D/E Lab -2

-. ,. .: ,. . . ,;:y. .::‘;:; ~:““:.“‘:.::‘.:.‘..::,y.~:,:,;.~ ::y,. _:., . . . . . . .:. . . : :_:.: 

.‘. ,. : ..: .: :_: _’ ..:y:i:;:jc:, ,,. :,: ,: .j ::I ,,.: . . ;., >( ., :; 

.;’ .., :;’ .: 

::.::‘:i;.i ‘::.:..‘?j i:.::.: :.;:.: 

., ,.. ,,.. : .: .P. :.,, 

,.:.‘...::I :..::. .,,_.. , _ _.i :.+ : :; ..: :., ,. ::::. ‘_’ ;.: .::.:..;,;j.j,::: ‘: .y ;:; :.. 

:, ..~~~,18: :.:., :.:i ,. :,:, ,: ,.... 

._.:.. .: .:: ..,... ::I:..: >, _,: 

,’ ..‘.~.‘,.:.. :_:..: _,.. . . . ..I :. y:. I.:..:, : ,:.. ,. .:: .: :,j”:::. '..> . 

.,.:.: .:: ','.. /j.,.: ,, :. :: .: CARREL INSTRUCTION - SPANGENBERG 

':. ':~.:::~,:~,::::::' :.,.,.,.,... ,;:: : .,.:,: iii.:' :' '.: . ...::::. 

.;::: ,:: :: . . . ..'.i:~.i:;i,::'i: : .>, ,:., .: .:, ..,:. .,_.A . . : :: ,:; ,:.: I,,.. . . . . . . . >: 

AUDIO-VISUAL SroRk BOARD 

: ,. . . . . . ..i:: AR&T 

. . .'.:.! ,.' .'.:. :. >,::. .:.:::.::':::,. . . . . . . . ,.:+.: . . . . :. . _,. . : :, 7.. .: ,.(. x:..::.:.:. .,:,:. _:., '. i.: . . . ,,.;:;;.i,.y ,:::j. :_ ,, ::;: z.1: ,:: : 

:.. :,.\ ,::,. j ::,g...:. _.. :_ ..,:, ..::. (1) carrel instructional materials must be 

. .'. developed, produced and validated locally, 

Carrel with two Carou.sels, 3nm .'I.:. 

: projector and audio playback 

unit. Random access slide 

selector is shown. 

1 1 

. . 

13 '.. 

'. .: 

, ,. 

INTERACTI'jE CARREL 

Interactive carrel with txo 

persons viewing an Audiscan 

presentation 

14 

., . . . 

(2) carrel instruction is not necessarily 

performed in isolation. We feel students 

frequently should work together as in t;lis 

interactive carrel which is designed to help 

two students work together - or for an 

instructor to help a student, 

IThe Audio Visual Instructional 

1 

(3) the third conclusion is that a carrel is 

Display System module being' : not a piece of library furniture. This portable 

demonstrated by a secretary carrel can be rolled to actual pieces of equipment 

and provide hands-on instruction. Tousa 

carrel could consist simply of a set of earphones. 

1 5 

.' 

:ARREL 1!6TRUCTIOEI 

.‘.. ,.. 

': .., ,;:, .' : ,. 

: ._:'I $':.. : y ] ..;; _, ..I 

.,. : 

ltudcnt working at standard 

.arrel. 

16 . ., ,, ::.‘I. y-:: 

- 

Lw-4-- 

. ,.: ” ‘.._ 

. . 

i You can conclude carre 1 instruction may be many 

,: different events. However, there are three areas 

1'1' of evaluation which will receive increased 

emphasis due to the use of carrel instruction. 

_._-- - --A.--a*a.&d-- HRL Media D/E Lab -2 , 

:. ,.. 

157

1.’ 

.; 

:,: 

‘. . :, 

.,.. >.:..;;: 

q:,:.: : 

,,Y 

. . .’ (. . . 

.... 

. . 

:_. ,’ 

,” 

CARREL IZiTRXTION 

QlJAL ITY CC:ITROL 

Student working at standard 

carrel. 

17 

::,RREL IWTRXTIO:! 

Q’NL ITY co!moL 

t?OD’JCL: VAL 13ATIO:I 

Student writing ai standard 

carrel. 

18 

Cl4RREL 

I!;ST2lJCTIOiI 

LEAR:iIi

. 

.,. ., 

. . 

:...: ,; : :.,. 

‘, . :,.: . . . 

. . ._.’ 

.' ,' 

,. 

'.: 

.: 

:. 

..... 

,:: : ." CARREL INSTRUCTION - SPANGENBERG 

AUDIO-VISUAL Sl’DRY BOARD ‘5. AFKN,/TI 

“.. 

‘. 

(1) Pylon wing stations are 

nulilbered : 

r 

I . 

22 

(d) 1, 2, 3, 9 

3.v g3s nrefssre 

(bj Gy hyc..~lic pressure 

(c) Gy electrical solc;7oi:! 

(J) Fnem:at!ca;I:: 

1 

(11) Dcrinzj grolJnC tof cation . . . 

(3) Turning cff :,draulic 

f)rxssuri- 

(b) Installing a lof.kcut I 

Ltol t 

(c! 1i:stallicp a ;lround 

safety pin 

(dl i;cversinq breech 

SlWVCS 

23 

I The student is given about 30 seconds for each 

test frame. 

(i) Pylon k:ing sta:iom arc Then after he has completed the test.he will 

ntimixred : 1 then check the answers. 

(a) 1, 2, 3, 3 

(b) 2, 4, 6, C 

(c) 1, 2, 4, ; 

(d! 1. -7, s. 2 

c 

24 

. 

; 

159 

..I 

.) 

( HRL Media D/E Lab -2 

. ,.. 

. ..- . 

. -. 

.’ 

.

(S) The carrying hooks... 

25 

(cj By electrical solenoid 

(d) Pneumatically 

(11) During ground o?erationc . . 

' (a) Turning off ;l:W,dratilic 

pressure 

(b) Installincj a lockout 

bolt 

(c) Installicn a ground 

eweversinq kt-cccil 

jsleeves 

26 

27 :: 

PSM 6 Multi-meter ,:ith red 

ead beinn inserted 

. . 

e 

:I:; 

J 

I 

‘.> 

M~ CARREL INSTRUCTION - SPANGENBERG 

AUDIO-V1SU.y 9'0RY IX?RD luml..frr 

(; 

z 

The sound track t;o:rld say the answer is (c). 

During proJnd operations the accidental release 

of loaded stores is prevented by installing a 

grourl safety pin. 

Fo 

to 

WI1 

Ilowing iach self test the student is invited 

ask the; instructor to explain any confusion 

ich he zay still have. 

I 

Mr. RaiFord, anot;,er Lowry instructor with whom 

we have/worked, developed a refresher module for 

making 

i 

'continuity checks with the PSM 6 Multi-meter. 

Using a mediated nrogram and a special board in a 

performance carrel, the student was walked through 

a PSI? 6 continuity, check. 

160 

HRL Media D!E Isb -2

: 

., :, .I ., 

, , ., ., .:. 

: 

.:. 

. 

.,. 

:. 

.,:. :,. . .: ,. :. 

,: .:. ., .: . 

; . 

. . 

:. 

., 

_..... :;?, j. :y...:: 

:,,: .._ ..::..;.:‘. ,“S 

,; ._.;_ :: . . ..I ,.... 

:’ 

‘y,‘:y;:i .‘K, 

., :,y::.:.. 

.‘::..:: ‘. : 

,.:.,, .: ..:,. ‘. ‘...I 

.:,. :.,: 

. 

:.,. 

,_ 

,, 

.‘. ,.i.; .,/. . ::y::,; :,:. :.:.::. 

‘:, :,:.y, ;:..+.::.::.: 

. . . . . .. ,... 

:‘; ._.. ,:.;,,: .:.: . . 

. . . ..y..:.:.: ‘:.I 

1,: 

. ,, _. 

1,: :,: > 

‘:), ,.::.‘y: . j:‘j,::::. ,:;.p. 

‘, .,.. 

. 

.‘. .:y: : ” ..:; “.” ” 

.,,... 1.‘:. ,::.:::,. ..,:.: :v..y:‘... 

. _; 

:: 

: . . . . 

Student performing continuity '.i;t: 

f check with Make/Break switch 1': 

open. 

c 

29 

; ;: 

.:sck with Ibke/Break switch 

:losed 

30 

inger pointing to wires on 

ontinuity Check Inset-t Goard 

heck on wire 6 showing PSM 6 

i:~~ 

3 2 I:, 1, :::~,,:; :: ;'1 $$; 

. . . . . . . .I ..: . ..&., ., :,: 

. . . :. ..: -'::.::x,:i.; ._ ,. 

- -. 

t 

J 

i CARREL INSTRUCTION - SPANGENBERG 

ALKUO-VEUAL STORY KMRD -/l-r 

The student was shown an open circuit check 

which he then performed. 

Then he was shown a closed circuit check which 

he perfomed. 

After both open circuit and closed circuit 

examples were shown using wires, the 

'_ student then performed additional continuity 

: checks. The results of oroper checks were show 

.:: him. Successful comoletion of the program 

;. required the self-evaluation designed with the 

program. 

161 

:_ 

..:c .;: SRL bi$x?ia D/E Lab -2 

-. ,: . . 

. . 

_- ._. -. __ -, . . .._ _ _ . _ _ 

b 

.

:: . . 

i: ,:>::. . . / : ,. :... . . :.::.- .: 

.y: ,, .:.:_:.. : ,I, : 

wnnLL A 1q.3 8 hyc I . vtI “. . .,.Y_..__. ._ 

: ,. 

‘. 

: .+, 

:. .,. 

;; ',: 

.." 

L '. 

. . ,....._. 

.:’ .= :: 

.' :.., 1 

AUDIO-VIZ&U SIURY BJiWD AlNRL/Tr 

~ENSITOMETRIC 8EASUREIKNT In another area a program was developed by 

Ron Filinger on densitometric measurement. 

: 

33 

26. 21 STEP b/ET&E - 11/11 I 

I 

Final frame in Oensito~etric 

I*leasurer.lr?nt Program. The 

student will mcastlrc 11 out of 

11 steps on a 21 sten wedge. 

I 

I 

34 

Sensitoiiictric Strip No. 0395 

35 

I 

After the student is shown the correct procedure 

he is given a sensitometric strip on which he 

performs densitozetric measurements. 

ensitasetric Strip laying on .The s&dent records the various readings which 

ensitmatric ilcasuru:;ent Answer arc then checked by an instructor. We feel that 

heet :evaluation should be integrated into each module. 

36 

: 

. . . 

. . 

'I: 162 

. . ,.,.I$...,, ..I.$ 

: HRL Media D/E Lab -2

:ARREL INSTRUCTION 

QMLITY CONTROL 

MO@L;LE VALI3ATIOZ 

;tudc..t working at standard 

:arrel 

!r;t;:r~c:or focusing !/c?' video 

3l3"k? 1.1 r-3 . :.:A:.!- 1 ; 

30:2 Rack. Pro?! 2~71 cues kritten 

3n ~311: board in tack:1round 

39 

S!id?T 

- 

: I 

1r:s '.ructor shooting Sony Ravel 

l/2" bide0 camera in developing 

XXl ,122 bomb rack !iodllle 

39 

43 ‘_ 

- . . 

‘I 

':I CARREL INSTRUCT!ON - SPAKGENBERG 

AUDIO-W+M STORY' BOARD m/I-r 

Learning module validation will become increasingly 

objective. KoduJ& will be systematically evaluated 

to insure optiaal lc3rning. The area of module 

evaluation will probably exercise the innovativec ;s 

and creativeness pf military testers to the 

greatest extent. f 

i 

In module develognent and validation we have been 

experimentally zlaring technology in the hands of 

erqerienced instructors. I will show you two of 

Yit? techniques we have examined. The first 

technique is dynamic storyboarding using l/2 inch 

video tape. 

A3 instructor shoots the desired perfornLnce of 

the joblbeing trained. After rev;e\< and feedback 

from ot)ler instructors, and possible revisions, 

iI 

the &deo seqrrence is shown to a small number 

of s&dents. These students are _ 

163 

HF.1, Media D/E La% -2 

-_ . , 

- 

.-.. . . 

_ 

. . 

I

Ji 

&“l.:~ 

: 

i ; 

r 

t 

.,... : : ..,., 

: :,. ;: .’ 

,, ::.' 

.:.. 

iVALUATE 

: 

41 

. . ;,,.. J. ..:.: ./. : bmNt~~ A~V~B nut., AUII - 3rtwutwxnu 

: :. 

( . ,::i . : . . 

,., .:. :. . . .. ALIMO-VI.SUAL SMRY BOARD 

'. . . . . ,.._.. : .: m/l-r 

:.;. 

Instructor evaluating student 

ilerformance 

on Xii 12 bomb rack 

nodule 

ISHOOT 

Instructor develcr~inq :'Jl module 

is sltooting with Irlsia-matic 

camera 

42 

SEQUENCE 

Instructor seouencing slides 

on multiplex slide sorter 

43 

H01.J 

nstructor projecting slide / 

ram M 2 Iqodule for students 

44 - 

. 

- :i: i. _ - : HRL Media D/E Lab -2 

then taken to the equipment and their performance ir 

evaluated. Any difficulties or confusion on the 1 

of the student would necessitate some revision of ~#IE 

storyboard. kJhen the module is informally validated, 

the media format for student learning is selected. 

We do not recon?nend the use of l/2 video tape as a 

primary teaching media but normally would convert 

this dynamic storyboard to a prenarrated slide or 

Super 8 oresentation. Revision or validation is 

completed before any production costs are incurred. 

\!hcn the instructor learns to develoo modules which 

teach visually, another storyboard technique can be 

used. 

Using am insta-matic or 35mm camera he can.stioot 

selected critical vicars. 

Then when he receives the develooed slides from 

the photo lab he selects and sequences a set 

of slides portraying what is to be taught. 

tie will t!len present the slide sequence to 

students - developing also the narrative 

associated with each visual. 

164

">.. 

. 

.: 

. . ., :, ., . . . . 

.: . . :.. ,:i 

:,: 

: ., ,' ..,'. ; :.. 'j . . ',;,,;,; . ..;. .: .,.; : " 

.: .,. ., 

..'.,,.' 

'. 

: 

. ',.. . . 

"._.. " CARREL INSTRUCTION - SPANGENBERG 

: ‘:'.':,..::::.: ..:.; ., ',' 

,, ..,.;.:. .;,. .-., 

AUDIO-v1suAL SroRY KMRD 

:. . . 

". 

Alm/Tr 

- 

SYNTHETIC JOB PERFORMANCE Evaluation of synthetic job performance can Fake 

many forms, 

Student operating switches 

in LT38 front cockpit 

49 

Ft Carson, Colorado 

Video camera setup irhich records 

panel during Vulcan I~lcapons 

System operation 

50 

IQUALITY COZTROL 

tudent performing arresting 

uplock assembly check on 

ELIAGILITY 

tudent making continuity check 

n performance carrel 

52 

i 

For example at Ft. Carson, video tape is used to 

evaluate the perforflance on the Vulcan Weapons 

System. 

Quality control can be built into many modules. 

The instructor is able to make a performance 

check on each member of the team of four students 

who are learning the A/C preoaration procedure 

for loading an AIM 70 guided missile in this 

quadraphonic sound module. 

It seems quite important that the reliability, 

-, : .., 165 

- . . .;:, ..: 

. .:.' . . :' HRL Media D/E Lab -2

: 

._ 

EVALUATE 

Instructor evaluating student 

performance on Mi 1 Module 

45. 

ARREL INSTRUCT102 

QUALITY COIlTROL 

tudent working at standard 

arrel 

46 

'SYNTHETIC JOB ENVIRONMENT 

Student sitting in front cockpi 

of LT38, a photo mockuo with 

some live switches of F4C 

aircraft 

47 .. 

..‘. 

‘.... 

‘. 

.,, -, 

: 

,. . . . 

OCKPIT PROCEDURES 

tudent at dummy rear cockpit 

48 :.. : ,: .y, ..: 

. . 

. . '. : : . 

: 

I 

:. LMRRCL 11‘(3lKULlllJN - SPANGENBERG 

AUDIO-VISL& STORY BOARD 

I 

I 

/' 

-/?T 

: Students can then be evaluated and the storybbar 

validated.. 

I! 

The third area of increased emphasis in evaluation 

related to carrel instruction is quality control. 

Quality control of critericn performances will be 

increasingly emphasized. Student sampling and 

extensive s%udent performance tests can insure 

the ubjectivity and validity of student performance 

checks. 

Synthetic j’ob environments can be readily 

created for, comparative purposes. The use of 

synthetic performance tests, such as 

this cockpit procedures check, should increase 

because of their potential ecoraomy. 

166 

!J 

Y' 'HRL Media D/E Lab -2

.( 

.., 

,.I .A. : .:: ,.: . . . . . . . 

. . ..:. :y’::‘):::. . . .j::, j.: ,: ,, . . :. 

:i 

‘. ‘., 

.'.. : ,:. 

:.:,,2 

.: .._. : 

.?; 

., .:.i:. ::. ., ” :’ .’ .:,:!'i: . . . 

.':'f., :Y' .,:,;. _,... .:.:I, ; ,.:..I :: [..:,;j,)..y. ~. ', ;. : 

,,: 'j. 

. . . . . 

:: 1.. . . '.. 

..y’ .:... I :. .:.;:..: .:.. . . . . ‘y’:. :, 

. . : -. _: ;,:j$~~. . . . : ,, 

.,_ ,.., :::., .+ ., ., .( : ,. : ..’ . . . 

.j .:,. ,., .,: _;..: 

,. ,.. ,, . . . .y,. . . : ..:. : :i' CARREL INSTRUCTION - SPANGENBERG 

AUDIO-VISUAL. S-ORY BOARD 

,‘. ., 

. . 

mm 

'RELIABILITY 

valjdity and 

. VALIDITY 

:.. ,. 

' 

Student making continuity check 

in performance carrel 

53 

RELIABILITY 

VALIDITY 

OBJECTIVITY 

Student making continuity check 

. in performance carrel 

54 

4 Kose Gun ammunition handling 

ysten. 

ointing to link 

55 " ., .; Y,:::..;:. .I" "j. ; ,: 

'. 

..: 

INDI'JIDUAL TEST 

PERFORMMCE- 

. . 

Instructor testing student on 

4J 1 parts location 

56 ,:: .,.;';, " _' 

. . -' .: ._ 

: .:. :::. : .: '..' .. 

i , 

, : 

_’ 

“T 

: 

‘/ 

objectivity of all tesis be established through 

student cerfomance testing. An example of 

what I mean is now being conducted at Loiqry AFB. 

2e are askinq the quest 

test can subititute for 

certain kinds of tasks. 

of instruction we are c 

testing. 

on of whether a picture 

a performance test in 

Using the same modules 

moarlng two ways of 

Following exposure to the learning module 

i .. Sgt Osborne takes the student to the actual 

I ,.. equipment and asks the student to show him the 

i selected items. The other way of testing pcrmitS 

I group testing using pencil and paper. 

.: 

167 

'. 4, . . .' 2 HRL Media D/S Lab -2 

.z. - -. 

/. 

;/’ ! . -. 

. 

- -. .- : 

; 

. . 

.

_ 7.. ;,;,,-;;,T-. :,;,,-;;,T-. ..’ ..’ :. : ,,.I,( .,,.:., ..,;. c:.:.:.:::“.:~.. :.:.:::“.:;-i ,,., ,,., .,>, .> -_ -_ ,__ ,__., .. - ^ . . . . . . . . . 

..::;‘:::::. . . ._ ..~)x::~~:‘.::i.i:,::::: : .5 . . . . .:.:.. .: ..: 

i,. I *y.:y : .:.:>y : ..: 

:. . . ,.._. : .I.. .:..: .,. . . .., . . ._. .,._ . . . . 

: ::, .j:;;.:‘,: ,‘.> 3 j::,:..~.:::;:;-: :.pc ,,;:..: ,.,:: ._: ._: :, :, . . . . . . . . . . . ..y . :.:,: .+:.:.., .:._, > :......: ., :.,. ,.. .; .; . .._>,.. ..>... ‘i-y ‘i-7 :?.‘:.‘x..: :y:: :.:::: 

,,,,: .;,.: >. p< ~z.‘.:.:.: _: ‘; ,. ,. .;. ,, .,.,,,, 1: ..::“:.:::‘~f :...:.:, :...:.:,:. :. ? . . . . . 

‘. :, .,:.>:;: ..> -,“::l;i:i:‘i~~~~::: .::::. .I:. ‘f ,: :$,:: ..: ,. . . . . . ..L..... .._, .,.,., :.:.z+:.:...:::.. i,,:.:,,,,, i::. ..:__,. ,_ . . . ,...,,:.., :y:: : ,, : . ,.:. ..> ., 

‘-‘i ‘.‘i .‘. . . . . . . . . .:. :: :....: :..,.: .,.,._ .,,,._ >,.. :.:...,. : .. . _ ,.,:.,, ,,,:.,, ,,, ;:..;.:“.. ‘F, :. ._.’ ;.,~‘,,.+f.~ . . . . :;.:; :.:,... 

: I.‘.: ..::. ‘...i ‘..i.. ::.:. ::p::.,,:. .,..,. ,,. ::, ,, .,,; .,:..:: .; :,> .,. .._. ,. : : ., ., :, . ,, 

::: .: ; .::,::ic .:::: ‘.. ,. :::’ ” ‘.‘.‘.’ “: ..:, 

: :.::y,: ? : ,. ,., .y::j:::: ‘._ .., ..‘, ,,:., .::.:: I . . i : ‘:>.,.: _.., ,:.Ll. : ., .., 

..... ,- .:,. ;:: ._: ., : 

‘_.. 

.; : :.: :.;, :,:: :: . 

,.,a., 

. . . . . . . 

:: 

.,. 

j;.::.:: 

,..... ._ 

.‘. 

,. 

‘. ,. :.:,, :;. ..:, .‘I. ,,; ,,:....,: _...,: :::y: ;:..:.: ..::. 

,, ;:. : ,,::.. 

‘,::.:yy::. : ,;: . a:: ..‘j ,,:::: ,,:y~::,:,.::;:,., .::,:,.::;:,., 7.:’ y.,” ‘-: “: ,._,, :..::,_‘:.x:,.:,f ,.- : ; ‘, .’ . . . . .:.?: :‘..‘.“’ :,: : 

.;.,... ..::. ,, ,,: ,.: .:.:..> I. .,>> _..;,:,i:.,. 

..: . ..v.... .> ., . . . _. :.:,. .._ ,‘.’ ‘r’i. f .f .., .:j:. :, ::... 

:.;:::: “..z:;> :,. .“. ,’ ,_ .:. 

.:::.::‘;:~:i:::~::. : ,,: .,: ,._ : .._ . . : . .’ ..,. 

/. ..-,: 

:: :.: i.,, : 

9 j 

” .;. .; 

:.!:‘::;z;:.: GROUP TEST 

. . 

‘_ ..i 

_, : 

,_ 

.’ ,. 

1 

: 

. 

57 ‘.! :. ,’ .‘.’ ;’ ‘, ‘: : j, 

: . . ,. 

; :. ..: . . 

1 

I 

CARREL INSTRUCTION 

I 

Student !qorking at standard 

carrel 

c 

58 '( 

'CARREL INSTRUCTION 

: 

. 

:’ 

. 

: QUALITY CONTROL 

:.y. .: . * : 

: ..: :::. : 

1 .I.. 

,c. 

j ,.,. ‘, ARREL INSTRUCTIOJ 

. 

..:.-: 

:, .‘, 

,.. 

..,, : .,# .:,'.I- 

‘c” 1”“” UALI TY TY CONTROL 

:, ::. :. '. 

.,, 

i.:., 

;> IODULE VALI DATIOR 

. 

:'. 

' tudent working at standard 

arrel 

: 

.: ; 

: 

CARREL INSTRUCTION - SPANGENBERG ', 

AUDIO-VISUAL mRY BOARD A.FbYRL/Tr 

The student is shown a picture of the item and 

.'.i;:;;.:;; .:.. :.,. y>, -PENCIL PAPER- 

asked to select the correct name. These ways of 

.' ., . . . .j: .' : testing are being assessed as to whether they 

; ::)~;:~~ : : 

: can be considered equivalent, 

I: : ..,.,. . 

.., 

. . :. 

..:, .:, 

,' ,’ .'::,J- ._ : Instructor testing students on 

..: &I I&I 1 1 parts identification 

:) 

'.-.:t 

The USC of carrel instruction appears as an 

opportunity for military testing 

to provide better quality control of student 

performance, 

.'-: to systematically validate learning modules 

. ..i 

168 

HRL Media D/E Lab -2

z ‘. 

,..... 

,:i ,.,: :, 

i : : :::::’ ‘ij::‘- ..y. :;:; .:i: :.. . ..:..':+$:::'. :::r:‘ii:~~~:i;i;::,‘;:‘:i . . . . ..~.~.‘.-.:..:.~~ .., .>:‘: .A.:.:...: ,_:,.y::j. . . :, .,, .: : ., ,.::,. :,::. ;' .',,. :, " :.,, T'::. . .!,i.& . . ,: . .:::.::.: 1: ._.: .:_ ..‘.., ‘, . . . 

.' CARREL INSTRUCTION ,' SPANGENBERG 

i : : ,.,::, 1, :_:.~ ,.'.:.'::: -; i.......:. '.~:a;i:~,.':::: 

,, 

':... 

: .. .: ::. : ,.::: .'::'y$. y \,:::, :. :': .~ :, ..:> ., 

; ,I;.;. ., . :, '.'.,‘:. . _,, . .: ,,:, .:,, ':. ,, ,( . ;., . ..,j::.:. I..:- 

:: .' :',;:::, ..: :. :..': 

> ,_' :.. ~UDI0-VISUAI.;S?DRY KlF 

; i :. -. .:,:, : :: ,:: ,: ;,.,.,.:... ,..- .:.,:, ,. ,, ;::$ .: .:,. .y,:. :.;: : .:... ,.; :.;:i ::.:. ..:;.; ,... ':.:.:, ..'.L .., ;. :: :: '..' ,. ,,. ., .,, :, ,,; :yj:' :.:: . . :. (.. :.:, . .._. .::_ :.. :;:,".'..,. .:... .__ 2 :,:,__ .:. . ,.: ..(. y:' ;. " ., : .j ., ., 

/ 

. '. ,:.. 

AmwIT 

. I 

CARREL INSTRUCTION and to increase Iceming effectivolcss. '-' 

:. :,.::i: QUALITY CONTROL 

5, ,:j::. 

::y. ;, :: 

.;:,,: MODULE VALIDATION 

.' 

. . . 

'.. :' '. LEARNING EFFECTIVENESS 

: 

t working at standard 

,.,: .( .; . . ..'.. 

'.. 6 1 ,,, " ':::,: ';:;.I; :;; _ "y ., .' " 

::. ,,, . . . . . . . . ;,.' 

'.'. 

AFHRL 

TECHNICAL TRAINING DIVISIOi: 

MEDIA DESIGN & EVALUATIDX 

BRANCH ' 

62 ,‘. 

..: 

. . . 

. . 

, -. 

I! 

169 

I 

r 

t 

t 

1 HRL Hedfa D/E Lab -2 

- . 

, . . 

‘L.’ ,_. 

. . 

I 

i l. 

i i

PROPICIE::CY KEXS:J.Scr*.-. """UT ID FLIGHT SIWLATOhS 

Edwin Cohen, Ph.D. 

Slmulatlon Products 9lvlslon, The Slnrer Company 

Mnrhacton, ::ew York 13502 

An lncreaslnr crOnCrt.iCn of the Work of mlIlt3ry personnel 

Involves partlcl:eti-n in man-machine systems, ;UCh aS 

aircraft, txks, s!-.l~)c, xfssiles, and command and control 

systems, t.kat are xe;r7~> le tc rezi-tic?, mm-in-the-1ooD slmlatlon. 

in sue!: r.17.ulntir.n . 1llust.rnt.e~ In Tir,u-e 1. the man 

1s located ln a renlfca cf kls ncrmnl work statIon, and provided 

b;lth the sa."e ::tir.lrli -- Vis’d2i, aural, grO?rlOceptiVe -that 

cor?nrlse lr.r~t:: tc r.1.m durlnrr combat or otl,er operations. 

lie reacts to thef.e stl:2i 1. renerntlnz suitable outputs or 

Tenponses -- contrcl movements, switch actuatlcns, verbal communications, 

lor entries --; the simulator, which cenerallv 

Incorporates a dlrltal ccxuter, receives and acts upon these 

responses to modify the stimuli on a real-time basis. Some 

slm,ulatlons are deslrnod Por crexs, teams, or units, rnther 

than olnrle :nc!lvlr!uals. car some command and control or mlsslIe 

control slrulatlcns, tL-e operational 2;ork stat!r,n !r 

used, with sgnt?.etic rather than real-world Inputs L v~I., - 

plied. This Is sometimes referred to as stimulation of t.., 

display system at the work station. 

Posltlons amenable to simulation, although reoresentlng 

a small. mlnorlt:: of arme2 forces personnel, contribute dlspropontionatelp 

to the effectiveness of each of the armed 

services. Eecause of t5.e crucial nature of t!:cse pcsltlcns, 

substantial efforts arc Invented in the selectlcn alld traInInK 

of men to fill them, anti ln the assessment and verification of 

their combat re?dlness. ~lculatlon plavs a lnrre and 

- 

lncreasing 

role in tralnln~., ?cr alrcrn:t pilots, simulator trzinlnm 

Is far less costlv tK;Pn .a . tralnlnm 

in the alrcrnft; for declalon 

makers In command and cor.trol systems. it AIIO';IG practice of 

combat sltuazlons not ct?.ert:lse available In pe:icetlne. SirAlators 

are also used for sue!; civilian positions as afrllne 

fllF;ht crews, nuclear r,CXf?r nlant operators, air traffic controllers, 

and automobile drivers. 

Simulators have been a locus of measurement for a variety 

of purposes, Indicate= tv the follo:~llnF: sampllnr , xhlc'n Is 

meant to be ll:Lustratlve rather than comprehensive: 

1. Selectl%n. AE??i,'s Personnel Research DlvlsZon has 

a procram under I:?? to r?etcr- ..lne t!.e utility of the Slnrer 

CAT-1 simulator for selection of pilots for underGraduate 

pilot tralnlnrr.

2. Tralnlnc Feedback. All current tralninz sIm. '. 

have provisions fcr .Curnlshlnc the lnstructcr wltn dzt;" .:. 

trainee performnrce. Soze , such as the ASUFT, a 1' ~. sls: 

tor to be used by 5% at Xllliams &?3 for trzlnlnf? i~3e;??'6r 

and the 2924 a slzclatcr of the LF-1 helicc3tm usec for 

transition a;d lnstrnzent trnlniw zt Ft. %c%er, fu~.fsk ~b 

data directly to the trainee in th.e dockplt- 

3. Debcrrinin- the ??fec% of 3vircnrentn1 V,?rl?bles - - 

on Human ?G?or7-zncc. k.r.2~ ies kere incluke t:".e effect, of 

spncecrz:'t-Jl;

generally t!lcse In which the tester Is comfort&bly superior 

to the testee. 'I'hus the basic methodolop of personnel testing 

and measurczent has developed around gaper-and-pencil Ins truments 

to the rolnt where no one thinks It Inconkruous that 

"Tests and 4casurements" textbooks devote upwards of 005 of 

their space to paper-and-pencil tests. t 

2. >!u!tInlIcltv of Yeasures. %'hen a osychometrlcian 

----_ 

finally de?i,:e:; to racker .h.r:,nn rerformnnce'data on a slnulator, 

he su:':‘crs an embarnsscent of riches -- scores or even 

hundreds of !r.easures are available, all bossessinrz face 

validity. ?I:-ure 2 shows data obtained hurlnc the lendlnrr of 

a 707 slmul:ttcr; these items reyrcsent a selection from a 

much larpzr n::aber of measures available. 

3. L!1Ck of Parallel Items. in a landInz, there Is just 

one Indicated alrsreeti at touchdown: to pet a parallel Item, 

another lnn,llnr must 'ye conducted. 

. 

This lack of parallel 

Items wIthIn :: riven crtneuq:er contrasts wit:? tke easv avallability 

of p3rnllel Items for most paper-and-pencil test 

domains. 

4. I~I?:‘Icult~.@ ?n Comblnlnr Item Scores. With naner- 

.-A.-. ---_____~ 

and-pencil tests, the ncmter of correctlv ansirered Items 

usually s e rv c s 2 s an cverrlll or summary ne3scre; occasionally, 

differential &Ir-htln- Is emnloved. 031:; rarely Is as coc?lex 

an al~orlt~;:~ as successive hurdles used to derive a summary 

measure from Item scores. 

1 

Nuck more connlel al~orlthns are reculred to 

characterize Froficiency In tasks such as landlnr: a 707. For 

that landin;: task, Items such ns Indicated airspeed, sink 

rate, distance from runxzv threshold, distance from runway 

centerline, r.r.c 3lrcr3ft attitude (Ditch! e keadlrz, and roll) -- 

all measure4 r;.t tcuch-33.n -- cannot L?e e aluated Independent 

of each ot%** .L ., as Darer-and-nencll test btems are ceneraily 

evaluated, Put r.:lst !'f cozSIned, usln;r 7 rather complex, yet 

to be developed nl~orltt-.n. 

5. In!?rn?IcabIlIt~- of Conventional Item Analvsis 

Techniques?,tcz anaivs:s Is most >iItiqlly usec for muitiplechoice 

Items. Kach multiple choice item renerally recresents 

a separate kc!zr?vIcr snmnle, and the alternatives are discrete. - 

When, as they often do,'ceasurements on a simulator cover 

various nspccts of a sIn;?le behavior sample, rather than separate 

behavlo? sazoles. and use continuous scales, rather than 

discrete altcmntlves, conventional Item analysis Is lnappllcable. 

6. p~~~v?Ila!~IlItv for :*!ensurement of Pe.h.avIors :!ot 

Directly Hc!‘Ic.?tc:! In ‘.:?chlnc Innuts. in the simulator, the 

�� .~n--??cr. ?r: t'-e-* . . effect. the machine, are 

;,xC?::rL' .; c t.-\ :'P-..':;:'C'.cI:t , :.1:t t!.e :':':,::':'.:e:- .- -_- :!*? * -..:!:ic.'. t?c-c 

172 

I 

I 

.

Inputs are made are not. For example, in a flir,ht simulator, 

Ii we would norm:,.?ll:: have no k'sy of knowina whether the pilot 

used an efflnicnt visual scan oattern in checking instrument 

readlnm. T!:e snrne cS’cctive ocrformnce could be achieved 

by two pilots xlt5 ver:.lnifr'erent instrument scan patterns -one 

usinK an cfficl.ent pattern and thus minlr:.lzin~ his work 

load, the other compensatlnc for a less efflc.lent pattern by 

worklnr harder. Xhen dcall,nr irlth n sinulaton that enables 

measurement.of z2ch a 13r;e nur.Ser of aspcctn of vehicle or 

system perform,?nce b:: merely speclfvlnf? the parameters to be 

measured, it i:: all too e3sv to for17ct about those aspects of 

human perforcnnce not directly reflected In machine Inputs. 

DS shiftin:: our ennt;nsls fro3 paper-and-?cncil testInK 

to the kind of perfor:?nce tests for which t5-e SlxIUlztOr is 

such -in appronrizte vehicle, we would qo far toward ceefinr: 

the objectives of crltlcs of our current testinK posture, 

such as PlcClell?n:! , ~50 tells us to test for competence, 

rather t%n intelll~ence, and t5:nt "t5e best tcstlnr: is 

criterion s~.cnlln~" !YcClelland, 1073, p. 7) and O'Leary, who 

advocates the "';‘:-.e ad*ziantnres of job simulation aptroacfies to 

selection over techniques lackinp content validity (O'Leary, 

1973, P. 149). 

In conclusion, :ie have a lcn:: :qny to TC, ln exploiting: 

simulators IS ce2surezent tools, before t:elr one basic 

weakness -- innt:6illt:: to zensuye bc!:avlor net directly reflected 

in machine inputs -- becmc?:; a truly llnltinlr factor. 

By refusing: to tc s!-iackled to psvc!:ozetrlc practices appropriate 

only to paper-and-cencll tests, we czn exploit this 

powerful tool to zeasum a wiee ?nnpe of Individual snd team 

behavior, obtalnlnrr relevanti relin!)fc, valid, and socially 

useful data at rr.odest cost. 

1

REFERENCES 

1. McClelland, D. C. "TestInK for Conpetence Rather Than 

for tIntelllgence'," American Psyckolorist, 1973, 28, 

1-14. 

2. O'Leary, L. R. "Fair Employment, Sound Psychometric 

Practice, and Reality. A Dilecma and a Partial Solution." 

American Psyckolorlst, -- 1973, 20, 147-150. 

. 

r

s 

iJl 

-STORED PROGRAM 

VEHI CLE DYNAI.11 CS 

SYSTEM OPERATI Oil 

DEt4OilSTRATIO~iS 

PROGRAlZlED I i;STRUCTIO:I 

PERFORM&\r4CE t.!EASUREMENT 

STIMULI 

' FIGUP,E 1, FLIGHT SIilULATOR CLOCK DIAGWI 

COCKPIT 

RESPQNSES 

INSTRUMENTS STICK 

INDICATORS PEDALS 

MOT I ON THROTTLE 

VISUAL LEVERS 

COiITROL FEEL SW I TCHES 

Al1310 VOICE 

INSTRUCTOR STATION 

D!LLSUUS !iLw.RD.S. ugL!aLs 

-u _ 

INSTRUMENTS X-Y RECORDER SWITCHES 

I t4DI CATORS STRIP CltART KEYLiOARD 

TAnULAR CRT RECORDER 

GRA?l{IC C R T TELETYPt3RIlER 

AUDIO LI;4E PRINTER 

MAG TAPE 

J 

.? 

1. --

i 

ILS LANDING WC1 RiL”l 3 6 

INITIAL COND R&F DATA 

OAT 6.1 DEC C VREF 128 KT 

BAR0 . 29.91 IPJ HG VUCA 117 Kf 

VlND DIR 36 DEC AL? 2602 FT 

VfND VEL tt XT LAT 39. t 230N 

TUR8 0 LOUC 94.8286V 

WY ICE 0 IN 

GY 206364 LB 

cc 30.4 t nix 

DEC EL 

STD TOL DEV 

FLAPS’ 2s DEGREES 

IAS 140 KT 5 3 

ALT 2582 FT 2600 SO 0 

ALT DEV -0 

LOC INT 

ALT DEV so 050 0 

UAX ROLL L21 DEG ~30 0 0 

LOC TRK 

LOC DEV RT .lDOT 0 1 0 

DEV LT .I DOT 0 I 0 

CROSS 0 0 

DEV WS 0 DOT NA 

IAS MAX 147 KT NA 

MlN 140 KT 0 -3 

ALT b!f:J 2620 FT 2500 -SO 0 

UAX ROLL L2 DEG ~20 0 

PITCH XAX 5 DEG k’:;\ 

. UlN 2 DEG NA 

GS INT 

GEAR DO’.Y 

CS DEV -1.7 DOT -2 1 0 

LOC DE’.’ ODOT 0 05 0 

IAS 142 KT s 0 

FLAPS 4 0 DEGREES 

GS DEV -t.2 M T - 9 1 10 

LOC DEV 0 DOT 0 05, 0 

IAS 140 KT 3 0 

Fl.APS 50 @fG.RIS 

GS DEV -.I DOT 0 .3 0 

LOC DEV ODOT 0 05 0 

IAS 135 KT 5 0 

ALT LO?! 2497 FT 2500 SO 0 

’ 

G!j TRK’ TO E!M 

GS DEV tit 01 DOT 0 03 ‘0 

DEV Lo .lDOT:O .3 0 

CROSS 

I 0 *o I 

DEV R?(S ;1 DOT NA 

LOC DEV RT 0 DOT 01.5 0 

DEV Lf 

CROSS 

92 DOT � � � 

� � 

� 

DEV R.‘?S 0 DOT NA 

IAS UAX 133 KT 5 0 

MlN 124 KT 5 0 

UAX RC)LL 

t-2 DtG 0 

PITCH MAX 2.5 DCG NA 

PIIIJ -07 DeG N A 

SINK MAX 7 2 5 Ffwe999 

XIN 4 6 9 FRI NA 

es TRK 

GS DLV K .2DOT 0 

DEV Lo .tDOT 0 

CROSS I 0 

DEV n?fs .2 DOT fJA 

LOC ULV R T .l COT 

DLV LT .1 Dot 8 

CROSS 1 0 

DEV RMS .l COT NA 

IAS rva 129 KT 

t-l111 124 KT 

rst ROLL R3 DEG *S 

PITCK YAX 2.9 DEG NA 

n 1 N 1.1 DEG NA 

SXNK tcax 573 FPU~BOO 

KIN 371 FPM NA 

FLARE 

MAX RCLL 

PITCH ZlAX 

UlN 

SINK KAX 

XlN 

DEV RW KDG 

LAT DRIFT 

C/L ALIC‘I 

TOLJCHDOUJ 

IAS 

ROLL 

PITCH 

SINK 

ACT Sl’JK 

D&V R’;YilDG 

C/L ALlG:J 

DI ST T.5RESH 

THR POS 

GRD ROLL 

SPOILF?S 

DRAKI:iD 

RESPO!.‘S E 

NV CO:.“iACT 

IJV STEER 

R&V FY 

MAX VI 

C/L CROSS 

C/L DZV MAX 

C/L CZV R?fS 

UAX R3LL 

RWY USED 

RS DEG

Y Y Y Y 

b 

I 

I

TlIE PERSOXU'NEL ASD TRAA..T"IX:G EVXCXTION PROGRAM: 

A Workinq Program fcr Improving the Efficiency and 

Effectiveness Of Fleet Ballistic Xissile Weapons 

System Training 

Part I Scope and hchicvement 

Lcdr Clarence L. h’alker 

Central Test Site for the 

and Training Evaluation 

INTRODUCCIC'N 

USN 

Personnel 

Program 

In April of 1969 the Chief of ?:aval Operations issued an instruction 

which brought into beinq the Fleet Ballistic Wissilc Weapons System 

Training Pro.Jram. This Program was aimed at improving the effective- 

ness and efficiency of training in the areas of the missile, missile 

launcher, missile fire control and navigation on FBX submarines and 

I 

at their support activities. Specific qains: looked for were the short- 

I 

ening of the replacement pipeline, provisioni for advanced, vice 

I 

refresher training at the s?ip's homeports ynd standardization of the 

advanced training available. h built-in part of the FEY Weapons 

System Training Program was a monitor Ca!.le 

the Personnel and Train- 

ing Evaluation Program (PTEP). The proqram: 

4 

"provides the organization, procedures, /and responsibilities 

i 

required to accomplish the qualitative pssessment of personnel 

knowledge and skill levels for officer and enlisted personnel 

during replacement training and while assigned to SSBNs, FEM 

Tenders, and FCM training facilities.' 

4 

The PTEP also provides 

for evaluation of training facilities, hardware, documentation, 

and courses of instruction for use as the basis for inpleraenting 

improvements in training and in all elements of the FBM Weapons 

System Training Program." 

178

Ii 

Various aspects of PTEP have been presented at the last two meetings 

of this association. For these presentations PTEP was in the 

speculative stage. This paper addresses the early working stages. 

To understand PTEP, however, it is necessary to look at the climate 

in which it oper.ates. 

PTEP is presently limited to the Poseidon Missile community, which 

consists of: 

1. A replacement training site - fIava1 Guided Nissiles Schoo? 

Dam Neck, Virginia 

Dam Neck, Virginia 

2. Two off-crew training sites - Naval Submarine School, 

Groton, CT. 

FBM Submarine Training Center, 

Charleston, SC. 

3. Wo submarine tcndexs 

4. Twenty-six FS!4 Submarines each vith 2 crews 

The submarine crews comprise the majority of the personnel and absorb 

the vast majority of the effort. 

FFiM submarines utilize a two-crew concept: One crew mans and 

operates the submarine while tte'otber crew is available at an off- 

crew site for training. Planning revolves around a six-month patrol 

cycle. This consists of one patrol/refit period and one training/ 

leave period. Each man remains aboard for approximately six patrol 

cycles or three years: he is then rotated to shore duty, frequently 

at a training site; follcwing this he returns to sea again. All 

f 

personnel selected for FBM Wiapons and Navigation Training are of 

above average intelligence. The submarine pcrsonncl have an initial 

six year obligation, with the exception of the Torpedomen responsible 

for the launch tubes. First-term reenlistments run in excess of 40%. 

179 

. ,_- , 

:

-_... ..-..-. “._..--_-- --_ . .._._-... __.. _._ _.. ._ __ --.. . .._ _ ,. . . .._- ~ -,--.. “_- .,.._- i.--* .-_ 

.- -7 

These then are the personnel involved in the FBM Weapons System Train- 

ing Program. The training program itself is not unique in its 

academic concepts, but it is innovative in the area of program 

management. i\s with any viable training system, the FBM Weapons 

System Training Program first provides training objectives. It deter- 

mines whtrc, when and by whom the learning is to take place; it 

provides the learning opportunity, and then assesses the quality of 

the end product. 

PERSONNEL PERFOR.'?XICE PROFILES Ah'D TRAINING PATH SYSTE!4 (PPP) 

- 

The Personnel Performance Profile (PPP) is the basic building block of 

the system. The profiles are for the m&ost part developed by hardware 

contractors and state in a. standardized format the skills and 

knowledge necessary to operate and maintain a system or equipment. 

A system called The Training Path System assigns to each Fl3!4 Navy 

Enlisted Classification (XX) the profile items and I.evcls of 

achievement appropriate. This' same system, by use of profile items 

and lcvcls of achievement, assigns material to replacement, advanced 

and on-board training. Curricula are prepared using the profiles 

and Training Path System. Informal training materials in support of 

the profiles have been developed, and watch qualifications require- 

ments have been keyed to the profiles and levels of achievement. The 

FBM Weapons System Training Program provides a system, whereby, the 

required knowledge for an individual or the specified content of a 

course can be convenienriy identified by iisting profile items and 

achievementievels.

_ 

PTEP AD?!XNISTiGTIO!J 

t 

The Personnel and Training Evaluation Program is administered by 35 

military personnel and a contractor civilian staff of about 10 people. 

One officer and an enlisted staff of expcricnccd technicians are 

stationed at each training site to support the testing and evaluation 

tasks. The bulk of the enlisted staff is at the Central Test Site in 

Dam Neck, Virginia. The program has a broad .:hartcr of evaluation. 

Evaluation requires facts. PTEP gathers its facts through examina- 

tions, collection of persor.al training information, ship and equipment 

operational history, and review of curricula with their supporting 

material. Examinations are the primary source of data. 

~0 separate types of examinations are administered by PTEP. One 

/ 

I 

measures the knowledge of an individual against knowledge required of 

I 

him by the Training Path System. Fhis is called a(System Achievement 

I 

Test, or SAT. The ocher type of examination measures the performance 

of a course against its Training Path System requirements. This is 

called a Course Achievement Test or CAT. examination 

content is based on PPP items and 1s prescribed by 

the Training Path System. 

Course Achievement Tests 

Course Achievement Tests are desigr.ed to measur, both the performance 

f/ 

of an individual in a coarse and the performance of the course against 

its prescribed content. The instructor does not see the test until 

it is time to administer it and so has no opportunity to "teach" the 

test. Standardization of training between sites can also be measured

with a Course Achievement Test. Few CAT's have been produced to date 

primarily because the standardized courses they support have not been 

in place. No meaningful analysis has been possible with the results. 

System Achievement Tests 

The greatest field of endeavor, and heart of the Personnel and Train- 

ing Evaluation System from the testing point of view, is System 

Achirvement Testing. The tests were implemented incrementally by NEC 

over a period of 9 months, starting in August 1971. To date there have 

been more than 4,000 administered. Each of the several tests in use 

is designed to cover the full scope of the knowledge required for a 

specific Navy Enlisted Classification. Content is determined using 

the Personnel Proficiency Profiles, and the Training Path System. 

The tests consist of two parts, a knowledge part consisting of from 

180 to 360 multiple choice questions and a skill part whM uses a 

paper work fault isolation technique which has been developed by the 

Data-Design Laboratories. Depending on the NEC involved the tests 

take 3 or 4 three hour testing sessions. Each of the testing sites 

is connected by a data link w.ith a central computer complex. The 

tests are scored, training recommended; and exam question information 

stored for future analysis by the computer. Test results are returned 

to the testing site by teletype. From the outset these SAT's have 

proved to be a reliable indicator of individual knowledge. The 

primary measure of test validity has been a comparison of test 

re.s-uits -wit> . the subjkctive arr

vehicle to confirm opinicns readily derived by asking supervisors. 

This can be seen from the nature and construction of the tests. 

(Figure 1 Sample Test Result) 

Each test is structured to provide information on the sub areas which 

make up an XEC and to recorrmend specific remedial action where indi- 

c.itcd. Analysis of individual areas provides information on the 

effectiveness of training in each of the areas. 

Personnel Data System 

The Personnel Data System is the part of the Personnel and Training 

Evaluation Program which changes a useful testing scheme into a 

valuable evaluation tool. This system picks up a man as he leaves 

replacement training. It records each man's trajning and duty sta- 

tion history and provides the information required for measuring the 

effectiveness of training at various locations and on personnel 

with varying training backgrounds. It also provides the statistical 

base against which achievement or lack of it can be measured. At 

present there are over 3,000 records in the Personnel Data System. 

(Figure 2 Personnel Data Sheet) 

One can thus see that PTEP provides the capability for collecting infor- 

mation on the training and personnel performance for which the FBM 

Weapons System Training Program is responsible. It also provides a 

broad source of background material against which to evaluate this 

information. 

183 

,” 

., 

I , ‘_ -.--.” 

! 

j . 

: .

Y PTEP ACCEPTAXE 

Uhan first introduced to the fleet, System Achievement Testing was not. 

mot with ovcrwhclming enthusiasm. Guarded skcptisn is about as close 

as anyone cane to a&eptir.cc. The rigorous schedule of examinations 

already participated in by FBM submarine crews in the arcas of nuclear 

?ropulsion and n*;clear weapons safety contributed to this unenthusi- 

astic reception. 

t3inir.r acceptan& for the System Achicvcmcnt Test therefore was not a 

sample job. A strong public relations effort was launched which 

ctartcd with the indoctrination on the F'Db! h'eapons Systen Training 

Prorjraz of cvcrycne taking the exam as well as the administrative 

pqrsonnel responsible for using the results: To further aid both in 

acceptance cf testing and the use of test results, a policy was 

adopted whereby cnly the command examined and not its superiors in 

the chai?l of cormand was supplied c;ith the test results. This gave 

each commanding officer thc‘capability of evaluating examination 

results in the light of his oxn knowlcdgc of his ship's training 

needs and to make judicious use of the results. The practice 

removed from both the commanding officer and the PTEP organization 

the requirement to defend results from an untried test instrument. 

Perhaps the most important step toward acceptance was the procedure 

uhcrcby each set of test results was returned to the cormand by an 

officer assigned ?TEP duties. The results were discussed with the 

commanding officer and department heads in the light of individual 

and overall crew performance. Czreiully kept records allowed these 

PTEP officers to review trends and to point out consistent low and 

184 

.

high performers as well as anamolous results. The personal and 

individual interest shown by the PTEP organizations at the training 

sites effectively sold the testing system to the ships. 

Training recommendations were at first reluctantly used by the sub- 

marines. An early sampling showed that courses recommended by PTEP 

were used at about the same rate as randomly chosen courses. A 

graphical analysis of successive test results presented a convincing 

picture that use of training courses improved performance in the 

area where the course applied: and that the most gain for training 

time expended was realized when the time was applied to weak areas. 

This seemingly self evid.ent conclusion, convincingly demonstrated 

to management personnel, has much increased the utilization of PTEP 

course recommendations. Thr use of self-study material recommended 

by PTEP was another area that gained slow acceptance. As with 

courses, however, PTEP has been.able to demonstrate that men who use 

self-study material show im provcment in their overall pexformance; 

consequently the use of this training material has increased 

considerably. 

ANALYSIi ASD EVALUATION 

YJ!EP*s business is much more than improving training utilization. 

Since January of 1973 sixteen evaluations have been completed on such 

diverse topics as: curriculum management materials, fire control 

software training, training of submarine tender personnel, and the 

restructuring of a sonar training course. Evaluation of test 

results of individuals and crews have shohn that personal and leader- 

ship problems are reflected in performance trends and anomalies; 

I 

185 

.

. i 

2 

" : 

these results could give early indication of personnel problems. It 

must be pointed out that to commence and continue ;h program such as 

PTEP rigorous and continuing internal evaluation faf material, pro- 

cedures and current objectives are necessary. Much attention has 

been focused on this facet of PTEP. Due to the nature of PTEP, 

I! 

external suggestions for improvement are not tardy in coming either. 

OFERATIOSXL CRITERIA 

The Personnel and Training Evaluation Program doscribed thus far is 

one which closes a neat academic loop--do poorly on our test, take 

our instruction and you will do better on our next test. 

The education provided, however, is not an end in itself, but a 

means to an end. That end is the effective operation of the Fleet 

Ballistic Missile Weapons System. Correlation of training with 

! 

actual operations is of paramount importance in neasuring the effect- 

iveness of training. Since PTEP can accurately reflect training, 

V 

I 

comparing PTEP results with operational information should link the 

training and operating worlds. FBM Submarine patrols and equipment 

operation are meticulously documented. E uipmcnt 

4 

i 

I 

failures are 

analyzed, as are operational procedures and repair procedures by 

sources outside PTEP. From this wealth o c data, information should 

be available to approach criterion testr 

.j 

g backed by operational 

statistics. The problem, in this continuing effort, is sorting out 

the multitude of difficult-to-relate va&ables in order to correlate 

9 

operational information with testing information. To date, PTEP 

has found correlation in test and operational trends. Spearman 

Rank Difference correlations have been positive and have run as 

186 

-

high as +.83. By July of 1973 correlation of test results with 

operational inforcration was sufficiently strong to warrant the 

release of test results to the Subrr.arinc Group Commanders to assist 

then in c\*aluating the personnel and training needs of their subordi- 

nate corr.zacJs. PTEP can ir.dicate crews with an increased probability 

of difficulty in handling Eaterin problcns arising at sea. It is in 

no way able to predict casualties. 

CIXRENT STAX'S 

Whnt has been the net impact of PTEP on the ships supported by the 

FDPt Weapons Systcn Training Prograri? It has provided a positive 

means of identifying the best means of employing training time. It 

has indicated whcrc serious training deficiencies exist, particuiarly 

in supervisory pcrsonnc~. it has provided a tool for identifying to 

highor conmands personnel and training deficiencies beyond the 

immediate control of the submarine Commanding Officer. 

For the training camand, PTSP has standardized definitions of objec- 

tive and insisted upon cocpliance with these objectives. For 

managemnt cocnands, PTEP has derrohstrated the effectiveness and 

ineffectiveness of various management tools built into the FBM 

Weapons System Trainin? Program. It has provided a quickly accessible 

pool of infomatim for analyzing crew, rate, or KEC problems. PTEP 

has the capability and has made beginnings towards measuring the 

program against the product of, the previous program. Both the 

Naval Examining Center and the Bureau of Naval Personnel have been 

provided with PTCP products. 

187 

: 

. 

. . 

.

PT&P has made a constructive contribution to the development and imple- 

mentation of the FBM Weapons System Training Program. Its importance 

in maintaining a responsive training system continues to grow as the 

Training Program grows. With the addition of the automated access 

to information files, which is expcctcd before the end of the year, 

the possibilities for analyzing many facets of the complex personnel 

and system inter-relationships of milit--p L-l training beccme awesome. 

188

06/16/W G%OUP SAT REPORT SSBN 695 BLUE CR 

&c 3306 FIRE CTRL TECH SSPN TEST NUMBER Flf3134 

I! 

TESTED 73108106 

ICJOt'tECCE AREAS-PART 01 AVERAGES Ksow SHILL 

A FDM L.3X'OS SYSTEY I EOUIPYESTS 1 FLEET 50 50 

B COST cc::s3L.z 6 PYX SUOSYSTEI\:S A E7/E6 57 52 

C.PLATFORM PGSITSCI~~I~~G EIUlF:-'.flJi El E6 56 54 

D WL 6 GUID -iXTI:iG E?;IITXENT C ES 51 50 

E DIGITAL CCXTi;OL CO!

0. i 

., -, L- .- . * m 

i! . .‘.c’C,“d 

. . < i. . 

: ;‘- . au.dE 

4 ;:-a ‘.;c” 

I . 

I I i 

i 

II 

I 

i 

I 

j 

1 

I

TEE PERSONNEL AND TRAINISG EVAJXlATION PROGRAM: 

A Working Program for Improving the Efficiency and 

Effectiveness of Fleet Ballistic Missile Weapons 

System Training 

Part II Program Development 

bs 

, 

Frank B. Braun 

Data-Design L&uratorics 

Norfolk, Virginia 

The purpose of this paper is to discuss some of the problems and 

decision points encountered in the development and implementation of 

the Personnel and Training Evaluation Program WTEP). As in the case 

of any system development,, not all the decisions made in the heat of 

battle were right or the most appropriate. Many of our solutions may 

be of intcrcst, however, since we do have a working program for test- 

ing and evaluating military technical speciaiists. 

PTEP is basically the product of system engineering. The personnel 

involved in the development of the program are engineers, instructors, 

and technical specialists: a systems analyst and a programmer applied 

automatic data processing (ADP) procedures to almost all aspects of 

the program. 

Since "kho" we were going to evaluate was pre-determined, i.e., the 

FBM weapons and navigation technicians, the first problem we had to 

tackle was what we were going to use to pcrfcnn the evaluation. We 

started with an advantage, since a job task analysis alreadl existed 

in the form of the Personnel Performance Profiles (PPP) and the 

Training Path System (TPS). The PPP and TPS provide the knowledge 

/ 

191 

. _ 

. 

-.

and skill requirements for each technician in the program< and are used 

as the standards for all elements of the program. 

I 

t 

We began our task by searching for the most appropriate types of test 

instruments to test the PPP requirements. Review of the many reference 

sources on the subject lead us to thc"decision that the four-alternative, 

multiple-choice question would be the zest satisfactory for testing 

)mowledgc items. Since the questions Gould be acquired from many 

different sources, i.e., equipnent manufacturers and various Navy 

school instructors, a specification was developed to insure test item 

standardization. Along with the question, the test item writer is 

required to provide various information to relate the question to the 

PPP/TPS standards (Figure 1). For the skill items contained in the 

PPP/TPS, it was determined that performance tests would be the most 

desirable test instruments, but were impractical for a large scale 

testing program. However, Decision Development System (DDS) exercises, 

I 

reported on at the 1970 and 1972 MTA nettings, cobld be keyed to the 

w 

PPP/TPS standards and appeared to be a satisfactoky means of testing 

the tschnician's application of knowledge. We decided tc utilize the 

DDS as the skill test instrument. 

The next task was to develop tests from the PPP/T bS 

requirements which 

would measure each technician's knowledge and skill 

1 

level. The problem 

that became apparent in this area was the wide range of requirements 

Eor each technician. In the case of the Fire C"ntro1 Technician, for 

i 

example, there are 28 primary equipment profile tables, 10 secondary 

eq.fpment profile tables, and two system profile tables. Each profile 

contains approximately 35 knowledge and skill items with varying 

192

. 

OG52-004-s ‘, 1 1 1 6 P -I-5 73 01 06 11 0 6 

I 

7309m 

A/D COSI’ERTER. \\7lERE IS TJJE PRE-OPERATJOSAL CHECKOUT i 

I PROCEDURE FOR TIJE X,‘D COSVERTER DESCRIl3ED? 

I 

I 

I 

1. A/D COSVERTER TECJJ. X~SL’=iL 

2. Fl3SI S-J-D. >IhJSTESzlSCE PROCEDCRES 

3. Ff)SI ST!Il. S>\1’JG:\TJO$ OPERr\TTSG PROCEDURES 

4 . NAI-IGATIOS SL-IJSIS TEN XASUAL 

Fiyrc 1 - SAMPLE TEST ITEhl 

: 

193 

3 

; -. 

-,. _

numbers of sub-items under each item (Figure 2). Obviously, we could 

not test each and every requirement within a reasonable length of time. 

Our solution to this was the System Achievement Test (SAT) which con- 

sists of both a knowledge and a skill part. The knowledge part of this 

test is generated using a weighted random sampling method. Each tech- 

nician's PPP tables are assigned weighting factors based on their 

relative importance to the overall requirements. Each PPP item within 

a table is also assigned a weighting factor based on complexity and 

importance in relation to the other items in the table. When a test is 

generated, the computer uses the weighting factors to develop a sampling 

bank of all applicable questions. Then,a random generator selects from 

the sampling bank and generates the knowledge part of the test version. 

The skill part of each SAT is generated manually after a thorough 

review of the DDS exercises applicable to the l?PP standards. Each 

test version undergoes a thorough engineering review by system experts 

and is administered to a pilct group for further refining prior to 

. 

issuing as a new test. 

Another type of test was developed to use in the training courses 

administered at the FBM Training Activities located at the Guided 

Missiles School, Dam Neck, Virginia, the Fleet Ballistic Missile 

Submarine Training Center, Charleston, South Carolina, and the Naval 

Submarine School, New London, Connecticut. This test, named the 

Course Achievement Test (CAT), presented a different set of problems. 

The PPP items covered by each curriculum are indicated in a chart 

which relates course objectives to specific PPP items (Figure 3). 

This chart, the Objective Assignment Chart (OAC), becomes the basis 

for designing the CAT for the training course. It was decided to make 

the CAT a criteria-rcfcrenc& test since the standards could be defined 

194

010 

100 

105 

1OG 

107 

109 

109 

110 

111 

112 

113 

114’ 

115 

116 

118 

120 

121 

122 

124 

126 

127 

131 

133 

\ 135 

137 

140 

145 

146 

151 

155 

189 

200 

204 

205 

2 0 6 

310 

315 

400 

500 

508 

540 

541 

600 

610 

620 

‘TRAINING PATH CHART FOR FIRE CONTROL TECHNICIAN (SSEN) PbSEIDON, 

FCS UK 88 (NEC FT.3306) (SHEET 2 OF 2) I TPC-Fl 

TABLE INDES 

FBM \Vcnpon System (Svstrm Level) 

Fire Control Svstcms - SI!i SS Slwis 0 nriii 1 rind Ilk J4 &xi 1 (Sul)system Level) 

Alignment ~ul~systcm - .\Ik 3s Sleds I) and 1 and Ilk %I 31~1 1 

Erection %lwvstcm - .\Ik -* Sloris 0 and 1 3nd 111; Y-I Nod 1 

Slissilc >lolic~n .-;ul~s~stcm - 111; 5- .\Iods 0 and 1 rind Nk Gq Mod 1 

Digital I{c:ltl-In hbsvstrm - ilk 53 Mods 0 nnd 1 and .\[I; 34 SIcd 1 

Master Clock nncl Timin z 5ubsystem - .\!I< SS .\Iods 0 :lnd 1 and .\lk 84 SIod 1 

Comm3ntl Slc\v Sul)svs1cm - !bIk >lr 1Iods 0 rind 1 

Control 3nd Displny 5ulw5tcm - .\lk S* Jlods 0 nd 1 and Nk 84 Mod 1 

Rate Compensation IAcctronics - .\[I, 83 Mod 1 

Automatic Fnll!t I:czistrxtlon Subsystem - Ilk 98 Mods 0 and 1 

~Iultiplcxers - .\lIi .:- .\lw!S 0 and 1 and Zlk 3-I Nod 1 

Digital Control Computer - Slk >* Mods 0 and 1 and .Uk S-I Mod 1 

Digital Control Computer to Equipment Configuration Switching and Patching - 

hlk 88 Mods I) rind 1 

Servo Group Mlsyqtem - ?Jk g3 Mods 0 nnd 1 

Fuze Set Pubsystcm - Xlk :i XIods 0 ant! 1 and Xk S-l .\ldd I 

Pnmllcl-To-Serial Converter - .\Ik SS: Mods Oand 1 and Mk d-I Mud 1 

Events Tr.nnsl:lto~ - Slk 2s Nods 0 3rd I 

Kcpbonrd antI Keybonrd Display - .\Ik 93 Mods 0 and 1 

Intercomputer Link - Nk 35 Mods 0 and 1 

JIngnctic Dir;k File - .\Ik 55 Nods 0 rind 1 I 

Com,Mcr Printer - .\Ik S3 ZIods 0 and 1 and Ilk S-I ~Iod 1 

Computer Tape Render .\Ik 1 .\Iods 0 and 2 - Slk SS Xlods 0 and 1. JIk 80 Mod 2, 

and .\Ik 9.1 Nod I v I 

I I 

Tape Ilcndcr Test SC? 

Power Distribution Suhsystcm .\Ik 68 Nods 0 and 1 and >Ik 84 Mod 1 

Test Subsystem -’ Slk SS Mods 0 nnd 1 

Test Instrumcntntion Adapter - XIk 5.J .\Ic& 0 and 1 

Digital Transmitter i?eccivcr - 3Ik 8s 3Iod 1 I 

Temperature .\lLmitorlns Potter Supply Ilk 141 %xl 1 - .\Ik 84 Mods 0 and 1 

Optic31 Alignment Group Sik :! Nod 0 - .\lk $J .\Iod.s 0 nnil 1 

Nxiulc Tent Set .\Ik -II:! Uods 0 rind I - Slk 86 Nods 0 a d 1 and .\Ik 8-I Mod 1 

Missile Systems (Subsystem Levei) 

Cuidnncc Subsystems ? i 

Gimbal :\sscmi~lics .\lk 3 Xod 0 and .Vk 2 Mod 1 and As’socisted Electronics 

Guidance Computer XIk 3 .\lod o 

Nissilc Tcs: rind Ilcntlincss Equipment \[I; 7 Nods I. 2, and 3 

Missile Test :mtl Itcndincss f:quipmrnt Ilk 6 Mods 1 and 3 

3Iissilc Iaclnchinz Systems tSubsvstem Level) 

Savigntion Suibsvstcms tSu!,system Level) li 

Multispeed Rcywter \Ik :1 .Uod 3 

Frerpcncy ‘I’imc :?nx!::rd ;I.\:. l?CQ 3, -‘3X, ;nd -2;; 

Optical Equipment 

\Vmpon System Ship Support Subsystem (Subsystem Level) 

Ventilation System 

Elcctricnl System 

Figure 2 - SANPLE PPP TABLE ISDES 

195 

. 

-

t 

- . . *. 

. 

.: _. . 196

adequately to support this type of test. Our biggest problem to date 

has been in obtaining stabilized, approved curricula for which tests 

can be designed. These curricula are now becoming available, and the 

CAT program is finally gathering momentum. 

Many problems arose in the actual administration of tile tests. Three 

distinct environments existed which required differw.t approaches to 

administering the tests. First, WC had the submarine crew during its 

off-crew period. I: Vas decided to use a dedicated Navy team at each 

site to administer the tests. This solved the potential problem cf 

having civilian engineers give tests to military personnel. Every 

attempt was made to avoid the "Big Brother is watching you" image in 

order to obtain the best possible,positivc attitude towards the tests 

by the examinees. 

The eecond testing situation was with the-submarine tender personnel. 

Since the tenders were located at remote sites, test packages were 

developed which could be administered by the tender personnel them- 

selves. This meant providing clear, easily understandable instructions 

for each test section and directions for return of the packages upon 

completion of the test. 

The third testing situation involved the CAT This test is administered 

by the instructor at the scheduled time in the course. For the CAT, we 

developed a simple proctor guide and a punched, overlay type answer key. 

These materials, along with the test, are provided by the local Navy 

team to the instructor on the scheduled examination day. This procedure 

prevents the instructor from teaching the test. 

197 

.- _‘. 

7 

i 

I’ 

_I 

@$s

. 

The System Achievement Test, because of its objective to test the 

technicians' total system capabilities, is a relatil-ely lengthy test. 

The SAT ranges from one J-hour Session to fez such sessions, dept>ding 

on the particular PPP/TPS requirements. Obviously uitb a test this 

long, we were concerned witi: the attitude CF -2.e personnel required to 

take it. Surprisingly, after the initial gr-amblings, cbe technicians 

appear to vie2 the SAT as an integral part of their training progran. 

Some technicians :..-vc even provided rccornendaticns to improve various 

aspects of the tests. 

The DDS exercises require the use of the sa=e technical docoinentation 

employed by the techn,cians when troubleshoczing similar casualties 

onboard ship. h'e decided early in the develcpment of the program that 

close monitoring of the change status of'thc publicaticcs would be 

required to insure currency of the tests. Tt was also decided that 

the local Navy team at each site should maintain its own technical 

library to insure positive control of the change status of the publi- 

cations used with the tests. This decision created ex-.rc: work for the 

teams but reduced the cries of "this won't Lark" during t!le test sessions. 

During the first year of testing, the raw results \;ere mailed from the 

test site to the central site for scoring. This created a delay of 10 

to 15 days from the completion of the test to the time the results 

were returned. The pro+jran managers in the S*Jategic Systems Project 

Cffi;a ji:act& that tiiis turnarOund time be reduced CO enabie commands 

to schedule remedial training, if the test results indic-ited a signifi- 

cant deficiency, prior to departure for patrol. An optical scanning 

device and a teletype machine were used to solve this problem. The 

Navy team processes the examiners' answer sheets thrcugh the optical 

2 

.

scanning machino and associated data set into the computer. The 

computer is used to score the tests and the results are ret~rncd to 

the appropriate team vi,a the tcletypc network (Figure 4). Turn- 

around time has been reduced to one or two days, depending on computer. 

availability. 

One of the largest prob1crr.s WC encountered in program development was 

how to best score snd analyze the test results. One goal was to deter- 

mine individual strengths and weaknesses while another goal was to 

identify and correct deficiencies within the training program itself. 

Ke developed a "quick-look' SAT report to meet the first requirement. 

This report is based on tbc.exam.inecs' results compared to the existing 

fleet, or total, results contained in the answer file. Individual 

training recommendations are also contained in the SAT Report and are 

based *on Z-scores greater thdn 0.5 bckw the fleet mean. This criteria 

was picked art" .:ly but has worked well in practice. 

A more detailed analysis is conducted to meet the second evaluation goal. 

This analysis includes individual and group evaluaticn (i.e., submarine 

crew, tender crcx, instructors, etc.). PPP table evaluation, teat 

instruxent evalcation. These evaluations are conducted after each test 

version is retired and replaced with a new version. The FEM program 

provides a uniqac envirxzent for many of these analyses. CoLlmun i. t y 

idiosyncrasies include (1) relatively stable crew composition, (2) 

detailed hardware pcrfornance data, (3) close liaison among all nctivi- 

ties, and (4) extensive personnel history data. These attributes 

enable crew Fcrfcrmance and empirical test validity evaluations beyond 

those which are normaily Fracticable. 

I 

199 

:. _ 

-..

c 

. 

Ccn1raI 

rcj: 

me 

(crs) 

LA- 

. 

Exmnfncc Tmt 

Answers- T”,“,“,“f,,“l t 

. 

Volco Phone, MscJ, Handcarry Service . 

FIgure 4. PSEP EDP timnunfcation Nehvork 

4 

-- 

-a _ 

.% 

-_ 

\. --. _

One fact becane readily apparent when crew scores were compared ini- 

tially. T?,e difLcrenccs in higher and lower rated technicians in the 

various cre\.s .::i*azLy ar‘fected the group overall sccre. A method was 

devised to determine an "expected" score for each group based on what 

the group scoLe ucufj. have been, had each examinee scored average for 

his rate. Actu31 3"d expected scores are plotted on crew trend analy- 

- 

sis graphs anJ assAst greatly in spotting significant deviations. 

During test item analysis, those test items whose characteristics do 

not met f2stat~isi.e~ discrimination criteria arc automatically identi- 

fied for rr~i~.~ .tr,d prir:tct otit from the camp-zter's test item file. 

Thcsc test items are than reviewed by Central Test Site personnel to 

deter,:ine if ti.cy ;hould rc deleted or revised, or if they provide 

an in2icatloi: c“). ir :rJining problem. 

b-k rcccgnizcd i;.tr!y in the PTEP devclcpcental phase that exter.aive 

computer data rlocessing xould be required to handle the quantity of 

data Lo be ~out.~r~:oJ within the system. Accordingly, the following 

five major ccmputrr program sublsystems were developed (Figure 5): 

1. Test G!::eration S-!.svstem. The test generation subsystem 

includes the programs necessary to produce the tests. 

a. Test reference data &i 

F'le load and maintenance program 

creates and maintains personnel records. 

0. A L)eisor.;;el :report program retrieves and prints out 

the conterlts of:selected personnel records. 

I 

C. A pdreornel survey program prints out key information 

from all personnel records. 

. 

: 

201 

.- - --- . 

. -_. . 

. .

. . i 

f : 

. - 

! .-----L - _ _ _ .- .- 

- __ _--- ..- 

_--

I 

3. Testing Scoring and Reporting Subsystem. The test scoring and 

/ 

reporting scbsystem includes programs for data transsission, 

for scoring and reporting of test results, and for update of 

required pernanent files. !' 

a. An inptit transmission (teleprocessing; program enables 

remote transmission cf test data via optizal scanning 

equipment. 

b. Test scoring and reporting programs assemble test data, 

score the data using stored ans'**er keys, update fleet 

norm files, and prepare reports. 

C. An output transnission (teleprocessing) program transmits 

test result reports back to the originating test sites. 

4. Test Analysis Subsystem. The test anai.y:is subsystem manipu- 

lates acccxulated test results data to piovide infonzation 

used in evaluations of t,Csts, personncl,\and training. 

a. 

b. 

C. 

Item statistics probrams compute at&activeness indices 

for eadh test iten response, discrimination index for 

each test iten within an overall kno 

ledge test part, 

and suary difficulty and discrinini: tion indices by 

knowledge area, TOS level, etc. An/engineering 

I 

rev~2w 

report is also prepared to list thoke test items failing 

to meet a specified value of the discrimination index. 

A score analysis program computes /i; ttndard T-scores for 

each exaninee, computes actual and expected average 

scores for each examinee group, and identifies exm,inees 

and groups for later, manual evaluation. 

A skill test analysis program recomputes skill exercise 

scores based on final performance data accumulations and 

computes summary statistics for USC in evaluation activities.

5. Query subsystem. The query subsystem provides flexible access 

to the various files to select, retrieve, and report 

information for evaluation and management. 

The majority of problems experienced during development of the ADP 

support systecs were cormnonplace; the normal qrowinq pains, problems 

with formats and record contents, and so on. Pertaps t2.e most 

interesting (and frdstrating) set of problems occurred in connection 

with transmission of test data from the test sites to the computer 

faci'ity. Our cquiFnent provides Wrrrcte access via “BTFJlI”, Basic 

Teleprocessing Access Method. Ice prcscmed, zs did the vendors with 

whom we worked, that the remote access would be the least af our 

problems: you hook the pieces togethet, write a Little program, and 

everything works. Uifortunately, it did not fall together that easily. 

If you have been in u similar situation , you probsb1.y have experienced 

the problems, sometimes hunerous but more often maddening, that we ran 

into trying to obtain information, and to purchase and coordinate 

hookup of equl;:-ent from the telephone company, the vendors for the 

mark-sense reader and the data setst and the computer manufacturer. It 

appeared that we xere reinventing the wheel. The system finally jelled, 

and today we have a fairly smooth remote access capability. 

Most of the PTE? ADP scpport programs ale fairly typical COBOL appli- 

caticns. One, however, is not: and I will describe it briefly. As 

data tecji4.r. LO acc~i~ulhitl, +oyir,naiuraiiy began wanting to use it. 

We had planned to create a xbspstem similar to the Management 

Information systems so much talked about in the past four or five years. 

204 

: 

.‘* ‘. 

- :

-. . 

Unfortunately, we could not precisely define our information require- 

ments: we were accumulating quantities of data to which a seemingly 

endless number of questions could be addressed. 

We settled eventually for a general purpose "query" system. This 

system can access any computer readable file of fixed-length records, 

including, of course the PTEP files. It works in three stages, start- 

ing with a request for information: 

1. A man processes the original question to define a logical 

procedure for extraction of information from the file, 

or files, that contain relevant data. His function is 

essential; he designs the algorithm to extract the 

desired information. 

2. A COBOL program reads the algorithm and writes the imple- 

menting program (also in COBOL). 

3. The new program, which is unique for the specific question, 

is compiled and executed, producing a report tailored to 

answer the original question. 

The system capabilities are extended by allowing the output from one 

question to become the input to a second question. Basically, if we 

have accumulated the relevant data, and if the question can be resolved 

to a set 0E logic equations, the query system can extract the needed 

information. One major advantage of this system is its flexibility. 

We can change our file structure around without losing the ability to 

readily access all data. 

205

Looking to the future, we are working on new analysistechniques which 

will be required as the CAT program develops. Since thgse tests are 

criteria referenced, 

i 

the traditional difficulty and discrimination 

formulas contained in our test analysis programs will not apply. 

Several new types of test instruments are under investigation in the 

hopes of decreasing the testing time and lovering the cost of acquiring 

new test vehicles. It is anticipated that the lessons learned in 

development of the PTEP for the Poseidon program will provide a sound 

footing for the forthcoming Trident program. 

206 

. . . 

! 

I 

.

USITEI) STATES ARW VAR COLLECE 

~csrninLq, vaiues , attitudes, Lbili: l Lxod qrndtng acadecrc sys:.?m at the Arm-; War Coile?pc, the 

-;,‘,-,501 .iL t;lc to? of the !.iZy SChOl system. Tne rationale ior :he 

L;I;.CCR (eac’r. year less thsi. 4;: of i!le el?gibLe officers are seicc:cd to 

;::czd :1-.5 W-r College). J:ese sruZer.?s do not rcr;u:re “he motivational

I 

.._ 

Lilt group oi hip:i achlcvc;S t!lere ij obvious and ?OSiiivZ corraljtjon 

t,etveen grading with resultant “rl?lative standjng” and student ar.xiety. 

ihc W;rr Collcgc experience prov idi% Rn educationai opportunity for 

ilie stujc~~t to rcilec: and lenr:] in an unstressful cnvi ronmefit wi LII 

3 cinfmm amount of szute;lt Anxiety-hence, no Eorzsl:zed grzlin; 

System and no relative stanGing- cr ‘?.oncr g.-Gduate.” 

Since the mission cf the t‘s Army liar Coilege is to educate and 

since learning cannot be asco_:,Li~Iicd k’i;t:ou: feedback, the :nsk is 

:G pro-ride inventory ar.2 feedback !>ut nt the same time minimize student 

nnxiecy. The 11 ew inventory 3nd assessment program nttcnpts to provititi 

;~er~onal and confidentiai ieedb.lck to the student without :he inf(:r- 

r.ation bring part kf his cfficinl rccorcis, 

:xmbcrs of the experimental R:oop ;o oarticinate ii1 Lhr new Invcntorv 

and Assessment Program. Four xacukry scn’oers have been assigned ;c 

work wjth the 60 studen:s wha wiii participate in camprchcnsivo personai 

inventories 2nd professional nssesssents. T”ne assessment is accorql ~shel 

through the use of five separa:e asscss.;.ent technjques: 

a. Baseline lnv2r.tory 

b. Opt ions I ?ersor.al IWCtOry 

2. Trofessi~ncl Se;“-.:ssz:.sz.:nr 

! 

. 

: :‘. 

” I.

. . 

E;se~rne !i:ventorv 

'Tllis is a battery of measures designed to describe!for students, 

di*vciopmcntal coaches, faculty, and the Army the skfllJ, abilities, 

f 

and characteristics of the senfor army officer at a point in tine just 

before he enters the lzs: anl ili.giISt echelon of fcrmai executive 

, f 

acvf~opaect, the i~S.hKC. 2:e student must be given e fairly comprehen- 

sive ;;icture of his owir. skills, abi'iities, and characteristics. If 

ire is to be encouraged to develop himself proiessionally and personally, 

he needs at least R qtiasi-rstionai basis for determining which areas 

to work on; and in the interests of attenuating anxiety, the student 

needs to know how he stands in skiils, abilities, and characteristics 

in reiation to his student con:cmporaries. 

A second an~illnry rat ionrlc for the baseline Inventory relates 

ciircctiy to t!le sta:cd goal of "tailoring" 

!’ 

I 

the educational experience 

:o fit' tile iniividual studesit. The 3aseLine /inventory provides, in 

I 

cssencc, a recoxaissance Of :ha incoming ciass and of its individual 

members. The reconnaissance permits--assuming a reasonable degree of 

pedagogical flexibility--adjustments in the ;Irriculum to meet the 

measured needs of the :iass, or a,f specific indlviduals. In tiie p.ahe 

. . 

0: tar,orin:, for t!12 indivl

variety of areas related directly 0: indirectly to the curriculum, 

The inventory can answer, for exampie, a question such as: "Does 

this student know the field of human reiarfons weli enough to write 

a book, or, are the prl*lcfples and terminology of tfie field so unfami- 

liar that he'mus: cxer: extra effort to read or c!iscuss the coursp 

material?" . 

The Baseline Inventory can achieve several inventorv and assess- 

n.ent 0hjectivc.G. The s tx!ent , conpnri::g his inc'isidual profile to the 

competencv range envelope of the ciass, has a rational basis for 

piannin;: his own program for proiessionai and personal develo?ment 

cc .g., attend workshops for "->eak" areas). The studen:'s counselor 

0: co:1ch, StudvinE the profiies of his counselees. has a more o\,jcctive 

basis for understanding and coaching his counselees as unique inciividuais. 

The faculty , studying the group profile and comparing it with baseline 

measures of yevious student classes, h3S an objective tooi for.curri- 

culun modifications which recogntie dliference in classes from year 

.'o year. I.S\W , studying individual. profiles can acconpiish (with 

I 

cornouter support) several desirc5le objectives: (1) :aflcr s:udcnt 

committees toc;ard equivalent Skiii composltlori; (2) identify students 

who are weak in .? number of areas and require careful coaching programs; 

(3) identify, for recruitzcz: ES perzai.e;lt fzc*d:y or for use as associate 

1nstructcrs, rixse s:

?hc first Hasc1ir.e ;nve;rtory was xailed to A’. ?L students early 

in the year prior to the time ttey reported to the College. The 

inventory was scored and returned to students is “Feedback.” Addi- 

t ional Iv :!K i.1culty was p rovidcd with pertinent data as a group result 

cjf ti;t* ir.vcnrory to obtcir. 1 (‘i;;ss profile. 

-7’ 

. r;e re are sc~~.~crai rcco ;sizea shortcomisgs of the Sssel ine Lnventor1 

izscruxbzt. Lath al-e :aiated ro tI7a fat: that :!IE ins:runent ts a 

self-invenrory-- the individuni assesses his own leveis of skill in 

areas as deflncd by the instrument. WE can assume that some individuals 

will intentionally oversta:c their own capabilitles. We can assume 

T.iSO ttls: SOrne indivlciuais will zisintcrpre: or no: fully comprehend 

the impiica:ions f-f ttie area descripticns. These possibilities are 

1>,>rze in lind when intcrprct:nx .GivlduaJ profiles. tiowever, for 

.iy,~if2i',iltr prOfilL?aS* the .L,ortcom:xfis a r e Less sl ct’icant l~eca:~sc 

.:rror.s occur in both direct ions; e.g., so:e indivluuals will understat 

ir,tentjo:ally their oxn ab,lltIes. The variatioxs also provide the coach 

with one indicator of tSe stcdent’s seif-concept. 

Fur:hcr . while we have thus far (for purposes of illustratioc) 

discussec t;lL 3a;eline inventory is a sfagle instrcment, we shouid, 

in reaiity, consider it as serres of reiated data collection. tests 

and sur;eys . Initially, we beEan with the 8zse:ine inventory designed 

aro;;nd currlcuiun objectives. 2ver time, we wi>l substitute validated 

and es t ~:lis;Aeti measureme;.: inscrcxen.cs Por portiox cf the Baseline

tr**is avail,;h:tb for measuring rcac!!:\.: :Jmpre’!:cnsion, listeni?.; ,.,~~.pr~- 

!l,*n::Ion, and writing skills. Ad ~!\a Educational Testing Service, given 

~le.rrlv-stated learning objec:ives .ntd supporting course material, 

,*.\;I ‘love1 op a c;RE-~ ike conprehcns tw \%smtna:ion with alternate ;ram, i s cesigcec t5 :iSSiSttne stuc!er,t Ln this dif:‘icult 

i+lCrox;xctivti task. The rr.~~t.doioj:~ of the OF1 is far iess strsctured 

:n.w that of the Raseline Lnvcntor::. Ctcreas the Saseline Invent,~ry i s 

\wwuctcd prior to tne student’s arrlvs! at the LSXKC, the Opticr,al 

i’ttrsonal inventory is dependent for 

t 

:CS success upon a sound student- 

co.r,:h relationship and canno: coxexc cn~il s.uch s reiationshfp hss 

be\*r; initiated. A ht:ery cf inve,::o:les are avafLb;e to students 

,I:> :;n o,,;zonal basis. ix;u~ed :n :;ie :a:tery acz ixtr-menus to zca- 

;:I;,*~\ :~:.tituces. F;)ezkir.q, :e..~:.:;; :cC; u:;cL~g ability, pers.onaL~ty, need.+ . 

\\rc,\t:ona; interests, (::\I. ‘i.- -.-AC& '--al cc.x~~sg s~sskcs with incczing 

I 

212 

3

?-hcrc arc twa sources of evidence which suggest the strsr.6 

potentid of the Wi concept. 

Itt the executive development field, one of the most pronis inE and 

rapidly-&rowing :d:hods of developing executive potential at higher 

lC!VC’%-. i s the assesszen; center. Esecctives v i s i t these cer.:ers .I::d 

particrp.z:e i:: 3.. “3ssessmnt week,” whereix aeasLrcs,fobscrvnt LXS 

isire rak c f varims skiils and characteristics whkh have b~::e:: fmnd 

tL: be ,:los.aiy re:ared to esecucive proficiency. An incezsivs “I*tid- 

back” stxssfon :‘uila~s the assessment week. &ring :i:e feedbac:; se8 ion, 

tLe zpccu:ive is 

. . . siven objective and professional in:crpretacion cri 

t?.e t%‘ilSUL cs made luring assesscent week. This is privileged inior- 

. 

c;lt:cn; t:.lc staff ss’essor is Cot a n?emoer of ;he parer.: organiz.:lon; 

azc :h+ .ndividu;rl csecutive is free to do what hc wistles wi:h the 

reed3z:ck iaforrzaticn he is give;: (writter. and grap.:ic Z;iCrlai. LU$jo 

ad vldt-o:ape cassettes). The L’S Amy icvesrigatec the ?c

WUork-up” (scoring, interpretat ion, grognosk, pr$cription) ai the 

avsi Inblc Jat.1. The s:u&nt grou? was highly enthuslastfc .i?~out this 

Y 

in-dcbrth objective excrc Sac in self-rkvelopaent . ‘{here was virtually 

t~;~~~imous a?:rcwwnt that it was ext rmely valual,L-t il. :crzx iri ;rerso:ln; 

;7roiessidn.ii self-assessrrent is to izdividualizc feedohck to the :;tuticr.: 

re,:.?rd;r,g t;:s attexnpts :L* iearn, cind his personal, Trofess :k-:,.r. ar,d 

4 edlershi; Li-:ciopert: to ?:OV:JC ftAt*ack to :hc instructor rcca:dinr 

.

and ~413. if presenzed with :he examination ques ticns -I:, an official 

capoc i:y, would respond as indicated on the solution sheet. 

'111~ studtxt, hy self-assessment, compares his answers with those 

cn the solution sheet. lie i3 then given the opportunity to discuss, 

cA?ha:e, Or.SUppJrt :he coapc:.e2Cs Of ttie expert Solution--oreferably 

with ;!le cxgert himself. 

From this cocpari son and Discussion prozcss there ur;doubtrdly 

!n: indivi,! ualized disc-ssion betwee> student and faculty. 

A: ;;he compietion of the exaaina :ian-coaparfson-discussion 

prGC?SS. tile stcdmt turiis his paper i? to the fzculty member. There 

is a clt>ar stipLiati02 :hat r.o paper be identified with the individual 

atudca:':. name. (An "apt ioiiri cr. Sipiing exacinatioc papers wou!.c! have 

cocrc ivc :mplicacions). The Turpose 0f this turn-in of csanination 

papers is to provide the facclzy with enpirfzal subject!ve feedback 

on ho= veil leaminp objectives were achieved. 

Experic::t.nl Assespxent 

Experiential assessment cz;:si.;r.s of Lhree s+arata assessment 

tasks conducted x!lfle the stbcen: iS levo:fn,: his entire educational 

e i :’ 5 t t t\j +rtlci?a:fon in r. ira-2 problem-solving exercise. The 

, 

hhsashxr.i .dr c,lir ?;las, : 2d ;2r.-22 ' 

i 

"experiential assessment" because 

iiii! CeS2trcl7 djectivc ai.d ct.2 grxp problem-so1vii.g method represent 

a ::od~l G: 2i.e ?rofesSion;i requfremen:s :ca; the student wili xost 

‘\ , 

* 

.- _-- . 

: 

_ , _ . . . -.- 

Q$i;ib

, 

I 

I 

. . . 

-‘- .- -. 

216 

\

\’ 5 i ‘cl e ;..: : .1 ?d invcarczv 

. vc :xiv, good evidence that change Occurs witbin the student 

Y 

drrr ir.2 t 1~ I’.SAKC \‘CB , r - we assume that choqcs ociur also in other 

t 

;trcns--in funct~onai, academic, ceographiz, ntxt intrrperson.31 abilities, 

rc r w.n,~;c. i:ut WC 2;~ ixrd,?ressed :o s a y jus: what hss change;, 

a+ kow xus!~ , .lnd i n W;U: Iizect;or,. 

3lC assessment cask of u!le Value Gained ;nvrntorv is an attempt 

fir ieZdr:i;;C tke io:al develo?menza: rff ect (t:poa the studenr) cf the 

yecr spnt at ;he US Arzy Liar Col?egs. The s:uleat needs this infor- 

. ..- -: A^ .-. Ordzi co r*v;cw cn,‘ 

. . . ..CIU.. -.. _ zaqrecizce tk2 p;c:essioca: :nd personal 

2veioyxcr.z broil&: 3.x,: oy cl:‘= cotoi rduc.3::0~31 experience. Sucn 

iievelc-- 

,...c;.:. ex:cnd;ng fsi tWOIUi ZAXC q&La: iox, 

fnc \‘nwiue ikir.ed Ixrntor!. ;s ever.+ .:,;t ical :o :irr War Co;iej!e 

i:S*ii. t cu: educational 

I 

sysich nas acr.;evec. 

. . 

* it!; currfcular objectives. 

;.,-.p reef fnccs ';a snswer wi:!-. 

“i;;?at sort of mar, does the JSdh’C >iOdUCe?' 

i 

i 

.I +: 

sane 3recis23.: ,.I: of:c.?-as Led qucscio-i, 

217 

I

%tLitlS:iCdll)’ significant Gilfctexes, on gairincs af data for the 

sam.~ individua1S. strongly suggest effects directly attributable to’ 

I ilc* i'::Aedi: \‘f.‘ilT. 

1~*.3;lrt;lKki,r, i f rc3 :: be .?.OGif:dj to include a larj7er Se,-;r.e:it ,Ir t!lc 

.s:iaer.ts a; t!w A:zy War CoLIege. Those portions of the grsAr:m 

ueter.r.ir.ad to be of marginal value wiil be eliminated. 

During ik? ,*cr.duct of :;le ;ro~;;as certain basic principles :.iuSC 

!!C :'oI:;w&. i'irst, :he stucien: parCicfpa2Cs xuai realize the pm- 

:; !‘ .-A -I i 'I,; ic>:..:,! :

� 

official records or reports which Identify toe individual. 7-h c 

relationship betvcen the student participant and the faculty coach 

must be a privileged rciationship. 

The ot!wr principies e.g., honesty, empathy, respect, concretcntss, 

.seLf-disciosurc, and inxdiacy, of an effective aelpin; program arc 

:r;so important. Since t!ie program is on a trial basis this year :ilel-e 

is SO,ZW skepticism and an~rehension by a few of the observers. For i::c 

most g~cr:, everyone participt:!ng in the inventory, assessment and 

feedback c:ior-c is enthus& and confident it vi.11 enhance the cduc:;- 

tional indcavors a t Lhc’United States Army War College. 

219

SUBORDINATE MTINGS: Iv’liP NOT? 

W. H. Githens and R. S. E-lster f 

U.S. Naval Postgraduate School I 

The notion of having subordi??tes rate their‘seniors is one 

that typically engenders a great amount of feeling. These 

authors became interested in the topic while teaching graduate 

level psychology courses to’military officers when we found 

that a quick way to pravoke class rtiscussion was to assert 

that a system for gathering subordinate evaluations of seniors 

seemed deserving of experi=lentat iO,l. Over the last several 

years, therefore, we and a r.umber LI tiur students have conducted 

some studies concerning subordinate evaluations. It 

is to these studies that we will now turn. 

TWO GENERAL SUKVIXS COSCERSIXG SUEORDINATE RATINGS 

ln 1971, Lt. J. G. Bloomer. i’s:;, working .with one of the authors, 

sent 1100 survey questionnnircs lo t!ic U. S. officers attending the Xaval 

Postgraduate School. Over 350 officers responded to this survey concerning 

subordinate evaluation. Lt. Bioomers’, survey Focused on the 

/ 

acceptance of subordinate evaluations wizen t!le,y are for the senior’s 

use only. Thus, his instructions read, in parit, ‘I.. .The purpose of this 

survey is to discover whether the s:udents of /;;PS feel that the military 

officer might be benefited b> a program such a’s this, in which he i.s 

evaluated by his subordinates, with the ratings submitted directly to 

him solely for &s own use. Ti;k responses obtained to this survey are 

summarized in TableT 

TXBLE I 

Results of Lt. .Johm Bloomer’s i urvey 

o f Sa\*al Postgr;lduate Stud ‘nts 

B 

Attitudes Concerning Subord;inate 

Lvalunt ions (X=350) 

i. 20 you feel such a periodic survey , y his enlisted men 

P 

would be a signif ican: help to theAjunior officer? 

Yes - - - :hybe If no, go to part III. 

Note: Lt. 31oomi:r grouped the number answering “Yes and “Flaybe” 

into “those iinding some merit in the progran,” since in the majority 

220 

-

Ii 

of cases the “Elaybe” answer seemed to indicate concert with the nechanics 

of the program, not necessarily the value. 

Total finding some merit in the program: 

Overall (all officers responding) 74.5% 

LT ;tnd below 83.1:: 

LCDK and above \ 9 

Army (of 10 responses) 80.0% 

Ftarines (of 22 responses) 45.5% 

2. Should srrch a progrxz be restricted to the junior officer only? 

Ses 15.4% NO 83.6% 

Xote: The most common response Lo "levc: it should be restricted 

tom wa; al: 

hel.pful all 

3. 

4. 

5. 

DO 

on 

below flag rank; however, m;lny felt such feedback would be 

the k“iy to cm. 

y,u feel that suctl :I system \;o:rld 11nve a significant effect 

enlisted morale? , 

scs 61.52 so 15.x:; Don’t know 22.7% 

If you -acre rated by your subordinates, would you considtl 

these reports: 

Quite seriously 33.49e Casually ?. .z 

9G.RX 

Seriously 57.4? 0 Other 5.3:: 

- - 

If you found such reports adverse, would you attempt any changes 

in your le;ldcrship techniques? 

Ses 72.72 X0 0 Don't know A27 32 

Note: This question elicited mxty comzxents, and in most cases the 

"Don't know" answer was a qualified "Yes". That is, after consideration 

of the source, her: LIIC program was run, whether they were isolated reports 

or a trend, etc., a Leader,ship style change might be attempted. 

6 

6. Do you fcei that Com&nding Officers might gain by having a 

similar evaluation m!tdc of then by their junior officers: 

Ses 76.4;: X0 14.0% Don’t kr.ow 9.6% 

221

Note: Lt is interesting to note that many who had held command 

commented that this ieedback would have been especially useful to them. 

7. Do you feel that such feedback might have been of help to 

you at any time in your career? If yes, when? 

Yes 87.9% :;0 12.1% 

::ote: The “when” in t.!:is case was generally as a LT and below, 

but many responded that it would help at any time. 

8. Do you feel that enlisted raters would conscientiously attempt 

an accurate evaluation oi their superior officer? 

Yes 74.2% SO 7.22 Don’t know 17.6% 

9. If such an evaluation system were adopted by the military 

services, do you feel that the raters should be restricted to: 

;:o restrictions (all levels of subordinates) 58.3% 

Petty ‘Officers only 34.6% 

0)ther 7.1% 

So t e : l-Ill2 “Other” iz this case generally was accompar.ied by comments 

suggesting a graded firading system; that is, officers and CPO’s grade Ci) & 

SO while Clli.ets and First Class rate their Department Head, or some similar 

groupings. ‘i’here is certainly some merit to this idea, if only to reduce 

the amount oi paperwork. 

_ .- _ - . -- - -.--_-_--_ .._ 

The results of this survey appeared Lo indicate two points conclusively: 

We quote Lt. Bloom~:r, “(1) There is a general concern for improving 

. . 

officer/enlisted 1 ines of ~rJmaun:cat lea. (2) The ovcrwllelmin~ majsrity of 

respondents indicated a high degree of respect for the judgement of the 

enlis:ed man of today.” The resider should keep firmly in minci, however, 

that 1.t. Bloomer’s survey had as n z&Jor premise that the subordinates 

evaluations would be for tile use of only their senior. 

A L about the same time, Lt. Clooaier was conducting his survey, Githens 

surveyed 68 USS officers ;lt tile Savy Postgraduate Scl~ool. Two of his 

quest ions involved the issue cf raLinps by subordinates. The questions 

� nd the responses obtained are presented in Table Il. -ihe reader should 

notice thaL in the instructions to Githen’s survey, unlike those to Lt. 

Bloomer’s survey, the use of Ltw data from subordinates ratings were not 

explicitly restricted LO USC only by the senior. 

222 

/ 

I

Results of a Sur.ey by W. Ii. Githens of 68 USX ’ 

Officer/Students at the i 

U.S. Naval Postgraduate School 

As part of an overall evaluation, what do you think about being 

rated by your subordinates? :: 

An excellent idea 3 

A good idea 13 

A fair idea 17 

A poor idea 34 

Assuming that peer, subordinate, self, and the 

by superior” were in cperation, what should be 

to the total evaluation? Assign a weight from 

system. The sum of L&W weights must add up to 

Rating by peers 13.g7* 

Rating by subordinates X.00* 

Rating b> superiors ?2.23* 

. 

Rating by self 5.79” 

I 

b i 

* ?ieans computed over sample of 68 offic,er/students. 

-.__ -. 

I 

present “rating 

their contributions 

0 to 100 each 

100 units. 

The results of Git!len’s survey seem to show that officers are less 

sanguinc.nbout subordinate r.rtings than did the reisults of Lt. Bloomer’s 

study. We feel that this difference in results stems from the psychoiogical 

sets given by the questions and instructions fo the surveys. Lt. 

Bloomer’s survey addressed using subordinate ratings as a feedback vehicle 

for the cognizant senior, while *:i::hen’s survey asked the respondents 

/ 

about using data from subordinate ratings :~hen evbluating officers. These 

results lead to the rather obvious cone-usion that / the use to which subordinate 

evaluations are put will influence the reception they receive 

from those who are evaluated. Thus, it is necessary to understand the 4 

fears and concerns that officers have concerningasubordinate ratings. 

lJ . _ 

In his survey. Lt. BloLjmer asked those who were opposed to the 

type of subordinate rating system he proposed to comment on ;heCr reservat 

ions. Table 111 contains a representative set of, t1.c ccmments that 

were obtained. 

._ - 

, 

:’ 

223 

I 

!’ 

:

TABLE III 

Representative Comment:; of Officers 

Opposed to a Subordinate Rating System 

as proposed by Lt. Bloomer 

LT "This system oi evaluation would turn into a popularity contest 

between'junior officers." Xote: This conxzent was the most 

coumion reason give? for the negative response. 

LT "1 believe face -to-face discussions are more productive." 

CDR "ll~~d~rrxint2s dixip! i3t2. .A cru:c!l for poor lenders who don't have 

the abili

I , * \ 

LCDR “I feel a junior officer is very formable in his early years and 

should be guided only by iiis seniors, not his juniors.” 

LCDR “The evaluation would seem to put too much emphasis on molding 

the junior officer to suit the needs of subordinates, when in 

fact the molding should come from the top -- providing of course 

that the superior is capable himself and takes advantage of 

opportunities to observe the junior cf f icer.” 

LCDR “One would tend to cater to the wishes and xhims of subordinates 

if only to protect his ego when he receives their ‘biased’ 

evaluations.” 

LCDR “Feedback should be welcome in any form. Lt enables one to see if 

points of emphasis are coming across. It should be entirely 

optional on the part of the enlisted men and should not go throug!k 

any chain of command, i.e., leading PO’s or Chiefs.” 

LCDR “The Lypical junior officer is too sensitive to the opinions of 

his subordinates. I+ trying to please a11 of them, having no 

way to differentiate between serious comment and sarcastic, 

he could ruin himself, in !lis oh-a image. He could lose whatever 

self-confidence he ilad built up.” 

LCDR “ I iee? the success of such fecdbisk would depend upon whether 

the recipient could take criticism, and if he could, effect 

changes where necessary, There is .i Jonger of an impersonal 

system such as that described bxoning d routine, meaningless 

exercise if the recipient canntit communicate its worth by 

viewing the feedback seriously and making changes where nec2-sary.” 

LT “Problem in m.iintaining anon>-mity Tomes ~JF when the officer has 

very few subordinates. ?lay IlPc’Ci t c> propose a system in wliirh 

evaluations are only made wfnn the officer has, for example, five 

or more subordin:ttes;” Xote: Some form of this comment was often 

made and it is probably true that maintaining true anonymity in 

a small unit would be difficult, it not impossibJ.e. 

CDK “The evaluation strould not be forced upon the subordinates. It 

should be optionaL whether or not you evaluate your senior.” 

LT “Since seniors 113762 a hard lime evalualing juniors and our fitness 

reports are a highly controversial topic. don’t you t Aink nn 

uneducated individual ..:ould have even a more d?fficult time 

efficiently evaluating seniors? 

225 

/

LCDR “I rton’t: t~~:l a .PO is qualified to tell me how to do my job!” 

Li “I do ttot hclieve that enlisted men, in genertil, have the background 

to evaluate tit? perforrx\ncc, since personnel tx~nojiesent 

is ot~ly j);trt of IJY job as a Saval Officer. Ziost cou:d ilL\t 

evaluat,\ my shiph:ndling ability since most have no experience. 

They c~u\J not ev>..u;?te my n3tr’riA. and f innncial mana~r‘ment 

since t\trt\’ do not i.ax*c the trail:ing I h;tvc, PLc., etc. 

LT “L do t~crt i~rl most younger Fti’s ctnd non-rated cxn would be honest 

Jnd objc~tivt> i n nuking out such cv.rluaLi,)ns. Also, I t-CC1 :L 

would lro ,i ifiicult to keep theso evaluations irom tk cixxerned 

oificer’ll superiors.” 

J 

Li “I brlic\*tib i11,tt Lhc average sailk-r working for a yortng .lcJ is not 

i’lturt! ~*no~~gh to tzake the type of comments which uould be beneficial 

.” 

LCDR “I think ;\rk2 whole idea is a bunch of ----! If you c.111 subject 

tht U’S. ,O the same pressures, hold then co the S~PZ responsibilities, 

~t2., Till3 they are in .A poai:ion co rate hit objectively, 

and not Iw!ore.”

Two Critical IncLient Studies Concerning 

Subordinates’ Views of Officer Ef fectivtness 

p,,lh of the studies to be discussed used the crit.ical incident method- 

,,logy (Flanagan, 19jS) to gather responses from enlistad men concerning 

effective and inrfftXtive officer perforxnxc. 

Tlw first study used enlisted personnel as.signLd to two Savy Jet Aircraft 

Atr xk Sc,a.idra:.s. Onto inntired sis XC:I rcprfsentii:g pay grades E-2 

fhr~uglk E-9 were included in the sample. T;w questionnaire form asked 

the r~~Sp~It\leltt t ,,--describe an oiiicer’s bch.lvior. in tlie situation during 

your !iavv experience r;lxn you considered XI xt ion on : he part of the 

-t)ffict?r 

for whom you vxked to be the besr resample of :~n effective Naval 

officer.” Essentially, the sxx quest ion I;AS then later presented, asking 

the respondent to describe an iwffectivci Xdval officer. The reader should 

notice that the questions address eftectivtiness and ineffectiveness in 

rather broad terms, and :‘xi such as unit ~,x~ls or mission accomplishment 

were not provided. These questions to the respondents should not hsve 

servtd to preclude responses concerning. say, discipline by the officer, 

or tl,c lack of discipiinc by the officer. 

s4 

‘0 

Effective 

Sav;31 Officer 

Human :ietds 

Ineffective 

Saval Officer 

34 

Profession31 

Compet encc 24 

227

9 

5 

18 

Effective 

Naval Officer 

106 respoc.!ents 

i 

Trust in 1 

Subordinates 13 

Involvement 

if in Job 

I’crsonal 

Characterist its 

Su Itesponse 

Ineffective 

Naval Officer 

.’ 

? 

7 

- 23 

iFrespondents 

To give the reader a flavor of the incidents obtained and categorized, 

the following paragraphs list sons r?scrrpts of representaLive incidents, 

Sensitivity to fluman Xc&s, fifeciive: 

An enlisted man was having problems with drugs; this 

CO set up a drug abuse programlwhich really helped 

. this man and some others. I 

I -. 

Sensitivity to Human Sc*&s, ineffective: 

m 

i 

After a plan: crash in which the pilot was killed, 

this officer conmentt*d, “z-en can be replaced”. 

Professional Competence, Eiicctivk: 

Ship was anchored in Hcng Kong when action had to be 

taken due to a typho

Trust in Subordinate, Effective: 

The department head gave me the assignment and left its 

completion up to me. 

Trust i n SuSordinatc, fncffect~ve: 

:.;y divisisn oificor caused me to’ lose a launch by ,insisting 

rit;~t the gyrs)s on an air--to-air missile wouldn’t 

be ~n~ag~.i when power was applied to the plailc, oven 

thoug!l all of us in the shop were csperienced ;wJ. told 

him ile k-,35 mong. He let us know he didn’t feel ilo 

could be? icw us. 

involvement in Job, Kiioctivc: 

During iarrier qualifications, pilots in three planes 

in a rrx Laid bra&s w!lile goi,lg ofi the catap.rults. 

‘iiw .iivi=ion officer SJV WC’ n~dcd !~lp fixing the tires 

so hc! pi:i iled in to get the job done. 

i’r’rsonal Ci;.3r;l~‘tcristi\-s, !neffective: 

i:, z.:: last ~qu:ulron, my CO wts a ~:t‘.ruy drinLen an officer, whom you worked directly for, 

did something that contributed directly to the successful nccompliahmrJcnt 

of your units mission. L?>;actly what did this person do that was heipiul 

229

to you or other parsons in the branch/division/department?" The critical 

incident question addressing ineffective performance was similar to the 

one above, but asked about something that an officer did that actually 

delayed or hinder& accouplishnent of the units' mission. 

The content analysis of the 269 critical incidents received in this 

second study was conducted by a group of officer/students working independently 

from the group who content analyzed the data from the first 

critical incident study. The results of the second critical incident 

study are summarized in Table V. (h’here it would not do violence 

categories found in the second study, their labels were changed to 

thdse found in the first study.) 

TABLE V 

Broad Categories of Responses Plade Concerning 

Effective and Ineffective Qaval Officer 

Performance - Study 11 

(X=289 Savy Enlisted ?len) 

to the 

match 

Percent of the Total 

Catecorv Responses in this Category 

1. Trust in Slibk\rdinates 26.8 

2. Professional Competence 24.3 

3. Involvement in Job 13.4 

4. Sensitivity to Human Seeds 1l.h 

5. Training of Subordinates 10.4 

6. Conmunicatiens 6.5 

7. Safety 5.0 

8. Discipline 1.5 

9. Over Famili.rrity with Subordinates @.7 

A comparison of Tables IV and V shows that there is a high overlap 

between the categories derived from the critical incidents gathered during 

the two studies. The second study (Table V) yielded five categories not 

found in the first study (Table IV): Training of Subordinates, Communications, 

Safety, Discipline, .ind Familiarity with Subordinates. The first study, on 

the other hand, yicld& a Personal Characteristics category not found in the 

second study. The nsthors suspect that the difference between the questions 

in the two s;urweys -- Kk second addressing mission success while the first 

,_ .._. .-.. 

i 

-.‘-. - -- . . . .._ .’ ._ . . .._... .

did not -- was the n~,jor reason the categories found from the two studies 

did not map even btttrer onto one another. 

Two categories in I‘able V probably require some additional comments, 

as Lhey might seem to s+rbstantiatr? scrm~ of :!le reservations held concerning 

subordinate ratings;. these are the categories labeled: Discipline, and Over 

Familiarity with Subordinates. The incidents under Discipline typically 

mentioned punishment of a group. For instance, “Punishment was given to 

the whole radio gang by rcscinding,spccial liberty when one individual was 

at fault far not delivering a Ceisagc’,” was one of the incidents in this 

categoi y. Inciden:s in the category labeled Over Familiarity with Subordinates, 

all roierred fo cases in Gricil junior officers bypassed senior 

enlisted r.:en in the ch.lin of comm;ruc! Uiicn dealing with enlisted men. The 

individtul reporting such an itzzide:!; app.-lrt?ntly felt tl~e chair. of command 

V;JS bypassed bacause the junior ufficr?r WS too familiar with some of the 

junior enlisted men, and that this hindcred mission accomplishment. 

_ AS w.rs the case oi t!,c first study, none of the critical incidents 

gathered dl*ring the second study sccotxd of a petty nature. Instead, they 

usanl iy address& periorn:uxcs : !~.tt thi> .;avy uould ci:her r;ish to reward 

or cxt Lnguish. S+ith.zr study providc‘d support for the no:ion that subordincLe 

rarillgs would be greaily inilucnced bv officers adapting strategies 

of ingrar iatiun in de.11 kg with Lhrbir sLbordi:latrs. 

Discussion I 

scars. 

Tile use of suhardinate ratings h.ts been discussed for mAny, many 

Dased on armchair analzrses oi the topic. it hss bren pointed 

out that subordinsccs hre inpa position to observe some of the performance 

of their supervisors. In m:uly cases titk subordinate is in a unique 

positiott which ai fords him opportunity LO observe supervisory performance 

that, is inaccessible to orhers. Assuming that more iniormation available 

c’oncer:ting various .~spc?cts of a supk*rvisors p+rformance will permit a 

better ev,jluaLit)n r*i his over nil performance1 it follows that subordinate 

rat ittgs s!~ulJ b+: used. But evc~ Lttot~~it this w:rs all pointed out long 

.tgo, subordinate r;f~i::gs have nut bwn utiliz *d in practI.ce. This non-use 

is cvidcntly based on the conceptions of a “ pularity game”. limited 

perspective, etc. 

II 

?‘Iw most signiiisant finding of our critical incident studies in 

this area, i s 1 hat Ltw bases used by subordjnat cs ior their evnluat ions 

of their superiors were not consistent witf d such conceptions. They were: 

for t!tc nust p;irL, based on aspects on which mo:-t supervisors t;ould want -. 

to be rated. The n.3vzl oif icers. who as part of their academic training, 

Sathered the critical incident information for us were originally suspicious 

.\f, fearful oi, and gcncrally opposed to, subordin3te ratings 

prior t0 their w,>rk in this area. iitcir switc!l in attitude after having 

gather-4 and nnaIysed r!le critical incident information was dramatic. 

231

Subordinate rating% of a sort, have recently received an upsurge 

in use in academia. Students’ evaluations of teachers or professors 

are no longer an uncommon practice and at least one firm, the Apex 

Corporation, uses subordinate ratings. The main resistance to their use 

seems to be based on possible misccaccptfons about what the subordinates 

would actually be racing. If this is so, what information could be 

generated to correct these possibie misconceptions? k’ould.it be necessary 

to have all supervisors do studies in which they gathered critical incidents’? 

With out recent renewed attention to the “people” aqect of organizations, 

subordinate ratings may be mortl important and possible now than ever before. 

Since our critical incidc\n: studies argue strongly against the 

xeasons usually given for not halving subk>rdinate ratings, the question 

should again be asked: SuiZrdituttc Katinrrs: Lay Sot? 

, 

f 

i 

1 

232 

3

References 

Flanagan, J. The Critical Incident Technique., Ps~cl~olpgical Bul.l.etin, 

1954, 51, 327-358. 

Lutrell, H. Performance Appraisal at Apex Corporation: Discovery -- Or 

Delusion? Sational hsiness, >!arch 1?72. 

. 

233

Thc~ Kecslcr Study - Electronic Technicians Four year 

Evaluation or’ Turce Types of Training 

r’ 

Virginia Zachcrt. Ph.D. Consultnnt ’ 

Fledicnl College of Ccorgin 

Augusta. Georgia 

r 

I 

(For tin> 1973 Military Testing Associ.!tion Conference 

San Antonio, Texas. 31 October l9i3) j 

..i : 1. 

i f- 

IXTRODCCTIOS 

A Dcp:irtmcnt o f 

:! 

Dciensc servicetc’st program W’.IS cst;lblishc~d i n 19S5 a t Reesl(tr 

Air Farce Bnsc. Xississippi, to explore ::.lys to reduce, the* tin,1 and cost of training 

non-prior service airmc>n to p,lrforn as cltictronic cquipmcnt repairmen. mo occup3t 

ion3 ! specialists and two narkcd ly di ffercbnt ;Ipprc.lchL\s to training were compared 

with a control group. During the year 196b. 441 gradu:it~*s of the four expcrimontal 

course’s were m.rtchccI with 294 control graduates .,i corresponding tx~gulnr courses and 

assigntad to field units in four commands. Job periormince data wt’rc collected throughout 

their first active duty tour and annlyzk-d to coap~rc~ their success on the job. .. 

There ~t’re f ivc major evaluation steps as fotlo6: 

TECRSIC:UES I’SED 

Xi tar-Course Profir icncy Test nt Xecslcr Tcchinicnl Tr;Lining 

Ccntc>r the first crc~ck following graduation. (Appendix A) 

First-Job Profic’iency Evaluation - first :cn’ weeks on the 

job. (Xppcndis 6) 

e 1 i 

Delayed Field Evaluation - nine months after completion cl 

resident training. (Append is C) 

Second-Job Proficiency Evaluation - n[trr two 

years on the job. (Append ix D) 

First EnIisLncnt Termination Survc*y - at 

initial active duty tour. (Appendix E) f 

Th-o following tecllniqucs were used to gather e 

Pcrfcrmnnce and written tests administered to graduatt-s. 

Job Proficiency appraisals by iicld supervisors. 

Nailed qucstionnlires completed by graduntcs and their supervisors. 

Field uisi.ts by cvaluntion ccam members. 

Survey of records at Air Reserve Personnel C~~ntc~r.

FINDINGS 

I. After over one v,t;tr oi graduate field cspcricncc. iindings indicated that: 

,\ccordittg’ LO ~~:~II;ICCY and thchir supervisors, training provided in 

Y the control (regular) courses is very closely nligncd with the needs of 

: field units and is compatib;c with cstnbli>hcd unit OJT programs, career 

dcvelopmcnt COIIL-SCR. and specialty knowlk*ds:c tcsLi.ng r~~quircmcnts. Field 

supervisors almost uncnli~tousl~ oppos~b � ny rrduction in the current scope 

and quality of residcbnt technical training. 

The shorter scrvicc tc st courst s \;ith drastically rcdttccd clcctronics 

fundantcntal s tr:iinittg (“:;” cocrsc’). vhil~~ ::i)t c~~~pi~*eely unsucccssfitl, rcsill 

ted i n b~.Ver;ti i:nporc;tnt tr3inin.C ~~iici~~~ci~~s rv vocitl i n their ndvcrsc 

crit icism of tit‘. “S” typl* cxpcritx*nL:tl training tilcty recrived. 

lht* SIIJrtL.: scrvicc> test co~trS\*s. A: icti cmp!tasizc~d ~‘i~ctronic 

flltld.l:Th’ll~;l~d Lr.tinittg bit; de-ezpit:ts iZ,‘d ? pltntt’t wubst;tnLi;tl dif ficul tics 

;tmong Lhcsc d~~ii~i~;tciC~s in g;radtt;tl,* it~)h proiic icattcy wc’rc: 

L;tck of sttfficicnt initi.11 skill in using published 

tcchniC.il d3Ln (tr~citnic,tl ,,ord~rs. m.titttc’n;tnc~* mnnuals. etc.) 

Ton 1 iLtlc vcrsntiliig rind skill in using stnnditrd 

test cqltip”cvlt. 

T o o limitc~d a knowlc~d~,~ t>i ~*lcctronic systems 

,>pcr;iL ian 3nd m-1 i ittc-nnncc t,.citttiqu~*s (this wns very 

pronouncr.d for nlrcr.?ft cqttiprr*nt. rcpairtttcn becnuse 

ctwy t;,,rca rc~sponaiblc f o r n.tint.tit;ing st*vcrnl diffcrcnt 

L\‘ix’s 0i cqtlipment ~vet3 during: ini tin1 job assignments). 

Gr;tdu;tLcs wt*rc niso v0rv crl.ti:.*r of tl*is training. 

Evidc*ttcc L O date, Al thou& inro3pictc. indicates that if field 

condiLions r~~ntsin tltc sxx~ ,zdoption ei Lhc shorter courses would: 

~cxtttcc ittiLi;tI j d b proficik%ncy. 

KcflccL ndditinnai t:;tininl: worklo;td nnd loss of 

produc L iv,* work into opcr.ation.tl ttnits. 

~ntl,~r prcst.nt st.?nd;lrds. irxreasc SET failures. 

the 

‘. 

t 

i 11.5 

per- 

. .-. ._ 

,- - : 

.

II. After approximately thr\bC _vcars of the Service Test Program. findings indicated 

that: 

The cxpcrimcntal graduates comperrd favorably with the control graduates 

after three years of job espericncc and considernblc formal training and 

job oriented tr:Lining. 

Experimental gr:lduatcs and supervisors wk*r‘c\ still critical of the nbbrevintcad 

type* resid~.nt trchnic;ll trilining rcccivcd b y tilt. cxpcrimcntnl 

graduates. 

If iicld conditions remain thr same. adoption of the shorter courses 

would apparent 1;: bL* Lhc same as stated c;lriier. 

ISFEKESCES THAT CAN RE.?I;U)E 

Tbc servic

I 

Append ix A i 

Sample of After Course Proficiency Trst (Pcrformancc) 

. ’ 

.,I 237 

..I 

-. 

i I. 

, i 

.

. 

! 

. a 

Appendix A 

Sample 0: After Course Proficicncv Test (Written) 

SA:.:f: At-SC 

C!.;\SS PATt: TEA.\1 . 

I 

30 ?X,T IR*TE 

:s ‘-PACE 

H t LO iv 

. 

:0 --.

. 

’ 

, 

.r 

I 

’ I ,I’ 

239 

Append ix I3 

S.1:np1~ of First-Job proficiency Evaluation (10 uwks)

Appendix C 

SXlp IC of Delayed Fit? I d Evalucltion QucsLionnaire (Graduate) 

__-----e---e.- --- ._I_.-- _ --__---.- 

._ __-.--‘v-- -----.- .- .-.- -.-- -_._. --.---.e---t 

i 

! 

sour -. ,,:.a’ ---- 

I: !, I __y---T.-.---. 

1 . . . 

. .._a ..- - - - - 

: ;

Appenarx n 

Sample of Delayed Field Evaluation Questionnaire (CrAduate) 

N-X = 137 N-C = 137 .I . 1 

� � �� ␛� �� rmripl*r 

DC Fundamentals 

Vacuum Tuber 

Solid State Devices 

Indicators 

Test Equipment 

Troubleshooting Logic 

Use of Han ,to.>ls 

IUse of Maintenance Publications 

Spare Parts Acquisition 

Other All omissicns not shown in totals. 

241

1. GRADUATE’S JOI PROFICIEMCI: bd,c.*. (k. 

,,.du’.t.‘e dr..sr.r.d r.fa.mcy I . ..&I.# 

*I+. ,rinr,,l.. ‘4 wch .Y D ,.ev - C.1.r 1. lh. 

Timetm 

tify) All ominsicm 

. . * 

.A --. 

242 

-,. 

.I. 

c

i 

Appendix C 

Sample of Delayed Field Evaluation Graduate-on-t~!eTJob 

Trainability Questi.orwaire (llnlt ProJcct Tecbn~c~an) 

-.s ; x-1:@? ::s = c-1 11 

b. Check tht* appropriate squar1.s in thcs table bylaw to rfilcct your best rsti;natc* 

0~ the gradL:a;e’s performance in the TECHSICAL TRAISING progratns oi paragraph 4a. 

243 

‘> 

> 

- ._ , . .c- _ - 

,. .. 

-:

8. In your judgmentwMch of the foUm COUTB~S in the DCD,~SLF Smvlce 

Tyt Prcgrm would you mwxmatmd for tmlnZng ftxture redar repaIm? 

1 

tell 107 ‘I cl "C" (Regular Cc&r01 Course) 

Amount of Uaef'ul i;ork 

Produced 

Qudityofuork Pro- 

Practlc 

Ekwlc clr 

ApDucati 

sh 

u3 

of Ten1 

08s 

i 

I I-J I - I l-2 L-2 1 - L.-i J 

I - I - I - I 

. :_ . . .- 

244 

_ 

F .: ,_. 

,

Appendix D 

Sanplc of Second-Job proficiency Evnlu;ltid? (Supervisor) 

*N=S-134 c-135 

/ 

1. SW~VISOR’S EVALUATION O‘i GZAMJAT”; J!B K?!Or&EDciE 

INSTRUCI?CNS 

2 Please check each kncwledfp item llsted’belou in the approriate 

column to Indicate your jua@Wt oi? the gr8dmU’8 subject knmledge 

1f .evel. Base the rating on yaw obaemtion of this grapte’e psrforaance 

)i rhile under your supervision. I. 

Scale 

Valup,D%INITION 

I -j 

1 

Xot applicable to or has not demmstrated his 

NA knowledge ir, his current assignment. 

f D ,m.LUATION. Can evalusts conditions and make 

rdecisicms about t.be 8Ubject. 

AKALYSIS . Can anal+yze facts and principles 

C and draw coriclusims about 

the subject . 

PxmCIPLES. Can exp:3In relation- 

B 

ikip cf basic facts and state 

general principles about the 

subject. 

FACTS. Can Identify 

A lacts and terns about the 

s;;bSect. 

TR;UI!TNG S’XiDA~D I ?‘!2!4 

1. . A c CfrCtit8 

s. 

----9. 

I 

sclid Slate Pc3vkes 

- - _.-__ y--.---, 

4. ,‘Vacum Txbes 

5. Oscillators 

_--- 

5. iieceiver3 Principles 

t- 

’ 7. Xotors and Servoxechti;m 

4 

8. Zaveshaping Circti ts 

___ -___-- 

3. Hicrau3ve Print i$es 

------- 

FJX’LIO:; ASP TkCTPXL U3E Oi-’ Acjl iU%R 

ZUIl’? 91 T 

13. Search Radar 

r 

1 11. Height Finder ?Mar 

I 

Il2. Cap Filler Radar 

:, 

; i

-- ----- - _--_- ..__ _____ 

Append ix D 

S.mpIc~ of Second-Job Ptoiicicncy 

Evaluation (Supervisor) 

COMPETENT. Can do a 

-- ____ ____~______ 

.l . MiitlEl9t9rs 

_. .._._ _.-_--- -.‘. _- -_ __._.. 

0 0 

.- _ _ 

.2. Vecum tube vgitmtsrs 

- 

0 

- 

0 

.3 � �� genbrst.'rs 

.4. Oscll2oszopes 

-.---- ---_-__ ._ ..__.____ ___ 

5. * spec*.r:2n an(rl32-“3 

3 2 

-I I 

___-- _____._.__._ __ ..___. _. ---_ ) ._-._- 5 

.6. Power amtars 

---.-I_-_ -_ 5 -I 

.7. cays+~l checksrs I, I 

- - - - - - 

.P. Tri‘nsistor ts.:terl 

- -__-...-- _._ .- -_ -_ 

‘0. Frequexy meaeurlng de’tices 

-__-_-- -_--.-------- 

4 .? 

‘1. 

- 

HTI ewiha%crs 

-__---._--.-. - i . .._... --__- 

‘1 

- - 

z 

- 

‘2. Z’ransponder t.eat a n t s ’ ,’ 

_-._.___ -_-_-_--.- __ _____ -_-_ -.;:- h 3 

---_ I- 

‘3. Select& L,he p:-oper tast eqtipmx,t and 

repair too'rs iLn mainkhir.kg Kii 

equlpn9nt 3 1 

_- __.___-- --.- __...- _ - ._._-_- - - 

'3 . 3ecognizes dafeztlve test 6q.iipaant 

� %rough indications cbt.a: r.c3 dur:ng 

operaticn > 1 

-.‘. - - 

246 

12 15 50 

3 

.--- - 

‘1 84 

- - 

18 79 

0 53 

4 65 

d-28 

8 5.1 

2 22 

9 

7 48 

4 13 

- - 

3 15 

8 58 

h 59 

!3 ‘16 

‘/I 

I2 9 

I.3 10 

15 8 

I.? 9 

!9 2’ 

I9 1-I 

52 4 6 

58 5 5 

!7 18 

- - - 

47 80 

77 7 5 

12 0 

I5 9

Xppccdix E 

Sample of First Enlistment Survey (Graduate) 

2. Lhte separated from the Air Force: 

3. Base of separation: 

day month year. 

. 

7. >lilitary pay grade a: thy tin.4. of reparation from the Air Force: 

OE- 1. C)E-2, (JE- 3 , OF.-4, OE- 5. OD.hrr 

5. Marital states: 

a.0 Single h. Cl M.arricc! c. 0 Divorced d.0 Widower 

6. S:rmt;er oi dcpcbrtdc-nt s: 

i. Do you h.avr any plans at prvsent fur rc~turning to active military duty’ 0 Yvs, 

OS& If yes. plc.ase lrst date. 11.1v mont~n year. 

8. arc you now a member of’an .Ic‘tivr rescrvc unit ? OYCS, OSO. 

a. . In year judgment. did the* t.*rhnical training yau received at Kreslcr AFB adequately 

prepare you for satisfactory career progression in the Air Force? 0 Yes, (J No. 

If no. please explain briefly. 

IO. Please check d the following statement conccrnin.. your employment: 

0 F‘uIl Time. OPart ‘lime. OL’nemployed 

11. If you are now self-cm~loyrd. give a brief description of your work. 

I’ 

-. If pan arc now employed 1,:; someone other than yourself. please complete the followi-g 

Items: 

b.0 

Type burine.ss 

c.0 Drirf dcscrip:ion of work 

13. If you are employed. does vswr job require any know!edt and e-xphin 

bricf?y. 

247 

.

I Observation: 

I 

c 

Appendix E 

; 

1 

Smplc of First Enlistment Termination Survey [Supervisor) 

N = X-83 C-80 

Is your rating hasrd upon a personal observatton 

performIn< this tar;k ? . 

Y 

of the graduate 

DF.r;REF.S OF SfPERvISIOS : 

1. 

_ . _ _ _ ” - . . 

PERI-;,RS!.\SCE I..ASKS i 

x’c !X2C x3c x4c 

2. Uses of t.-chr.ir.*I >lxblil.atinnr =:zch as: 

I IL 

I 

f 

t 

a. 1Virir.L: dit

PROCESS VERSUS PRODUCT XEASURES IX PERFORWCE TESTING 

William C. Osborn 


Consider the f ollovizg si:uations. 

titer having undergoce training ia task gunnery a soldier's proficiency 

is beicg tested 0;1 a g~~nr.e:y range. During the course of this test he will 

fire several nain gun rozds a: targets varying in size, shape and distance. 

In each case his score is deternixed by whether he hit the target within 

some specified-time limit, arid he is certified a tank gucner if he scores 

above some ninirnrul level required for qualification. 

Under other circumst2-aces, a soldier having uadergoce similar training 

may be evaluated differectly. Let's assume that azzunition is scarce or 

that adequate range facili:ies arc not available. Here the soldier might 

have to be.tcsted under dry-firing conditions. He would be required to 

take actual or miniaturized versiocs of targets under fire, and's tester 

would assess in each case whe:her the yzner 1) acquired the target with 

smooth manipulation of tte hand coztroller, 2) correctly ra;lged on the 

target, 3) achieved the proper sight pfcture, 4) squeezed the firing switch 

wfthou: losing the sight piciure, and 5) fired within some allotted time. 

Here tfre gunner is qualified if he performed each of the five procedural 

steps correctly on some zinimun nutber of targets. 

In the first situation described, a task outcome or product measure -- 

tar&e: hits -- is th6 basis for evaluating gunners; whereas, in the second 

i 

instance correct task p:&ed&xe or a process measure is ehe basis for evalua- 

iion. Tnough somewhat oversimplffied, the contrasting approaches to 

249

performance testing dram in these two examples illustrate the focus of this 

paper: the use of process versus produc: neasures in performance testing. 

I am chiefly interested in the us2 of performance tests to evaluate the 

results of training, and :o properly set the stage for what I have to say 

today let me first summarize what the training eval&tor considers to be 

the ideal use of product and process measures. Performance tests are used In 

training evaluation to serve trio purposes: (a) to certify student achievf- 

mat, and (b) to dis&nose weaknesses in th2 instructional system. In the 

use of such tests, Profici2ncy measures which focus on task outconco (products) 

normally provide data relevant to the first purpose, whereas measures of her: 

the tasks are carried out (process) pertain to the second. For example, the 

number of targets hit by the tanker trainee r=ould be the product measure by 

which his qualification ss a tank gurner is assessed. However, if he fails 

to qualify we would also Eke to know W;I~ - where was his training weak? 

This is where process measures are useful: if the gunner consistsntly 

. . 

missed :argets, was it because he ran&ad incorrectly, or obtained an irqroper 

sight picture, or r-asn't able to maintain the gun lay during firing? This 

type of data is useful in diagnos9.g areas of training deficiency, and Is 

essential in efficiently rsmediating trainees. 

3x1s we sac the roles piayed by product and process measures in training 

evaluation. Both are iqortant -- even critical -- when used for their 

respective purposes in evaluating the results of training. 

Product and Process as Xeasures of Stuiant Xchicvencnt 

k'fth this background i wsoitld now like to narrow the focus of my comments 

to the us2 of these types of p2rforLance measures in serving just the first 

’ 

_--. .-. -. . 

./’ 

250 

. I . ,s-. VII 

- 

- : 

a. 1 

'

. 

of the two evaluation purposes stated above - that of certifying student' 

achievement. ,Xn testing a student to dcteraine ifi,hc: is qualified to 

advance to the next level of training, or ultimately out of training and on 

r 

to the job, we Gould, as mentioned, normally prefer;. to use a product score. 

Before a man is certified as a gunner we would likk tohnve him demonstrate 

that he can hit targets; or before certification as a navigator he should 

actually demonstrate that he can get frcm point A to point B; etc. 

Although it may safely be said that every task has a purpose -- the 

fact of the natt(:r is that in practice a great many perfomance tests are 

used chich em?131 process measures o.:fy in evaluating student achievement 

or job readiness. Khy is this the case? Is the substitution of process for. 

product measurement justified? If so, when? If not, how nay the test 

developer inprovt: his methods? These are the questions that I will be 

addressing today. 

Product, Proces s and Tvpes of Tasks 

Before exploring in yore detail the i&sue of why process measures are 

I 

so widely substi tu:ed for measures of task produc:, it will first be helpful 

to consider three categories of tasks: 

I 

1. Tasks in cbich the product i' the process. 

2. 

-f 

Tasks in which the product a'kdays follows 

from the process. 

! 

3. Tasks in !:hich the product cnv follow from 

t!.c prucczx. 

P 

Relatively fev tasks are of the fir&t type, thGse in which product-and 

process arc one and the same. These arc normally tasks which serve an 

aesthetic purpose such as g}mnas:fc exercises or springboard diving. Close 

251 

_. I 

. 

‘. 

..- . . 

b 

._ * :: 

.-L- ---- . . . ,- ._ 

;’ .: 

: $. p: pt 

, 

:r 

i /- 

4

1: 

order drill is a good military exanple. Here we see that the outcone or 

product of the task is no more or less than the correct execution of steps 

in task performance -- that is, the process. 

?!ore tasks are of the second type mentioned, those in which the product 

invaria3ly r'ollows from the process. Fixed-procedu:e tasks typically fall 

in this category. Troubleshooting an electrical circuit, disassembling a 

rifle, and implanting a land mine are ex&nples. In tasks of this type the 

procedural steps are knob-n, observable, and comprise the necessary and 

sufficient conditions for task outco?.e; so if process is correctly executed, 

task product necessarily folious. 

A great man; job tasks are of the third type >*here the product is less 

that fully conditional on the process. In other words, with these types of 

tasks the process nay appear tc have been correctly carried out but the goal 

or product not achieved. This can happen for one of two reasons: eitf;er 

(a) because w are unable to fully specify the necessary and sufficient steps 

,. 

in task performance or (b) becaze we do not or cannot accurately measure 

then. In aizl-firing a rifle, far exaqle, WC are interested in knowing if a 

soldier is s:anding with his body properly oriented to the target, face 

properly positioned on stock, rtfle sling in correcr position, lead arm 

I 

perpendicular, acd firing arc1 parallel to ground; if be is breathing correctly, 

has a good sigh: picture, and squeezes the trigger. Presumably, if this 

process is followed the rifleman will hit the target. Assuming that we have 

identified all esseztlal steps in rifle iiring, and, further, that de can 

reliably measure their c,orrect execution, the:1 tbe task is of the second 

type described above and dock zot belong in Category 3. 

1 

However, in practice, 

we know that ou:: best efforts to cvalucte execution of this particular task 

are not sufficient to warrant substituting process for product ceasurezen:. 

; .! . 

, 

- 

_ ’ ----_ - . L 

t 

, 

I * 

-‘: 

: 

*( 

. . .

In ether words, somez1mes the target is missed even though in the judgment 

of a skilled evaluator the rif1es.n did everything rig%. Therefore, either 

because we are not absoluteiy certain tSat we have idez:ified nil recessary 

steps in the firing process or because ;re cannot accurrrely assess rhe execu- 

tioa of some of then we ultimately qualify s rifleman on the basis of w-nether 

he hits the target. 

In reflecting oa the r.nture of these three types cf tasks as iqortaat 

implication ezcrgcs rebsrding the role of product zeasurezcnt ir, tes:icg 

task performance: Because of the interchangeability of process ;;d prsduc:s 

for tasks of the first two zypts, it doesn't really mat:er which measure is 

used to assess proficiersy; but for tasks of Type 3, product zeasurezeat is 

very iiaportan:. Ix; 2iie of this., in praztice,perforzce tests for zany of 

the latter type of tas'ks do no= attccp: to nensure product. h-h;- is this so? 

Problem in Product Veasurecent 

The reasons largely stem from practical considerations in uhi?h the 

ceasuremnt of task pror’,uct is viewed as either too costly, too daqerous, 

or for other reas0r.s si-qly too iizpractical. In testf-zg such performances 

as hand-to-hand co-oat, for exazplc, rhere task prcc!u:t would t&z the fox. 

of disabling a hostile eceay, the fest developer is normally E&ted to 

requiring the denons:ra:icm of tack process. Similarly, in a first aid task 

like controlling the bieedixg fro= aa external wound, the person teizzg tested 

is, f;r obvious reasons, asked ol;ly to demonstrate the process. Or, in 

removing a jai2i2ed r0ur.d iros a vcnpon, it is cocsidered iz.?ractical to 

actually jm a round ic order to create a valid test situation, so agaic 

only the steps in task perfoxacce are measured. Man,- sudh exazplea are to 

, 

-. .-‘-- __- 

253 

- .., - . , 

.- 

. . 

-_

e found in the area of in:&personal behavior. One has to do with instruc- 

tor training, where at leas: traditionally the military instructor trainee 

is evaluated by having him prepare and deliver a block of instruction during 

which he is judged on such process fac:ors as: “stood erect," "had good 

eye contact with audience, )( "conld be heard in the back of the room," "used 

visual aids effectively, (t "covered all points in the lesson plan," etc. 

Although thc.pro&ct of instruction clearly is student learning, I believe 

i: is scldon if ever used 3s tke critericr: for qualifying an instructor 

trainee -- probably because 1. '* vould icvolve a more time cons-x,ing ar.d inprac- 

tical method of- evaluation. iu?other very sinilar exaqle which cmes CO 

mind pertains to a recrui:er's task of delivering a persuasive speech to a 

student audience. If the product of this task could be measured it would 

be in terns of the number in the audience echo later contact the recruiter 

with an interest in enlisting. i3ut, again, because of its implaussbility 

as a measure of student achievcent, product gives day to process and the 

. . 

recruiter traxze's persuasive speech is evaluated in -zxh the same way as 

va.s described above for instructor trainee. 

Dealin< with Pr&lerzs of ?rcc?ust Yeascrecmt 

I'm sure that those of you involved in performance testing can think of 

many more instances in which product measurement is not used. Clearly, some 

of these are justified by cost or safety considerations -- but others are 

not. hhich bricgs us to tke ccnrral point of my rcanrks today. I bcllcvc 

that test developers often faii to see the inportamce of measuring task 

outcome; or perhaps they merely slight chc iqortaxe when fated with prnc- 

tic31 litzi:a;,ons in its LstxCn:cnt. h%.l:cvcr tlrc motivation, I bclicvc 

they do not strive nearly kard cslough to overcome rcsL;‘yce problems which 

254 

3

constrsin attempts to measure task product, and too easily give in to the : 

simplistic approach of measuring task process. The overriding cjuestion that 

a test designer should ask himself in this situation is: If I use only a 

Trocess xasure to test a xxx’s nchievezzent on a task, how certain con I be 

frcn: this process score that he would also be able to affect the product or 

outcme of the tr;ok? 1;here the degree of certainty is substantially less 

than that to be expected fro3 coml ceasuren;ent error, the test designer 

should pause and reconsider whys in r;bich time sud resource limitations cm 

be cozpro=li;ec! in schicveing at least aa approxitation to product neasurecent. 

Although there will remain instnnces in which product measurement sinply 

camot be achieved, ue will discover oacy others where, through sorze inagina- 

tive thinking, we can devise s-.., i-ulatiocs that will enable us to assess task 

outccne in a note relevant fashion. 

Ir. testing the student instructor, for instaxe, I see no compelling 

reason c;hy we shculdn’t get away fro3 the "chdrn school" 2pproach to evalua- 

tion. Cny not sisply have hi-, corxduc: 2 brief instructional session for 2 

snall group 05 stndr?n:s iperha;ls his peers), ui:h his achievement beicg 

neasured in term of whether his students have accoqlished the instructional 

objactive? In the c2se of the recruiter traiaee's speech, evaluating task 

prodcct is core difficult; but surely a masure closer to task outcome 

could be achieved -- perhaps a ptid student panel representing the potential 

audience could be ezqloyed to v kw and rste the np?eal of videotaped trainee 

speeches. In zeroing in on critical cotor skills, such as those involved 

in extracting a ~axzcd round fro3 a weapon or fn coatrolling bleeding from 

a wound, it uouid see3 that relatively low cos: sizulators could be devised 

for use in tcs:ir.g :ask outcome. Had-to-hazd combat very likely represents 

a case in whicS ultixte task product simply c2mot be xzeasurcd. However, 

255 

,

in a similar vein, the Amy is now experimenting with an intriguing method 

i :, 

of assessing the outcome of an infantry squad combat exercise. The principal 

feature of the method entails each participant havfng'a number printed on 

-his helmet and an inexpensive scope mounted on his rifle; then, during the 

, 1 

course of the exercise a soldier may "kill" by correctly reporting an 

enemy's number, or "be killed" by allowing his numbe! to be sighted by the 

enemy. :;umber size and scope power have been carefully calibrated from 

empirical data SD that the probability of a simulated "kill" is highly 

correlated uith t'ne expected ou:come in acturl battle. Ynis is an excellent 

example of an innovative method of achieving product neasurrEent on a task 

that here:ofore had betin subject to process evaluation. 

Obviously, from these eszcples ve can see that the accomplishment of 

product measurement is no: always a simple matter; but it is a demanding 

and essential goal to be pursued by the performance test developer if his 

products arc to be relevant to real world behavior. 

I 

256 

i 

I

BRIEF CF 

FCL’R RESEARCH STUD?ES USItX 

THE SFLF-FVALUATICN TfCHNIQUE 

(SET STUDIFS) 

US Army Crdnancc Center and School 

Doctrine and Training Development Givision 

hJrFOSc of t.he SET Studleg 

The purpose of the four SET stxeies was to determine if student stlfevaluations 

of their performance tests improved student ptrformar.ces on 

required gerformance tests. 

Procedure of the SET Studies 

This section descrihrs the subiects, course requirements, testing 

it,;trumrnts used. hou the tests were administered, and how the scores vtre 

obtained. 

The subjects used in these SFT studies numbered three hundred and 

fifty-four Ftudencs vho performed tutnty-seven hundred and four performance 

tests. Tt-ese studtn:s vert enrolled in the L&C20 Vtldtr Course, 61ClO/20 

Fire Control instrument Repairman Course, 6LE20 Hachfnist Course. and 63C70 

FIJQ~ and Eler’tric Systems Repairman Course. These courses were conducted 

at the US Army Crc’nancr Centrr and School. 

Course Recuiremcnt s 

Gnc of the prereouiFites for each of thQ courses IS as fsllovs: 

Weltier. General “aintenance (G?!) score of 90, Fire Control Instrument 

Repairman. General ?!aintenance (G’I) score of 100, Machinist. Central 

Maintenance (C!f) score of 100. an8 FUPI an+ Electric Systems Repairman, 

Yotor !reak t?ous several of the performance tests so that some of 

the require? tasks coulci he tested at a prescribe+ ccsttnp station. A 

COPY of thic instrument is attache8 as Appendix E. 

257

For the evaluation of tht Fire Control Instrument Kepairaan and the 

Apprtnt ice Hachinlst a separate performance evaluation Instrument was 

designed to cover each of the performance tests. Copies of there instruments 

art attathtd as Appendix C and D. 

Testing Procedure 

Several classes were used to validate. all self-evaluation instrumtnt 

5. After the instruments wcrt validated, students who did not make 

self-evaluations were dtsignattd as the control group and students making 

self-evaluations were considered to be the experimental group. 

The SET test project dirtctcr matched each student in the control 

group vith a comparable student in the txptrimtntal group using the GM 

or .% scores as a basis for equating these matched pairs. In some 

instances it was impossible to make comparable matches for each of the 

matched pairs vher. matching the students by classes. The overall results 

on the total of matches for each course showed a very close mean for the 

matched pairs. 

The experimental groups made a self-evaluation for each of Cht ptrformanct 

tests they performed. k technical ly qua1 if fed grader evaluated 

each performance test performed by both the experimental md control 

groups. ‘rhtst evaluations wert forvardtd to tht SET study project director 

who scored the self-evaluator’s score sheets. the grader’s evaluation of 

the self-evaluators’ score sheets and the pradtrs’ evaluations of the 

non-self-evaluatiqns. The student’s self-evaluation and the grader’s 

evaluation sheets were a’ttachcd toEtther and returned to the students 

4th tither critical or complimentary remarks which were influenced by 

the scores they made on that performance test. X:0 score was rtcotded 

on the returned student or prac’er’s evaluation sheets acd the comments 

reflzc:cd only his weak or strcnp tasks on rht Ftrfornance test. The 

Fral’ers’ scores for the non-self-evaiuators wtrt recorded by the project 

2irectar opposite his match from the self-evaluators. 

The performance test scores for each of the matched pairs were used 

to test for a sign! ficant difference between the means of the control 

and experimental groups. 

For the purpose of evaluation the SET studies were designed to test 

for significant tiifierenct bervetn tSe grac’crs’ scores for students uho 

made se1 f-rvaluat ions and the praders’ scores for students who did not 

make se1 f-evaluat ions. 

258 

.

The secret of the set studies vhich I am going to present can be 

attributed to direct communication and automatic feedback. 

r 

To show you the importance of proper and correct communicatfvt 

procQdure T vould like to rtac! a copy of the registration form 1 filled 

out when I registered for this military tQstinp association symposium. 

I 

sow 1 will briefly cover the need for conducting set studies. John 

R. Carrolls article “XQRlQctrd Areas in Educationai Rrscarch” prrsented 

in numhcr 42. Flay 1961 issue of :tt:& Phi &lta Rappan s:atQd that 

“Research has told us littlt about the role of motivation in school 

1Qarninp.” ‘Lrt us take for Rrantcd that notivat ion is the sense of 

villinpnQss on the part of thr Learner to rppage in learning.” Studies 

of both intrinsic and rxtrfnsic motivation arQ nerdec’. 

S. L. PrrssQy and F. P. Rohinson StatQd in their article “Psycholo} and 

the New F&cat ion .‘. Informing? students rrpardinp :teir work is QSSWtit. 

for intQlliRtnt learninp. If rral lift tests are brought to the point 

where both stutient and teachrr can SQ informed about proRrQss in thr 

devrlopmcnt of social adjustments, interests and attitudes this would 

he splrnded. 

I 

The only study which I could find that ‘rven vaguely resembled the 

SET studret was located in thQ twcntp-ninth yQarbook of the Narional 

Society for the study of Qducation and was published in 1930. 

I 

THE DESICX M! RESULTS OF THE STUOY ARE AS F;OLLGXS: 

Two Qqui\*alQnt proups?of 35R Fourth grade pupils vert given identical 

arithmetic drill cxercisos for 15 minutes a!week for twenty-one uecks. 

The only variable was sprcific knovledcr of itorovement. The menbcrs 

of.the control Rroup never rccQivQd their veekly scores. Those in the 

PYpQrfmQntaL ClaSsPs kept int!iVidual ;roprrks charts and pooled their 

losses or Rains in a graphic rcpr.*srntatfon of improvcmcnt of the class 

as a vhole. ThQ rccor? of actual frnrrcvrm, dt 

was made possible by using 

r’rtll units for whfc?; rompara’\lQ stanc’ards $a2 been provided. T!:is 

scoriny c’evicc compcnsa:Qd for dtffQrencQ 3; difficulty of the tasks. 

ThQ Rronp yahict. had contfnuous informa: ion /conccrninF improvement 

mar’c sipnificantly prra:rr pains than the rcntrol proup. 

This in$tcatts chat if %Luitn:s are $novl~&able of \;hat resuiremcnts 

are Qxpccted of then ant’ rhev are made avpre of their progress there 

should bQ an improvQmQnt i? their overal 1’ pcrformancc. -. 

?lcase refer to the self Fvaluaticn instrument for the r-elders. 

(Arpc”c’ix A; 

Several classes wcrc USPC! to vaiida:e this instrument. ihc first, 

second an2 third instruments vcre discariec’ and this instrunen: Las 

259 

- . -. , 

: !, 

; .

adopted for the evaluation. illis same instrument can bc used for all 

pi$-ht wQl

Please refer CO Appendix E which shows the final results for the four 

studies. It shows that the self evaluation technique vas successful for all 

the studies conducted. A copy of the complere study can be obtained upon 

reai.es+ ty vritinp: 

Dr. John J. Ilolden 

US Army Ordnance Center h School 

Al-TN : ATSL-CTD-Df-P 

Aberdeen Proving Ground, ?!aryIand 21005 

261

_ 

AI’PFKDIX A 

! 

EVALUATION CF STUDCNT’S 

VELCING PRCJECT 

!’ 

STUDENT’S NAME 

HAkK AN (X) IN THE 

PROPER SPACE 

CLASS NO 

PATE 

I 

STUDENT SELF-EVALUATIC?; 

CRADFR F”ALUA&CN 

- - 

7:ZRCCUCTlCN : ihis is a romplct’e velding project vhich will indicate how well 

you feel you performed the important tasks for a 

VPld. 

The purpose of this experiment is to obtain illformation that vi11 

help us to improve the vrlding course. You c;iIl not receive o - 

grade on your velding project but an honest evaluation of your 

performance vi 11 be of a great help to US. 

DIRFCTICW : Read each item on the task list and mark an (Xc) in the column 

which shovs how well you feel you performed that task. 

I 

rolunn A Column 3 Column 2 Column 1 

Sot quite Far Rclow 

I 

Eoual to Performancc 

of 

Ecual* to 

Ferformance 

Fqual to 

Performance 

I the Perfor- --I 

mancc of 

TASKS PFRFCKNFI’ . Fetter than of Average of AveraRe Average 

Avrrapp Stu- Student; Student . Student 

r’rnt !.+c I r’er ‘Ie I de r .‘Jrlder a Kelder 

1. Cbtained rorrert flame 

adfustment (current sef t inp 1. 1 

I 

2. Chtainrd pnod fusion . 

(tfnnfnC act ion). 

3 . . Cbtafr.eZ pooti crmm ora 

wrlf! face. 

(L. Cbtainr’d poo0 penetration 

(bonding). 

I 

5. Rept head free of cracks. 

6. ‘Chtained even unifor,n 

/ 

head shape. 

, 

7 . Kept head free cf undercut 

fdrople:s). 

P. h’ert head free of overlap. 

ii 

Q. Kept bpaE free of burner’ // 

or Crvstall ttec’ meta!. 

10. Kept brad free of holes. 

Could finish projert in time allotte Y’r s :

APPESDIX B 

PERFORMASCE TEST 63c20 

DKLIVFRY VALVF 5i’RIKT. TEST 

STATICS 1 Checking I%1 iwry Valve 

Pate 

0 Student Self-rvaluation 

17 Instructor Fvaluatton 

1. Connectre the nczxle tcstrr at?? prrcate? tbr ~II-p for testinp the delivery 

va!ve sprtnps wi:h: 

D Ko trouble - c 

Received instructor’s brlp to ;rerfcm operation: u 

i-i::!* trouble u Lots of trouble 

‘ITS 0 SC 

if I’Zi is rhrckc?, ho-.. many times t’i8 tt*e instructor help? /I 

1. Chservec’ 0penir.r pressure of all $01 ivery valve springs. recorded reddin?s 

and determiner’ ronditior. nf vaives: 

9 . 

m C valves corrertty 0 6 cr 5 valves correctly u 3 or less valves 

corref t ly . 

Rccrived inst.ructor’s !brl~ to perfcrs operation: D“ xl. 

lf YT5 l= cbcckcc’, ho0 many tines Z.iZ tbr instrurtnr help? 0 

iverall performance for *i>is ctatiop: 

0 ASow averape stur’ent perfcrmance D 

Fqual to average student 

performance 0 blo:’ avcraj-ctstu2ent performance D Far l~rlow average 

atuc’ent pcrformancc 

.-%, .-- . . 

. 

f . 

f .’ -, . 

‘I 

, .a 

.I. ., 

.d ~b 4 i f 

263 

3

Student’s same 

Class 41 

7. 

3 . 

1. 

L. 

APPESDIX I!

. . 

Student ’ s Same 

Class I’ ratp 

1. 

1. 

3. 

AI’l’l?‘SCIy E (Cant) 

PERFORMASCF TFST 63C20 

I 

f 

STATICS 7 Timtnr the L’nit Injectors 

.I 

D Stut?e?L S e If-Evaluation 

n Instructor 

rosttionec’ cnpinr f o r timing of the! selected injectbr correctly: 

rvaluat ion 

n ho trouble m Little trouble D Lots of trouble 

Keccived instructor’s helIe to perform operation: D YES D :ic 

If YES is checked. hw many times did the instructor help? n 

Selected correct timinp gape and ad.justed injcctor CO specifications with: 

n So trouble n Ltttlc trouble n Lots of trcuble 

Eeceived instructor’s help to perform operation: 

If YL’C is checked, hov many times did the instructor help? 

(:veraI 1 performance for this stat ion: 

0 Above Bvrragr stuCenc performance u I!qual to average student 

I ..’ 

performance DP~lov avrragc student perfoknce a F a r belw 

m I 

avrrape stu*ent perforranre 1 

Tic! you have cnouph t i-e to cot,*lete the test required at each of the 3 sta:ions:’ 

0 YiC 0 SC. Tf ancvrr i s l+-rrtc ::O. reason(s) on back of this sheet. 

265 

I 

1 

-.

APPENDIX C 

SIXTH PERFOR~NCE EXAMINATION 41c10/20 

Maintenance and Adjustments of Panoramic Tclc~~~p~ HI15 

MARK AN (X) IN THE 

I: STUCENT'S NA!! PROPER BOX 

CLASS NUMBER STUDENT SELF-EVALUATION 0 

PATE 

GRADER EVALUATION 0 

PURPCSE : The pxpose of this self-evaluation experiment is to obtain information 

that will help us to improve the fire control instrunent repair course. 

You will not receive a grade on your self-evaluation for this project 

_ but an honest evaluation of your performance will be of a great help to 

us Co evaluate the course. Your course grade will be determined by the 

score you obta’n from the department’s performance examination. 

DIRECTION5 : Mark a J in thr TOLFRANCE RASCI: CC!lJ'MN for the TOLERMCE which YOU 

obtained on each of the listed dimensions to be checked. 

,“L,,nT\,.“L ..a..-.- 

CIHENSIONS 

TO BE CHECKFI' Column 4 Column 3 Column 2 Column 1 

Perfect ;;ithin Uithin’ Over 

1. Parallelism of 1 Mil # Hi1 

reticle and FW 0 

Perfect 

0 

Over-Under 

I 

Cver-Under 

E 

ever-under 

l/S Hi1 h Hi1 # nil 

2. Mapnificetfon o f FCV 0 0 

Perfect Within Hitnin . Cver 

1 Mil # Mil -. # Hi1 

2d. Collimation 0 Cl-J cl 

So Paral 1 ax 1 Hi1 # Mil Cver 3 xi1 

4. Paral lax El 0 El I 

Perfect Within Vithin Over 

1 Diopter # Dtoptrr Ii Diopter 

5. Eyepiece Focus 0 1 0 I I 

Within Within Within Over 

6. Adjustment of 1. Mil 3 Nil 3 / 4 :*I i 1 3/t Mil 

Counter Htchanism El El I 0 

Perfect Vithin 1 Mil ‘Lithin % Mil Cvcr # fiil 

7. hack1 ash 0 0 

R. Sideplay 

F 

El ! I 

9. Lift I n 0 

ver,Q;d 

Good Fair Poor 

10. Cleanliness of ‘0 I 0 

Optics No Assistance Assistance ksistance 

Ass{ st ante 1 T i m e 2 Times More Than 

11. Instructor 

2 Times 

Assistance 0 0 1 CJ 

Above Eelou Far belov 

12. Fvaluation of 

Completed Pan Tel 

.?verape 

0 

AveraRe Average 

El 

I 

Average 

0 

> 

.

.\PPmDIX D 

FIRST PEZFCR.WCE EXMIr\‘ATICH 

EVALCATTOH CF SfUCE:.‘T’S 

?W-ZHIKIST PROJECT 

SHh.I’m TERFGRMCE TEST 

!+ARK AH (X) IB THE 

PROPER B(;X 

CLASS NUMBER STL!EKT SELF-EVALUATIGS 0 

PAW 

CRACER EVALUATI- D 

PURPOSE : The purpose of this sclt-evaluation experiment is to obtain information 

that will help us to iqrave tne makhfnlst course. You will not receive 

a grade on your self-•vaiuation for this project but an honest tvaluation 

of your perfomancc vi11 be of a great help to US to evaluate the 

. c o u r s e . 

PIRCCTIONS: Yakc a J i n the T3’Z%Z;CE RAKE COLl?G f o r t h e TCLERASCE wh?.ch you 

obtained on each of the listed dimensions to be checked. 

cI?fFNsxcss 

TC BE CHECKE!: 

2. ‘;tdth 

3. Cepth from Top 

of v to Clot 

L’idth o f Ptrst 

Step 

5. :Jfdth of Second 

Step 

6. Length o f Block 

7. Linear Pitch 

P. ‘didth of V 

9. Drpth of Teeth 

IO. Grncral 

Appearance 

Could you finish pro 

you i;dn’t have enou 

DIXESS 1c:: 

1,750” 

1.75(3- 

0.625.’ 

0.625.. 

0.625- 

2 - ?/L” 

25/6L” 

1 - 3/!c’- 

0.770” 

ct in ti-e 

time. Ii 

Column 5% 

f 0.010” -~ 

I 

: 

I 

0 

etccr Than 

Averape Averare 

I n 

- Colc!an 3 1 Colurn 2 

-I- 0.012” 

0 

I 

0 

I 

t I/32” 

- C-006 

- 0.001 

El 

0 

I 

7 0.007 

- o.OO2 

I 

Belaw 

Average’ 

I 

Column i 

Above or 

Below T 0.010’ 

cl 

0 

El 

cl 

I 

Above or 

below + 3161” 

0 

I 

I 

Above f 0.007 

Belov - o.oc2 

EJ 

Far Below 

Average 

I I I 

Iilo::r2 Y-zn :1-. if answer is SC; tell why 

you finish aiieai of time YES=. 

267 

I

1 LLI:70 ?lar!Iinic: curse 

I- 

. 

- - - 

7 . . 

S o . Xa:ch,l fairs 31 : L er.l‘or.rancc ‘Lcsr S Iotai So. i’erf o*mancc? ;‘P St S {a?& 

fipnif icanr 

1 iffcrencc 

!‘T 

.if tr. 

:;.: 1 

Signif icartf 

i if ferv+?ce 

!OC.~. .Cl -S!>F El’!. 1 4C.h S o n r ;I 

268 I

. i 

_ . ..__.....__ “__ ._ _ ____._..._. . ,.., . ..y. -,-..-I . . ..-. -.-...,- -“-.-l.l.. .C..,i_.__C.-.U--_.. 

-._.- _I I - -\ . 

Cr ..- ~.I- .-... 

i. ._,_ __, ~_, ,,. 

/ 

. . . . ,.. . _. . . . 

I 

. . , . . 

.’ : s 

., : 

i! 

A MODULAR At'PROACh 10 PKuFICIENCY TESTING 

Roticrt W. Stcphcnson, Ph.D. 

Warren P. Davis, Col. USA (Ret.) 

llarry I. fladlcy 

AMERICAN INSTITUTES FOR RESLIRCII 

. and 

Nrs. Bertha tl. Car) 

U.S. ARM RBSEARCH INSTITUTEiFOR THE 

BEHAVIORAL AND SOCIAL SCIENCES 

September 1973 , 

From 3 paper presented at the Fifteenth Annual 

Confcrcncc of the Military Testing Association, 

San Antonio, Texas, 28 October to 2 Sovcmbcr 1973 

. 

269 

/

44 

INTRODUCTION 

A new, more specific laqguagc for describing work activities is being 

designed for the Army on an c-xperimental basis.1 The new language is 

based upon a concept called the “duty module”. Lkty nodules are clusters 

of tasks that tend to go together occupationally and organizationally in 

meaningful Kl\‘S. 

The need to evalunte the feasibility of personnel information systems 

1~1sed upon clusters of tasks smaller than a Military Occupational Specialty 

(NIX) \;as originally suggested by personnel at thr k-my Research Institute. 

The development of the duty module concept was a team effort involving 

staff pnmbers fr& the Army Research Institute (AHI) and the Arzerican 

lnstittces for Research (AIR).2 

Duty modules are rationally dcrivcd clusters of work activity based upon 

a detailed examination and grouping of task inventory or job analysis data. 

Attention is then given to ways in which these tentatively identified job 

content modules can be tcstcd against various available criteria of operational 

utility. One relevant consideration is uhcthcr job content modules 

can be used as field assignment noduIcs. Xnothcr possible application is in 

the area of requirement planning 3nd unit cffcctivcncss. A source of 

information here is data that can lx gathcrcd in conjunction with unit 

training and unit effectiveness excrciscs that arc performed in the field. 

The xord “module” -was chosen bccausc job activity clusters, like 

equipment components of the same nzmc, are meant to bc largely selfcontained, 

independent units of work. For purposes of occupational classification, 

a duty module is a cluster of tasks that apply without modification 

in a number of occupational classifications or specialties. 

1 The work MS carried out under Contract No. DAK-19-71-C-0004, “A 

Taxonomic 6392 for Future Information and Decision Systems”, and Contract 

Xo. DAK- 1%is-C-0041, “A Comparison of Officer Job Content Modules with 

Activity Grouping Implicit in Course Design”, awarded by the Army 

Research Institute to the American Institutes for,Rcsearch. 

2 Key pcrsonncl st XRI:wcre Nr. Cecil Johnson, who provided the initial 

guidance as the Contract Officer’s Technical Rcprescntativc, Dr. J.E. 

Uhlancr, Technical Dircdtor, and Mrs. Bertha Coy, who succ-eded Mr. 

Johnson as the Technical Rcprcscntativc. 

Key AIR staff members, in addition to Dr. Robert W. Stephenson, who 

was Proicct Director, included Dr. Robert Miller, who served as 

Consulting Scientist. Colonel Warren P. Davis, >lr. Harry I. Hadley, 

Dr. E&in A. Flciskman, Dr. Albert S. filicl\mnn, !4r. Clifford P. Hahn, 

Dr. Ronald P. Car\‘cr, and Mr. Albert Farina. 

t 

270 

2

.-- -.. _.. .-_.. . ..- .__ _ 

,’ 

Another form of construct vdlidntion is avnilable through examination 

of mission elements which arc activities thnt are used to design and evaluate 

the performsnce of drgnni z3tional unit. 

This presentation \*ill describe wo different kinds of modular cvaluation 

devices -- sets of t3sks performed by individuals, rind sets of tasks 

performed by, orgnnizntio1131 units.5 

INDIVIDUAL PROFICIENCY TESTS 

Every indiuidunl proficiency test and cvcry Arm)’ training test is 

already divided into spccinl component sections \iith ycparutc scores. Before 

going into detail ahout wh3t modular :omponcnt scores arc supposed to 

?o and supposed to look like, it is necessary to describe wh3t thcsc cxisting 

systems 3re like. 

ENLISTEU PERSONNEL PERFORMAKE EVALUATION 

This discussion .:hould’bc prcfaccd by noting that cvnlu3tion of the 

pcrformnnce of enlisted pcrscrnncl is 3n import3nt responsihility of every 

commissioned and senior noncommissioned officer in the Army. The rewards 

and punishments 3ssoci:ltcd wit!1 such evaluations give commissioned and 

noncommissioned officers the ncccssary control over enlisted personnel to 

maintclin and improve cffcctivcncss. In addition to this important supertiisory 

function, howcvcr . there 3rc 3 number of formal proficiency cvaluation 

procedures for cnlistcd personnel thnt nre conducted by various 

hcadqwrters. The most important of these, for purposes of this paper, 

is the U.S. Am\- Enlisted Evnlwtion Ccntcr, located at Fort Benjamin 

llarrison, Indianapolis, Indiana. 

3 These developments :lre described in detail in Miller, R.B. A Titxonomic 

Base for Future M;w:tigcncqt Infolm;ltion and Decision Systems: Thcorctical 

Background to the tksign of IkIt\’ Modules; American Institutes for 

Research, Kashington, 1b.C.) Technic31 Report AIR-23500-7/71-TR-2, July 

1971. (U.S. Army Behavior rind Systems Research Laboratory, BURL Technical 

Rescnrch Note, in preparation.) ; and Stephenson, R.N. (Amcricnn Institutes 

for t;escnrch , h&hi ncron, D.C. ) A Tnxonomic Base for Future Management 

Informntion and Decision Systems: A Common Languurtge for f&source rind 

Requirement Pltinnlng;-KS. An!~y Behavior and Systems Rcsc3rch Laboratory, 

Arlington, Vs., Technical Rcscnrch Note 244 (AD-757-794)) October, 1972. 

271 

/ . . 

, 

‘,

The Enlisted Evaluation Center is the major oper%\ing clcmcnt of the 

formal enlisted evaluation system. It ~3s established in 1958 as a Class 

2 activity of the Army. Its primary purpose then was to help the Army 

manage the proficiency pay program, which had been esttiblishcd in response 

to the rccommcndations of the Cordiner Committee--the LDcfcnsc Advisory 

Committee on Profcssionsl and Technical Compensation-Fin 1956 and 1957. 

Monctnry incentives wcrc one of the Committee’s proposals designed 

to improve personnel retention and job motivation among trained technical 

specialties and, at the same time,, stimulate higher quality performance 

among 311 enlisted personnel. Proficiency pay, as a concept; emanated 

from this Committee recommendation. Ilowcvcr, an underlying principle of 

this concept was that proficiency pay must bc directly rclatcd to the 

dcmonjtrntcd lcvcl of proficiency and must bc contingent upon periodic 

checks to ensure maintenance of that proficiency. The Army enlisted 

evaluation system was developed to meet +.his rcquircmcnt. 

ARMY ENLISTED EVALUATION SYSTEM 

The cnlistcd evaluation system consists of two major components: 

(a) evaluation of the enlisted man’s knowlcdgc of the various duties that 

arc rcquircd at his skill lcvcl in his NOS. as, indicated by MOS evaluation 

tests and performnncc tests; and (b) evaluation of performance in the 

currently assigned duty position, as indicated by supervisory ratings on 

the enlisted evaluation report (see Figure 1). , A rating system is applied 

to the scores obtnincd on thcsc instruments, and it is used to compute a 

composite sccrc for taking individual personnel actions. This MOS cvaluation 

score indicates the individual’s relative standing nmong thdse evaluated 

in the same MOS and skiJJ level and in the same py grade. It is 

used to verify MS qualification, to assist inj determining promotion eligibility, 

to award proficiency p3y, to guide remedial training, and in a 

variety of other pcrsonncl 3ctions. 

PROFICIENCY TESTING "AREA &ORES" 

The characteristics of the MOS proficicndy testing program will not 

be detailed hcrcin, but one particuiar aspcct’of the ?X)S‘proficicncy test 

program directly rclcvant to this paper vi11 bc considered--the MOS “major 

area” scores. Ikuzh MOS evaluation test is or,ganized into six to nine 

major nrcns; that is, six to nine subscorcs .I The six major 3rcas for an 

Infantry scnicr sergeant, for cxamplc, arc wcapors, tactics, field activi- - 

ties, unit dcfcnsc, administration, and personnel accounting. Study 

references from Army regulations, pamphlets, field and technical manuals, 

and other manuals arc coded to each of the major areas in an accompanying 

study guide so that each soldier can locate the print.ed materials upon 

which the test is b3scd. Ile cay study these reference tcntcrials to improve 

his knoulcd!:c and performance. 

272

,c 

..,‘, r 

TEST EER 

scol~1s SC’O!W 

.\lINI.\!U.\I MULTIPLE 

--I_- 

HURDLE SCORE 

Figure 1. MOS Evaluation Score.a 

RAW CON I’OSITE EVALUATION 

SCOR 13 S C O R E ,#.I\. 

PrctPay -Combat MOS 

- - 

CONVERSIOS 

a From U.S. Army Enlisted Evaluation Center, Briefing Supplement; Indiana, USAZEC, 1971. 

SCORE 

,

The mcijor iIrc3J arc wcightcd nccording to the rclrttivc importance 

of the functions in the missions of 311 units thst are authorized dut) 

positions in the blOS skill level. .~nd nc; on the basis of the time rcquircd 

to teach the subject m:lttcr in form31 clnssroom courses nor on 

the basis of the number of pcrsonncl assigned or clxthorized for specific 

duty positions. The number of questions allocated to an 3rea out of the 

tot31 number of items in an NOS proficiency test indicates the weight 

assigned to thax area. Subscorcs for thcsc mnjor 3rc3s 3rc useful not 

only to the soldiers tcstcd, uho c3n USC the information to improve their 

performance, but. 31~0 to v3rj ous headquarters, to ccntrn 1 i-cd managcmcnt 

progr3ms, and to commanders for nnnnging 3ssignmcnts 311J training progrnms. 

DIFFERENT APPROACHES TO DUTY AREAS 

Unfortunately, there is nc consistent theoretical basis or consistent 

approach to the definition of these 3rca scores by test dcvclopcrs, trnining 

personnel, or rcquirwcnt ItIanncrs (see Tnhlc 1). Some of the 31~3s 

for which the Enlisted Evrnluntixi Ccntcr has dcvclopcd scores cnn bc 

classified 3s duty arc35 e .Thcs~ 3rc3 scores roughly correspond to subject 

matter are3s within 3n ?IOS. They nrc used to provide informntion as rcgards 

3n cnlistcd m3n's strengths and wcakncsscs in sclcctcd subject 

areas, and they arc associated *iith specific. subject matter rcfcrcnccs. 

Table 1. Differmt Approaches to Duty Areas 

Approwh 

MOS proficiency tests 

Army training schools 

Kcquircment plnllncrs 

Duty :\rcas 

Sclcctcd by test dcvclopcrs 

Idcnti ficd by systems 

engineering of trilining 

, Associntcd with ndditionnl 

skill idcntificrs 

If subject matter refcrcnccs hnppcn to be organized in terms of dut) 

areas, it is easier to find the apl~ropriatc references that need to bc 

studied; 110wcvcr, it is not csscntirtl. A snmplc list of major arcns in 

an NOS is given: 

1 Kcapons 

2 T3ctics 

3 Field hctivitics 

4 IJnit Dcfcnsc 

5 ,\dministr3tion 

6 Personnel Accounting 

The study guides list the rcgulntions and technical manuals for the vorious 

iwxs, rind no grc3t amount of effort is nccdcd to find the rcfcrenccs 

/ 

274 

: . 

,

that correspond to the area in which a low score was received in the 

proficiency evaluation test (see Table 2). 

Table 2. Sample Study Guide 

Rcfercnces 

Army Regulaticns 

65-75 

210-10 

DA Pamphlets 

600-S 

672-2 

Field Manuals 

5-1s 

7-10 

Ma-j or 

Area 

It should he clear tha; the rise of area scores for proficiency tests 

was an important development in the design of MOS proficiency tests. Such 

feedback systems are an integral part of any sophisticated testing program. 

What, then, would be questioned in the design jof these area scores as they 

are used by the proficiency testing system? The point raised is not so 

much how the proficiency testing subsystem works, but the manner .in which 

thcsc area scores interface with other personnel subsystems in thc”Army. 

I 

The Army training schools, for example, identify major duty areas at 

great expense and with great difficulty, as part of their systems engineering 

of training process (see Figure 2) .I Systems engineering of 

training is a long, drawn-out procedure, involving detailed job analysis 

and the application of systems engineering pr)nciples and approaches in 

order to break a job into components and then,! select components for training. 

The job is also organized into “areas” !when a Program of Instruction 

(POI) is prepared. There is no consistent r&lationship bctwecn these PO1 

arca scores designed by the training people ‘and the area scores as used 

by the proficiency training people. 

Requirement planners are also interestid in a different kind of duty 

area. For example, the Army has additional skill identifiers (ASI) that _, 

are authorized for functional skills, which arc not consistently required 

of all the job incumbents in an NOS. An example of such an additional 

skill identifier is the ability to maintain a specific type of system 

(e.e., maintenance on the Hawk Guided’Nissilc Simulator, or the ability 

to work with specially trained scout ‘dogs). Thcsc different approaches to 

duty areas are not necessarily incompatible with each other, but they are 

all different. Khcn you have three different parts of the same organization 

-- and the Army is one orgnni:ation -- using three completely different 

275 

5 

6 

6 

I5 

4 

2 

1 

Y

- 

step 1 

PERFORM JOB ANALYSIS 

Identify job (Overview) 

-3 Develop task inventory 

step 2 

SELECT TASKS FOR TIL\ISISG 

Criticality to job 

Percent Performing 

> Frequency of performance 

1 Learning difficulty 

step 3 

PREPXRE TRAISING A.YALYSIS I 

Identify job conditions and standards 

Develop training objectives and criteria 

Sequence training objectives 

Identify evaluation points 

I Stev 4 I 

PREPARE TRAINING ?L-\fERIALS 

L e s s o n p l a n s 

step 7 

PROVIDE QUALITY COSTROL 

Intcmal feedback 

External feedback 

Step 5 

DEVELOP TESTItiG ZL4TERIXI.S 

-xEq 

Figure 2. 'Simplified Flow Process of Systems 

Engineering of Training.' 

“From Southeastern Signal School Briefing Supplement. 

Systems engineering of training at USXXSS. tindated. 

! 

276 

2 

- . . - - . _ ._ . . ..- 

. 

_ 

-. 

w.

I . * 

language systems to describe the same kind of work, it is likely that 

there will be some unncccssary duplication of effort. This could be 

avoided if a common language could be designed for all three parts of 

the organization (e.g., the Army) to USC. 

THE DUTY MODULE 

A duty module is a group of occupationally interrelated tasks smaller 

than an occupational specialty. It is modular in the sense that it can 

be used as a plug-in unit to a variety of different occupational specialtics. 

Table 3 defines the module group, number, and title. Table 4 

shows an MOS duty module matris for Army military occupational specialtics. 

As one can see, a relatively small number of duty modules can 

account for seven different MOS. Notice that each of these military 

occupational specialtics has demonstrable similarity with other MOS. 

A ADMNISTIUTION 

Table 3. Module Group, Number, and Title 

A-l Performs general administration at unit level 

A-2 Performs unit supervision and control of personnel 

A-3 Establishes and operates a unit mail room . . 

A-4 Types, files, and performs general clerical 

operations 

B TRAINISG 

B-l Conducts or participates in unit and individual 

training 

C COMlUNICATIONS 

C-l Operates unit tactical communications equipment 

(excluding use of Morse code) 

c-2 Installs and maintains unit tactical wire 

communication systems 

D TRANSPORTATION 

D-l Operates unit combat support vehicles 

(continued) 

277 

.’ 

! * . . . 

. .

. . . ‘* , . . . . . 

.- . . *. . . . : * 

.*. 

Table 3 (continued) 

E TACTICAL OPEKATIONS r 

f 

E-l 

E-2 

Prepares and employs maps, charts, and instruments 

in land navigation 

1 

Engages enemy with tank and Armor vehicle 

mounted assault weapons 

E-3 Drives tanks and associated Armor combat vchiclcs 

E-4 Emplaces, reports, and neutralizes tactical obstacles 

E-S Performs in mounted, dismounted, airborne or long- 

E-6 

range pat&-01s 

Engages enemy with mortars 

E-7 Participates in ground tactical operations 3s 

mcmbcr of a maneuver unit 

E-9 Engages enemy in close combat with individual 

weapons and machine guns 

E-10 Engages enemy with recoilless rifles and direct 

fire missiles 

E-11 Functions under CBR warfare conditions 

F STAFF PtkXAGEblENT 

F-l Performs tactical operations support duties 

F-2 Performs tactical intelligence support duties 

G MAINTENAXE . 

G-l Performs user maintenance 

i 

on individual and unit 

G-2 

equipment and wea$ons (excluding motor vehicles) 

Performs organizational maintenance on track 

and wheel vehicle mechanical systems 

G-3 Performs organizational maintenance on track 

and wheel vehicle electrical systems 

G-4 Performs maintenance administration 

ti FOOD SERVICE 

11-l Establishes and operates a fieId,mess i! 

11-Z Prepares and serves meals 

11-3 Operates a mess facility 

I SUPPLY 

I-l Establishes and operates a uni 

J PERSOSSEL 

J-l Initiates, posts, files, and retrieves information 

from personnel records 

J-2 Manages individual enlisted personnel and carries 

out manpoucr and personnel management programs 

J-3 Processes personal affairs actions for individuals 

. ’ 278 

I

Ii 

A-1 

A-2 

A-3 

B-l 

C-l 

u-1 

E-l 

E-2 

E-3 

E-4 

E-5 

E-6 

E-7 

E-9 

E-10 

F-l 

F-2 

Table 4. MOS -Duty Module Matrix 

Plilitary occupational Specialties 

11B 11c 111) 11E 11F 11c I It! 

x x 

x 

- x x 

x x 

x x 

s s 

s 

s x 

x 

s x 

x 

I 

x 

x 

X 

s 

x 

x 

X 

x 

X 

x 

X 

x 

X 

x x 

X 

X s 

X x 

x 

X x s 

x 

X 

x 

X 

x x X 

x X 

“See Table 3 for definitions of duty modules. 

It is also possible to use duty modules to express personnel requiremcnts. 

The list of work ‘activities in Table 5 has not been formally approved, 

but it i llustratcs tj& type of approach that can be used. Given 

a data processing group, there is a need to supervise, to plan the anslysis 

of the reporting, to keypunch, and so on. The number of full-time 

+ty positions needed in the organization is ten. You can also s!)ccify 

tnc requirements in terms of the number of people qualified to perform 

each work activity. If you have a computer activity with a lot of night 

shift work, you arc going to need at least three people who can supervise. 

279 

. - 

X 

x 

X 

. :-, 

x 

X 

X 

X 

X 

X 

X 

x 

. .

-. ..- - .~... _ . _ __, 

, 

. . ..- . . ..m- _-_, . ..-. . . - . .._ . ._ 

Work Activity 

Supervision 

P13nning of cinalysis 

.’ and reporting 

Receipt and verific3tion 

of input data - 

COBOL programming 

Keypunching 

Comput cr operation, 

including peripherals 

Interpretation of output 

Frcpsrntion of reports 

Table 5. Work Activity Requirements 1 

Total Number of Kork 

Activity Rcquircmcnts 30 

Minimum Sunbcr of 

Fcople Required 

Xinimum Sumber .of 

Feoplc Se&cd 

with This Skill 

m i 

1 

5 

3 

I 

i -1 

/ 

I 

fiumbcr (or 

Froportion) of 

Ful i-Time Dut) 

Fosi tions 

1.00 

9’ 

.-a 

1.00 

2.00 

1.50 

5.00 

1.00 

3* 

.-3 

I 10.00 *- 

I 

You may want one or two people for hnckup, in case of illness or vacations. 

Even thou&t: there m3y only be three duty positic$,s involving supervision, 

you may w3nt five people to be qualified 3s shift supervisors. Simi.larly, 

in pl3nning the analysis rind the reporting, >‘ti~ m;ry w3nt three qualified 

persons, but it is only a quarter-time job. 1 d .othcr words, defining work 

activities in terms of duty moJuies Gin provide 3 more efficient use of 

personnel with all functions coverll, using 3 minimum number of personnel. 

I 

TESTS OF EFFECTIVENESS OF DUTY FIODULES 

Resource and requirement pl.uxning expert s must first 3grce upon the 

qualification requirements. Secondly, compatibility with work practices 

in the field is in\zolvcd. For this, actusl survey datn regarding the way 

in which tnsks are nssigncd in the field cnn be emI>lpye,l. A third test 

is to cvalunte the userulncss of the module in plnnnlng 3nd evaluating 

the requirements for rind pcrformxxe of orgrwixtional units. 

. :’ 

: 

- “T 

i

TEST 1 

! 

t . . . 

.\ first test is to ask experts to Jcsign job content modules. Typical 

job content modules 3rc: , 

(1) Operates unit tactic31 communications equipment, 

esrlluding Norse code. 

(2) installs and maintajns unit tactical tcire communic3 

t ion system. 

,In this case, the people who arc asked to operate unit tsctiz:\l sommunications 

cquipmcnt arc usually different from those kho inst:rll .ind 

mairtain it. Scithcr of these work activities, hoxevcr, is ;I iull-time 

posltion for :rnyonc. These are modular things rhat can be assigned to 

diffcrcnt ~wzoplc. 

TEST 2 

A second test of duty modules is conpatibilit: uith assi$nncnt 

practices in the field. Some data have already been analyzed (see ,igurc 

5) . Data WI-C us;ed that were already in csistenCc, and that h;td been 

collected riith task inventories that xcrc administered by the Army Office 

of Personnel IQwra t i on5 . The- data IMSC is called the Xi1 i tar?’ Occupations 

Dsta Cank, or >lOl~U. The original tasjr statements in !XX)B arc organized 

in terms of fuwtional areas of performance (see the administration and 

:raining colr~:~:ns on the right side of Figure 3). Ttrc row zorrespend to 

task clusters that were identified in an cqiricnl clustering of tasks. 

ss 15 

10 95 

Figure 3. Test 2, Compatibility with Assignment 

Practices in the Field (in Percent). 

's 

.5 

281

The Comprchcnsivc Occupational Data Arullysis Programs (CODAP) 

system, dcvclopcd by the hir Force, was csxnincd to dctt - Gnc it applicability.4 

This system was dcvclopcd to cluster pcoplc rnthcr than tasks, however. The 

Army Rcsenrch Institute modified the COIW system to tl:c problem at hand.5 

People were not clustered togcthcr on the basis of their similarity in terms 

of task performance, instead, tzsks wcrc clustered togcthcr on the basis of 

the probability that the tasks would be assiKncd to the sxz pcoplc. On the 

left side of Figure 3, clustering is compared with some duty modules that wcrc 

analytcd in terms of the “Obvcrst Cluster :\nnlvsis” system designed (see 

Tnblc 3 for definitions of uuty modules). The computer run suggcstcd that 

there wcrc seven task clusters in this particular group of tasks. The perccntages 

shop in the duty module columns on the left side of Figure 3 indicate 

the percent of tasks in each of the cmpiric,?lly identified clusters that fall 

into duty mcdule cntcgorics A-l through !t-2. The percentages shoxn on the 

right-hand side of the figure indicate the pcrccnt of tasks in the empirically 

identified clusters that fall into each of the administrstive arcas u .rd to 

group tasks in the Military Occupations Data Bank. 

It is il,,portant to note that the design of the duty modules is not 

complctc, nor is the preparation of the task statcmcnts. These task 

Statcmcnts vi11 be revised to reflect additional job data and evidence 

concerning the organization and application of command authority in field 

units. Task statements will also be revised 3s the duty module system is 

further dcvc loped. 

It does not necessarily follow that tlrcre will bc complctc agrccncnt 

with the computer runs. For cxnnrp lc , it is possible that the first three 

cmpiricnl task clusters xould bc considcrcd as really one cluster rather than 

three. Before’ that conclusion cnn bc mndc, many other occupationdl spcci altics, 

in addition to the one that thcsc data arc ba?scd upon, will have to be reviewed. 

Essentially, duty modules arc dcrivcd from many different occupational 

spccialtics rather than just one, lzhich is the cast with this particular 

ccmputcr run. The decision as to whcthcr these first three clusters arc one 

cluster or two or three clusters must be based upon data in other occupational 

specialtics as well. 

TEST 3 

A third test of the utility of duty ~.lulcs is the applicability to the 

evaluation of unit pcrforcancc in the field. Task checklist items have been 

4 Bottcnbcrg, R. A., and Christnl, R. E., “An iterative technique for clustcring 

criteria which rctnins optinnm prcdic:ivc efficiency”, The Journnl of 

Expcrinmtal Educcltion, 5 6 , (-I), Sumner, l!WS, 2S-34. 

5 Edison, I

devised based upon the duty m~dulcs. Each duty modulc,,consists of a collection 

of task statcixnts. A study of Army Training Tests (ATT’s) revealed that 

task statements could be .convcrtcd into checklist it&s, This provides the 

means by which information can be collected about thei performance of individuals 

during unit training tests. Separate scores arc obtained for separate 

practi cc maneuvhrs . 

Scparntc scores can ‘bc obtained for movement, defense, 

and attack, as noted in Figure 4. Each cxcrcisc will have a scenario, the 

possibility of casxllties, and so!‘forth. Duty nodules will be looked at during 

the appropriate phnsc of the Army Yraining Tests. They would not be tested 

during. ever)- sing lc phase. 

Unit Tc*st I’hasc 

Plodlllc Task Chccl, 1 i st I t cm blovcmcnt t _ fcnsc At tacti 

c- 1 Establish and opcratc 

field communications 

rclny station x s 

G-l Make cntrics in 

cquipmcnt lox books s s 

Figure 4. Step 3, Relationship to vrrit Performance. 

SUMMARY " * i 

I 

! . . 

These individual duty modules are dcsigncd to be dcrivcd from a variety 

of specialties rather than just one. They arc cconoaicol in the sense that 

many different task inycntorics can bc designed and jobs described with a 

small number of duty nodules. They can utilike task inventory data already . 

collcctcd on assignwnt and work assignment pkactices in the field. The) 

meet an apparent need fbr consistency at a lckcl of gcncrality between the NOS 

.z?d the task, at an optimum lcvcl of detail.1 Ultimately, if duty modules can 

provide a language xccptablc to both resow& planners and requirements 

planners, they will facilitate COmlmiCatiOJI and pronotc better matching of 

men and jobs. 

I 

PERFORbWCE STANDARDS .4NDrSKILL LEVELS 

It is not possible to talk about testing in terms of duty modules without 

first t.*lking about tcstins standards. I’crformancc standards in the 

Army arc pre,lared sysrcmatically as part of the systems engineering of trnining 

process iDr the dcsiy of Army school COWSCS. .At one point in systems 

engineering, task and skill analysis sheets arc prcparcd. lhcsc result in 

evaluation plans _ It is ;I standard practice to indicate specific performance 

standards for training purpos:cs for each terminal or facilitating objective 

on these cvalunt ion plans (SW Vi gurc 5). 

+ 

283 

\

~hc evaluation plans only cover those areas of interest to the schooi, 

hL,:ever. They usually do not cover al 1 the tasks in 3n NOS, especially 

for the more advanced tasks, which the pcrsonncl arc supposed to learn on 

the job. The Enlisted Evaluation Center, therefore, has to supplement this 

school-oricntcd inforrxtion with other sources. Usually, they use the judgmcnts 

of knowledgeable KCOs who have csrcricnce in the particular MOS, and who 

formulate proposals rcgsrding Kh?t tasks are appropriately included in an MOS 

proficiency tcs t. 

1. Criteria for the Training Cbjcctivc dcvcloped for the task: 

Action Troubleshoot AS/?‘RC-24 . (A-??-1) 

Cond!~ion In addition to Standard Training Conrlitions, the 

- - 

student is riven an XK/TRC-24 with one major componcnt 

containing a DS- part dcfcct as weli as OS-8, 

N&30/U, TN 11-SS.!O-‘57-12, TN 11-5820-257-34, and 

AY/‘iRT-24 Block Diagram. 

St nndnrd The stuJcnt is qualified if, when given two defcctive 

XS/TRC-24s with a Z-hour tire limit on each, 

he can isolate one of the dcfcctive parts. 

Figure 5. Evaluation Planning Information Sheet. 

The proficiency testing system in the Army, as it is present!? organized, 

provides scpnrstc tests for each skill lcvcl within each MS. Occasionally, 

one test may bc used in tuo or three skill levels with different score lequircments, 

but the principle is the same. .3m reason for providing scparatc 

tests for diffcrcnt skill levc,ls is that an NOS is a broad collection of duty 

areas that cover many different duty positions. Providing separate tests for 

each skill 1~~1 makes it possibIe to provide items that are more appropriate 

for the positions being filled by those uho take the tests. 

This skill level approach, which is not incompatible with the duty module 

concept that KC ha\-c described, is appropriately used in connection with 

duty modules, and, further, it iilustratcs how useful duty modules can br: to 

those who design tests. Table 6 lists the tasks for 2 duty module of patrolling, 

cithtr mounted or dismounted. One’s skill lcvcl is dependent upon whether one 

supcrviscs a task, doe’; the task and also supcrviscs it, simply dces the task, 

or rshethcr one just assists in doing it. The tllrcc skill levels in the 1lB MO.5 

(Light Weapons Infant~nan) are indicated. The ski1 1 levels are numbered 1, 2, 

and 4. Thcrc arc only three skill levels in this particular blOS, so there is 

no skill lcvc 1 numbered 3. Table 6 indicates that people who 3rc at skill level 

4 are more like 1y to supervise. People who are at skill level 2 do not supcrvise, 

and arc much more likely to assist somebody. People who are at skill 

level 1 carry out the orders and requirements of their superiorc. This kind of 

information about skill Icvcl profiles could be cstremcly useful to Army 

organizations in dcsignins proficiency tests. 

! 

284 3 

. c . ..-*.- I. - . . 

,- -..’

Tasks 

Table 6. Duty Module E-5: Patrols, Either Mounted or Dismounted 

� 

(1) Plan patrol operations 

(2) Assemble, inspect, issue patrol 

order, and lead patrol 

(3) Operate listening or 

observation post 

(4) Serve in combat patrols 

(5) Serve in reconnaissance patrols 

(6) Serve in ambush patrols 

(7) Mark route or serve 

as guide for unitb 

(S) Participate in air search 

operations or air delivered 

patrol 

(9) Estimate charge, emplace and 

fire demolitions 

Performance Expectations for 

Skill Levels 1, 2 and 4 of 1lB NOS" 

Super- DO and 

vise SLlpervise 

I 

4 

4 

4 

4 

4 

Do 

Assist 

4 2 

4 2 

1 

2, 1 

2, 1 

2, 1 

2, 1 

2, 1 -' 

2, I 

;LThere art? only three skill levels in the 1lB blOS: 1, 2 and 4. 

b Task No. 7 (rsllen performed by an 11B NOS) is normally supervised b) 

someone in another NOS. 

ADVANTAGE OF A MOCULAR APPROACH IN TEST DEVELOPMENT 

Previously mentioned were the ways in which the Army military occupational 

specialties were modular. Refer to Table 4 and note that with 

this matrix one can account for many different tasks with a relativcl) 

small number of duty modules. Moreover, this matrix is just the corner of 

a much bigger matris. To date, 31 enlisted duty modules have been developed; 

they completely account for 16 different s-digit enlisted NXe. 

285 

.

To appreciate the possible savings it is necessary to translate this 

information on duty modules and NOS into test items. Say that these 16 

blOS would rcquirc 100 items apiece to account for them if they were developed 

independently by various test-dcvelopinp, agencies. J%at would make 1,600 

test items, if you used an independent approach to test development. Ke 

estimate, rhat it takes only 10 items apiece to desckbe a duty module. 

In other words, it is possible that 310 items can do essentially the same 

job as 1,600 test items. To be able to prepare 310 items rather than 

1,600 items reflects a considerable savings. It is contingent upon dcfining 

modules that cut across, and have the same meaning in, different 

occupational specialties. In giving these figures, WC have not discussed 

skill. lcvcls; but different tests for different skill levels would bc rcquired 

in both systems. Thus, multiply the number of items in our cxample 

by three or four to get the number of test items that would actually 

be nccdcd by the people who design these tests and work with them. 

MODULAR AP~Y TRAINING TESTS (ATTS) 

At the present time, the Army has several hundred Army Training Tests 

(;\TTs) for use in evaluating the perfornnncc’ of organizational units. 

Each of these tests has a scenario and provisions for referees who arc 

trained to follow people and take notes as regards their performance in 

the lmit test. The question to be posed her: is this: Kould it kc Jcsirablc 

for Army TraininS Tests for organizational units to be organized 

in the same ,modular fashion that has been proposed for indiy:idual proficiency 

tests? 

i 

. . 

P 

PURPOSE OF ORGANIZATIONAL UNITS 

Before further discussion of the feasibility of modular ATTs. the 

accuracy and consistency with which the i?tcbdcd purposes of organizational 

units have been specified must be considered/. The Army has several Jifferent 

terms for describing the intcndnd purposes of organizzational units: 

a primary mission, some functions, and a ca P ability. 

A primary mission is defined as the principal purpose that an organization 

is designed to accomplish. The funytions are the appropriate or 

assi.gned duties, responsibilities, mission:$, or tasks of an individual 

office or organization. A capability is dhc abi lit)’ to cxccutc a spcci- - 

ficd course of action. Further detai Is will not be discussed except to 

state that after studying the various mission, functional, and capability 

s:atemcnts in Army documents, it was concluded that the capability statement 

was the one that should be used as the basis for structuring organizational 

unit testing modules. 

286 

. . . . _ - _-. 

,

THE CAPABILITY STATEMENT 

A possible capability statement for a theoretical unit is shown in 

Table 7. It- is clearly possible to analyz c a capability in terms of specific 

component functions and operational criteria, as shown in the table. 

1. Title: 

Table 7. Theoretical Capability of a Unit 

_ Transport supplies and resupply itself. 

2. Essential Component Functions: 

a. Load, move, and unload unit loads of rations, 

FOL, ammo, and repair psrts. 

b. Repair minor vehicular failures enroute. 

c. If unable to make minor vehicul3r repairs, tow 

inoperable vehicles. 

d. !4ove unit loads on the road or cross-country. 

c. Pick up and issue supplies. 

., 

3. blinimum Operational Criteria: 

a. 

b . 

C. 

d. 

e. 

f. 

6h. 

. 

1. 

Sufficient vehicles on hnnd in condition to 

move unit lo3d. 

Sufficient vehicles on hand in condition to 

pick up and deliver supplies. 

Trained drivers. 

Authorized m3ps and comp3sses on hand. 

Supply platoon trnined 3s a team. 

Trained supply personnel. 

Trained vehicular and radio mechanics. 

Satisfactory status of equipment maintenance, 

Satisfactory completion of ATT and FTX. 

4. Standards (To be developed): 

Includes minimum personnel, skills, operable 

equipment, ond training necessary to be considered 

C-1, C-2, or C-5 as defined in AR 220-l. 

StaRd3rds below C-S arc C-4. 

The capability of the unit is to transport supplies and to resupply itself. 

h’hnt is involved in this cep3bility are the functions of loading, 

repairing vchic lcs, picking up initial supplies, and so forth. The minimum 

operation31 criteria 3rc slso indicated. A lkpnrtmcnt of the Army 

287 

2

m. 

study of output measurement conducted in 196S6 shows the same conclusion 

that is arrived st here--that capabilities are a good way to structure and 

organize ATI’s. 

CRITERIA FOR MODULAR ATT EVALUATION DEVICES 

A number of criteria for an improved system of evaluating performance in . 

Army Training Tests have been formulated. Details are beyond the scope of this 

presentation, but the criteria and related recomn?cndations arc given. 

’ A series of clear, quantitative statements specifying the 

capabilities of n unit. 

� � taxonomy of unit capability statements. If possible, 

these statements should be modular. 

’ Criteria of unit effectiveness based upon and relatable 

to the taxonomy of unit capabilities, and consistent with 

and relatable to criteria for the performance of individua.ls 

in the unit. 

* Both kinds of criteria (individual snd unit) based upon 

performance standards rather than relative standing in test 

performance. 

� Varied performance standards depending upon situational 

conditions, such as terrain, percent casualties, and 

resource inputs. 

� Aggregation of standards for organizational components and 

generation of an overall indcs for the unit as a whole. 

� The emphasis in criteria statements upon end results rather 

than methods used to achieve the results. 

� Output measures allowing for the possibility in evaluations 

of corrective mcasurcs that may have been taken by command 

personnel, and that would permit the unit to meet standards 

in spite of some departure from cxpectcd procedures. 

� Scoring that provides specific information about the lcadership 

of 3 unit. 

6 Department of the Army. SJmprovcnent of Output Measuremen& Report of a 

Special Study by the Army Staff Coordinated by the Comptroller of the 

Amy, January 196s. 

. 

288 

.

SUMMARY 

’ Scoring that provides comprehensive evaluations of the 

unit when tested 3s 3n entire unit. .f 

w Quantitative weights assigned for sntisfnctori performance 

and deducted for in3dcquate performance that rcflcct the 

probable seriousness of the actions. 

� A method for rcl3ting output messurcs to input mc3surcs 

logically. 

A series of clcnr, quantitntive st? ‘emcnts are need4 to specify 

the capabilities of a unit. There is slso 3 need for a tzonomy of unit 

capability st3tements. I\ taxonomy is a theoretically-bnscd language 

that implicitlcv clnssifics or categorixs 3 capability statement at the 

same time t.hAt it describes it. 

Capabilities should nlso be modular in nature; that is, the capability 

statement for an Infantry battalion should be the same, if possiblc, 

3s 3 highly similar, closely related cnpability stxtcment for an 

Armored Cavn 1 ry squadron. If you can design the capabrlit!. statements 

thusly, and organize the testing accordingly, i;t is possible to design 

one modular unit trclining test component that would be useful for infantry 

battzllions and for kmorcd Cavalry squadrons. Ii.Kllll;~I~S Of possible 

modules 3rc night ground Gt3ck, rctrogradc movement, 3nd stat_ionar) 

defense. Since many different capabilities of ,differcnt kinds of battalions 

sre common, having similar tests proviJe many economic features. 

CURRENT WORK ON MODULAR ATTS 

Having proposed the design of modular Ar 1 ly Training Tests and proceeding 

to design some, field survey Icork was conducted in conjunction with Army 

Training Tests at Fort LeEis, K3sbington, in /August 1973, to pre-test the 

first versions of thcsc modular evaluation devices. These devices and data 

have not been analyzed or evaluated as yet, thus comments nrc based upon what 

was learned during the design stakes. 

d 

PERSONNEL CAPABILITIES 

In designing tbcsc evaluation devices, two things quickly bccoac 

apparent : (3) devices could not be designed for units based upon enlisted 

duty modules along; and (b) there was 3 need for officer duty modules. ,I 

contract already undcrr

jobs. Table 8 defines officer duty modules by area and module number. 

Table 9 accounts for all the major duties of an officer’s job with just 

a few duty modules. Each of the positions shown is completely accounted 

for by the officer duty modules listed. (The letter refers to the 

“areas” from which the nodule was taken in Table S, and the number 

identifies a specific module within that area.) 

Area Title 

A 

B 

C 

D 

E 

F 

G , 

II 

I 

J 

K 

L 

bl 

x 

0 

u 

\\’ 

x 

FF 

1111 

Table 3. Officer Duty Modules by Area 

Command Management, General Management 

and Administration 

Personnel 

Intelligence 

Operations and Plans (Staff) 

Organization, Training 

Logistics (Staff and Consumer Units) 

Communications and Electronics 

Civil-Military Affairs 

Comptrollership, Budget and Fiscal 

Army Aviation 

Research, Development, Test and Evaluation 

Operations Rcsekch nnd Systems Analysis 

ADP Nanagencnt and Programming 

Education, Instruction 

Information Activities 

Tactical Direction of Combat Units 

Miscellaneous 

Individuiil Functions and Special Qualifiers 

Logistical Services 

Supply and Maintenance Support Operations 

290 

3 

Sumber of 

Modules 

9 

4 

5 

4 

3 

9 

2 

3. ,*. 

2 

5 

2 

1 

1 

2 

1 

5 

9 

4 

9 

9

Table 9. Application of Duty Modules to Officer Positions 

Position 

Cdr., Infantry Rifle Co. 

CPT 

Duty Modules 

A-l, A-3, A-4, A-6, A-8, 

E-l, F-l, X-l, X-2 

Cdr., Rccsption Station Co. A-l, A-3, A-S, F-l 

. LT 

Asst. ?my Agtachc A-l, A-4, C-4 

LTC 

EQUIPMENT CAPAillLITlES 

In addition to information on the personnel capabilities of a unit, 

there was a need tclr detailed information about the capabilities of 

equipment. The nwd to think about equipment capabilities became apparent 

when prepa~lng a unit capability table, which described different 

types of capability S’or each component in a platoon (Table 10). 

Item of 

Equipment 

1. Antenna (AT- 

784/PRC) 

2. Armored Rcconnaissance 

Airborne 

Assault 

Vehicle 

(Fl 551) 

Table IO, Equipment Capability Table for Armored ',. 

Cavalry Platoon (TOE 17-107H) 

t!oJjs of Issue Capability Reference 

Z-$cout Section Determine the direction ST-24-18-1 

S-Rifle Squad to a specific radio 

transmitting in the 

frequency range 30.0 

to 75.95 MHz. 

d-Light Armor Negotiate almost any ST-17-l-l; 

Section terrain at speeds ST-17-15-l; 

from 4 miles per hour FM-17-36 

in water to 43 mph on 

roads, including 7foot 

spans, 33-inch 

vertical obstacles 

and 60% grades. 

291

. . 

It Kas not possible to dcscriI)rl !?& fire-poxer capabilities of a 

squad that was equipped with3 ccrt:ljIJ fyi+e of machine gun, for example, 

without knoliing what ty-pe of machine ;:kr :t wls. One machine gun might bc 

cap3ble of 3 sustained firing of 40 T‘~P.vI~:% per minute, while another 

might fire 100 rounds per minute. ‘!)I*: t.~r.zc of thd first machine gun 

might be 6,000 meters, while the r;ure:l’ r,l tic second, faster machine gun 

might be 3,000 meters. Clearly, 3ny CifJ+f 1 native statement of the capabilities 

‘of th3t squ3d, 3nd hcncc t t/c : +rbjbiIitics of the w!~olc platoon, is 

greatly affcctcd by rihich of the tr*‘tr filLI;!I.;nc guns is being used. Simiiar constraints 

upon unit capability St:ltcvwli(; zst-c imposed by the kind of trsnsportation 

th3t is rtv3il3ble. This t :/::t’ c.f inforn3tion !i3s obvious implications 

3s regards :hc capabilities of’ .J rf:;t, 3nd :nust be incorporated or 

dclineatcd in unit cap3bili ty statcnc:/l~.;, 

PRELIblINARY WORK ON r:VA!l:r;.T:ON DEVICES FOR ATTS 

As stated ccirlicr, one of the rx’~!‘~#t’cmcnts for the individual cnlistcd 

modules is th3t they bc meaningful in t+ r:\,s 

as tie11 3s individual evaluation pro~‘c~~~tcs. 

of unit evaluation procedures 

Ever, cnution must be 

cxcrcised in the using or modifying ci:~t/ ::~A~lcs at this stngc. The task list 

for a given ZIOS may or may not adcc(i~:ltcj, r:tmple the tasks pcrformcd by 

jncumbcnts. l’hC task 1’ 1st 31~0 may (lr 1~) not include 311 the tasks which 

make up the duty modules for that W)!i, ?,,.:I /fin311y, dutv nssignments nsy csist 

rihich arc in3ppropri3tc to 3 duty poc/tjr,!r Jcsignation within nn YOS. 

9 

i 

APPROACH TO THE PROBLEM ' 1 

The first :lppronCh Was t0 dCVc*lolJ F ~:a.mplCS of criterion behavior for 

a particular duty module, but it cl id IN! !:ckm to hc \t*J, /.(n exampic of ovcrnll unit pcrformnnce 

rating procrdurcs is show in Figure I’, 

. . i f: 

, ?3E 

I 

-.. . ..-. --. __. 

I

. . 

Checklist Item 

1. Replacements properly 

received and assigned 

2. Losses and casualties 

properly processed 

3. Leadership of platoon 

and squad SCOS 

4 -* Duties of subordinates 

properly allocated 

ATT Phases 

Figure 6. An Approach to Unit Evaluation Devices 

Based upon Task Statements. 

Activity Rated: Rifle Platoon 

Phase 1--Daylight Attack 

Proper actions:and preparations in 

assembly area? /’ 

Proper organization, formation, and 

dispersal? 

Platoon’s use of cover and concealment? 

Firing on objective--good volume, 1~11 

directed? (Scored only for live firing’ 

score Comments 

- - 

Figure 7. Evaluation in Terms of Unit Performance As a Whole. 

t 

/: 

- 

.-.:

RECOMMENDED APPROACHES AND Pl.ANS !'I 

APPROACH .y 

I I 

Test; have usually been designed by taking a p+rticulor criterion situation 

and designing a test for that one situation irrespective of how it is 

structured for that particular case at that particular point in time. If, 

for cxainple, a proficiencg knowledge test is being designed for somebody 

who repairs automobiles, you determine exactly Aich tasks pooplc in that 

NOS are supposed to perform, then you design the individual proficiency tests 

to measure knoulcdge of those specific tasks. 

The same is true of unit trninin~ tests. You look nt the capability 

statements, study the terrain in \

The Army has already embarked upon a large-scale program to overhaul all 

of its unit training tests. The Army recently designed a systems engineering 

of unit trsining programs that is p3tterned after its systems enginrzring of 

training procedures for individual training. Several organizations are 

currently harking on the redesign of unit training programs in terms of the 

Army’s systems engineering of unit training programs. Thus, 3 nuder of new 

evaluation devices will definitely be’ designed. These new approaches to 

modular unit training tests have been discussed with those uho arc responsible 

for this type of work, and it is bclicvcd that they will consider a modular 

approach to AITs 3s an alternative when the tests are revised. 

USES OF DUTY-MODULES 

This paper has stressed the economy of duty modules in terms of test 

preparation costs and the importance of consistency in language. Other 

possible 3dvantages are given. 

(1) Duty modules can improve occupational research, its 

description, and utilization. 

( 2: They have the potential to reduce training time and 

lower tr3ining costs. 

(5) They can provide a better use of individuals in 

assignment substitutions. 

. 

(4) They can simplify automated assignment and control 

procedures. 

(S) They c3n improve proficiency evaluation. 

(b) They can improve c3rcer guidance and planning. 

(7) They c3n improve utilizstion of personnel at a 

10~31 level. 

(S) They can improve unit training evaluations. 

PRESENT IMPLEMENTATION OF PROCEDURES 

It xould take many years before this kind of modular system could be 

established. If everyone agreed that it should be done tcmorrow, it Kould 

still take several years before such a system could hccome operational. 

Sevcrthelcss, the advantages of a modular npproach to test construction are 

so great that people should start thinking about it now. The problem arises 

because test developers cannot proceed by thcmselvcs. The whole occupstional 

structure needs to bc revised so that occupational specialties would be 

defined in terms of duty modules. 

295 

2

‘. 

Nevertheless, there arc several procedures that could be started now. 

For example, change the way in uhich’test design work groups are established. 

Instead of assigning test development to a group of experts in the same 

specialty, create 3 work group comprising representatives from several 

specialties, and ask that group to design a single test component that would 

be useful to thea all. A similar approach (i.e,, assigning groups of experts 

from different types of units) can be used with those who design unit evaluotion 

devices. These preliminary steps, taken now, would have many immediate 

advantages, and hould greatly facilitate a conversion to modular testing in 

the future. 

/j 

I 

296 

, 

‘.: -;

THE GROWING DEhIAND FOR HUMAN PERFO+IANCE TESTING 

-- J. E. GERBER, JR. -j 

ABSTRACT 

This paper reports the human factors involved in moving United States 

A&y Infantry School instructors away from norm-referenced, know- 

ledge sampling testing and into cri: erion-referenced performance 

testing. It sketches the gradual convergence of and increased com- 

munication between subject matter esperts (instructors) and test 

psychologists (quality controllers) from’group orientation through 

stylized esample to specific problems of performance testing, and 

I 

compares the dynamics with those obsefved by educators in civilian 

schools. m 1 i 

INTRODUCTION 

Parallel innovations have been occurrir!g and recurring in different 

I 

types of schools in separated parts of the country. 

In April, 1973, Doll, Love and Levin i.i) reported as foollows on their 

esperiences in implementin,n a new instructional model in Louisville 

clemcntary and high schools. / 

“In Loui5xille, Kentucky, a brave new world of educational 

innovntivn was recently attempted. Bright ideas and good 

.’ 

I 

297 

_.

JJ 

intentions, a situation ripe for change, and a golly-gee, 

whiz-bang attitude at many levels of decision making 

augured well for its success. Education was to roll 

for&ard. History was to be made. But somewhere along 

‘the way the wheel had to be rediscorered and history not 

only was made, it was remade. It 

Two years ago, Kramer and Kneisel (2) showed you a model for 

course design as it was then being developed at the US Army Infantry 

School, and related to you a dynamic plan for imposing (2a) this model 

onto on-going courses of instruction. Their model is depicted on this 

slide. 9 

(SLIDE 1 OK) 

SLIDE I, SYSTEMS ENGINEERING PROCESS 

The two-headed arrow labeled quality control in the middle of the slide 

implies that quality control takes place at every step of the systems 

engineering process. Our concentration today will be on criterion. 

testing. 

(SLIDE OFF) 

, 

i 

I 

298 

_ - 

.. . . c 

. 

_-.

Thus, we at Infantry School had a model and an implementation plan 

for training adults to perform specific jobs. 

Nean\vhile. Getz and others (3). who were changing froi,> a tradition- 

ally based to competency based programmed instructional model for 

teacher education at Illinois State University, provided the following 

admonition in eal.ly 1973: 

“\Vhile the rationJe for any program can be developed 

logically, the actual implementation invariably creates 

many problems that cannot be anticipated. . . The results 

are rewarding. but adjustment by staff and students is 

slower and more agonizing than most would suspect. ” 

IW R POSF2 

The purpose of this paper is to compare human esperiences at the 

Infantry School with those at other instituiions in changing instruc- 

tione models; to focus on the impact of changing from norm-referenced 

tests of know!-dgc to criterion-referenced tests of abilities; to discuss 

some of the dynamics implied; and draw some Inferences. 

Implementation of the systems engineering model at the Infantry School 

has been a tremendous undertaking, both from the technical and the 

human side. 

299 

,

,, - 

Systems engineering documentation for the first of some 22 active 

courses of instruction began in June 1968. We expecd to have the 

final one completed in June 1974. 

Ct-ltcrion testing was continued in those ;lr~as where it altyndy es- 

isted, that is: 

Pure performance trsting, in whicn students ;xrformcd job tasks 

against pass/fail (-t’iteria, continued in tcrts of physical fitness, 

n-capons firing qualification, parachute jump qualiiication. and land 

navigation. 

Yet-iormance testing under conditions siynulating job conditions 

and against subjt\ctb:ely t-a:ed criteria 1va.s darried on in tactical map 

problems. 

n 

r 

Bu;, to a large degree. students \~cre g:‘adJatcd f r o m our cout*ses on 

their ability to answer multiple c*!wice required recogni- 

tion, t-wall. or comprehension of principles, and 

which were graded “on the curve. ” 

On 2i February lBi3. v,xz determined to make all tests pct*formance - 

arriented so that all would have the student +ply job-required skills, 

300 

. ’ s. 

, 

?

I 

. - 

knowledges, and attitudes to an identified job task. \L’hile the rever- 

berations of this decision were felt throughout the School, they were 

experienced most acutely by the instructor \vho retled most heavily 

on multiple c.hoice items testing r-ecognitlon or recall in his arc3 of 

expertise. In olwrr to relate subsequent events, observe the stylized 

functional diapganl of the School or � h -,lanizational activities on this 

slide: 

(SLIDE 2 ON) 

UX

\ _ . ‘\ . , 

The test psychologists or quality controllers are the principal guard- 

ians of the instructional model presented earlier. Obviously, they, 

through their actions and judgments, are focal. points for convergence 

of differences in rationale, procedure, and requirements. 

The instructors, of whom there are approximately 1. 050 in the School, 

have multifaceted roles under the systems engineering model. Lectur- 

ing a class of students, as he might have done daily in earlier times. is 

only one of the instructor’s time consuming activities today. He works 

closely \vith systems engineers in documenting his area of expertise. 

He prepares his lesson plans with all that this entails. IIe works with 

quality controllers in his design and construction of tests and exnmina- 

tions. In fact, his role is not unlike that of the Illinois State 

faculty member described by Getz and others (3a) and shown 

next slide. 

(SLIDE 2 OFF) 

(SLIDE 3 ON) 

SLIDE 3 TftE FOUR AREAS OF RESPONSIBILITY.. . 

Un?ve’rsity 

on the ’ 

In addition to those marked on the slide, the USAIS instructor has other 

instructional duties which are physically and mentally demanding such 

. ./ .. 

/ 

302 

.

_ 

as field problems and weapons firi~t;. The instructor iaces massive 

prohlems of priority. 

(SLIDE 3 OFF) 

The multiple demands upon the ins!ructot .‘s time and attention account 

in pa1.t for the spectrum of response to the requirement for perform- 

ance testing. 1 en?phasize the word spectrum, for a spectral responsr 

did occur, varying from, “Great, I’m dc,ing real world performance 

testing already, ” to “NO way. ” !Ve applaud the former and the many 

who flocked to it in short order, but it was those on the “No way” end 

of the scale who required the most attention. 

. 

This paper deals with dynamics of tmhange. If a given reaction or 

I 

attitude was encountered at 41, it is discusse,d here for its dynamics, 

I 

1 

not for its frequency. 

Quality controllers conducted a set ies of fre -wheeling meetings with 

groups of 10 to 20 instructors at a time to 

f 

SL irface 

problems and to 

establish the kind of dialogue and openness,which Doll and his Louis- 

ville associates (la) belie\-e essential. 

i 

if 

i 

The problems surfaced bore a -’ 

marked resemblance in kind if not in content to. those reported by Gross 

303 

. . . : / 

: ;.

and his Harvard colleagues (4) in their attempt to implement a modu- 

lar change in a small elementary school. These were: 

1) Staff resistance; 

2) Alisconceptions of the model and roles within it; 

3j ‘Lack of expertise; 

-1) Lack of materials anti 1.csources; 

5) Incompatibility with the model. 

It became apparent to us that not only were there some problems of 

implementation. many of them were problems that had been raised 

elsewhere some years before as will be related in part below. 

We determineri that, ha*.-ing uncovered our own choke points or bttle- 

necks, we should press on while hceding the following speculation of 

Doll and his associates (lb): 

,, . . . the wheel does have to be eternally re-i:lventcd.. . ; 

- - 

unless you go through every agonizing step yourself, what - 

ever you accomplish ultimately will be superiicial and 

unimportant.. .Each of us must discover for ourselves 

where it hurts the most, but the experience of others may 

be helpful in leaking how to cure the wound. ” 

304

In this endeavor, we determined to work toward changing old concepts 

and practices into new ones using a proactive-retroactive transitional 

rationale described by the present author (5) in 1964. On,the way we 

noted that Popham 16), at least as early as 1968, warned the educa- 

tional cotnmunit~ i!lat many arguments against “stating instruction31 

goals in terms of measurable learner brhaviors” were being voiced. 

Popham felt that all these arguments KCI’C invalid, SO, in order to 

refute them, he c-odified these arguments into eleven reasons why a 

behavioral model \vas or is said to be improper. Innovators at the 

Infant t-y S~*hool, at Illinois State University, (3) and Louisville Public 

Schools (1) encountered several of these “reasons” when changing from 

one instructional or testing model to another. Therefore, it set-ms 

appropriate to compare experiences and management techniques at 

these three schtwls, using the negative arguments as focal points. 

Perhaps others planning to roll toward pedagogical innovations may 

profit by our rt-invention of the wheel. 

Popham’s negative reason number 1 is shown on this slide. 

(SLIDE 4 OS) 

SLIDE 4 ~EG\TIVE KEASON 1 TRIVIAL LEARSER EEIIAVIORS 

(SLIDE (1 E’F) 

,$ 

*. 

305 

-I_ 

: 

‘;... 

- 

.._ 

. . __ -. ,., .‘; 

‘. ‘1 

-.,. 

.’ 

-*- ’ . 

,” 

*. . 

“I..--.- _. . 

-- .,, 

: : ! I? . . . ,. 

: . \ 

I 

‘* 

; : 

. 

j : 

Q~$;.”

i I 

/ 

A comparison of experiences and management at the! three schools 

which I have related to Reason 1 is shown on this slide: 

(SLIDE 5 OX) 

. 

r 

I” 

j - 

SLIDE 5 REASON 1: “TRIVIA” 

We see that US.4IS and ISlj (3b) encountered the problem, L*vhereas 

Louislrille (lc) apparently had a more positive esperience even tllough 

some of the sample objectives presented seem to be taxonomically low. 

In our experience, operationalizing at the lowest level was easy. It 

took the form of, “At the conclusion of this’ instruction, you will be 

able to list, state, compare, etc. ” The stydent needed only sufficient 

mental capacity to recognize, recall or understand instruction in order 

/ 

to pass tests. 

(SLIDE OFF) 

b 

I 

The mental capacities and the elicited behaviors are at the Louver end 

I 

of Bloom’s (i) taxonomy of educational obj,ectives illustrated as a 

stairstep or hierarchy by this slide: 

(SLIDE 6 ON) 

/ 

li 

SLIDE 6 HIERARCHY OF MENTAL REQIJIl’\EhlEXTS.. . 

-..‘- _ . -. 

._ _. 

306 

I

This slide, which is a modification of Towne’s (8) illustration oi 

Bloom’s tasonomy, cotnpares mental skills with job performance. 

Our difficulty arose in trying to raise the taxonomic level from the 

first two levels--knowledge and comprehension- -to the appticatory 

level and higher. In the case of knowledge, we want to say. “Given 

a situation such as you at-e likely to encounter in your future job, you 

will be able io apply your newly acquired knowledge well enough to 

analyze your situation, synthesize all pertinent factors, make an 

evaluative judgment, and behave nppropri ately. ” 

I\-e found solidly imbedded defense of subject matters as ha\-ing intrin- 

sic value--knowledge for knowledge sake--regardless of the j

.f . 8 

.,d *. I .*. 

\ 

. ‘,’ 

*., 

\ I’ . 

answer had been generated more r\tYectively than cognitively, the 

I 

technique succeeded where logical :\I *stunlent a and kogent reasoning 

I 

had failed. Some instructors were nwre concerned with the minutia 

of goal achievement than xvith the :wllie\Vement itself. It was more 

imp&ant, for example. for the student to know the fact that a hole 

must be dug to a certain depth than it was to specify acceptabie 

behavior for determining hole dep’.I\. This attitude was amenable 

to suggestion that a slight change of teaching strategy would present 

lwth the doctrine and thr terminal Iw!1avior. 

I+qlxum’s reason 2 (Gb) is sho\vn On this slide: 

(SLIDE i OX\‘) i 

I 

I 

SLIDE ‘i REASOS 2: PRESPEC,tW-$TIOS OF EXPLICIT GC\:lLS.. . 

I 

(SLIDE OFF) 

The con-2arison of the three schcx:ls i 1 

(SLIDE 8 ON) 

I 

shotvn on this slide. 

SLIDE 8 REXSOS 2: “USEXPG si TED OI’R~RTGNITIES” 

We see that esperi&crs among the three schoois differ. The differ- 

ences may stem from the different frames of references, i. e., adult,

face-to-face instruction at LiSAIS; adult self-instruction at ISU (3a); 

and elementary child face-to-face instruction at Louisville (Id). In 

our espericnce, serendipity in the classroom is rather tightly con- 

trolled bu;h rtdministrntivcly and by the students themselves. There 

is a grooving demand by students ;o !w informed of the precise goals 

of i mpendiug instrurbi i~)n, aided t)F :dministrati\‘C demand for precise 

statement of both immcdinte and ultimate goala. Novel, spontaneous 

student rwponses in proach. Tests, ob\*lctusly, must test the trniningAob- 

jectives. not the “old \var story” whic*h might tv used to illustrate 

solutiotl ?a ., job task dilemma. 

t 

(SLIDE C i’E.1 

Popham’s w.:son 3 (b’c) is show-n O:I this slide. 

(SLIDE 9 C,N) 

I 

For the prcscnt purpok, I shalt compare and discuss only changes 

in the proG?ssionaI stnff. 

: 

309 

- 

‘> 

>

(SLIDE OFF) 

‘f . .‘.J. 

Gomparisons of staff attitudes and behaviors are shoed on this slide. 

(SLIDE 10 OX) 

SLIDE 10 REASON 3: “ATTITUDE CIIAXGE” 

The reaction to proposed change was spectral at all three schools. 

so we may concentrate discussion on the resistance or opposition end 

of the scale. 

Some members of the USAIS professional staff espressed themsell-cs 

as follows: “\t’e have gotten along \vitl:out job knowledge performanct- 

. 

testing for nearly 200 years and we have won many wars duringthat 

time; \vhy, all of a sudden, do we have to change L!o\v?” 

Those instructional staff members who held this or related views 

presented the greatest challenge because%. unlike the Louisville plan 

(le), our implementation plan provided no immediate escape for those 

opposed to change. Aloreovcr, neither \ve nor the Louisville schools 

(la) had the option of closing down while we sorted ourselves out. 

Some viewed systems engineering and resultant 02quiremcnts for 

performance testing as cllnnge for sake of change. others as schemes 

,’ 

310 

:* . 

. 

- 

. ‘. /- / * . . 

I _ ‘.l-

: 

.- . . .-.\. 

.- :. \y ,:. . . .- I. . 

dreamed up independently by qtlality controllers who, as a result, 

sustained a certain amount of personal denigratioi to \\hich Doll (If) 

also refers. Some viewed the changeo\-cr as diitct attacks on their 

areas of espertisc. pedagogical approaches, or organizational divi - 

‘. 

!: 

sion ‘of labor. Some saw the change as a plan worthy of resistance 

or subversion. as f.he Louisville administrators foresaw !Ig). 

Even so, these members of the staff responded positively to three 

kinds of suggestive challcn~e in the military setting: 

First--Look at the kind of testing \-MI are doing nvw. If your 

lifs depended upon thcl ability of one of Four students to perform in 

your area of cspertise, would xou say he/ \~as qualified on the basis 

I 

of the test you give now? 

Y 

‘Second--If you cannot show how youl, J subject applies to the gr,x!u- 

ate’s job, I\!-;. not delete yo’lr subject from that course? 

Thit-d--If sppl;calion of the facts ycu I 

teach is taught by someone 

I 

else, why not stop trying to tes: rnplicyion noiv? --\Vhy not get 

together with the other instruction and il.-. later? 

(SLIDE OFF) 

/ 

Popham’s reason 4 (6~) is shown on this slide. 

311

! _..“. , 

(Sl,IDE 11 ON) 

’ _’ *./ ‘, . -. . . , \ 

SI.IDE II REASON J: “MEASURJIBILITY IMPLIES BEH.4VIOR.. . ” 

(%lDE’OFF) 

t’ljnlparison of the three schools is shown on this slide. 

(SLIDE 12 ox;) 

SIJDE 12 REASON 4: “DEIIUXI.4NIZING” 

.i 

‘l’hc ISU esperience (3b) makes the case for criterion testing of overt 

Iwhnvior at the teacher college lcvcl. 

‘1‘Iw Louisv.ilte experience (lh) makes a case for contingency planning 

unticr a whole person interaction concept. 

..\s previ&s remsrks on Infantry School experience with Reasons 1 

through 3 imply. we did not run head-on into cries of reductionism, 

rwbotism, or similar charges of dehumanizing either Instructors or 

students. In fact, part of our underlying appeal for performance 

oriented testing has been on grounds of human interdependence. We 

;rskcd for and r;ceived imaginative replies to this generalized ques- 

I 

lion: “\~hat is the/minimum behavior you will accept now from the 

312 

\ 

.+’

student that will convince you that he can perform in your area of 

expertise? ” The replies were translated into criterion statements 

of desired behavior and hence into performance tests. 

In the so calied “soft skills” areas --the leadership and attitude areas 

addressed by the present author in 19;2 (9)--subject area specialists 

working to the point of exhaustion with test psychologists progressed 

from a condition of no validating tests, through multiple choice know- 

ledge regurgitation tests, to case study type situational judgment tests 

of application of principles and knowledge. In addition, they have 

developed leadership peer rating scales on the premise that leadership 

qualities depend, at least in some degree, upon the perceptions of 

those who are to be led. In addition, they have developed a judgme’;ltal 

rating chart for race relations instructors to use in judging fitness 

of students to graduate as group facilitators. Significantly, here, 

according ‘to Rogers (lo), one of the appropriate behaviors is keeping 

quiet, remaining still, and allowing contentious group members to re- 

late to the group. 

Experiences at the three schools seem to point to the desirability of 

stating performance objectives in advance and testing them by some 

/ 

* 

313 

- 

‘.: ,_

means other than the objective type. norm-referynced paper and 

I 

pent il test. 

1 

(SLIDE OFF) 

Popham’s reason 5 (6d) is shown on this slide. 

(SIJDE 13 ON) 

SLIDE 13 REASON 5: “IT IS SO’IEHOW UNDEMOCRATIC.. . ” 

(LC’JDE OFF) 

Comparison of the three schools is on this slide. 

(SLIDE 14 ON) 

SLIDE 1-l . REASOS 5: “UXDE110CRA$IC” 

I . 

� 

This charge does not apbear to he a pro&em at any of the three 

schools (I, ‘, 

I 

Our students are encouraged to demand t know in advance the objec- 

tive of an impending lesson or exercise. / .\lost of the studerlts are in 

I 

courses of instruction by choice. planning thetr futures around thetr 

to-be-acquired ability to perform in a iven military job field upon 

/ f 

graduation. They tell us, sometimes vociferously, when an examina- 

tion fails to meet their expectations to allo\v them to demonstrate their 

314 

�

ability to perform future job tasks. True, there are some who com- 

plain about having to make decisions on examinations, but not m&y. 

(SLIDE OFF) 

Popham’s reason 6 (6d) is shown on this slide. 

(SLIDE 15 OX:) 

SLIDE 15 REASOS 6: “TII.AT ISN’T RE=\LL\- TfIE \V.AY ‘I’EACIIING 

3 

r3.. . ,t 

(SLIDE OFF) 

The comparison is on this slldc. 

(SLIDE 16 OX) 

SLIDE 16 REASON 6: “REALIShI” 

Discussion of our experien

Relatively few, including those of the United States Military Academy, 

have been trained and educated to perform a specific job. Now, 

suddenly, these graduates, who are now faculty members, are re- 

quired to write and teach toward behavioral objectives and to devise 

tests *requiring appropriate student behavior at a given criterion level 

under. controlled conditions. 

They were not taught like that. 

Teaching was not like that. 

Teaching is now like that. 

And, as we are seeing, the change from one frame of reference to 

the other spawns collisions. 

We have managed the overall change through successive approxima- 

tion. Quality controllers at first urged. helped, required instructors 

to USC action verbs in their teaching or training objectives, and later 

required more and more specificity of the conditions and standards 

of performance to be elicited from the student. 

Quality controllers are continuing to help instructors develop tests 

of identified job tasks which are as closely related as possible to 

job actions, conditions and standards. While some of the tests 

. 

316 

L . 

.

emain in the multiple c!loice mode, in which the st’udem selects one 

of the stated actions as his solution, we are pulling qway from these 

i 

as we develop the capability, resources, strategies,, and instruments 

to observe and grade outdoor and indoor situational performance. 

(SLIDE OFF) 

Popham’s reason 7 (Ge) is on this slide. 

(SLIDE 17 ON) 

SLIDE 17 REASON 7: “IN CERT.qIN SUBJECT AREAS.. . ” 

(SLIDE OFF) 

The comparison is shown on this slide. 

(SLIDE 18 bN) 

SLIDE 18 REASON 7: “~U;\IANITIES” / 

Obviously, all three schools encountered the problem (Id, 3b). 

In our early experience, 

:! 

1 

ing and performance testing were fine for/everyone else except him, 

because of the nature of his subject area. 

1 

an instructor wou d feel that systems engineer- 

A few went to great lengths 

to “prove” that performance testing thei b areas was impossible. Their- 

cases were strengthened, spuriously, by early inability of test 

. .- 

317

psychologists who were unfam.iliar with specific doctrine and subject 

areas to come up with instant test items, schemes or scenarios. 

1Ve managed this by continuin g to iterate performance testing as the 

goal under a stance we termed ”urgent evolution, not instant revolt - 

tion”; by frequent, open, and sometimes protracted test item develop- 

ment sessions between instructors and test specialists. These took 

on a Socratic aura in that the testers continually asked questions of a 

“could you do this” type until the instructor himself eventually con- 

structed the test. N-e, in the words of Doll and his associates (ii): 

“held firmly to the notions that people support what 

they help create, that people affected by change must be 

allowed acti\*e participation and sense of ownership in the 

planning and the conduct of the change. ” 

(SLIDE OFF) 

WC believe that the approach was successful because both the test 

t 

psychologist and tho instructor gave freely of themselves in order 

that the instructor might succeed. If this sounds humanistic, it is 

so intended, for it fits rather well with Buehler and Allen’s (1 1) dis- 

cussion of Bugental!s humanistic ethic: 

I 

! 

I 

318

“Integral to his scheme is his emphasis that the humanistic.. 

educator -person hopes that via his interventions and inter- 

actions he himself and the individual with whom he is inter- 

acting will emerge from that experience as societal change 

agents themselves. ” 

This Socratic: tedium is stil!. going on, but, as Bugental would have it, 

the conferences are more and more frequently now between instructor 

and instructor in the questioner and responder roles. 

At this point I should say that some of our best constructed paper-and- 

pencil tests are the military tactics planning and movement exercises 

laid on terrain maps. However, these tests have no precise sol,$ions. 

They are graded subjectively and serially by a team of 3 tactictans 

who use their own judgments as to how good a tactical move the student 

made and how :;vod the student’s justification of the move i:!. t’ailure 

rates are traditionally high even though norm-referenced grading has 

not been completely eliminated. Quantification of tactical judgments 

in order to determine why this is SO may indeed prove a difficult task. 

.. _..- *---I.-,--....,.---...-- ---. -_- -.._- _._.- ___.._,_.I._.,___ __. se'.. -.- 

. 

- . ..dbP‘. rri-..,-.rU.--LL--l,.,.~.U_l. 

, ;. 

-..r-. --* dr* _____ :;,,.&%uru;&&&-&,& 

1 

:< 

'\ _. * 

',. *.. ,' 

*a 

/- '.. _ _ yeeL.;- . i *' ' ' ! 

' ..,' .a: \ 

*\, a\ J) ..I' 

',*. '.,;":x.:\, 

: : ; \ 3; y.. ,; 

'.) 

,", .;, 

;. _ . 2. .:. 

,:. . . . .' , . :. "'-: 

-.a , . .\.I, . * . 

/ 

319 

.

. 

Popham’s reason 8 (6e) is shown on this slide. !” 

(SLIDE 19 ON) 

SLIDE 19 REASON 8: “WHILE LOOSE GENERAL STATEXIENTS OF 

OBJECTIi:ES. . . ” 

(SLIDE OFF) 

The comparison is on this slide. 

(SLIDE 20 ON) 

SLIDE 20 RIMSON 8: “IKNOCUOUS OBJECTIVES” 

As skills in identifying and writing behavioral objectives are being 

developed, it is simply axiomatic that poof-, weak, or innocuoustiQbjec- 

. . 

tives will be included (Id, 3b). The prospect is allowed for in the all 

i 

pervasive quality control’provision of the bystems model used at all 

three schools (1, 3). Infantry School instructors, and probably others 

I 

as well, found that trying to construct per ormance tests on innocuous 

r 

or poorly constructed behavioral objectives was essentially impossible. 

This impasse also held true when constructing tests on objectives which 

did not relate to the job for which the St&lent was being trained. 

320 

lJ 

i 

/ 

.r 

f 

\

Ii 

During conferences with an instructor who was having great difficulty 

applying his subject area content to the course graduate’s job, the 

test psychologist would help the iastructor define the problem but 

would never point the finger or try to dictate course or lesson content. 

As a-result, instructors sometimes drastically revised objcctir-es and 

instruction in oi-de;- to teach and test critical job skills. 

(SLIDE OFF) - 

Popham’s reason 9 (6f) is on this slide. 

(SLIDE “t OK’) 

SLIDE 21 REASOX 9: “MEASURABILITY IMPLIES ;\CC-

: 

) 

‘1‘. ‘. 

,---. .’ \ 

: 

i . 

‘:t 

( 

, 

:-.- I 

P 

. _a 

- 

! 

, . . -. __ 

. 

* 

. .__ 

Illinois State (3a) reported a fragmented accountz&lity framework 

which is also similar to ours as iiiuslra:ed on px-:i-ious slides 

(Slides 2, 3). t 

our faculty members do not fear x-countabilit?- In fact, most demand 

it. Some argued tenaciously for the right to giw tests of knowledge 

and comprehension. saying, “I \\-st;i to ~I>O\V tl\;it the student knows 

what 1 taught him. If he fails the < 3i?rs.e later, it Will not be because 

I didn’t teach him my subject. ” 

This stance \vas understandable. \i’e managed it by sny;ng in effect. 

“you may use \vhatex-er instru&icrnzl s:ra:egy you desire. including 

i 

tests of fa&ual knowledge. But when thy! student is certified in.youl 

a.’ 

area of espertise, he qust ‘be ce:.trfied on a performance test. 

I 

If 

you teach only facts. 

1 

and so~wone else teache s application of the facts, 

the test must be ori the application. He&Its of your in-cIass Informa- 

tion or practice tests can be @x-en 20 ih student’s faculty advisor for 

consideration if the student’s progress (pppears in doubt. ” 

We found one area which might ha\-e 

7 

more likely a liberal arts compassion for the stl;dents. The instructor 

I 

en feal- of accountability but was 

proposed an examinatior) of se\-eral problems wherein the student could 

322

omit one or more. This plan was abandoned when test specialists 

pointed out that it did not assure testin g ~,f all the training objectives 

in the instructor’s area of expertise. 

(SLIDE OFF) 

l~opham’s.reason 10 (Gg) is on this slide. 

(SLIDE 23 ON) 

SLIDE 23 REASON 10: “IT IS FAR 110iZF: DIFFICULT.. . ” 

(SLIDE OFF) 


(SLIDE 24 ON) 

. 

SLIDE 2-f REASOS 10: “GENERATE OI33SCTNES” 

Whereas the ISU self-instruction packages C3b) are said to contain all 

the compctencies considei-ed necessary fr\r teacher training, XC noted 

previously that not all the higher order cotupetencies were specified. 

At Louis\-ille (lh), workshops in systems language and in writing 

behavioral objectives were run initially but were not continued after 

the first year. 1nster.d. the school entered a plateau phase--a time 

I 

during which concepts, ideas, and motix-ntions hopefully are being 

internalized. 

’ t 

323 

� � 

_-. 

. . 

. .

In our experience. some instructors raised the questio:l as to who 

properly should write the training obje’ctives. Outside agencies? 

Systems engineers? Instructors? Some instructors believed very 

strongly that their job was to teach, and nothing else. Some felt that, 

because of their junior officer grades, they lvere singularly unqurili - 

lied to construct and conduct tests. 

We managed to gain their cooperation in helping to write objectives 

by stressing: 

1. iVeed to certify graduates; 

2. Graduate ca’pability to pei*form in the instructor’s area of 

expertise on tltc job; 

3. The log:icallJ inseparable relation of teaching and testing; 

4. Teaching and verifying or testin,‘5 as two aspects of the s;\me 

behavioral modification process; 

5. Qualification to teach carries v:ith it the obligation to test; 

6. Writing the action, conditions and standard of the training 

objective as tantamount to writing the test. 

IIere again, the approach was by successive approximations to the 

ideal product. 

324

: 

: . : 

(SLIDE OFF) 

.I 

Popham’s reason 11 (Gg) is on this slide. t 

(SLIDE 25 ON) 

SLIDE 25 RE.AX)S 11: “IN EVALUATISG TIiE !VORTH OF IKSTRUC- 

TIONAL SCIIEMES.. _ ” 

(SLIDE OFF) 


. 

I 

Unforeseen events puched all three kchools. Xcntioned earlicr~v~~c 

I 

I 

Louis\-illc’s (lh) underemphasis on objectives for classroom manage- 

merit and Illinois State’s (3~) facuIty\ and students’ slow, agonizing 

adjust merit to competenq r-based teaIher 

education. 

I 

In our csperience. the role of the 1 ittle 

10 minute quiz became the 

focus for philosophical c.lash. On the one h-??d were those who wished 

/ 

to verify learr?ing of facts witho&application of those facts to the 

learner’s future j*>b. There were also those who viewed education 

I 

325 

I / 

b 

t:, 

‘4 p ’ . . . 

1. 

. 

_ 

v, \‘ ._- ,.

t . ’ -.* ., 

.! 

as punishment and the quizzes as a whip, an outside motivator or 

disciplinary measure necessary to assure that students read their 

homework assignments. These proponents would penalize students, 

academically, for poor answers to pop quiz questions. 

The problem was overcome by e.‘ttensive revision of school testing 

policv and by separating thesr britf tests into t\x;o categories: 

*’ - 

1. Instructional techniques,. progress checks or feedback devices; 

2. Verifying techniques, or perfor.nance tests of ability to 

perform job tasks. 

(SLIDE OFF) 

Hopefully this paper has anslvcred, at least in part, the following qucs- 

tions about criterion-referenced measurement posed Lx? Dziuban and 

Vickcry (12) in February 1973: 

-’ 

“HOW are teachers to make the trasition from the more 

traditional practices and what are the consequences?” 

“Can present instructional material be adapted to 

criterion-rieferenced measurement?” 

.! . :( ,/ 

326 

‘> 

Z

“Will a new system ultimately result in substantial 

additional demands upon teachers, many of whom 

are presently operating on overcrowded schedules?” 

CONCLUSIONS 

I believe that’we can draw the following general conclusion from the 

experience at the various schools as presented and discussed here: 

Experiences with radical change from one frame of reference to 

another have been about the same at liSAIS as they have been at other 

schools that \VC know about. In the words of Doll and associates at 

Louisville (lb). this conclusion 1s stated as follows: 

11 . . . problems which have arisen in bringing about 

. 

educational reforms. .exemplify and reinforce findings 

of previous research. . . ” 

CLOSE 

This paper is not intended to malign 

the words of Doll and his associates 

“It . . . is intended as a salute to 

L ’ . 

i 

/ : 

,‘.I ..’ 

any body. Instead, and again I use 

(lj): 

people who tried boldly to 

make a significant impact on problems which bedevil every ‘.W 

large . . . school system. Their mistakes stand out clearly 

: 

I( 

_ -.__ /’ 

-. - . 

i. 

:. 

.,’ 

‘.. 

--:

- -.. . 

bEcause they attempted more than anyone else rb date. 

Their successes may seem unduly minar becaqe they 

I 

attempted to conquer a whole mountain range yd now 

only control some of the peaks. But they have at least 

gutten past the foothills, atid from the peaks they can 

see more clearly than before. They can continue to 

climb. Hopefully, they can help those of us at base 

camp who may try to join them. ” 

* 

328 

I 

-2. .- -. , . 

I 

_‘. 

>

REFERENCE 

1. Doll, R~,~ssell C. ; Love, Barbara J. ; and Levine, Daniel U. 

“Systems Renewal in a Big-City School District: The l.essons 

of Louisville. ” Phi Delta Kappan, April, 1973, pp 524-534. 

la. Ibid, p 527. 

lb. Ibid, b 530. 

LC. Ibid, pp 526, i. 

Id. Ibid, pp 5Pi, 8. 

le. Ibid, pp 525.7. 

lf. Ibid. p 528. 

lg. 

ibid, p 525. 

lh. Ibid, pp 526-E. 

li. Ibid. p 529. 

Ij. Ibid, p 524. 

3-. Kramer, Bryce R.. and iineiscl. Richard S. “Systems Approach 

to Evaluation and Oynlity Control of Training. ” C;S Army Infantry 

School, Fort Bcnni:ig, GA 31905. Pr*e~cntctl to 1lilitat.y Testing 

Association Conference. \\‘ashington, D. C., Scptcnlbcr, 1971. 

2a. Ibid, p 6. 

3. Getz, IIownrd; Kcnncdy. Larry; Pierce, ivaltcr; I’dwards, Cliff; 

and Chesebro. Pat. “From Traditional to Coml)ctc~!lcS-Rascd 

Tcache r Edwat ion. ” Phi Delta Kappan, January, l!K3. pp 300-302, 

3a. Ibid, p 302. 

i 

-. - __ 

: 

.- 

_.- 

.- 

. Ji 

329 

3 

_..’ L 

. 

. 

. r 

__- -’ 

. . _ 

,

3b. 

3c. 

1. 

5. 

6. 

Ga. 

6b. 

(iC. 

6d. 

Ge. 

Gf. 

6&t- 

i. 

8. 

Ibid, p 301. 

Ibid, pp 301,2. 

Gross, iVeal; Giacquinta, Joseph R. ; and l3ernstein, Marilyn. 

“An Attempt to Implement a Major Educational Innovation: A 

Sociological Inquiry. ” Cambridge. Harvard University. Center 

for Research k Development on Educational Differences, 1968, 

Chap 6. 

Gerber, J. I:. Jr. “Proactive and Retroactive Effects in Programmed 

L,earning. ” In Ofiesh. G. D., and Xleierhenry, W. C. 

(ED), Trends in Programmed Instruction. Washington,- NEA, 

1964, pp 232-234. 

Popham, W. James. “Probing the Validity of Arguments Against 

Behavioral Goals. ” In Kibler. R. J., Barker, L. L., and Miles, 

D. T., Behavioral Objectives and Instruction. Boston, Allyn 6r 

Bacon, 1950, pp 115-124. 

Ibid, p 116. 

Ibid, >p lli. 

Ibid, p 118. 

Ibid, p 119. 

Ibid, p 120, 

Ibid, p 121. 

Ibid, p 122. 

Bloom, Benjamin S. Taxonomy of Educational Objectives, Handboo_k 

I; Cognitive Domain. New York, rllcKay, 1956. 

Towne, William B. Sr. “Differentiating Between Situation-Oriented 

Items and Setting-Oriented Items in 110s Proficiency Testing. ” 

5 Seymour Pi. , Xorth Augusta, S. C. 29811 (Unpublished manuscript, 

personal communication), October, 1972, p 3. 

330

9. Gerber, J. E. Jr. “Evaluation odleadership and Communicative 

Skills. ” In Haines. R. E. Jr., and Hunt, I. A. Jr. (ED). CONARC 

Soft Skills Training Conference. Fort Bliss, TX, US Continental 

Army Command, December, 1972, Vol IV, pp 28-34. 

10. Rogers, Carl R. Carl Rogers on Encountek Gro*. New York. 

Harper & ROW, 1970, pp 48, 66 et seq. 

11. .Buehler, Charlotte, and Allen. Xleianie. Introduction to llumanistic 

Psychology, Monterey, CA, Brooks/Cole. 1972, p 78. 

12. Dziuban, Charles D., and Vickery, Kenneth V. “Criter;lon- 

Referenced Measurement: Some Recent Developments. 

Educational Leadership, February, 1973, pp 483-486.

“V’ 

SYSTEAfS ENGINEERING PROCESS 

___-_ e ..___ 

W WIv 

-. _. ._. --__ 

-_ _._-- ._ ____ _c_7 

CONDUCT Of TRAINING

, 

c3 

tx 

0 

e 

ur 

N 

= 

cn 

a0 

i 

I 

5= 4 

!2 

% 4 

- I 

041 

% 

0 

. 

; 

I 

I 

I 

4 

: 

0 

I 

\ \\\I

. . . - 

‘-..’ ‘. 

. . 

- - A 

, 

i h 

2 ,,_ 

:. 

,. 

.*./. 

‘. j 

THE FOUR AREAS OF RESPONSIBILITY FOR EACH 

PROFESSIONAL SEQUENCE STAFF MEMBER \% 

i 

A rGB WllHlN 6 INOlVlOUAl IfAM C StOUfNCf GUI01 

0 COMMIlTtf 

1Hf SYSltM PARlICIPAlION R1VISION 

MtMBlR 

* ’ MAINlAIN LIBRARY 

IASKS ., * * Bull0 NtW * i AlTtND SUBltCl 

MAlfRlAlS * AlltNO ltAM PACKACtS O R 

MffllNCS 

. COqRDlNAlt IUNIOR 

PARTICIP’ ,:ON tXPtRltNCt * ; 

M!!?INGS 

CJRRtCl PAPERS 

RtVlSf 0 1 0 

PACKACtS 

* . PROViDi ftiDBAiK 

f O R CtNTRAl 

* . PRfPAt.t INSlRUClIONA~ 

SiSSlON SCHtOUltS 

* 1 SUPtRVlSt M I C R O - 

itACHINQ!B 

- UPOATt M~SSAG~-$@ITI-- 

� 

* 

: OfftR ftlOBACF 

10 SlUOfNlS 

1 PRGVlOt PiRSONAl 

AND ACADEMIC 

.-A 0 UC... _ 

* I’ Bull0 NtW 

!iSlS O R iXf’-AND 

OLD 1iSlS -- 

* I B, l1O NtW M[U,A 

0;; RtVlSt 0 1 0 

MEOIA 

* ? 

POllCY COMMllTtt 

VOlt ON ACCtPl 

A.;lCi Of NEW 

PACKACtS AND 

1tSlS I N AREAS 

O f lXPfRTlSt 

TV tOI1 PROf 

StOUtNCi 

LSSIONA! 

CUlUb 

* ;, INlllAlt A N 0 

~ONOUCl 

INSTRUCTIONAL 

1 ilC 

1 

-./ 

flC 

--. 

-- 

,-‘. 

-. . . 

* OVtRSft IHt SURVtll SiSSIONS 

* a; 

IANCi SYSltM 

OPtHAlt 1Ht ItSlING 

� r SiRV! AS lfA# 

CHAIRMAN ON A 

* APP[ltS ALSO IO USAIS INSlRUCfOR 

PROGRAM 

ROiAlING BASIJ 

* 9 COOROINAft CRfiJlf NU t ir v, 

CRfDll f’OR GRAOt C A R D S 

t: 

‘0 i IL _. _. 

Fi 

cc

* 

: . 

.‘. 

\. ,. .;’ ‘... . 

- 

. 

335 

c

, 

336 

, 

L 

0 

A 

u 

a 

> 

- 

a 

. 

. 

u 

a* 

= .A - 

- -- 

- CY 

. 

W 

3 a0 

0 

. 

I 0 

-rli 

rl! 

1 

) 

D 

- 

i3 

u 

e 

awn= 

-

IC’ - 

1 

I 

I . 

HIERARCHY OF MENTAL REQUSREMENTS 

FOR PERFORMANCE OF JOB TASKS, 

- .__ 

COMPREHENSIOFJ 

EVALUATION 

HIGH

. 

S:,IDE 7

REASON 1 UNEXPECIEO OPPORTUNITIES" 

EXPEIIIENCE AND MANAGEMENT 

I111 i Ulll\bli! i 

i lpiki!r(: ik IN PASSING 

SIUCENTS DEMAND PRECISE 

C.OAl STATEMENTS 

AOULT STUDENTS OlSClPllNE 

THEMSLLVES AN0 

INSTRUCTORS wHWiJRA~\, 

fROM C,OAlS 

VAMAuivihl REOUIRE 

CRITERION TESTS FOR JOB 

PERfORMANCE 

!XPtHltNI,fll .YES 

COURSE WRITERS 

,INSTRVClORS RESTRICTlO 

BY PREClSElY SlATi 

OBJECllVES 

'+ANACIM1Nl CONTINUE 10 

REOUlRi FRECIS! WRITING 

ix?ircii hl io IN REVERSE 

OBJECTIVES CENTERED ON 

CROUP PROCESS AN0 

SlUOENl CENTERED 1EARN 

INI. NEtllECiEO STRUCTURE 

AN0 AOU!T GUIDANCE 

-. .I 

-‘_ 

MAhACiMfhl fACUlTY -. 

Sflf MANACEMENT 

1---.- 

I - 

’ , 

3‘ ‘\ 

rx 

r 

3 

M 0: 

-.\

-- 

‘ . -. 

..’ ._:._. 

. . _ . . !. . 

. ..- :-- . .-.\‘.\ \ 

: \ 

, 4 _ . 1

*. 

. t 

. . 

‘aj. 

i ‘.. 

r 

i .,I. ; 

I . 

i 

i 

: 

‘r’ 

I 

REASON 3. “AIIITUDE CHANGE. 

IISAIS 

i r Pt Hlf NCi YES 

CERTAIN PROFESSION Al 

STAFFERS FAVORED STATUS 

OUO 

MANACLMtNI CHALLENGE 

PROFESSlONAllSM 

EXPERIENCE AND MANAGE 

ISU 

- 

__---~~~txPtRltNCiO 

YES 

VARIOUS LEVELS, OF 

PROFESSIONAL STAFF 

ENTHUSIASM TO CHANGE 

I N IRADITIONAI. 1EACHER 

ROW 

MANACiMtNT CHAllENGE 

PROFESSIONALISM . 

ACTIVE INSERVICE 

PROGRAM FOR GRAOUAL 

C H A N G E O Y E R 

1 

I !xPtRltNLtO YES 

:l, SOME TEACHERS & 

PRINCIPALS OPPOSE0 

CHANGE TO “SYSTEMS” 

CONCEPT 

12) T E A C H E R 

PARAPROFESSlDNAl ROlE 

CONFLICTS. 

MANAGtMtNl TRANSFER 

OUi WITHOUT PREJUDICE 

‘. _I . .’ 

i 

,. ..“-d ” d 6 . , - . ..I

342 

. ,I 

i / 

.-. - . _ 

. 

3LIUC 11

RLASON 4 .'OEHUMANIZINC 

EXPERIENCE AND MANAGEMENT 

;IP!i;ljN i: YES 

- -_ _ 

SCHOOLCIIURSES AN0 JOB! 

STRESS HUMAN INTFROtPihS- 

INCE- CONlRlVfO TlSTl ).- 

RESTRICT MlASURABlE 

STUOENT BEHAVIOR 

vi+: :iUIrr' DEVELOP TESTS Di 

APPllCATlON OF JUDCMENT IN 

JOB SITUATIONS 

REDUCE RELIANCL ON MULTlPlt 

CHOICE TESTS 

COMPETENCIES INFERRED BUT 

NOT MEASURED BY PAPER ANC 

PENCIL TESTS INCOMPETENT 

STUDENTS LET BY ON NORM 

~ REFERENCED TESTS 

~~H::,!M!NI TEST OVERT 

BEHAVIOR REDUIRINC SKI11 lh 

PlANNlNC IOR. ANALYZING 

lNlE,RPRETING. AND EVALUAT- 

ING SITUATIONS OR BEHAVIORS 

E~LIMINATE CONlRlVED 

MUlTlPlE CHOICE IESTS 

DCVElOP TRANSITIONAl NORM 

, ;O CRITERION TEST SlR.ATEClES 

: I P [ k‘ I ! N’! i r IN REVERSE 

WORKSHOPS ON NEW 

HUMANIZED BEHAVIORAl 

SYSTEM" 1EfT TEACHERS 

UNPREPARED FOR PUP11 

BEHAVIORAL PROBlfMS 

v:~cl,tu:rc~ SElf 

MANAGEMENT 

, 

. . 

.._ . 

-t

344 

U 

SL:IDE 13

REASON 5 "UNDEMOCRATIC 

EXPERIENCE AND IVlANAGElViENT 

STUOENTS DEMAND'10 KNOW 

IN ADVANCE THE INTERMEDIATE 

AND TERMINAL BEHAVIORS 

EXPECTED 

STUOENTS OEhkND ---\., 

OPPORTUNITY 10 DEMONSTRATE 

PROFICIENCY 

vi+!::iv!ti: lEVIlOP TESTS IN 

A PERFORMANCE MODE 

OPERATE 0UAIITY CONIRO! 

MODEL 3llDE 1 

NOT A STUDENT COMPlAlNT 

ABOVT SELF STUDY PROGRAMS 

PARENTS. TEACHERS. PARA 

PROFESSIONALS. AND PUPllS 

AR[ GOVERNING BOARD 

U!N!i,i Vi R! AGREEMENT IN 

ADVANCE PARENTS AN0 

PUPllS ARE FINAl AUTHORITY 

ON NEED FOR PROIECT 

MODIFICATION 

I

3 

_ . _ 

, : . _‘, 

. . 

346 

SLIDE 15

! - 

i. 

’ ‘. 

, . 

. . 

.a , 

: 

. 

0 

e 

REASON G "REALISM' 

EXPERIENCE RND IVIRNAGE 

:1~tRltNI.tn us 

INSTRUCTORS EMUlATE THEIR 

COLLEGE PROFESSORS IN THE 

6NOWlEOCt ACOUISITION 

WORlD 

INSTRUCTCRS RElUCTANT TO 

ENTER THE WORlD OF IOB 

PERFORMANCE ICSTINC 

MANAl;t Mf N' SUCCESSIVE 

APPROXlMATlONS 

SUCCESSIVELY MORL PRECISE 

STATEMENTS OF JOB iEST 

ACTIONS CONOITIONS AND 

STANOARDS OF PERFORMANCE 

- 

i IViF’liN 1. NO 

USE Of Sflf PACtD 

iNSTRUCTiONA! PACKAGES 

IN TEACHfH tIru~A11uFc 

CONSTITUTES THE REAl 

WORlD fUR THE STUOENT 

IN TEACHER 3REPARATiON 

t’!iNAi;j HI tji CRAO'JATES 

EMULATE COLLEGE 

PROFESSORS BECOME 

DEVELOPERS Of SElF 'ACED 

'PROFESSIONAL SEQUENCE 

KUIOES 

'J?H?GiH!N~ TRANSFER 

WITHOUT PREJUDICE

. . 

348 

/ 

I 

1

a 

0 

- 

W 

a 

c 

5 0a-07z- 

I 

0 II 

z -7- 

W 

d 

a 

_- 

z 

a 

W 

W 

W 

W 

0 

z 

u 

a 

0 

- 

0 

C 

0 

- 

W u 

W 

a 

a 

W 

L 

I 

z 

u 

VJ 

W 

c3 

z 

u 

z 

a c- 

z 

0 

v 

I: 

a 

0 

a 

0 - 

a 

- 

c1 

c/, 

z 

- 

349

REA SQW 80 

-0 

” WtlILE LOOSE GENERAL STATEMENTS OF OBJKTIVES MAY 

+-=N 

APPEAR WORTHWHliE TO AN OUTSt!IER, IF MOST 

~o

,c 

I . 

REASON 8 INNlrcUOUS OBJCCTlVt. 

EXPERIENCE AND MANAGEMENT 

- -_ _ 

I;OAlS OF-S!JBlfCl MATTER 

EXPERTS WERE EXTRANEOUS TF 

NEEDS OF COURSE GRADUATES 

uivi!,iwi~: TIGHTEN SYSTEMS 

:NCINEERINC 

DEVELOP JOB RELATED TESTS 

irP!kltNl!L. PERHAPS 

HIC;HER OROfR COMPElfNCIEj 

N!!T SPECIFIEO 

~ih~i,iwitii ANALYZE AND 

UPCRAOE 

.,OMt BfHAVlORAl OBIECTIVFS 

"DID 1lTTlf BUT PARROT 

POINTS OF VIEW STRESSED BY 

AOMINISTRATORS OR TRAINERS' 

‘J!hki,!Wihl FACUITY 

SELF DEVElOPMENT

.: 

_._’ _ 

. . . 

.:.-*..‘. . - 

- 

: _-. 

352 

SLIDE 21

; 

, 

i 

f ’ 

i t 

I. 

:, 

‘ 

c 

REASON 9. "ACCOUNTABIIITY 

ir(~jkjjh:,ii! IN REVERSE 

INSTRUCTORS WANTED TO 

TEST STUOENT KNOWlEOGE 

LEVElS FREQU&NJlY TO 

AVOIO BLAME FOR fAIr*S- 

h4AhAbtMtNI USE 

KNOWlEDGE TEST AS 

INSTRUCIIDNAI STRATEGY 

USE PERfljRMANCE TESl AS 

GRAOUATlOic REOUIREMENT 

EXPERIENCE AND IV'IANAGEMENT 

-- 

a 51: 

- 

ACCOUNTABlllTY 

FRAGsMENTED 

SELF PACED PACKAGES 

ARE GROUP PRODUCTS 

--- 

‘+‘AhAL1M1HI CONTINUE AS 

PlANNED 

! !I II I s v I 1 1 i 

: XP! Hli h! I!’ NOT A 

PROBLEM 

TEACHER WAS 

'FACtlIT ATOR 

HAhAC!Mthl IN PARENT 

TEACHER PUP11 "MINIBOARO 

GOVERNANCE. PARENTS b 

PUPiS ARE FlNA: 

AUlHDRlTY ON NEED FOR 

CHANCF 

. 

v 

7 

-_

: ’ 

. . 

1 

t 

I 

,! . 

f 

/ 

\ 

. 

SLIDE 23

. 

REASON 10: “GENERATE OBJECTIVES’* 

EXPERIENCE AND MANAGE 

USAIS 

EXPERIENCED: YES 

INSTRUCTORS DESIRE0 TO 

TEACH, LEAVING DOCUHENT- 

ATION AND TESTING TO 

OTHERS. 

MANAGEMENT: SUCCESSIVE 

APPROXIMATIONS: TEACHING 

AND TESTING ARE TWO 

ASPECTS OF THE SAME- 

THING. 

EXPERIENCED: NOT REPORTElI 

MANAGEblENT: ALL NECESSARY 

COMPETENCIES INCLUDED IN 

INSTRUCTIONAL OBJECTIVE 

PACKAGES. 

MEMORIZATION AND RECITA- 

TION OBJECTIVES WERE MORf 

BEMAVIURALLV ORIEHTEO 

THAN WERE “HUblANISTIC” 

OBJECTIVES. 

#ANAGEYENT: WORKSHOPS 11 

BEHAVIORAL OBJECTIVES. 

PLATEAU PHASE: RELAX 

STRESS ON BEHAVIORAL 

OBJECTIVES. 

INSTRUCTOR SELF 

MATURATION.

. 

‘. 

. 

. . I... _ . _ - 

1 1 . 

-0 

, 

” I# EVAlUATI#G THE WON4 OF I~SYRUCYIO#AL SCHE 

-; 

IT IS OFYEH THE U~At4YlCIPAYED RWJLYS WHICH ARE 

-+-+ REA~Y-l~38RYA~Y-,--~UY PRESPECIFIED GOAL! 

,I’ . 

..I 

MAY MAKE T H E EVALUAYOR IDBAYYE#YIVE T O THE 

-n 1, -.. _ 

--.- 1 

-. 

U#FORESEE#. ”

-.. 

\ . 

I._ 

. 

. . . 

L 

.\. 

i * 

c 

. 

.‘, ” 

T “7. 

-. , 

/. 

; 

,’ 

/ 

55 

+&. ‘y. 

c C.” 

;’ 

-- 

: 

\ 

.! 

,’ 

i, 

I.’ 

I 

I 

I 

:. 

. . 

I 

..’ 

.’ 

. 

“v‘ 

RfASON II: “UNFORESEEN EVENTS” 

USAIS 

EXPERIENCED YfS 

‘-PtiilOSOilil-CA1 ClASH OVER 

ROLE bf IOMNUTE QUIZ. 

T.- 

MANAGf Yf NT SEPARATE DUizZf! 

INTO TESTS OF COI4Tf.k MD 

TESTS OF JOB TASK 

ACCORaPLISHblENT 

EXPERIENCE AND MAFdAGEMEPlT 

EXPERIENCED YES ’ 

STUDENTS LACK SELF 

DISCIPLINE TO COklPllTl 

SELF-PACED PROGRAM. 

STUOENTS MUST ADJUST TO 

UNREIINTING PRESSURE FROM 

PROGRAM START TO FINISH. 

SLOW. A60MZINC ADJUSTMEN 

BY STAFF AND STUDENTS. 

uANAGtU[NT HllP STUDENTS 

PLAN WORK SCHEDULE. 

loulsvllll 

EXPLRIENCED: IMBALANCE IN 

SPECIFYING TEACHER AND 

CEARNER BEHAVIORS. 

b!ANAC~#ENT: TEACHER SELF. 

MANAGEMENT:

, 

. 

: 

CRITERIOX REFERESCED PERFOftWNCE TESTING 

IS COYIAT AR.% SKILLS 

, John F. Hayes 

UR,S/Katrix Company 

~iy presentation today represents.a progress report on the work that 

I-IRS/Matrix Company personnel have been doing in the area of performance 

proficiency testing. While we have been working in this area for over 

four years, it is specifically our current work for :he Army in the 

measurement of combat arms skill proficiency that I want to focus on today. 

This work is being done under contract with the Army Research Institute. 

Dr. Frank Harris, Dr. Robert Root, and Kajar Larry Cord are the AR1 

personnel monitoring this work. Other project personnel from URS/Xatrix 

include ?lr. Ray Griffin, Dr. Boyd Nathers, and Hr. Don Jones. 

As background to this description, I would like to present a general 

framework of performance testing that has evolved from our work and the 

work of others in this area. Performance testing itself, of course, is not 

a new concept. It has long been recognized as one of the more important 

techniques in the area of training assessment. It has, hol;ever, traditionally 

presented problems in terms of the administrative load that it imposes on 

the testing function and in terms of test reliability. 'It takes a great 

deal of time, support materials, and it requires numerous personnel to 

execute. Siill, it represents the most valid means of assessing proficiency,. 

This is particularly true with respect to training for specific job 

assignments which is a major concern of the military servica*s. 

Xost general training development models tail for the development of 

. 

criterion tests early in the training development process. Criterion tests 

developed from such models are based not upon the training content being 

developed, but upon the same job requirement information that was generated 

I . 

! 

358

_/’ ‘.., ‘/ 

I I 

for the training material development itself. The test development 

( 

effort, then, is a parallel and concurrent but fndependent effort. In 

this way the criterion tests measure whether the student’can perform 

I 

criterion job functions upon completion of train’ing, not just whether 

or not he learned what was in the course. 

“Criterion referenced” tests as we are employing the term means 

:I 

more than scores measured against fixed as opposed to relative standards. 

By criterion referenced we also mean that skills are measured in the 

context of the application situation as completely as possible. By 

criterion referenced we mean that the tests are designed so that 

performance on them represents a true measure of job performance, rather 

than measures of the possession of skills presumed to contribute 

positively to job performance. In the ‘area of Infantry and Armor combat 

skills, there are no acceptable job performance conditions against 

which to verify or validate evaluation’mcasures. In other areas in 

I 

which we have worked ,,electronic and mechanical repair maintenance, vork 

I I 

samples could be easily constructed. 

I 

,For the Infantry and Armor soldier 

1 

.: :t 

: 

this was not the case. The job of th combat arms soldier fs not to apply 

a specific set of skills on a repetit It is to recall and select 

required skills as needed by unique sJtuations and integrate and apply 

f I 

them under conditions of stress ranging from mild to drastically severe. 

What we were seeking then was to approach and approximate those conditions 

in our evaluation of his job profit 1rncy. 

In our development of criterion referenced performance tests, several 

guiding principles have become common td all of our developmental efforts. 

. 

359 

/ ‘,. ‘_ 

,” i /’ 

’ i’ 1, e 

“.

fie first is that we start by defining actual job tasks. These tasks 

come from the analysis of the performance of job incumbents. This 

information can be gathered in a variety of ways. It can be done through 

conventional surveys of job requirements and job duties, or by obser- 

vation of job incumbents in the performance of their duties, or it can 

be done through the “panel of experts approach”. The method used has 

to be selected based on the best source of information available. Such 

methods are not a central topic for today’s discussion; however, a 

fundamental premise of our test dev*lopmcnt apTroach is that through the 

use of such dat‘ .validitv is built into the test situation from the 

beainninq. The validfty of the performance test results, then, is not 

demonstrated by comparison with other indices such as supervisor ratings 

or other test score results that are more typically used for test 

validation. Content validity is estab?.ished as a precondition so that 

the final tests represent absolute criteria against which any other 

indices of job performance can be :hemselves validated. 

The second characteristic of our performance tests arc that meaning: 

ful size tasks are selected, c;hicb require skill integration and sclcction 

from a total repertoire bf application skills. In performing a job duty, 

it is important that the job incuzbcnt be able to handle all of the 

parameters of that job duty in the contest and sequence that he will be 

expected to perform on the job. i’nder our concept, performance testing 

is not just the isolated performance of defined skills, but would include 

the selection of’ skills to be applied at any given time based upon the 

. 

parameters of the sttuation. For example, a technician may be perfectly 

360

, 

capable of repairing a carburetor, and this could be tested specifically. 

It is also important in our view, that he be able to recognized when the 

carburetor needs to be repaired. That is, he must be able to isolate 

the carburetor as a malfunctioning component from among the myriad of 

other engine components. Further, the performance of this task should 

be done in the context of the operational environment. To the extent 

possible the individual should have to locate the problem, select tools 

and equipnent , use appropriate job documentation effectively, correct 

the problem, and, in general, demonstrate his ability to operate within 

the operational environment. This then, is uhat we mean by total ski-1 

integration. 

The third maijor characteristic of the performance oriented tests is 

that the measures that are used for gauging successful performance are 

product-oriented rather than process-oriented. When a man is functioning 

as part of a system he has inputs and outputs the same as any other 

element of that system, either man or machine. The functioning of the 

I 

system depends upon the quality and timeliness of the man’s outputs, 

These characteristics can be measured as can the outputs of any other 

component of the system. 

Some of the more compelling aspects of performance for measurement 

purposes are often processes and in nany instances the quality of such 

processes is often a matter of subjective judgemen: on the part of a rater 

D 

or evaluator. He will observe whether the performing individual did the 

task in accordance with prescribed procedures without questioning whether 

those procedures were in fact the only way of accomplishing the output. 

.- ‘* _--. 

361 

, -. - . 

..A ._. 

. 

,.’ 

/ 

.__. ‘/. 

_’ 

. 

- 

, 

4’ 

’ 

,,.’ 

I 

, 

: 

s 

/’ 

’ 

,,’ 

/- 

, 

: 

: 

.-.-a 

,. -c 

_.a. 

G$&‘! 

* 

__.

In the design of our performance tests, we have conscientiously selected :v 

,. 

criterion measures which do not rely upon the (subjective jcdgement of the 

? 

rater. The guiding philosophy of this test development effort was that we 

do not care how a man accomplishes his job or ,his task. What we are 

I 

concerned with is whether he got the job donep i.e., produced the required 

outputs at the proper time. In this way, the testing procedures can 

remain independent of the,,specific processes that are prescribed. 

Applying ithese principles to testing combat skills of the Light 

Weapons Infantryman (XOS llB), we began with a definition of tke situations 

in which the individual fs experted to perform. These can generally be 

defined iq terms such as: assault and defense, reconnaissance patrol, 

defensive patrol, night infiltration, and anti-tank actions. The next 

step was to perform a behavioral :ask analysis, in cue-rcc--zs ‘S * 

to determine exactly what the individual soldier is respondin, to in each 

of these situations (the cues) and whft outputs (response) are expected 

of him. 

1 

One of the firs? characteristics, that became evident was that while ,’ 

we were. talking in terms of individual proficiency and individual testing, 

the soldier seld~~m acts as an individual in the defined criterion situations. 

He is virtually always a member of a I larger unit, most often the fire 

team, operating as a member of a squ d. As we proceeded, it becarr.e clear 

b 

1 

that we had to test him in that context. , 

Another critical factor that b came clear is that the infantryman is 

% 

operating most of the time against an intelligent opposing force, *

is neutralized. Initially :his element was consider- : **:* dyn? . . 

t,oo difficult to provide to be considered for inclusion in 3 tL * 

tmre we exained the situation, ;rovever, the cleare’r f: c~-r;h. * +.dt 

without this element:, atteqts to tesf :he co&at efricirncy .-f the 

individual soldier were going to be Loefully inadequate. ‘Lhile Ye 

investigated hit-indicator systeas such as laser beazs and rsdfo actfvatr 

equip&t, w ‘felt strongly that., to be effective az~d usable, any system 

tha: ve dc.Jised had to be free of ccuplicated equi-,=ent. 

Eventri;?ly LT devised a sys:ez which uses a t’-o-Cigit nu;=ber pl.lcrd 

on each i*:3?v-idual’r; heloet aztd rifles equipped wi:h tcl~scspes. h’hen an 

inraotrpan L’an Gistingztsh an o?ponent’s m.mber through his scope and 

‘fires a blank round $, it is ass-d that he would have been able to kill 

that individual. by plcci~g test controller personnel =ith each of the 

opposing elements and equipping the controllers with a radio, individuals 

can be knocked out of action on each side as they are effectively engaged 

by -the opposing force. The protlen is thus allowed to piay itself out 

or. a realistic basis. Bg experfzenting wi:h combinations of color, 

background, and size of the numbers and :he power cc the scope, a :onbina- 

tion of number- ~olou and umber-size, and scope poser, vas achieved vhicil 

duplicates closeI~r the effegtive range of the Xl6 rifle. For 

practical purposes this range is 200 meters. Xhile the actual range cf 

the .X6 is nuch langer, the combat situation seldozz pro+ +es accessible 

targets beyond that distance. 

The develbping of this scoring system opened zew possibilities to 

the use of combat situat’icns for testing purposes. Tine situations could 

j ”

now provide realistic cues which ilI’C! highly similar to the combat situation. 

They also provide the opportunity for the individual soldier to select and 

employ, in concert r;ith his fellow squad members, what he considers to bc 

effective responses. The required responses are not pre-defined; he has 

to select them from his repertoire, under stressful conditions, at the 

most appropriate time, in order to bc successful. Further, the situation 

will .r~~ change as a result of the individual acts taken on both sides. 

Uhcn the individual perceives that he is affecting the outcome, he becomes 

a very interested and enthusiastic participant in the exercise.- When the 

situation does change as a result of actions one takes, it becomes a 

measurencnt criteria. One of the most important elements of t!,e infantryman’s 

life is’ that fact that he must constantly respond to an environment that 

is changi.ng under very stressful conditions. And his responses cannot be 

isolated in’ terms of 

coordinated with his 

Techniques were 

effects of the other 

his own situatjons as an individual, but nust be 

fellow squad mcmbcrs, if they are to be effective, 

:hen developed for simulating and measuring the 

weapons aVailiI!)lC to the infar.trynm. The tests 

now permit the use of grenades, claymore mines, booby traps, anti-tank 

weapons, machine guns !, and artillery in realLstic fashion. As a result, 

the overwhelming opinion of experienced combat infantry personnel is 

that we have, in fact ,’ created situations which call for very realistic 

responses in conditions very closely approximating actu.11 cocbat. 

Having achieved this, it was then necessary to apply :he necessary 

controls to achieve individual testing within this context, The basic 

data-gathering instruments during these situations are the test controllers

previously mentioned. In addition to being equipped with radios in order 

to make the situations play realistically and to assess casualties, 

control personnel have responsibility for recording product outputs for 

each of the individual particfpancs. The con;roller keeps track of who 

scored what kills or was kill4 wfth rifle fire, cho threw grenades or 

was killed by grenades,, who called in artillery or was killed by artillery 

fire, who employed booby traps, mints or claymores, and who was killed 

by these devices. The mechanisms developed for keeping track of this 

information rely primarily on the USC of the identifying numbers on each 

individual’s helmet. Points are accumulated for each individual on the 

basis of the number of kills that he has scored, plus a factor added 

for the degree of success nchiavcd by this unit. The unit score fs 

determined on the basis of a set number of points for achieving the 

objective,. iess a set number of points for each casualty incurred. The 

individual scores are then lo&cd with a factor based upon unit success. 

At the present time, both the defense and the offense are being scored 

simultaneously. Therefore, the degree of success of one is diminished 

by the score of the other. 

There are, of course, many chance factors operating in an individual’s 

score, just as in combat there are many chance factors operating to affect 

one’s Rortality. A man can get killed by his own squad members, hc can 

be killed by an ill-launched grenade which happens to bounce off a tree 

back into his own position, or by calling artillery fire on himself. 

Some of these are his errors,sone are the errors of others. To overcome 

these chance fat tors , repetition. of testing is employed. Going through a 

I 

/ 

365 

.: _. 

r

test situation one time will not necessarily produce an!indcx of the 

best members of a squad, or absolute individual proficiency scores, 

1 

Over the course of a series of repetitions of the test, however, it is 

hypothesized that proficient individuals will achieve higher scores 

than those who are not proficient. That is, the chance factors will 

gradually balance out for all participants. Furtherzmre, it is necessary 

for the situation to be run numerous times to provide exposure of the 

individual to different situations. Even rur.ling the same problem over 

the identical terrain produces an altered situation. The fact that no 

two problems ever arc run alike with exactly the sane results or events 

occurring is also assumed to be a valid construction in terms of the 

combat situation. EIach combat situation is, i,n fact, unique and being 

able to respond properly to unique situations iis the highest measure of 

a combat infantrynan's profic?iency. 

I 

This theri is what we mean by total 

I 

skill integration. 

To provide a better description of the s&ten in operation I have 

1 

some slides taken during our tryout of the Infantry tests at Fort Benning 

I 

which depict several of the criterion test s"tuations. 

P 

Testing situations were developed for each of the other defined 

criterion situations using the same guiding, principles as for the assault/ 

defense. The individual was to receive tb i / same inputs, perceive the sama 

environment and make the same response as would be required by the 

criterion job. If there was a non-predictable enemy present in the 

criterion job he was provided in tits test situation. In the reconnaissanct 

patrol, for example, a squad *-as given an objective to recon, a map and an 

_. ~ * 366 'i

operations order describing what they might encounter in trying to gain 

intelligence about the objective. The defensive patrol on the other end 

was given a sector to patrol and protect against recon elements. They 

. 

could employ any strategy they chose, such as a fixed screen with booby 

trap, a moving screen, or any combination thereof. The problem was then 

started at a prearranged time and it ended when the recon element 

returned or was neutralized. 

Upon the return of the recon patrol all surviving members were tested 

by having to report what they saw at the objec:ive. The squad ras awarded 

points based upon the percentage of intelligence returned by any members. 

The objective of a recon patrol is to get accurate and complete lnrelligcnce 

about the enemy back to the base

The experimental design plan called for the,total pool of test items to 

,’ 

be administered to a t:rained Infantry platoon that could be expected to 

know and be able to perform at a reasonable level of proficiency. The 

individual test items that correlated best with total test results were 

then to constitute the final proficiency test. Further, this tryout was 

to determine the feasibility of the scheduling and times allotted for 

each of the test items. 

As indicated earlier, no independent criteria were considered suitable 

as validation criteria, since no other criteria were considered to be as 

valid as the tests themselves. The elements of the test content were 

reviewed by numerous infantry experts and judged to be valid samples of 

the infantryman’s job. Therefore, agreement or disagreement with indices 

such as supervisors’ ratings or written test scores was not considered 

appropriate1 To determine whether the results did agree with commonly 

accepted judgcments, peer ratings were employed which asked for ratings 

of prcferende of individuals to do into combat with. 

Test Evaluation 

At this point in time we have only had an opportunity to take a cursory 

look at the results achieved during the tryout of the Infantry tests. 

Following completion of the Armor test tryout more complete analyses will 

bd conducted of all the data. 

The Infantry tests were administered to one Infantry platoon consisting 

of 35 people. Nine were Squad and Fire Team Leaders (designated as HOS llB.40) 

and 26 were squad members (llB.20). Formal test administration took one week. 

368 

.

. 

Prior to test administration pre-training was conducted to insure that 

personnel were familiarized with the proper employment of’ the rifle 

scope, since that was a non-standard item of equipment. 

:: 

Several weeks were also spent prior to the test administration in 

training control personnel (XCO’s) in the details of test administration 

and in insuring that the Infantry personnel had sufficient minimum skill: 

to participate in the tests. This was necessary since the platoon we 

were furnished was newly formed with personnel just out or individual 

training &o had no unit training or experience. Since the tests were 

designed for journeyman level infantrymen, basic:skills in map reading, 

artillery adjustments, squad tactics, etc. had to be covered so that 

these personnel could be tested. I 

After the week of testing $he platoon membe:rs.were requested LO rank 

each other i‘n terms of whom they would mclst pre her to have with them in 

combat.. While they had participated with an against everyone in the 

I 

platoon, no one had any knowledge of the test sciore results at the time 

of this rating. It was based on what the 

I 

indiv duals 

f 

month of training and testing. / 

had seen during the 

The raw scores for the 2 level personnel (n=26) (squad members)’ ranged 

/i 

from 75-227 with a mean of 214.96 and a standsrd deviation of 48.26. For 

4 level personnel (n=9) (Squad and Fire Team Leaders), the range was 

158-434 with a mean raw score of 335 and a standard deviation of 96.34. 

The correlation of total +.est score with the peer ratings, computed 

by Pearson r, was .697 f?r 4 levr.1 personnel and .420 for 2 level personnel. 

Both are significant at the 05 level of confidence. 

369

. 

The Combat Performance sub-portion (two-sided) of the total test 

correlated with the total test score at ,942 and .927 respectively for the 

2 and 4 level personnel. 

Due to the built-in content validity of the test situations, the 

agreement of test results with peer rating data, and the spread of scores 

obtained, we-are satisfied that these situational performance tests 

are yielding an index of individual combat proficiency for the Light 

Weapons Infantryman. 

Armor Crewman Tests 

A similar set of criterion tests were developed for Armor crewmen. 

The duties of the Armor crew positions are well defined and they combine 

with the t&k to make a single weapons system. This is both a simplifying 

and complicating characteristic. The end product of a tank gun round on 

target involves the coordinated actions of four people which makes the 

job of ferreting outindividual contribution and proficiency more 

difficult. The proficiency of the tank crew as an entity is, and has 

been, a common level of distinction for proficiency measurement. 

To break through to individual proficiency we went to a process 

checklist type of evaluation within each situation when it was not 

possible to get product 'measures on each individual. The tank criterion 

situations include tank-to-tank engagement, night bivouac situations, 

infiltration, a river crossing, overcoming mined obstacles, bunkers, 

ambushes, and maintenance. 

370 

3

The primary scoring system used in tank engagement is to sight through 

the main gun via a telescope as with the infantry tube weapons. This 

provides immediate verification to the controller as to whether a hit 

would have been achieved. Hc can then take the appropriate action to 

kill the other tank (via radio) or notify the crew that the round missed 

and the situation continues. Against enemy infantry, the same scoring 

system is used as before that of reporting helmet numbers. 

The Armor tests are currently undergoing field cvaluaLion at Fort 

Carson, Colorado, and no results are available at this time. 

Conclusions 

Our experience to date in this project has reinforced our belief 

that performance testing could be employed to measure larger and more 

_. 

meaningful segments of job performance than has generally been the case 

to date. Certainly the control and data collection problems increase, 

but the increased validity of the exercise and the resulting increase in 

confidence of the results appears to be worth the costs. The change in 

emphasis is from having the individual demonstrate that he knows or can 

perform each individual skill required, to having him recall and effectively 

apply such skills as the situation demands. hhile there may be less control 

over which specific skills are tested, the sampling can be handled through 

both repetition and situation design. 

These tests are not yet in the final format. Refinement and further 

research is required to improve the quality of the data collected and to 

increase the discrimination power of the results. h'e strongly feel, however, 

/ 

371 

I ., 

.:*.

that these are problems of technique and not the inheredt nature of perfor- 

1 

mance tests. The challenge is to pursue the refinemen! of techniques 

rather than back off to more manageable though less fruitful approaches 

to testing. Performance testing has the power of accurate job proficiency 

assessment and the challenge is to tap it. 

As to the increased costs in time and manpower, we feel that a change 

in attitude and commitment to the testing process is needed. If we now 

have the beginning of mechanisms for accurately determining the status 

of proficiency of our personnel, the resources should be made alpailable 

to employ it. Without the type of data that is avrilable from well designed 

performance testing, training efforts will necessarily operate at less than 

optimum efficiency. I 

372

THE.AIR FCRCE: WIFE: HER YSjChZEDGE OF, AND 

ATTIiUDE:S TOWARD, THE AIR FORCE* 

John A. Belt,Ph.D 6 

Arthur B. Sweney, Ph.D. 

Center for iiuman Appraisal 

Wichita State University 

This paper prepared fc’r presentation at the Military Testing 

Asscciation Conference cn'"Iiuman Resources - Growing Demands," 

Oct. 28 - Kov. 2, 1973, San Antonio, Texas. A more analytic 

treatment of the material presented in the paper may be found 

in technical report $103, "The Air Force Wife - A Study of Morale 

among Military Dependents" issued by the Center for Buman 

Appraisal, Wichita State University. 

*The Research reported in this paper was supported,by the Air Force 

Office of Scientific Research, grant fi 72-2001. 

z 

373 

t. _ _. - . . .-* ..- 

:. , -;:,r-,.

A major corporation gives gold charms in the shcipe of each 

new state to wives of transferred executives and it becomes status 

to point with pride at the number of charms acquired during a career 

span. R novelty? Public relations gimmick? No, rather a small part 

of the concerted effort in industry to make wives feel a part of their 

husband's career and sat.isfied with their place in this future. The 

industry motives are far from altruistic for as William Whyte stated 

in 1348 "as an econoic lever . ..companies have learned that there is no 

stimulus quite so effective as the wife if properly handled." Much 

research has gone into ways,to improve the recruiting and retention of 

civian employees and maximize their productivity. Eielfrich, (1965) 

concluded that "corporations arc increasingly interested in the wives 

of their executives." In an 197.1 study of business executives J.M. and 

R.F. Pahl typified the comments of their interviewees on the single 

most important factor influencing their career, "my wife, more than 

anything else." 

It would logically follow then that the military in many ways would 

not only follow this pattern but demand more from wives and family. 

And likewise, the commitment a man makes in choosing a military career 

would call for support and approval from his wife and family. 

As our society has become more and more mobile, the extended family 

of the past has been replaced by a more insular unit. No longer does 

a man have relatives close by or living in his home, for his many moves 

may take his family far from any familial ties. Without these other 

sources to rely on, the family tends to turn to each other and intcr- 

personal relationships attain a higher value in assessing each individual's 

f 

374 

_ . . . 

/ 

I , 

. _ 

I 

-..

. 

satisfaction. The demands of a military life emphasize& this 

1 

phenomenon. 

One wife in a recent survey of military families related that 

her eldest child had attended 14 schools before entering college. 

(Entrails, 1971) How she views this style of Life, the rewards it 

offers her and her family must be weighed against the deficiencies. 

And the conclusions sha reaches concerning these matters will play 

an'imprtant role in her Husband's life also. He has drawn his 

family into a life style unique in many aspects; a subculture of 

over 4 million dependents. Roger Little (1971j has remarked on this, 

"All military families have in common knowledge and experience in 

an occupational culture ( or subculture 1 which is more distinct 

than that of other occupatior@ in the larger s,ociety." Base housing, 

I 

I 

required mobility, the status in the community of the military man - 

these are only a few of the pressures a man choosing a military career 

must weigh. 

I 

Add to these the similar pressures his wife and f?nily 

meet and one can see either an effective cohe ive unit or a -point of 

f 

dissention for a man trying to fulfill a rold as a husband and 

father. 

in a recent survey of career attitudes d among Air Force personnel, _ 

it was found that the wife had twice the influence upon her husband'< 

career intentions as any other individual, including his immediate 

(Belt, 1972) 

supervisor or any of his peers. f A determinant as important as thj.s 

can't be slighted or ignored. ?n an investigation of junior officer 

retwtion problens, Lund (1972) found that wives were the key variable, 

in the decision to separate or'remain in the army. A 1971 study emphasized 

that any good junior officer retention program must inclucie efforts 

_. , - I 

375 

i 

, 

I

exyar.8 t5is to excmi::e I-.cu her position as wife and her dt:ltpzdcs 

a~ a ncr%er cf tk n:.lz:ary !wives ;:I respozdino to questiL:nc ter:d 

361st Ft:lCI:t.

plen in the wir.9 duriq the time of t!x study, ovrq half of which 

WX-~ married. 

Rata collection 

A sericn .o f 

/ 

,three questionnaires were adninintcrcd tc the 

subjects. 

The first questionnaire (SDS-l) vas developed as an csploratoq 

instrument to investigate t.L.c basic I-rcmise that wives of Air Force 

personnel do, in fact, have strong opinions about their husbarzd's 

military affliction. SD?.-1 contained questions from major c.xtago:-ies: 

Dcxoqrqhy; Rclatians wit!1 Facilities, I!~ncfitr, and Scr-.*iccx; and 

Ccneral Attitudes toward :Iilit;irv Life. ,111 open ended scc:ion for 

ccncral cnm..cnts *was alsc included. 

j 

As a pilct study, .Ci,3-1 was Ji*triAitrd 

a 

/ 

to the Iics~~itnl Squadron 

to check on its aFplicabiiitg to the :\r Force lift style. Q-n 

analysis, the instrument was determined to he applicable. 

I 

The first qucstior.i~irc was adniEistercd by distributing the forrrs 

to all the married IXI: ln the wing, 

the instrlzTcnts to their wives. 

i 

!’ 

I 

and rqucstinq t5e husbands to take 

I 

Returrj envelopes were Frovidcd to 

facilitate‘tke return of the completed forms. A total of 627 copies 

of SDS-1 were distributed on April lG, 1971. Of these, 264 - 

completed forms were returned, creating a reswnse rate of 31.9%. 

Wives Attitude Survey II (t;.\5 II) was generated for two major 

purposes. The first section was aimed at investigating in depth trends 

that were evident in the oFen ended response section of rhc first 

survey. The second aim was to measure more Freciscly the relations the 

wives had with the facilities, benefits. and services. Again, the 

instrument included a demographic section and an open rcspnse section. 

+-- I 

t, 

t, R 

+ \ 

,. .’

Ii 

A listing of al.1 hoxw addresses iJcre abtaincd throqh the 

Cor.solidatcd Ut~,:t Pcxsor:r.cl O:‘i~cc. The scmr.d survey was ~rril~? 

directly to the wives cn YarCh 13, 1372. Ikludrd in Khc2 nA1ir.g 

were S:anFCd, self ,Iddrcsc& return envelop::. 

t.hesr hustar.dn scrvir: t il: thcs Air Force as 6cllows: 

r!i s:;~I.L C:rqd ‘- rr:n Aat arc assigned tc cperat.icr.al Cc:::bat Mssiic c’rc~:. 

i 

!:on Xlssi ?e Crcx - r 4:n ttat are not assiqrxd to Conlt.3: Xissilc Crews. 

Career - nen w:th fi-.-c years or more of tint in service. 

,>ff icar - zcn of risk 0-l or above. 

!:nl asted - am-. of ra::k U-9 or below.. 

First Tern cfficer - ner, with four years cr less cf tine in service 

wi tk rank O-1 or akvc. 

2 

378 

: . ._ 

: -.:

Career Officer 

- - men with Five years or more of time in service with 

rank of O-l or abova. 

First Term Enlisted - men with four years or less of time in scrvicc: 

with rank E-9 or below. 

Career Enlisted - men with five years or more of time in service with 

rank E-3 or below. 

The dyads selected for analysis to determine intergroup differcnceu 

were: Elissile Crew-Non Missile Crew; First Term-Career; Officer-Enlintcdr 

First Term Officer-Career Officer; First Term Enlisted-Carccr Enlist&, 

First Term Officer-First Term Enlisted; Career Officer-Car'Mr Enlisted. 

Factor Analysis of Wives' Attitudes 

The Factor ,Analysjs performed on s'ection of KAS IT qcncratcd ten 

factors. The contributi::g variables and their loadings for each of the 

factors arc shohn in Appendix 1. The first factor was described as 

passive alientation/integration. Tt car.oted a passive role for the wivc$n 

of Air Force personnel. They apparently did not feel that their 

participation was rcquircd or even solicited, yet neither did they fcul 

that they were rejected or prevented from becoming involved. This 

dynamic displayed the disavowal of personal responsibility for inteqcntion 

into the Air Force lifestyle and an attitude of simply “floating along 

with the current." 

yactor II was identified as a desire for information versus apathy. 

This continum was between a desire for more information about how the 

Air Force affects her life and the apathy which is present in all talks 

of Life. These feelings were active in that the dynamic stretches from 

apathy to curiosity. It also appeared that there was a recognizable 

solidarity of interest in what the Air Force was and did. 

379

The dynamic at work in Factor III could rojt a,-tly !~c\ve been 

I 

labeled fanilial naturity and independence frm the Air I'crce/ 

familial imaturity and dependence on the Air +orce. The dynamic 

was one of qrowtt, erred change iz focus of attchtior.. tA.5 tt1c far.11: 

matuxcd, the wife bccane norc'!icterested in its devclo1,~~r.t and less 

lr,tcrestea . :n her relatiohshil: to the Air Force. 

f3ctar ;V provrucc an ihsiqht into the wives' ~ercc,!~tior; of 

the currcht societal trend of dista-ce for the. r?ilitarv. The cobtim: 

traversed the area bttwccn i:rideful identification with her s;;cuser;' 

jcb nr:d the couple's relaticmshig to the Air Force to apolnqr.tlc 

rezoz:itioh ar.d rejection of identification with the Air I'crce. 

T!;c dt:sCrilltloh 0 f Iactor V *das a facet of wives' atti;udc 

ti:at war r.cc 

I 

directly related to the Air F;orce as a fucctioning 

or+3r.ization. Instead, it was ar, indication of the grcu? identifi-atic!:,' 

I 

disassociatioc m..onq the bwives of Air For& ~trsor.r.cl. 

1 

This within 

Trcur bipclal-ity was ayparehtly a wry inrortant realit\* cf life ir; 

9.e Air Force corui,ur.ity. 

I 

The dynsxic cxy?oscd in Factor "I was t'ifficult to intcrprct. 

out rcf6xlir.q to causslity, it appeared t':ut Fhysical/'~sychological 

! 

separation CT proxicity/ideatificaticn attitudes were prcaeht in the 

sarplc. 

/ 

Factor VI: [*r&ably revealed an exFereatia1 attitude set about 

regulation/restriction by the Air Force. The dynamic bctvetn wives’ 

feeiincs 0,5 yersoml. frcedon of action and restriction of action was 

quite cbvious. 

380 

With

The eiqhth factor was interpreted as two varying perceptionsof 

the adage "rank bath its priviledges (power)". It was based on the 

perceived transference ol.c the hurband's rank to the wife. The 

dynamic was primarily cne of Ierccived lzower in the trancfcrence 

as opposed to no power. The younger, lower rank, wives felt more power 

was inherent in the transfer, while the aider, higher ranking wives 

vre frustrated by the erosion of their preconception with the 

realization of the lack of power that the bust-Ads' rank gives to 

his wife. 

Identificatic.1 with the source cf information about the Air 

Force was the interpretation cf Fxtcr IS. It became apparent that 

as tt.. wife lcarnczl mere ab0-t t!:e system with which she is involved, 

she identified mom closciy with the source. The poles of this 

continum were the .hir Fox-cc itrc-If ar.6 her SpCGSe. 

The last factor was vicvcd as the conscious commitment of the 

wife toward invclvcment or,coninvolvcmrnt. The frustraticn dimension 

did not enter into this dynamic, as participation was directly related 

to the personal decision of the wife. 

Discriminant Function Analysis of Selected Groups and Subgroups 

-. 

(Note : The na::urc of the data array required that separate 

8 

discrininant fundtion ,. analysis be Ferformcd on each of the 

three ir‘struments. Tke voluminous nature of the resulting 

analysis (21 pages cf tabies)precludes ar.y but a curscry 

treatment of the findings here. The interested reader 

is refcrrcd to the tech. report 3103 "The Air Force r;‘ife - 

A Study of Morale among Military DeFendc~ts" issued by the 

Cerrtcr for Hman Appraisal, Wichita State University for a 

t 

381 

. a . . - 

“7 -.-. - 

.’ 

f@@

detailed ar.d comprehensive analysis of this data.) 

hushrds were ic Cornhat !! issile Crews were discrimixbly different 

than the rest of the test pplation. The mai:: facto1 underlying 

t ki i 2; ;iiffere;lCe appeared to kc! a Lack of association bttwccn thr 

wife zd ti:e Air Force, cnphasized by a r.cnati.ve attituh CF. the 

;:art of tt.r &:insiJc Crew wives toward facilities, tcnefits, and 

services cffered by tht? Air Force. 

*t '!a;. * :",~p:-",~!bc< ,a 

. . - The ra::y chtairti diffcrccccs between the 

7-r . e-s t Tcrx-. and Career qx-cups cculd bc et~cctcd because of t!?c 

(;i>,Pj c::.: aqt- ar.d time iz t.hc scrvlcc differentials. The first 

r c. :: d

that the career wFvcs identified more ciosrlc *..-ith t!lc Air !'orce, 

knew more about it, ai. c:cnerali\q ndir.t&incd A stronger rclatioliship 

with the Air Force. The findinqs that the career wives were more 

knowledgeable was further cmph~sixd by the fact that tt;c i'irst Term 

Officer group was more influcncabl 5~ military cropaqanda. 

First Berw Fr 5?:sL&--lr Free. 

I 

� 

�� ␛ 

�� 

☺�� 

� 

i 

- The

Although numerous compiaints were voiced, the wives for the 

most part had =S favorable attitude toward the miiitary lifestyle 

and their participation in it. 12 fat:, most of the wives expressed 

a desire for more information about the Air Force. The study 

indicated the average AF wife knoxs very little about the 

facilities, benefits and services available to her. 

Nevertheless, the depende,lt briefings, designed to offer this 

type of information, seemed to be an irritant to some wives, 

especially the younger group. A fairly con%on complaint about the 

dependent briefings was the "imFersona1" manner in which the wives 

were treated. Many also disliked the te.rm "briefing:' The term 

seemed to accentuate the military atmosphere of the meetings. 

Because of this attitude, many of the wives said they did not attend 

the dcpcndcnt briefings regularly, which forced them to look for an 

alternate information source. 

This alternate source ttsuali$ was the husband. But he too was 

often an insufficiant source for information. For one thing, 

the husband doesn't know the kinds of things his wife wants and needs 

to know and doesn't apFaar to be interested enough in these areas 

to find out. 

, 

, 

The information factor shiuld be of major importance to AF officials 

i 

a~ the survey also indicated the wives, as a group, tend tc identify 

and form attitudes about the AF based on the information they received. 

However, the survey also indicated that coercion would NOT be a good 

384

means of attaining the desired goals. That is, requiring wives to 

attend dependent briefings, undoubtedly would have a very negative 

effect. 

As night be expected the wives of men with long tern associations 

with the hF tended to have a prideful identification with the Air Force. 

Some of the younger wives, however,seened apologetic about their 

husband's military affiliation. This nay be due to the recent negative 

societal view of the military amor.g some young people. 

The survey indic:ated that the wives who identified with the AF 

lifestyle tended to live close to the base, and those least interested 

in that lifestylelivedfurther from the base. However, it is r.ot 

possible from this study to deter&nine which is the cause or which is the 

result of this phenomenon. 

It was also found that wives with no children tended to depend 

on the military to give structure to their lives. However, as children 

came into the family unit, the wife changed her vie+oint and became 

more involved with her family and less involved with the military. 

An interesting phenomenon revealed by the survey was the concept 

some of the wives had about rank transference. It appeared that wives 

of lower ranking men perceived a great deal of power transferred to the 

wives of higher ranking men (both officer and enlisted.) However, when 

the men rose into the higher ranks, the wives realized there was very 

little real power transferred to the spouse. The frustration seemed 

to increase as the husband's renk increased and the realization that 

she had no real power became more evident. 

* 

/ 

. .-. I.- 

_ . 

. . 

r

The various group comparisons made in the study revealed that 

the feelings and attitudes of the wives tended to be gioup 

specific. 

For example, the group of wives with husbands in missile crews 

had the most negative feelings about the Air Force and its benefits 

and facilities. Thiswasnot surprising, however, as our previous 

Career Attitude Survey showed missile crew members to be among the 

&east satisfied men in the AF. 

One factor that may have contributed to the poor attitude of the 

“missileer’s” wife was the fact that her husband's duty required him 

to be separated from her overnight several nights a week. This 

factor may be compounded by the fact that most missile crew members 

I 

were first-temers and were relatively young. /The wife's younger 

I 

age may have tended to make her less understanding about her husband's 

* 

reoccuring -absence. 

Wives of the career group men were more familiar with the bene- 

fits available to them and had a more positive attitude toward them 

than the first term-wives. 

I 

In comparison to the enlisted wives , the i wives of officers were 

basically more socially oriented and participated to a greater degree 

in the AF centered community. 

t 

/ - 

The career officer wives identified more closely with the A?, knew 

more abut it, and generally had a stronger relationship with the Air 

Force than the first-term officer wives. The first term officer group 

also seemed to be ,I.ore readily influenced by military propaganda. 

. . 

, * 

386 

1

The difference between the career officer wife and the career 

enlisted wife appeared to be one of rank. The career officer wife 

felt more socially interested in the AF and generally was more 

associated with +he AF. Also, t.he officer wife did not appear to 

approve of a program or benefit merely because it was 'Air Force,' 

BY dividing the wives into four groups it was possible to rank 

them according to their feelings about the AF. The career officer 

wife was the most favorably inclined towards the AF, followed by 

the career enlisted, first term officer, and first term enlisted 

wife. 

IX?LICATIONS 

Overall, it would appear that, because of her obvious effect 

on the husband, the AF wife's somewhat positive feelings about the 

Air Force and her desire for more information are desirable effects. 

Her apparent desire to become more of a part of the military 

centered community seems'to be blocked because of a lack of an easy 

aveneue to do so. The results seemed to indicate that both she 

and the Air Force lacked the real initiative to draw her into the 

community. 

The apparent; failure of the dependent briefings in providing 

information to the wi'vcs should be given some attention. It seems that 

i 

a more "feministic" approach is desired by the wives. 

In conclusion, it seems evident that the AF, as well as the other 

services, is ignoring a segment of its community that is an extremely 

powerful factor in retention and possibly an egually powerful influence 

on job performance and morale azzong the men. 

t 

387 

2 

_. --. _ _- _, 

-: . .

REFERENCES 

Belt, John A., and Parrott, Gerald S. The Relationship of Satisfier;- 

Dissatj.sfiers In A Military Unit t0 Re-enlistment, paper prC?Sentec. at 

Inter-University Seminar, Chicago, Illinois, September 21,-2;: 1972. 

Eutrails, research newsletter published by Ce.?ter for Hunan Appraisal, 

Wichita State University, Vol. 1, NO. 2, NOV. 1, 1971. 

"Eternal Triangle . . . Man, Wife, and Work". Industry Week, 168:4. 

January 25, 1972. 

Farber, Leslie E. "He Said, She Said". Commentary, 123:24. March, 

1972. 

Helfrich, M.L. The Social Role of the Executive's Wife, Bureau of 

Business Research, Ohio State University, 1965. 

Little, Roger W. "The Military Fanily". Handbook of Military Insti- 

- 

tutions, 247-272. Beverly Hills: Sage, 1971. 

Lund, Donald A. "Problens'of Junior Officer Retention in the Volunteer-Army: 

The Case of the Military District of Washington". 

Paper presented at ,thc Workshop on Military Manpower - The All 

Volunteer Military. Inter-University Seminar, Chicago, Illinois 

September 21-23, 1972. 

Lund, Donald A. "Active Duty - Yes or No?" The Military Police 

Journal 20 (February): 15-17, 1971. 

McKain, Jerry L. "F,-eling of Alienation, Geographical Mobility,and 

Army Family Problems: An Extention of Thecry". Published Dissertation: 

Catholic University of America. National Technical 

Information Eezvice" 1969. 

"Navy Wives" Perceptions of Conditions of Navy Life". Naval Persohnel 

Research and Development Laboratory Defense Documents 

Center, Washington, D.C. March, 1971. 

Triebal, Joane. "Your Wife: A Prisoner of YOUX Success?" Nation's 

Business, 73:13, June 26, 1972. 

Whyte, William H., Is Anybody Listening? Doubleday, New York, 1948. 

r? 

/ * 

388 

-. 

; ‘-.

. 

: ! 

APPENDIX 1 I 

0 I 

I 

359 

- i... 

3, 

% 

_. . .- - 

f 

i ‘,

Factor I: Passive alienation/integration 

Variable Factor 

Number Variable Description Loading 

4. 

13. 

7. 

10. 

19. 

24. 

5. 

The A.F. doesn't care what the wive's of its personnel -0.689 

thick. 

The it.F. doesn't care what I think. -0.884 

Thcrc are no procedures for me to express my feeling -0.788 

ahut A.F. policies. 

Civilians don't respect military personnel. -0.570 

The A.P. keeps the wives of its personnel well informed. 0,559 

Life AS an A.F. wife provides me..many opportunities to 0.402 

gc t I nvolvcd . 

A.F. wives should be kept better informed of base act- -0.369 

;vitics. 

Factor II: Curiosity/apathy 



15. If I understood it ITore. I think the Air Foxce would 0.739 

be interesting. 

14. I would like to be invited to attend my husbands re- 0.707 

enlistment inter3ier.s. 

9. I don't want to know more about my husband's job than -0.701 

I nlrcady know. 

8. Base activities have a direct affect on me. 0.526 

5. A.F. wives should be kept better informed of base 0.374 

. . 

activities. 

11. Most wives think they have their husband's rank. 0.322

Factor III: Familial maturity/immaturity 

! 

'I 


Number Variable Description L,adincr 

51. How long have you been in the service (4-%yrs.:S.) . 0.860 

53. How many children do you have? (1.2) 0.788 

52. Do you expect your husband to make the A.F. a career -0.727 

(probably) 

22. The A.F. should not be a 24-hour a day job. c.590 

24. Life as an A.F. wife provided me many opportunities -0.480 

to get involved. 

1. Wives of A.F. personnel should be involved in formul- 0.360 

ating A.F. policy. 

11. Most wives think they have their husband's rank. 0.305 

1 

tt*****t**t*C*******~***~,b*~~~~~~~~~~~*~~*****~**~~**~***************** 

. . 

w 

i 

I 

. - IV: Prideful/apologetic identification 

I 

Variable .I Factor 

Number Variable Description Loadin% 

20. A wife should be proud of her husband's/profession. -0.889 

25. The A.F. should not be just another jo 4 . -0.553 

5. A.F. wives should be kept better informed of base -0.413 

activities. 

,. , * 

/ 

.’ 

391 

! 

I 

/ 

/ 

..~ :,. ., .

-. 

Factor V: Within subculture identification/disassociation 


Nunber Variable Description Loading 

18. A.F. wives have a lot in COtx!oII. 0.825 

12. A.F. wives have a number of siziliar problems. C!.722 

17. I often feel I am a 

2. I njoy associating 

21. I know the wives of 

fairly well. 

member of the A.F. 0.702 

with other A.F. wives. 0.565 

mcmbcrs of my husband's unit 0.540 

24. Life as an A.F. wive provides my may opportunities 0.474 

to get involved. 

. . 

Factor VI: Phjjsical d Fsychological prox*ity/scparation 

Variable 

Factor 

Number -. Variable 'Description Loading 

54. How far 20 you live from base? (3-5mi) 0.8C5 

1. Wives of Air Force personnel should be included in -0.370 

formulating Air Force policy. 

‘ 

Factor VII{ External restriction/freedom 

Variable 

Number Variable Description 

6. Wives of Air Force personnel are free to do what 

they want. 

. _ 

e - : 

Factor 

Loadinq 

0.790

Factor VIII: Rank transference aspiration/frustration 



55. What is your husband's rank. 0.632 

24. Life as an Air Force wife provides me many opportun- -0.336 

ities to get involved. 

11. Most wives think they have their husband's rank. 0.367 

Factor IS: Identification with information source 

Variable 

Number Variable Description 

Factor 

Loading 

26. Where did you learn the most about the Air Force. 0.680 

1. Wives of Air Force personnel should be involved in 0.270 

formulating A.F. policy. 

14. I hvuld like to be invited to attend my husband's 0.268 

reenlistment interview. 

17. I often feel I am a member of the Air Force. -0.250 

f 

393 

.‘. ‘. 

-_.

-I - 

-. 

. . 

. . . ‘. 

Variable 

Cumber 

,f 

iI 

/--cm< ____ ^ .-.- .____.__-’ -c- -,-, - -y - 

1 

Factor X: Personal latitude for involvamcnt/non-i::volvcment 

- 

Variable Description 

I I 

i 

1 

Factor 

Loading 

2 3 . Air Force benefits do not ir.terest me a great deal. 0.684 

1 6 . I don't care hut Air Force policy except as it 0.581 

affects mc. 

5 . Air Force sho;lld be kept better inf#*rned of base -0.442 

activities. 

1. Wives of Air Force personnel should be involved in -0.392 

formulating Air Force policy. 

, 

.’ 394 

:.

Technical b r Report - International Military Testing Association

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?