Approved For Release 2000/08/08 CIA-RDP96-00789ROO2900290001-9 STHB HTE DT-S-1030-SL g. :XX :;5 DEFENSE INTELLIGENCE AGENCY REVIEW OF PROJECT PROCEDUHES (11) 40 5 HPRIL 199 1 KoFomi rlnrlr2 STHR GHTE LRMS Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2900290001-9 Approved For Release 200OLD810 - CIA-RDP96-00789ROO2900290001-9 e-% r- ^ m r- -r- %-.00 L. W I A IM REVIEW OF PROJECT PROCEDURES Date of Publication 5 April 1991 This is a Department of Defense Intelligence Document prepared by the Technology Assessment and Support Office, Directorate for Scientific and Technical Intelligence, Defense Intelligence Agency. PREPARED BY Technology Assessment and Support Office (DT-S) NOT RELEASABLE TO FOREIGN NATIONALS REPRODUCTION REQUIRES APPROVAL OF ORIGINATOR OR HIGHER DOD AUTHORITY. FURTHER DISSEMINATION ONLY AS DIRECTED BY DT OR HIGHER DOD AUTHORITY. CLASSIFIED BY: DIA/DT DECLASSIFY ON: OADR F=- G E+10tull, MIS Approved For Release On f91 E RDP96-00789ROO2900290001-9 -40~ Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2900290001-9 UNCLASSIFIED TABLE OF CONTENTS Ea ~e 1. PURPOSE ........................................... I Il. SCOPE ............................................ I 111. SUMMARY .......................................... I IV. BACKGROUND ....................................... I V. SHORTFALLS/REVISIONS .............................. 2 APPENDIX A. EVALUATION FLOW CHART B. REPORT FORMAT C. SUMMARY OF INFORMATION FORMAT UNCLASSIFIED 1w Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2900290001-9 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2900290001-9 UNCLASSIFIED I. (U) PURPOSE: (U) Purpose of this report is to review initial. STAR GATE procedures and -to provide recommendations for revisions based on recent E."PerienCe il-W.JUding DESERT STORM tasking. II. (U) SCOPE: (S/NF) This report examines data development, eval.Liation and assessment procedures for data generated by STAR GATE sources. 111. (U) SUMMARY: A. (S/NF) Lessons learned from recent DESERT STORM tasking have led to several immediate changes in STAR GATE procedures as discussed in Section IV. Some recommendations far future revisions are also identified. B. (S/NF) Evaluations of future STAR GATE reporting will be 40 prepared in accordance with -the criteria and provisions of DIAM 513-13, Subject: The Department of Defense HUMINT Management System, dated 2B March 19eB. IV. (U) BACKGROUND: A. (S/NF) In December 1990 and January 1991, initial efforts were made to provide an effective and viable system to evaluate STAR GATE reporting. These procedures are illustrated in Figure 1. The INCLASSIFIED -1 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2900290001-9 -do Approved For Release 200 oft.., -1 1- DP96-00789ROO2900290001-9 initial proposals resulted in the implementation of procedures that attempted to assess both tile utility and the value of STAR GATE reporting. B. (S/NF) During tile Course of an off-site conference held on 7 and 8 March 1991 at Fort Dietrich, senior analysts from DT-5A, DT-3B, and DT-S, identified shortfalls in current evaluation procedures and prescribed revisions that would allow for the implementation of an objective and work--.able system. The senior analysts believed that assessments relating to the a.ccuracy. of STAR GATE reporting could best be addressed by applying applicable scientific methodologies to STAR GATE material. Procedures already exist and can be implemented available project ADP methods. Specific accuracy assessments would be determined by intelligence analysts in conjunction with appropriate outside scientific review. The same analysts were of the opinion that the intelligence value or utility of STAR GATE reporting could best be determined by select intelligence analysts directly involved with specific tasl.-. areas. C. (U) What follows are the results of a comprehensive group effort conducted to identify procedural shortfalls and recommend the implementation of the following revisions. V. (S/NF/SG/LIMDIS) SHORTFALLS AND REVISIONS: - SHORTFALL: Need to identify action officers with whom DT-3B can deal directly to expedite the evaluation process and insure that such evaluations are accomplished in the most successful manner. NOFORN/LIMDIS STAR GATE Approved For Release 2000/08/08~~CIA-RDP96-00789ROO2900290001-9 Approved For Release 2000- RDP96-00789ROO2900290001-9 SG1J other) scientific analysts should help in better task definition and in identifying ways for improving potential utility. - SHORTFALL: Procedures to date appear burdensome and need streamlining. - REVISION: Procedures developed by the R&D community Should be used for assessing the accuracy of project reporting. The intelligence value of STAR GATE reporting must be determined by intelligence community analysts working from an all source intelligence perspective and environment. This should be based an an agreed-upon definition of utility and what target data would be required to satisfy the utility criteria. - SHORTFALL: Need to assess whether our quantifiable answer categories would be expanded to reflect the range of uncertainty when one is evaluating a DT-S report. - REVISION: Specificity in the judicious selection of mission requirements will go a long way towards streamlining the process. Target selection must be based on targets against which STAR GATE has i ii,'NOFORN/LIMDIS STAR GATE -'3- 1W Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2900290001-9 analytical evaluation process. Direct involvement with these (and Approved For Release 200 inA - CIA-RDP96-00789ROO2900290001-9 a favorable track record. To this extent, limits to STAR GATE data (or project type), need to be provided in advance to the tasking elements. SHORTFALLu Need to re-accomplish certain original tasking, updating it as necessary to the current situation. - REVISION: Tasking should be levied within specific time windows and not be left open-ended. Requirement originators should be reminded of their responsibility to update requirements and changes of collection emphasis. - SHORTFALL: Senior analysts need to insure that the intelligence evaluations (ie, utility assessments) of project reporting are timely, objective, substantive and prepared in consonance with established intelligence community procedures. - REVISION: STAR GATE reporting needs to be evaluated in consonance with the DIA HUMINT evaluation scheme paraphrased Substantially as follows- RATING Of Major Significance Of High Value Of Value VALUE Having Impact on National Policy and Decision-Making Best Effort - First Report in a Field of Endeavor Supplements - Updates - Confirms Known Data - - OFORN/LIMDIS STAR GATE -4 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2900290001-9 Approved For Release 2 a a In 0 -_RDP96-00789ROO2900290001-9 Of Low Value Of No Value There is another consideration. Poor Quality - Not Timely Erroneous - Misleading Proper assessment procedures for either accuracy or utility require that criteria for evaluation be specific in advance. Consequently, it should be appropriate for a candidate user (or task originator) to clearly define what type of data Would h a v ep ot ent i a 1. utility, and how it would be used if provided and how it Would be evaluated. This type of data could include specific essential elements of information (EEI) and required -timing. A troublesome problem may be encountered if the specific EEI (if posed as a question) provides too much background information for the source that Could adversely influence session results. To avoid such difficulty, it may be necessary to conduct the session in several phases, time permitting. This issue is not easy to resolve and remains as a future action item. (A basic review of this overall issue can be found in DT-S-1010-S, 13 Dec 1990, Evaluation Methods). 4W - SHORTFALL: Need to reconsider acceptance of tasking based an availability and timeliness of known ground truth. - REVISION: The STAR GATE collection management system must be selective in its acceptance of requirements for which there is a likelihood of learning ground truth within a reasonable amount of time. A procedure needs to be established that allows for rapid follow-up for task refinement. &WOFORN/LMIS STAR GATE -5- Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2900290001.9 Approved For Release 2000/"" rla-RDP96-00789ROO2900290001-9 .1 - I - SHORTFALL2 Multiple Source reporting in a combined summary format is confusing precludes affixing responsibility to the reporting. - REVISION- The principle of "one source one report" must be strictly adhered to so that reporting may be attributable to a specific source. Coi-iseqUently, data from any source will be reported separately or in easily idpi-itifiable and separated sections of an overall summary report. - SHORTFALL: Current STAR GATE reporting is inconsistent with other established HUMINT reporting formats. - REVISION: Reports need to be prepared in accordance with the criteria used in the preparation of Intelligence Information Reports (IIR's). Each report will bear a numerical source designator that will protect the identity of the source. An example of the reporting format is at Figure 2. A Summary of Information (SOI) will be prepared by the source. It will reflect a project number, source number, session number, and date of report. The SOI will be attached as an addendum to the formal report. An example of an 901 is at Figure 3. - SHORTFALL: STAR GATE predictions are rarely accurate. - REVISION: Predictive reporting is indeed the project's weakest feature. The results of searches are conducted at a slightly higher rate of success. Historically, the project's track record - - - - - ROFORN/LIMDIS STAR GATE -6- Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2900290001-9 Approved For Release 2000/08/08-: CIA-RDP96-00789ROO2900290001-9 reveals a relatively high rate of success when mission tasking is targeted against fixed, static targets located in past or Current time windows. ConseqUel-Itly, project tasking should be prioritized according to estimated success probability. - SHORTFALL: STAR GATE project reporting should not stand alone. - REVISION: Senior analysts need to insure that STAR GATE is an integral part of the all-SOUrce collection management system, a key tip-off player acting in concert with other intelligence disciplines. SHORTFALL: Highlights of singularly outstanding reporting should be recorded and made readily retrievable. - REVISION- A "KUdo File" will be established to facilitate the retrieval of examples of high quality reporting. Factors that may have contributed to these results Will be identified whenever possible. OFORN/LIMDIS STARIGATE -7- Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2900290001-9 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2900290001-9 -T1 A~ ON OUTPUT Approved For Release 2000/08/08 CIA-RDP96-00789ROO2900290001-9 DATA DEVELOPMENT, EVALUATION AND ASSESSMENT Approved For Release 2000/C~/p8A: [PlAtRPP96-00789ROO2900290001-9 ..... ..... .... .... ..... ..... . .. ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... .... .... .... ... ..... ..... .. .. ..... ..... ..... ..... ..... ..... ..... ..... ..... .... ..... .... .... ..... ..... ... .... . "I" 1\1 L) IVI E'.3 E: IR I , ..: .:., 1. E) C) LJF` C', 1\1 LJ !1 1::~, 1:.:, j. j-- 1\1 1) A "1" 1-7. 0 V:' F;' R''T, 3 1\1 E.." !".I D C) 1\1 N Li IY1 B E, F` .. ..... .... ..... ..... .. ..... ..... ..... ..... .... ..... ..... ... ................ ....... ...................... ................. .................................. .... ............................ .. . .... ..... ..... ..... .... . . C 1-::'[,'3 F" C) LJ 1\1 D 2. Z1.3 / I'll F., IR UT F~ 1:-: 1 -1 Ei", hl 1" 'S / 1\1 F7 Vk` C. C.) 111,- EN 1 I Ac TrICALACIE-.-I ~.-Af-[V infc) not c-ontainem:1 i1-1 SUMM~.-AF"y 04: (SOT !:.,) I r. c) V j. cl c.1 L, yCS (D t..t r- c. c.". . 5. E V 01 L UT' 10 N E71 1AW DTAM 58- 1~ IS E:7 AII E.VAI (J () -1- T. C) 1',] 1\1 C.) t IR s a b 'I t c) -f (::) v- g ri n a iiat 1 C, L D"I'A u'r Ohl: MADV", Lim-itc.)d mIiWiWWWWW [,,.10 F:T) F-Z NI L T M D I S) APPENDIX ~ Figure 2 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2900290001-9 Approved For Release 2000/08/08 :j C~A-PDP96-00789ROO2900290001-9 E X A M P E C V:: ~'D, 0 L.J IR, CI E'. 1'.) ("1 'T* SLAIMARY OF::' BY -1-1 .1 ":3 LJ F::, F F . . ........... .... . ...... . .... .......... . ........... I., ..................... ................... . ............................ . .......... ............ . .......... . ..... ........................ ........... .................. ........... . .............. ... .. ......... .................. . ........... .. ........ GOT 410 APPENDIX CiFigure 3 40 Approved For Release 2000/08/08 : CIA-RDP96-00789ROO2900290001-9