Menu

ZeroFox Analytic Standards & Tradecraft

Purpose

This document provides guidance on how ZeroFox intelligence analysts implement proper structured analytic tradecraft in a finished intelligence product and is intended to promote a common ethic for achieving analytic rigor and excellence. 

It aims to serve as an informational resource to guide analysts in approaching their work in a thoughtful and well-prepared manner with personal integrity—thinking beyond already established and preconceived ideas and exploring new possibilities within the parameters of their subject expertise.

Adherence to these standards is safeguarded by the Vice President of Intelligence, who is responsible for concerns regarding lack of objectivity, bias, politicization, or other issues in application of standards within analytic products.

Intelligence Analysis

Intelligence is the process by which information important to security is requested, collected, analyzed, and—ultimately—provided to decision makers.1 

Intelligence and information are not synonymous. Information consists of pieces of raw,  unanalyzed data that may indicate the incidence or evidence of a security event. Intelligence is  generated by assessing a wide array of raw threat information for validity, reliability, and relevancy and ascribing meaning to it via the application of critical thinking and logical reasoning. 

The traditional intelligence cycle consists of the following six operational steps:3

  1. Planning & Direction. Establish the client’s information requirements and plan operational activities accordingly.
  2. Collection. Task collection systems and gather raw data required to produce   intelligence.
  3. Processing & Exploitation. Convert the raw data into a comprehensible format that can be used to produce intelligence.
  4. Analysis & Production. Integrate, evaluate, and prepare the processed information for inclusion in the finished intelligence product.
  5. Dissemination. Deliver finished product to requesting client and others, as applicable, and execute any remediation measures to counter detected threats.
  6. Evaluation. Acquire feedback throughout the cycle and evaluate that feedback to refine each individual step and the cycle as a whole.

Four Types of Data Analysis

The four main types of data analysis equip ZeroFox analysts with additional techniques to  build more granular findings into their intelligence products.4

  1. DESCRIPTIVE – Describe past results based on set of rules (metrics reports, data mining and aggregation, summary statistics)
  2. DIAGNOSTIC – Diagnose why past results occurred based on probability (principal  components analysis, sensitivity analysis, regression analysis)
  3. PREDICTIVE – Predict what might happen in the future based on probability  (quantitative analysis, predictive modeling, machine learning algorithms)
  4. PRESCRIPTIVE – Recommend actions for next steps based on set of rules (simulation  analysis, recommendation engines, artificial intelligence, neural networks)

Threat Identification

Threat identification is often the first step to providing analytical context around a particular  finding. The six Ws of investigation are a useful framework to apply here.

The Six Ws of Investigation

As production timing and client expectations allow, each ZeroFox Intelligence product  should consider the six Ws of investigation to ensure analysis of key individual(s) involved,  criminal activity, method of operation, geographical scope, threat actor motive and intent, and time frame.

  1. WHAT – What’s going on? What’s being used? What things need to be addressed  concerning the activity (incident, event, campaign, trend, etc.) ? If applicable, include the malware  being used, vulnerabilities exploited, and the identification of the target(s) or target vertical(s).
  2. WHERE – Where are the target(s) located? Does that play any significance in the  targeting? Any geopolitical ramifications?
  3. WHEN – When did the incident/event/campaign take place?
  4. TO WHAT EXTENT – What is the extent of the damage? What is the impact to the target(s)/target vertical(s)? Why should clients care?
  5. WHO – Who was behind the activity (individual, group, etc.)? Can they be placed in any of the following actor categories: cybercriminals, hacktivists, activists, foreign state actors, or cyber spies?
  6. WHY – What was the motivation behind the attack? What was the intent of the attack?  (Intent and motivation are different driving factors.) Were the target(s) opportunistic or  purposefully attacked?

This approach not only ensures the thoroughness of the analysis but also helps with judging the distinction between a threat made and a threat posed. Many ZeroFox clients rely on their analysts to provide contextual and synthesized threat information that differentiates between what is likely just talk and what indicates the potential for real action.

Analytic Standards

Analytic standards are the core principles of intelligence analysis and should be applied to each intelligence product in a manner appropriate to its purpose, type, scope of the underlying source  information, production timeline, and customers. Standards are intended to promote a common  baseline for the implementation of analytical rigor. Ideally, ZeroFox Intelligence products will address all five of the below criteria and their subsets:5

  • Objective. Analysts must perform their functions with objectivity and with awareness of their own assumptions and reasoning. They must employ reasoning techniques and practical mechanisms that reveal and mitigate bias. Analysts should be alert to influence by existing analytic positions or judgments and must consider alternative perspectives and contrary information. Analysis should not be unduly constrained by previous judgments when new developments indicate a modification is necessary.
  • Independent of political consideration. Analytic assessments must not be distorted by, nor shaped for, advocacy of a particular audience, agenda, or policy viewpoint. Analytic judgments must not be influenced by the force of preference for a particular policy.
  • Timely. Analysis must be disseminated in time for it to be actionable by customers. Analysts have the responsibility to be continually aware of events of intelligence interest, of customer activities and schedules, and of intelligence requirements and priorities in order to provide useful analysis at the right time.
  • Based on all available sources of information. Analysis should be informed by all relevant information available. Analysts should identify and address critical information gaps and work with the collection team and data providers to develop access and collection strategies.
  • Utilize analytic tradecraft standards. Analysis should implement and exhibit analytic tradecraft standards, as described in the section below.

Analytic Tradecraft

Analytic tradecraft is the start-to-finish process an analyst undergoes in writing finished  intelligence. The implementation of tradecraft establishes repeatability of method and an audit trail that demonstrates cognitive processes that can be reviewed and scrutinized, as well as used to justify analytic conclusions. Key analytic tradecraft:

  1. Properly describes the quality and credibility of underlying sources, data, and methodologies. Analytic products should identify underlying sources and methodologies upon which judgments are based and use source descriptors in accordance with intelligence sourcing requirements to describe factors affecting source quality and credibility. Such factors can include accuracy and completeness, possible denial and deception, age and continued currency of information, and technical elements of collection, as well as source access, validation, motivation, possible bias, or expertise. Scope notes, as described below, are strongly encouraged and should be used to provide a holistic assessment of the strengths or weaknesses in the source base and to explain which sources are most important to key analytic judgments.
  2. Properly expresses and explains uncertainties associated with major analytic judgments. Analytic products should indicate and explain the basis for the uncertainties associated with major analytic judgments—specifically, the likelihood of occurrence of an event or development and the analyst’s confidence in the basis for that judgment. Degrees of likelihood encompass a full spectrum, from remote to nearly certain. An analyst’s confidence in an assessment or judgment may be based on the logic and evidentiary base that underpins it, including the quantity and quality of source material and the analyst’s understanding of the topic. Analytic products should note the causes of uncertainty (e.g., type, currency, and amount of information; knowledge gaps; and the nature of the issue) and explain how uncertainties affect analysis (i.e., to what degree and how a judgment depends on assumptions). As appropriate, products should identify indicators that would alter the levels of uncertainty for major analytic judgments. Consistency in the terms used and the supporting information and logic advanced is critical to success in expressing uncertainty, regardless of whether likelihood or confidence expressions are used.

a. For expressions of likelihood or probability, an analytic product must use one of the following sets of terms:

b. Analysts are strongly encouraged not to mix terms from different rows. Products that do mix terms must include a disclaimer clearly noting the terms indicate the same assessment of the possibility.

  1. Properly distinguishes between underlying intelligence information and analyst’s assumptions and judgments. Analytic products should clearly distinguish statements that convey underlying intelligence information used in analysis from statements that convey assumptions or judgments. Assumptions are defined as suppositions used to frame or support an argument: assumptions affect analytic interpretation of underlying intelligence information. Judgments are defined as conclusions based on underlying intelligence information, analysis, and assumptions. Products should state assumptions explicitly when they serve as the linchpin of an argument or when they bridge key information gaps. Products should explain the implications for judgments if assumptions prove to be incorrect. As appropriate, products should also identify indicators that, if detected, would alter judgments.
  2. Incorporates analysis of alternatives. Analysis of alternatives is the systematic evaluation of differing hypotheses to explain events or phenomena, explore near-term outcomes, and imagine possible futures to mitigate surprise and risk. Analytic products should identify and assess plausible alternative hypotheses. This is particularly important when major judgments must contend with significant uncertainties or complexity (e.g., forecasting future trends) or when low-probability events could produce high-impact results. In discussing alternatives, products should address factors such as associated assumptions, likelihood, or implications related to U.S. interests. Products also should identify indicators that, if detected, would affect the likelihood of identified alternatives.
  3. Demonstrates customer relevance and addresses implications. Analytic products should provide information and insight on issues relevant to customers and address the implications of the information and analysis they provide. Products should add value by addressing prospects, context, threats, or factors affecting opportunities for action.
  4. Uses clear and logical argumentation. Analytic products should present a clear main analytic message up front. Products containing multiple judgments should have a main analytic message that is drawn collectively from those judgments. All analytic judgments should be effectively supported by relevant intelligence information and coherent reasoning. Language and syntax should convey meaning unambiguously. Products should be internally consistent and acknowledge significant supporting and contrary information affecting judgments.
  5. Explains change to or consistency of analytic judgments. Analytic products should state how their major judgments on a topic are consistent with or represent a change from those in previously-published analysis or that they represent initial coverage of a topic. Products need not be lengthy or detailed in explaining change or consistency. They should avoid using boilerplate language, however, and should make clear how new information or different reasoning led to the judgments expressed in them. Recurrent products, such as daily reports, should note any changes in judgments; absent changes, recurrent products need not confirm consistency with previous editions. Significant differences in analytic judgment, such as between two analysts or intelligence providers, should be fully considered and brought to the attention of customers.
  6. Makes accurate judgments and assessments. Analytic products should apply expertise and logic to make the most accurate judgments and assessments possible based on the information available and known information gaps. In doing so, analytic products should present all judgments that would be useful to customers and should not avoid difficult judgments to minimize the risk of being wrong. Inherent to the concept of accuracy is that the analytic message a customer receives should be the one the analyst intended to send. Therefore, analytic products should express judgments as clearly and precisely as possible, reducing ambiguity by addressing the likelihood, timing, and nature of the outcome or development. Clarity of meaning permits assessment for accuracy when all necessary information is available.
  7. Incorporates effective visual information where appropriate. Analytic products should incorporate visual information to clarify an analytic message and to complement or enhance the presentation of data and analysis. Visual presentations should be used when information or concepts (i.e., spatial or temporal relationships) can be conveyed better in graphic form (e. g, tables, flow charts, images) than in written text. Visual information may range from plain presentation of intelligence information to interactive displays for complex information and analytic concepts. All content in an analytic product may be presented visually. Visual information should always be clear and pertinent to the product's subject. Analytic content in visual information should also adhere to other analytic tradecraft standards.

Writing for Maximum Utility (WMU)

Know Your Customers and What They Need

Knowing your customers and their requirements is integral to current practices and the central tenet upon which WMU is based. Yet, WMU also requires the analyst to appreciate the totality of the audience of customers served. Although many customers are technically savvy, most are supported by staff or provide information and intelligence to elements that are not.

  • To facilitate and ensure a current and accurate understanding of customers' needs, ZeroFox will collect and validate our customers’ priority intelligence requirements (PIRs). These PIRs will be updated on a periodic basis to ensure ZeroFox continues to produce valid intelligence related to our customers’ evolving needs.
  • Understanding a customer's operating environment, to include information systems access and physical security restrictions, is crucial to producing intelligence that is of the greatest utility. Analysts shall develop a better understanding of how our customers receive products (e.g., briefings, written products, online pull) and how the products will be used by the customers.
  • ZeroFox shall aggressively seek and incorporate customer feedback to ensure that its intelligence products best meet the customer's needs.

Write for Tailored Reuse

ZeroFox’s intelligence products shall, when appropriate and feasible, be conceived and structured in a manner that increases their potential for tailored reuse, thereby enhancing their utility. Such efforts span the spectrum from writing multiple versions of a product to formatting paragraphs and characterizing sourcing information in ways that provide insight but do not reveal sources and methods. Where appropriate and feasible, intelligence products—to include text-based, multimedia, and other formats— should be constructed using tools and strategies (e.g., templates, ontologies, portion markings) that facilitate reuse and wider dissemination. These tools and strategies should be selected to minimize post-production manipulation, facilitate sanitization, and aid in tailoring products to meet the needs of multiple, diverse customers.

Products Must be Discoverable by Those Who Might Need Them

ZeroFox personnel or customers must be able to discover intelligence, information, and analyses that address their requirements or aid in mission accomplishment. ZeroFox shall create processes so that customers can search and discover intelligence products of interest and easily request and get what they need in a timely manner.

Tradecraft Essential, Not Expendable

Regardless of classification level, products must adhere to Intelligence Analytic Standards and Sourcing Requirements. Quality products that reflect strong intelligence tradecraft are fundamental to WMU.

  • WMU seeks to convey insight into the quality of the sourcing through a variety of techniques and means. Although full visibility on product sourcing may not always be possible, in order to broaden dissemination, ZeroFox shall—whenever appropriate and feasible—include source descriptors and other collection parameters. Sourcing insight can also be conveyed via numeric scale or broad credibility statements.
  • Products should never render judgments in a manner inconsistent with the supporting evidence, and both confidence levels and probabilistic language must be congruent.

Timely Dissemination is Always the Goal

With minimal exception, ZeroFox shall disseminate intelligence products electronically. This includes products that are produced at multiple classification levels. Such efforts will expedite customer discovery, "pull" of needed products, and make it easier and faster for customers to share intelligence with those they support and direct. The formatting or tagging of products shall facilitate automated posting.

Train to Think of Customers Inclusively; Write Differently

ZeroFox leadership shall review and, where required, revise training programs to explicitly address how to maximize product utility. Writing and production curricula shall address the way analysts and collectors think about their customers and how they construct and write their products. This curriculum shall encourage contributors to the production process—from planning through dissemination—to think inclusively about their customers, their operating environment, and how their various customers use their products.

Developers of training curricula are encouraged to focus on the way products, to include releasable summaries, are organized and constructed and how judgments are communicated. Drafting and presenting information in segments that enable tailored reuse and techniques for conveying sourcing information and confidence levels in more nuanced ways shall be part of this curriculum. This may include training that encompasses the art of sanitization and the purpose and procedures for writing sanitized products. Such training shall emphasize creating products that are informed by all relevant information; however, to the extent possible, analytic judgments or relevant information shall be presented at the lowest classification level at which the essential message can be conveyed to allow optimum sharing and maximize utility.

Psychological Pitfalls

Every analyst is human and thus prone to the psychological pitfalls that bedevil even the most  sophisticated and experienced operators. A collective operational awareness of the  misconceptions and biases listed below is essential to maintaining the analytical rigor of the  intelligence cycle at ZeroFox. 

Common misconceptions often arise at the stages of collection and vetting. Taken by themselves,  the below criteria are insufficient to determine or dismiss a finding as relevant:

  • Perceived importance (i.e., whether a finding is “a big deal”) 
  • Whether the issue is known or unknown to the client 
  • Whether the finding conveys a true or false statement 
  • Volume or repetitiveness of a particular finding 
  • Whether the finding occurs in the past, present, or future 
  • Probability that a particular finding will occur 
  • Motivation, intent, or role assumed from the finding 
  • Completeness of the finding (i.e., lacking in detail)

Cognitive biases and other tendencies easily take hold at the stages of analysis and production,  particularly when analysts are engaged in time-sensitive work. Cognitive biases are mental errors caused by our simplified information processing strategies that are consistent and predictable. Awareness of the bias does not, by itself, produce a more accurate perception; therefore, cognitive biases are difficult to overcome.6 The following list contains a few of the pitfalls that have plagued and continue to challenge intelligence services around the world:7

  • Confirmation Bias: the tendency to only accept what aligns with an established view (i.e., analysts only favor information that fits within their hypothesis and dismiss the  alternative)
  • Group Think: the natural impetus to arrive at consensus (i.e., analysts fail to examine their assumptions rigorously enough and lapse easily into agreement as a group)
  • Mirror Imaging: the belief that “They are just like us” (i.e., analysts assume the target  shares motivations or goals similar to those most familiar to them)
  • Stovepipes: a lack of communication between intelligence entities (i.e., analysts fall into separately managed silos and engage in little information sharing)
  • Clientism: the loss of ability to view issues with necessary criticality (i.e., analysts become so immersed after working a client for so long that they lose analytical sharpness)
  • Layering: the use of assumptions for the factual basis of additional assumptions (i.e., analysts ignore uncertainties in a previous analysis that then underpin another analysis)
  • Attribution Bias: the hasty assignment of responsibility to a threat actor (i.e., analysts  attribute threat activity to an actor based on weak assumptions and incomplete evidence)

There are many more types of cognitive biases that affect belief formation, reasoning processes, business and economic decisions, and human behavior in general— as well as controversies over whether they result in useful attitudes or behavior.  More reading on biases can be found in the footnoted reference below.8

Confidence Levels and Estimative Language

Confidence is a judgment based on three factors: (1) strength of knowledge base, to include the  quality of the sources and depth of understanding the issue; (2) the number and importance of  assumptions used to fill information gaps; and (3) the strength of logic underpinning the  argument, which encompasses the number and strength of analytic inferences, as well as the  rigor of the analytic methodology of the product.

Confidence levels provide assessments of the quality and quantity of the source information that  supports judgments and align with the percentages of certainty show in the chart below:

Estimative Analytic Language is used to help convey the analyst’s assessed likelihood or probability of an event or incident and the level of confidence ascribed to a judgment. Assessments are based on collected  information (which is often incomplete), as well as logic, argumentation, and precedents. Certain qualitative terms are used with specific confidence levels to ensure continuity of  expression. The following chart depicts the relation between confidence level, certainty/likelihood of occurrence, and the corresponding accepted estimative analytic language.

Sourcing

Providing accurate descriptions of sources is critical to determining the validity and veracity of data gathered and analyzed in support of analytic assessments, conclusions, and recommendations. Documenting sources throughout written products is required. Furthermore, it is incumbent upon analysts to provide a scope note that indicates the various types of sources from which ZeroFox Intelligence products are derived, as well as a collection cut-off date and time. This ensures the reader understands the basis for conclusions presented and the relative weight those conclusions deserve based on the combination of the confidence language used (see previous reference to confidence language) and the sources supporting that assessment.

Due to limitations inherent to Google Docs that do not allow for use of both footnotes and endnotes within a document, ZeroFox will use footnotes for both URL source citations and to provide text that clarifies or expands on a term or reference, as needed.

Information and Source Reliability

Source and information reliability are typically assessed by the collectors of information. Since longer, finished analytic products are derivative of multiple sources  of information, providing an overall assessment of the reliability of all sources/information used  in the report is important to inform the reader/customer.

Traffic-Light Protocol

The Traffic Light Protocol (TLP) was created in order to facilitate greater sharing of information.  TLP is a set of designations used to ensure that sensitive information is shared with the  appropriate audience. It employs four colors to indicate expected sharing boundaries to be  applied by the recipient(s). The following graphic depicts color schemes, how they are used, and  sharing guidelines.

1 Mark M. Lowenthal, Intelligence: From Secrets to Policy, 4th Edition (Washington, DC: CQ Press, 2009).

2 David Carter, “Understanding Contemporary Law Enforcement Intelligence: Concept and Definition,” U.S.  Department of Justice Office of Community Oriented Policing Services, 2004, hXXps://fas[.]org/irp/agency/doj/lei/chap2.pdf. 

3 “U.S. National Intelligence: An Overview 2011,” Office of the Director of National Intelligence, 2011,  hXXps://www.dni[.]gov/files/documents/IC_Consumers_Guide_2011.pdf.

4 “Types of Data Analytics and How to Apply Them,” Michigan State University, last updated October 8, 2019,  hXXps://www.michiganstateuniversityonline[.]com/resources/business-analytics/types-of-data-analytics-and how-to-apply-them/ 

 5 “Intelligence Community Directive 203,” Office of the Director of National Intelligence, January 2, 2015,  hXXps://fas[.]org/irp/dni/icd/icd-203.pdf. 

6 hXXps://www.ialeia[.]org/docs/Psychology_of_Intelligence_Analysis.pdf

7 Robert D. Kline et al., Intelligence and the National Security Strategist: Enduring Issues and Challenges (Oxford,  UK: Rowman and Littlefield Publishers, Inc., 2006) and Mark M. Lowenthal, Intelligence: From Secrets to Policy, 4th  Edition (Washington, DC: CQ Press, 2009).

 8 hXXps://en.wikipedia[.]org/wiki/List_of_cognitive_biases