NIEMOpen

NIEMOpen (neemopen), frequently referred to as NIEM, originated as an XML-based information exchange framework from the United States, but has transitioned to an OASIS Open Project. This initiative formalizes NIEM's designation as an official standard in national and international policy and procurement. NIEMOpen's Project Governing Board recently approved the first standard under this new project; the Conformance Targets Attribute Specification (CTAS) Version 3.0. A full collection of NIEMOpen standards are anticipated by end of year 2024.

NIEM offers a common vocabulary that enables effective information exchanges across diverse public and private organizations. NIEM is currently developing the NIEM Metamodel and Common Model Format which can be expressed in any data serialization that NIEM supports, including, but not limited to JSON.

Formed from an interagency partnership, NIEM has come to represent a collaborative partnership of agencies and organizations across all levels of government (federal, state, tribal, and local) in addition to private industry. The purpose of this partnership is to effectively and efficiently share critical information at key decision points throughout the whole of the justice, public safety, emergency and disaster management, intelligence, United States Department of Defense and homeland security enterprise. NIEM is designed to develop, disseminate, and support enterprise-wide information exchange standards and processes that will enable jurisdictions to automate information sharing.

Today, NIEMOpen is sponsored by the Joint Staff J6 Directorate within the U.S. Department of Defense, the Department of Homeland Security Science and Technology Directorate (DHS S"&"T), the FBI Criminal Justice Information Services (CJIS) within the U.S. FBI, Equivant, Georgia Tech Research Institute, the National Association for Justice Information Systems, sFractal Consulting LLC, the IJIS Institute, the US Department of Transportation, and the Virginia Office of Data Governance and Analytics. NIEM provides a working and collaborative partnership among governmental agencies, operational practitioners, systems developers, and standards bodies across Federal, State, Local, Tribal, Territorial, International and Private organizations.

NIEM has been identified as a key enabler for Joint All Domain Command and Control (JADC2). NIEM is cited in the JADC2 Reference Architecture (RA) Version 3.0 Enclosure D (JADC2 Capability Development and Analytical Framework) within the Application and services, Interface and Data & Information principals. JADC2 Reference Design (RD) Version 1.0, Standard View 2 (StdV-2).^[1]

NIEM most recently was referred to as the National Information Exchange Model. That interagency government project was an outgrowth of the United States Department of Justice's Global Justice XML Data Model (GJXDM) project. As an interagency project it was expanded to include other federal and state agencies such as the Office of the Director of National Intelligence, United States Department of Defense, Federal Bureau of Investigation, Texas, Florida, New York, Pennsylvania, and others.

NIEM is designed to facilitate the creation of automated enterprise-wide information exchanges which can be uniformly developed, centrally maintained, quickly identified and discovered, and efficiently reused.

The following key concepts are essential to understanding the purpose, architecture, processes, and other capabilities of NIEM, as well as to establish a common knowledge base with which to develop the ability to use NIEM effectively.

Data Components. The fundamental building block of NIEM is the data component. Data components are the basic business data elements that represent real-world objects and concepts. Information exchanged between agencies can be broken down into individual components – for example, information about people, places, material things, and events. Components that are frequently and uniformly used in practice are specified in NIEM and can then be reused by practitioners for information exchanges, regardless of the nature of their business or the operational context of their exchanges, provided they are semantically consistent.

Information Exchange Package Documentation. The information that is commonly or universally exchanged between participating domains can be organized into information exchange packages (IEPs) in the form of XML Schemas. An example of this collection of information is data associated with an arrest. The data to be exchanged includes not only descriptive and personal identification data regarding the individual arrested (i.e., the person component described above) but also information about the person's alleged offense, the location of the offense, the arresting officer, etc. The IEP represents a set of data that is actually transmitted between agencies for a specific business purpose (e.g., initiating a charging document by the local prosecutor). It includes the actual XML instance that delivers the payload or information. Additional information regarding this specific exchange can be further documented in the form of an information exchange package documentation (IEPD), which also contains data describing the structure, content, and other artifacts of the information exchange. An IEPD supports a specific set of business requirements in an operational setting.

NIEM Core. Data components within an information exchange that are universally shared and understood among all (or almost all) domains are identified as universal components (e.g., person, address, and organization). To become a universal component, consensus by all domains is needed on the semantics and structure of the component. The set of NIEM universal components is stable (once established) and relatively small.

Domains. For purposes of NIEM, a domain refers to a business enterprise broadly reflecting the agencies, units of government, operational functions, services, and information systems which are organized or affiliated to meet common objectives. NIEM domains are organized to facilitate governance, and each has some measure of persistency. Each domain traditionally includes a cohesive group of data stewards who are subject matter experts (SMEs), have some level of authority within the domains they represent, and participate in the processes related to harmonizing conflicts and resolving data component ambiguities.

Communities of Interest. Communities of interest (COIs) are collaborative groups of users who exchange information in pursuit of shared goals, interests, missions, or business processes and who therefore must have a shared vocabulary for the information they exchange. COIs reuse data components and artifacts found in NIEM to document their information exchanges. One or more COIs can coordinate to develop new domain content as they identify gaps in the data components needed for documenting information exchanges.

NIEM Conformance. There are NIEM conformance rules that serve as guidelines for agencies utilizing NIEM to implement their information sharing exchanges. Grantees developing inter-agency XML-based exchanges must comply with the special condition language contained in the grant, and follow the associated NIEM implementation guidelines.

The NIEM Management Office (NMO) operates to:

• Bring stakeholders, agencies, and the domains and COIs that they represent together to identify information sharing requirements in daily operational and emergency situations;
• Develop information sharing standards, a common lexicon, and an online repository of information exchange package documentation and data components that support information sharing;
• Provide technical tools, processes, and methodologies to support the analysis, development, discovery, dissemination, and reuse of exchange standards and documents; and
• Provide training, technical assistance, communication, outreach, and implementation support services for NIEM-based information sharing.

NIEM is a continually evolving program, and new agencies and COIs are joining the effort all the time. As new stakeholders come on board, they need to receive information to gain understanding and knowledge of the core capabilities of NIEM and how to engage in NIE information exchanges. In 2022, NIEM created a training video series on NIEM.gov/learn that includes a technical deep dive and is intended to support developers and implementers. Monthly users can sign up for virtual training sessions which includes the pre-recorded videos in addition to a live session with a NIEM expert who will answer questions in real-time. NIEM.gov provides written training materials, such as briefings and process-related documentation on NIEM GitHub, as well as other resources, such as the National Information Sharing Standards Help Desk and Knowledge Base. Training provides the knowledge and know-how stakeholders need to use the tools and other capabilities provided by NIEM. NIEM tools and training opportunities are further described below. Other training materials, such as executive briefings, marketing material, and briefings for conferences and workshops, are offered from time to time and are tailored depending on the audience. A selection of online materials are also available both from the main NIEM.gov web site and resources such as Youtube.com (search on NIEM training).

The NIEM Web site serves as a primary means by which NIEM can provide the latest documentation and downloads to those interested in NIEM. It also serves as a starting point for those wishing to contact NIEM staff with questions, support, and information requests. As related projects, tools, and support resources develop around NIEM, the Web site will expand as the hub for these supplemental resources.

NIEM adopts standard XML schema constructs and methods, such as roles, associations, and augmentation from industry standards, such as the World Wide Web Consortium (W3C) XML Schema language.

The NIEM reference schemas are a set of interrelated schemas that define NIEM data components. Each schema defines its own target namespace. Schemas in the reference set may import one another by namespace in order to use (or reuse) components they define. In general, domain reference schemas import schemas from the Core. The NIEM reference schema set represents the full set of data components in NIEM.

The following kinds of XML schemas are associated with the NIEM architecture:

• NIEM reference schemas: Schemas containing content created or approved by the NIEM steering committees are periodically released in schema distributions.
• Subset Schema: a NIEM-conformant schema, containing only the parts of the reference schemas needed to support a particular exchange.
• Support schemas: NIEM includes three special schemas, appinfo, structures and proxy, for annotating and structuring NIEM-conformant schemas.
• Extension Schema: a NIEM-conformant schema which adds domain- or application-specific content to the base NIEM model.
• Exchange Schema: a NIEM-conformant schema which specifies a document in a particular exchange.
• Constraint Schema: a NIEM-conformant schema which adds additional constraints to NIEM-conformant instances, but which is assumed to validate in concert with existing NIEM-conformant or subset schemas. A constraint schema need not validate constraints that are applied by other schemas.
• Codelist Schemas: a NIEM-conformant schema which provides a list of acceptable values that a data element will be constrained to.

The only mandatory schemas for validation are the NIEM reference schemas or a correct subset, however the IEPD specification requires that an IEPD include an exchange schema (along with the reference schemas or subsets) to be considered a complete IEPD. The NIEM schemas may import additional schemas, such as code table schemas, as needed. An optional extension schema may be used to add extended types and properties for components not contained in NIEM, but which are needed for the exchange.

The naming and design rules for NIEM are documented in the NIEM NDR, which specifies the data model, XML components, and XML data for use with NIEM and provides a basis for NIEM conformance. The current version is NDR v5.0, which was released on December 18, 2020.

NIEM is based on several concepts from the International Organization for Standardization (ISO) 11179, which provides guidelines for the naming and definition of data elements, as well as information about the metadata captured about data elements. Part 5 of the ISO 11179 standard establishes a methodology for naming items in data dictionaries.

The ISO 11179-based NIEM NDR naming convention uses object class, property, and representation terms to constitute a multiple-part name as shown in the figure below:

Object Class Term: Represents the object to which the property is applicable. In NIEM, we interpret that object to be the real-world object. (An object class refers to a group of objects that share the same attributes, operations, methods, relationships, and semantics.)

Property Term: Identifies the property that the data element represents (e.g., last name, expiration date, height, total).

Representation Term: Describes the form of the data represented. These terms have been derived from ISO 11179 representation terms, and include Amount, Code, Date, Time, Indicator, Measure, Name, Numeric, Percent, Picture, Quantity, Rate, and Time.

Qualifier Term: The object class and property terms can have qualifiers; i.e., a word or words that help define and differentiate the element name.

The NIEM technical modeling and schema concepts and mechanisms which support building new data components that meet specific requirements and reusing existing NIEM data components are briefly described below. More detail can be found in the NIEM User Guide and the NIEM NDR.

The NIEM data model uses concepts originating from object-oriented programming (OOP). OOP defines a class as a specific entity in the data model, which may represent a real-world object but may also represent any conceptual object, such as relationships and messages. An object's properties are said to describe the object. When the NIEM XML Schemas are generated from the NIEM data model, data model classes are represented as XML Schema types, and data model properties are represented as XML elements and attributes.

There are two approaches for extending the NIEM data model for use in information exchange schemas and documents.

• Creating new NIEM types to represent new concepts
• Adding new data to existing NIEM types, to extend existing concepts

Specialization is used when a base object class (type) contains or can be subcategorized into a more specific subclass. When this can be done, the subclass derived from the base class inherits the properties of the more general base or parent class. This mechanism is used to share or reuse properties between the general data component and its specialization. For example, a vehicle type (or class) is identified as a data component with properties of vehicle identification number (VIN), make, and model. Truck type (or class) is a specialization of vehicle and thus inherits the vehicle's properties but also has its own characteristic properties, such as truck bed length. Specialization is time independent and is generally used only when the base class and subclass always exist.

A role is a special type which represents a particular function, purpose, context, or activity for an entity. Roles are generally time dependent and, therefore, temporary. A new type can be created for a role when the role has specific data associated with it and its own life cycle. A role type has a property, RoleOf, which indicates what object is assuming this role. A single entity may assume multiple roles. For example, many different entities may assume the role of a weapon. Therefore, if a vehicle is used as a weapon (to attempt to injure or kill a person), then an instance of WeaponType would contain the property, RoleOf, which references the vehicle instance used as the weapon. The WeaponType (the role) might also contain properties that describe the persons and activities involved, dates and times of involvement, and how the entity was used as a weapon.

An association type is an object that represents a relationship between data components. For example, two PersonType instances, Abigail and Bob, could be referenced by a MarriageAssociationType to represent the fact they are married. The MarriageAssociationType could contain its own properties, such as date of marriage, number of children, date of divorce, death of one spouse, etc.

Augmentation of a NIEM data type allows the addition of domain- or model-specific information to the concept embodied in the NIEM type, without creating a new NIEM type. It would be impractical and unwieldy to include all possible domain model-specific properties in NIEM Core schemas for general use. Instead, domain modelers need to be able define data for their use, independently from common definitions. Furthermore, that data needs to be applicable to the NIEM data object itself, and reusable in multiple exchanges. The augmentation approach built into NIEM utilizes XML Schema constructs to reuse the existing XML schema representations for the data model, by allowing them to be augmented with the new information.

NIEM uses several techniques from XML Schema to allow as-needed element substitutions for pre-existing NIEM properties and into pre-existing NIEM types. Element substitution techniques allow the substitution of new XML Schema elements, representing derived NIEM properties that can be used where the parent properties are expected.

There are three XML Schema techniques that support the NIEM use of element substitutions:

• use of substitution groups
• creation of abstract, type-less elements, and
• use of abstract elements in reference schemas.

In addition to adding new NIEM types and properties to NIEM, it is possible to adapt existing external (non-NIEM) namespaces for use in the NIEM framework. This allows the use of external standards within NIEM IEPDs, without requiring that the external standards themselves be NIEM-conformant. The intent here is to allow use of external standard components exactly as they were defined.

Metadata, or data about data, defines information that supports the actual content of XML instances. The metadata feature provides a mechanism for attaching structured properties that describe the pedigree or source (when reported, who reported, how reliable, etc.) of instance data to any data component of the model (type or object, property, association, role, or augmentation) in any namespace. It allows sets of metadata to be extended with additional properties for local requirements and enables metadata properties to be repeated.

NIEM has introduced the concept of Enterprise Information Exchange Model (EIEM) dictionaries to allow NIEM implementers to position local collections of components specific to their own enterprise. Typically these may be drawn from an Enterprise Data Model (EDM) and then exported and refactored as XML components that conform to the NIEM NDR. See Tools section below for information on creating EIEM dictionaries.

NIEM provides a reference set of tools freely available with each NIEM release. The tools implement all of the structural and content features of the release, including the NIEM NDR. NIEM's well-defined interfaces and output products also support the development of independent third-party tools. Example supporting third-party tools are listed below.

The NIEM XML data dictionary spreadsheet is a tangible representation of the entire NIEM data dictionary. It includes all of the element names that are organized hierarchically under core data components (person, property, organization, etc.) with hyperlinks to related elements. This spreadsheet also provides information on the type of data being represented (date, integer, string, etc.) and a precise, context-rich definition of each dictionary component.

The Schema Subset Generation Tool (SSGT) enables users to search through the NIEM data model, view information about NIEM elements and types, and build a custom XML Schema subset of a NIEM release. The NIEM data model contains over 12,000 elements and represents over a dozen domains; subsets allows users developing an IEPD to select only what they need based on the specific requirements of the exchange. With this tool users can:

• Search and explore the NIEM data model.
• Build a schema subset containing the components required by the exchange.
• Download the current subset as a zip file (go to Generate Documents -> Save Subset Schema to a file).
• Resume editing a previous subset (go to Options -> Load Wantlist and upload the wantlist.xml file contained in a previously downloaded subset).
• Switch between any NIEM release (go to Options -> Change Release to).
• Download already generated subsets from each domain.

The Conformance Testing Assistant (ConTesA) enables the user to test NIEM XML schemas against the automated rules from the NIEM Naming and Design Rules (NDR). This is an important first step in verifying NIEM conformance. Users can upload a single NIEM XML schema or a zip file (typically representing an IEPD) and view or download the results as a conformance report.

Movement enables the user to search and explore the content of the latest NIEM release. It provides a user friendly interface and smarter search results. It is also open source: the community who inspired its creation can contribute to it.

The Migration Tool enables the user to upgrade a NIEM XML Schema release subset generated by the SSGT to the next release. Multiple migrations can be run to upgrade to a non-consecutive release.

See the NIEM Tools Catalog for other tools developed by the NIEM community.

As of December 2022^[update], NIEM 5.2 is the current release. It was released on December 23, 2022, and contains minor changes.^[2] Notable to NIEM 5.2 are these changes to content and tools. Changes to the specifications, and tools are reflected below:

• Biometric domain
  • Added and modified content to support State, Local, Tribal, and Territorial (SLTT) Agency Cyber Incident Reporting Requirements (CIRR) and Cyber-Physical System (CPS) Incident Reporting Requirements.
• Learning and Development domain (new)
  • Added content about courses, course sections, students, and employment positions.
• Justice domain
  • Updated the FBI's National Crime Information Center (NCIC) codes.
• Military Operations domain
  • Added Minimum Essential Metadata (MEM) content.
  • Added U.S. Message Text Format (USMTF) content as a subdomain, with information about military operations, missions, resources, readiness, and training.
• FIPS county codes
  • Updated FIPS 3-digit and 5-digit county codes.
• ISO 639-5 language family codes
  • Added the ISO 639-5 language family code set.

• NIEM Naming and Design Rules (NDR), 5.0
• NIEM Information Exchange Package Documentation (IEPD) Specification, 5.0
  • New name for the previous Model Package Description (MPD) specification, 3.0.1.
  • Simplifies artifacts and IEPD structure.
• NIEM JSON Specification, 5.0
• NIEM Conformance Specification, 5.0
• NIEM Code Lists Specification, 5.0

• Schema Subset Generation Tool (SSGT): Added the new 5.2 release
• Movement: Replaced the current release (5.0) with the new 5.2 release
• Conformance Testing Assistant (ConTesA): Added support for NDR 5.0 Schematron rules
• Migration Tool: Added support for migrating SSGT 5.1 subsets to 5.2 subsets

• NIEM 5.2 Production Release - December 23, 2022
• NIEM 5.1 Production Release – December 23, 2021^[3][4]
• NIEM 5.0 Production Release - December 22, 2020 ^[5][6]
• NIEM 4.2 Production Release – October 1, 2019^[7][8]
• NIEM 4.1 Production Release – August 1, 2018^[9][10]
• NIEM 4.0 Production Release – June 28, 2017^[5]
• NIEM 3.1 Production Release – May 29, 2015^[11]
• NIEM 3.0 Production Release – October 21, 2013^[12]
• NIEM 2.1 Production Release – September 28, 2009
• NIEM 2.0 Production Release – July 31, 2007
• NIEM 1.0 Production Release – November 1, 2006
• NIEM 0.3 – April 12, 2006
• NIEM 0.2.1 – February 24, 2006
• NIEM 0.2 – December 23, 2005
• NIEM 0.1 – October 11, 2005

• Rules for NIEM Conformance
• Global Justice XML Data Model (GJXDM)
• ISO/IEC 11179
• Metadata publishing
• Metadata registry
• Universal Core (UCore)

• [1]
• [2]
• The official NIEMOpen.org website
• The official NIEM.gov website
• Justice Information Exchange Model (JIEM) Tool
• Grid-ML XML Schema Viewer - A tool for viewing and searching NIEM Schemas
• Data Interchange Standards