Bundle Entities

In order to understand bundles, we need to take a step back and take another look at relationships.

Getting UUIDs for Relationships

Up until this point we have intentionally glossed over an important detail in the creation of relationships: how do we create them if we don't have the URLs for the entities we want to relate?

If a Relationship entity requires two IDs, and if IDs in CTIA are always URIs, then we need the URI for the source_ref, and the URI for the target_ref before we can specify a Relationship. As you might imagine, this can become burdensome.

Defining Relationships Without Bundles

This is how we used to assemble relationships, before bundles:

Search for (with a GET request) or create and POST the source entity in order to get its URI for the source_ref (1-2 HTTP requests)
Search for (with a GET request) or create and POST the target entity in order to get its URI for the target_ref (1-2 HTTP requests)
Once you have both of the required URIs, create and POST a Relationship entity (1 HTTP request).

It used to require 3-5 HTTP requests for every relationship. If you knew that both the source and target entities were brand new, you could get the job done with only 3 POST requests. However, even with persistent sessions, this was never going to scale to our needs.

And so, we added the bundle import API mechanism to CTIA. The purpose of the bundle import mechanism:

We needed a mechanism that would allow us to define entities AND how they relate in a single bundle, and POST all of the contents of that bundle in the same HTTP request, and let CTIA automatically handle the complications around wiring all of the relationships to point at the correct URIs for the posted entities.
We needed a way to prevent the accidental creation of duplicate entities. Particularly in the case of indicators, we did not want there to be duplicate entities that differ only in their id field, but are identical in every other way.

Defining Relationships Using Bundles

The new way: Using transient IDs
- transient IDs
- features of the CTIA bundle import API endpoint

Bundle Field Summary

In addition to the required "type":"bundle" field and the strongly recommended source and source_uri fields, bundles can contain lists of various CTIM entity types. Most of them are outside the scope of this tutorial, but they are all defined in great depth in the CTIM Bundle schema.

Optional Bundle Fields

actors - A list of CTIM actor entities.
attack_patterns - A list of CTIM attack pattern entities.
campaigns - A list of CTIM campaign entities.
coas - A list of CTIM coa entities.
feedbacks - A list of CTIM feedback entities.
incidents - A list of CTIM incident entities.
indicators - A list of CTIM indicator entities.
judgements - A list of CTIM judgement entities.
malwares - A list of CTIM malware entities.
relationships - A list of CTIM relationship entities.
sightings - A list of CTIM sighting entities.
tools - A list of CTIM tool entities.
verdicts - A list of CTIM verdict entities.
data_tables - A list of CTIM data_table entities.
weaknesses - A list of CTIM weakness entities.
vulnerabilities - A list of CTIM vulnerability entities.

Example Bundle

The following is the example bundle:

{
    "type":"bundle",
    "source":"Modeling Threat Intelligence in CTIM Tutorial",
    "source_uri":"https://github.com/threatgrid/ctim/blob/master/src/doc/tutorials/modeling-threat-intel-ctim.md",
    "sightings":[
        {
            "observables":[
                {
                    "type":"ip",
                    "value":"187.75.16.75"
                }
            ],
            "type":"sighting",
            "source":"Modeling Threat Intelligence in CTIM Tutorial",
            "external_ids":[
                "ctim-tutorial-sighting-7b36e0fa2169a3ca330c7790f63c97fd3c9f482f88ee1b350511d8a51fcecc8d"
            ],
            "source_uri":"https://github.com/threatgrid/ctim/blob/master/src/doc/tutorials/modeling-threat-intel-ctim.md",
            "id":"transient:ctim-tutorial-sighting-7b36e0fa2169a3ca330c7790f63c97fd3c9f482f88ee1b350511d8a51fcecc8d",
            "count":1,
            "severity":"High",
            "tlp":"green",
            "timestamp":"2019-03-01T22:26:29.229Z",
            "confidence":"High",
            "observed_time":{
                "start_time":"2019-03-01T22:26:29.229Z"
            }
        }
    ],
    "judgements":[
        {
            "valid_time":{
                "start_time":"2019-03-01T22:26:29.229Z",
                "end_time":"2019-03-31T22:26:29.229Z"
            },
            "observable":{
                "type":"ip",
                "value":"187.75.16.75"
            },
            "type":"judgement",
            "source":"Modeling Threat Intelligence in CTIM Tutorial",
            "external_ids":[
                "ctim-tutorial-judgement-4340e8cc49ff428e21ad1467de4b40246eb0e3b8da96caa2f71f9fe54123d498"
            ],
            "disposition":2,
            "source_uri":"https://github.com/threatgrid/ctim/blob/master/src/doc/tutorials/modeling-threat-intel-ctim.md",
            "disposition_name":"Malicious",
            "priority":95,
            "id":"transient:ctim-tutorial-judgement-4340e8cc49ff428e21ad1467de4b40246eb0e3b8da96caa2f71f9fe54123d498",
            "severity":"High",
            "tlp":"green",
            "timestamp":"2019-03-01T22:26:29.229Z",
            "confidence":"High"
        }
    ],
    "indicators":[
        {
            "description":"The IP Blacklist is automatically updated every 15 minutes and contains a list of known malicious network threats that are flagged on all Cisco Security Products. This list is estimated to be 1% of the total Talos IP Reputation System.",
            "valid_time":{
                "start_time":"2019-03-01T22:26:29.229Z",
                "end_time":"2525-01-01T00:00:00.000Z"
            },
            "producer":"Cisco TALOS",
            "type":"indicator",
            "source":"Modeling Threat Intelligence in CTIM Tutorial",
            "external_ids":[
                "ctim-tutorial-indicator-c56de1c94c1ce862c4e6d9883393aacc58275c0c4dc4d8b48cc4db692bf11e4f"
            ],
            "short_description":"The TALOS IP Blacklist lists all known malicious IPs in the TALOS IP Reputation System.",
            "title":"TALOS IP Blacklist Feed",
            "source_uri":"https://github.com/threatgrid/ctim/blob/master/src/doc/tutorials/modeling-threat-intel-ctim.md",
            "id":"transient:ctim-tutorial-indicator-c56de1c94c1ce862c4e6d9883393aacc58275c0c4dc4d8b48cc4db692bf11e4f",
            "tlp":"green"
        }
    ],
    "relationships":[
        {
            "type":"relationship",
            "source":"Modeling Threat Intelligence in CTIM Tutorial",
            "source_uri":"https://github.com/threatgrid/ctim/blob/master/src/doc/tutorials/modeling-threat-intel-ctim.md",
            "source_ref":"transient:ctim-tutorial-judgement-4340e8cc49ff428e21ad1467de4b40246eb0e3b8da96caa2f71f9fe54123d498",
            "target_ref":"transient:ctim-tutorial-indicator-c56de1c94c1ce862c4e6d9883393aacc58275c0c4dc4d8b48cc4db692bf11e4f",
            "relationship_type":"based-on",
            "external_ids":[
                "ctim-tutorial-2c1f3fcaf89d294bf7d038f470f6cb4a81dc1fad6ff5deeed18a41bf6fe14e4d"
            ],
            "short_description":"judgement ctim-tutorial-judgement-4340e8cc49ff428e21ad1467de4b40246eb0e3b8da96caa2f71f9fe54123d498 is based-on indicator ctim-tutorial-indicator-c56de1c94c1ce862c4e6d9883393aacc58275c0c4dc4d8b48cc4db692bf11e4f"
        },
        {
            "type":"relationship",
            "source":"Modeling Threat Intelligence in CTIM Tutorial",
            "source_uri":"https://github.com/threatgrid/ctim/blob/master/src/doc/tutorials/modeling-threat-intel-ctim.md",
            "source_ref":"transient:ctim-tutorial-sighting-7b36e0fa2169a3ca330c7790f63c97fd3c9f482f88ee1b350511d8a51fcecc8d",
            "target_ref":"transient:ctim-tutorial-indicator-c56de1c94c1ce862c4e6d9883393aacc58275c0c4dc4d8b48cc4db692bf11e4f",
            "relationship_type":"sighting-of",
            "external_ids":[
                "ctim-tutorial-0664295d5da504180b4f232a0d5e95908fcbd6eb052b6e97f294ddfb6a7b11b8"
            ],
            "short_description":"sighting ctim-tutorial-sighting-7b36e0fa2169a3ca330c7790f63c97fd3c9f482f88ee1b350511d8a51fcecc8d is sighting-of indicator ctim-tutorial-indicator-c56de1c94c1ce862c4e6d9883393aacc58275c0c4dc4d8b48cc4db692bf11e4f"
        }
    ]
}

POSTing Bundles to CTIA

As described above, we supply the external ID prefix to CTIA via the external-key-prefixes query parameter. So, in order to POST our bundle to CTIA, you must run the following command:

curl -X POST \
    --header 'Content-Type: application/json' \
    --header 'Accept: application/json' \
    --header 'Authorization: <your auth>' \
    -d '{"type":"bundle", \
         "source": "Modeling Threat Intelligence in CTIM Tutorial", \
         ...
        }'
    'https://localhost:3000/ctia/bundle/import?external-key-prefixes=ctim-tutorial-'

When the bundle is posted, CTIA will perform the following:

Search for existing entities that already have the external IDs starting with the ctim-tutorial- prefix.
If existing entities with matching external_ids to the ones supplied in the bundle are found in storage, then all relationship references to that entity in the bundle will be replaced with references to the URI for the existing entity in storage. Duplicate entities with the same "ctim-tutorial-" external IDs will not be created.
For each entity in the bundle, if no existing entity matching the supplied "ctim-tutorial-" external IDs is found in storage, CTIA will create a new entity and replace the transient ID reference in any relationships with the URI for the newly created entity.
Once all of the references are thus resolved, CTIA will POST the relationship entities to storage using its bulk route.