From Indicators to Insights: Automating IOC Enrichment Using Python and Threat Feeds

https://dzone.com/articles/everything-you-should-know-about-apisSecurity operations frequently revolve around indicators of compromise (IOCs): technical artifacts or observables that suggest an attack is imminent, underway, or that a compromise may have already occurred. NIST defines an indicator of compromise in these terms, and the IETF highlights that IOCs have lifecycles and operational limitations that affect how reliably they can be used for defense. The practical implication is that enrichment must be repeatable, time-aware, and automation-friendly rather than an ad hoc sequence of manual lookups. 

When an IOC Becomes Actionable

An IOC becomes actionable when it is expressed as an assertion that can be tested, time-bounded, and traced back to evidence. STIX 2.1 frames an Indicator around a required pattern plus an explicit validity window: valid_from is required and valid_until can state when an indicator should no longer be considered valid. STIX also permits the pattern to be expressed as a STIX Pattern or another detection language, such as SNORT or YARA, which helps align enrichment outputs with the detection engines that will consume them. 

Insight emerges when enrichment yields relationships rather than disconnected attributes. STIX defines an “indicates” relationship from an Indicator to entities such as malware, intrusion sets, threat actors, and tools, and it defines a “based-on” relationship linking an Indicator to observed data. This pairing supports a provenance-aware pipeline: the observation that produced the IOC is retained, the indicator assertion is versioned with time bounds, and the linkage preserves the “why” behind the verdict. 

Normalizing IOCs for Reliable Lookups

Enrichment quality is capped by normalization quality because provider APIs are strict about identifier shape. VirusTotal states that a file object’s identifier is its SHA-256 hash, which means treating MD5, SHA-1, and SHA-256 as interchangeable will cause systematic false negatives. IP enrichment has protocol edge cases as well: AbuseIPDB documents that its check endpoint accepts one IPv4 or IPv6 address and notes that IPv6 should be URL-encoded due to colon semantics in URIs. 

A practical contract between parsing and enrichment is a small typed record that captures a conservative classification and a canonical value. This record becomes both the routing key for source selection and the cache key for deduplication, reducing repeated calls and preventing representational drift such as uppercased hashes, defanged URLs, or domains with trailing dots. The normalization logic below stays intentionally narrow and returns unknown when classification is not stable, which reduces silent misroutes that look like “not found” outcomes downstream. 

def normalize_ioc(raw: str) -> dict | None:
    s = (raw or "").strip()
    if not s:
        return None

    if s.startswith(("hxxp://", "hxxps://")):
        s = s.replace("hxxp://", "http://").replace("hxxps://", "https://")

    try:
        ip = ipaddress.ip_address(s)
        return {"type": "ip", "value": ip.compressed}
    except ValueError:
        pass

    if re.fullmatch(r"[A-Fa-f0-9]{64}", s):
        return {"type": "sha256", "value": s.lower()}
    if re.fullmatch(r"[A-Fa-f0-9]{40}", s):
        return {"type": "sha1", "value": s.lower()}
    if re.fullmatch(r"[A-Fa-f0-9]{32}", s):
        return {"type": "md5", "value": s.lower()}

    parsed = urllib.parse.urlsplit(s)
    if parsed.scheme in ("http", "https") and parsed.netloc:
        return {"type": "url", "value": parsed._replace(fragment="").geturl()}

    candidate = s.strip(".").lower()
    if re.fullmatch(r"(?!-)[a-z0-9-]{1,63}(?<!-)(\.(?!-)[a-z0-9-]{1,63}(?<!-))*", candidate):
        return {"type": "domain", "value": candidate}

    return {"type": "unknown", "value": s}

Once type is explicit, connectors can route to the correct endpoint without scattered special-casing. VirusTotal documents distinct artifact endpoints such as /api/v3/ip_addresses/{ip}, /api/v3/domains/{domain}, /api/v3/files/{id}, and /api/v3/urls/{id}, and notes that URL enrichment uses a provider-specific identifier for /api/v3/urls/{id} rather than an unmodified URL string. 

Pulling Context From Threat Feeds Without Throttling Everything

Enrichment sources generally split into internal intelligence platforms and external services. Internal-first querying often reduces cost and avoids unnecessary disclosure of investigative interest when a local corpus already contains the needed verdict or related pivots. MISP is a common internal backbone; its automation documentation describes /attributes/restSearch and /events/restSearch, uses an API key via the Authorization header, and negotiates JSON through Accept and Content-Type, which makes it suitable as a first-layer enrichment and caching target. 

async def misp_search_value(client, base_url, api_key, value: str, types: list[str] | None = None):
    body = {"returnFormat": "json", "value": value}
    if types:
        body["type"] = {"OR": types}

    r = await client.post(
        f"{base_url.rstrip('/')}/attributes/restSearch",
        headers={
            "Authorization": api_key,
            "Accept": "application/json",
            "Content-Type": "application/json",
        },
        json=body,
        timeout=10.0,
    )
    r.raise_for_status()
    return r.json()

Handling constraints should travel with content rather than being bolted on during export. FIRST defines the Traffic Light Protocol with four labels—TLP:RED, TLP:AMBER, TLP:GREEN, and TLP:CLEAR—to communicate sharing boundaries and explicitly positions TLP as sharing guidance rather than a formal classification scheme. MISP’s automation examples demonstrate tag-based filtering patterns such as excluding tlp:red, which maps directly to automated enforcement of distribution constraints during enrichment and export. 

Standards-based feed ingestion reduces connector sprawl where shared feeds are involved. The TAXII 2.1 specification shows version negotiation via Accept: application/taxii+json;version=2.1, and supports incremental polling via URL filtering parameters such as added_after, limit, and next. The collections response can advertise supported media types, commonly including STIX content types such as application/stix+json;version=2.1, which helps constrain downstream parsing and validation. 

Converging Responses Into a Coherent Enrichment Record

External services expand coverage and provide complementary evidence. VirusTotal API v3 authenticates with an x-apikey header and emphasizes richer context in v3, including IoC relationships and sandbox-related analysis details that support pivots beyond a single indicator. AbuseIPDB’s check endpoint provides a non-binary abuseConfidenceScore and can bound report recency via maxAgeInDays. AlienVault OTX documents per-indicator endpoints with “sections” such as reputation, malware, passive DNS, and URL lists, which is well-suited to evidence-centric enrichment rather than a single boolean verdict. 

A stable enrichment product is not a giant merged schema; it is a converged record that remains small enough for automation while retaining the evidence needed for audit and escalation. STIX reinforces this by listing handling- and attribution-oriented common properties such as created_by_ref and object markings alongside indicator-specific fields such as pattern, pattern_type, and validity bounds. The record below keeps a derived score and compact labels for routing, while preserving source-level evidence and raw payload references for later drill-down. 

def build_enrichment_record(ioc: dict, evidence: list[dict]) -> dict:
    score = max((e.get("score", 0) for e in evidence), default=0)
    labels = sorted({l for e in evidence for l in e.get("labels", [])})

    return {
        "ioc": ioc,
        "score": int(score),
        "labels": labels,
        "evidence": [
            {"source": e["source"], "retrieved_at": e["retrieved_at"], "summary": e.get("summary", "")}
            for e in evidence
        ],
        "raw": [e.get("raw") for e in evidence if e.get("raw") is not None],
    }

Reliability engineering determines whether this record is produced consistently. External APIs commonly signal quota pressure with 429 Too Many Requests; RFC 6585 notes that responses may include Retry-After to indicate how long to wait before retrying. VirusTotal explicitly documents 429-class quota and “too many requests” errors, and AbuseIPDB documents 429 responses when request limits are exceeded. Treating these as control signals, coupled with caching keyed on normalized IOC identity and bounded concurrency per provider, prevents enrichment pipelines from oscillating between bursts, throttling, and gaps in coverage. 

Conclusion

From “indicator” to “insight” is fundamentally a modeling and engineering shift: indicators must be normalized and typed, enrichment must be time-aware and provenance-preserving, and outputs must converge into a consistent record that supports automated decisions without discarding evidence. STIX supplies a concrete structure for this outcome through validity-scoped Indicators and explicit “based-on” and “indicates” relationships, while TAXII offers a standard path for consuming shared threat feeds with negotiated media types and incremental polling. An enrichment pipeline built on these principles turns artifact-centric detection outputs into repeatable, evidence-backed insight suitable for both automation and escalation.