Failed to run Apache HTTPD importer

[TG1999]

Traceback (most recent call last):
  File "/home/tg1999/Desktop/vulnerablecode/vulnerabilities/management/commands/import.py", line 60, in import_data
    ImportRunner(importer).run()
  File "/home/tg1999/Desktop/vulnerablecode/vulnerabilities/import_runner.py", line 44, in run
    count = process_advisories(advisory_datas=advisory_datas, importer_name=importer_name)
  File "/home/tg1999/Desktop/vulnerablecode/vulnerabilities/import_runner.py", line 54, in process_advisories
    for data in advisory_datas:
  File "/home/tg1999/Desktop/vulnerablecode/vulnerabilities/importers/apache_httpd.py", line 40, in advisory_data
    yield self.to_advisory(data)
  File "/home/tg1999/Desktop/vulnerablecode/vulnerabilities/importers/apache_httpd.py", line 43, in to_advisory
    alias = data["CVE_data_meta"]["ID"]
KeyError: 'CVE_data_meta'

[johnmhoran]

Exploring the code and data only briefly, I wonder whether Apache might have changed the field names in their JSON files.

I don't have the version in effect when I created the code, but looking at, e.g., https://httpd.apache.org/security/json/CVE-2023-27522.json, I see the field names for CVE-2023-27522 are cveMetadata and cveId, while the current code reference is data["CVE_data_meta"]["ID"] and the error is KeyError: 'CVE_data_meta'.

Perhaps data["CVE_data_meta"]["ID"] needs updating to data["cveMetadata"]["cveId"]?

[johnmhoran]

Looking at issue #1006 -- I created this in November 2022, and at the time the relevant JSON structure was

{
  "data_type": "CVE",
  "data_format": "MITRE",
  "data_version": "4.0",
  "generator": {
    "engine": "Vulnogram 0.0.9"
  },
  "CVE_data_meta": {
    "ID": "CVE-2022-31813",
    "ASSIGNER": "security@apache.org",
    "DATE_PUBLIC": "",
    "TITLE": "mod_proxy X-Forwarded-For dropped by hop-by-hop mechanism",
    "AKA": "",
    "STATE": "REVIEW"
  },
  
  . . .

and in the example I cited above of the current structure (I don't know if it's representative but presumably it is), the related structure is

. . .

  "cveMetadata": {
    "assignerOrgId": "f0158376-9dc2-43b6-827c-5f631a4d8d09",
    "cveId": "CVE-2023-27522",
    "serial": 1,
    "state": "PUBLISHED"
  },
  "dataType": "CVE_RECORD",
  "dataVersion": "5.0"
}

[johnmhoran]

Actually, randomly choosing an earlier JSON CVE file, I see that it still uses the earlier JSON structure, so it seems they have not normalized the JSON across all their advisories. See https://httpd.apache.org/security/json/CVE-2022-22719.json:

{
  "CVE_data_meta": {
    "ASSIGNER": "security@apache.org",
    "ID": "CVE-2022-22719",
    "STATE": "REVIEW",
    "TITLE": "mod_lua Use of uninitialized value of in r:parsebody"
  },
  "affects": {
    "vendor": {
      "vendor_data": [

. . .