Adversary Import fails with Code 500

[rubentroncon]

I can import reports and but importing adversaries fails everytime:

[2025-01-14 16:39:17,544] INFO    processor/main       _____ _______  _____   _____   ______ _______
[2025-01-14 16:39:17,544] INFO    processor/main         |   |  |  | |_____] |     | |_____/    |
[2025-01-14 16:39:17,544] INFO    processor/main       __|__ |  |  | |       |_____| |    \_    |
[2025-01-14 16:39:17,544] INFO    processor/main       
[2025-01-14 16:39:17,544] INFO    processor/main       
[2025-01-14 16:39:17,544] INFO    processor/main         ____  ___    __ __    ___  ____    _____  ____  ____   ____    ___  _____
[2025-01-14 16:39:17,544] INFO    processor/main        /    T|   \  |  T  |  /  _]|    \  / ___/ /    T|    \ l    j  /  _]/ ___/
[2025-01-14 16:39:17,544] INFO    processor/main       Y  o  ||    \ |  |  | /  [_ |  D  )(   \_ Y  o  ||  D  ) |  T  /  [_(   \_
[2025-01-14 16:39:17,544] INFO    processor/main       |     ||  D  Y|  |  |Y    _]|    /  \__  T|     ||    /  |  | Y    _]\__  T
[2025-01-14 16:39:17,544] INFO    processor/main       |  _  ||     |l  :  !|   [_ |    \  /  \ ||  _  ||    \  |  | |   [_ /  \ |
[2025-01-14 16:39:17,544] INFO    processor/main       |  |  ||     | \   / |     T|  .  Y \    ||  |  ||  .  Y j  l |     T\    |
[2025-01-14 16:39:17,544] INFO    processor/main       l__j__jl_____j  \_/  l_____jl__j\_j  \___jl__j__jl__j\_j|____jl_____j \___j
[2025-01-14 16:39:17,544] INFO    processor/main       
[2025-01-14 16:39:17,544] INFO    processor/main       Start Threat Actor galaxy cluster alignment
[2025-01-14 16:39:17,545] INFO    processor/main       Retrieving all adversaries.
[2025-01-14 16:39:18,057] INFO    processor/main       Got 257 adversaries from the Crowdstrike Intel API.
[2025-01-14 16:39:18,451] INFO    processor/main       Retrieving all adversaries.
Traceback (most recent call last):
  File "MISP-tools/misp_import.py", line 505, in <module>
    main()
  File "/MISP-tools/misp_import.py", line 497, in main
    import_handler.build()
  File "/MISP-tools/misp_import.py", line 401, in build
    self.import_new_events()
  File "/MISP-tools/misp_import.py", line 387, in import_new_events
    self.importer.import_from_crowdstrike(
  File "/MISP-tools/cs_misp_import/importer.py", line 342, in import_from_crowdstrike
    self.actors_importer.process_actors(actors_days_before, self.event_ids)
  File "/MISP-tools/cs_misp_import/actors.py", line 184, in process_actors
    cluster_result = self.misp.add_galaxy_cluster(get_threat_actor_galaxy_id(self.misp), cluster)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/misp-modules/lib/python3.11/site-packages/pymisp/api.py", line 1825, in add_galaxy_cluster
    cluster_j = self._check_json_response(r)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/misp-modules/lib/python3.11/site-packages/pymisp/api.py", line 3978, in _check_json_response
    r = self._check_response(response, expect_json=True)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/MISP-tools/cs_misp_import/misp_safe_check_response.py", line 55, in safe_check_response
    raise MISPServerError(fail_msg)
pymisp.exceptions.MISPServerError: Error code 500: SQLSTATE[01000]: Warning: 1265 Data truncated for column &#039;galaxy_id&#039; at row 1

In the MISP error.log I see:

2025-01-14 15:34:03 Error: [PDOException] SQLSTATE[01000]: Warning: 1265 Data truncated for column 'galaxy_id' at row 1
Request URL: /galaxy_clusters/add/698774c7-8022-42c4-917f-8d6e4f06ada3
Stack Trace:
#0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(502): PDOStatement->execute()
#1 /var/www/MISP/app/Model/Datasource/Database/MysqlObserverExtended.php(162): DboSource->_execute()
#2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(1132): MysqlObserverExtended->execute()
#3 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1943): DboSource->create()
#4 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1761): Model->_doSave()
#5 /var/www/MISP/app/Model/GalaxyCluster.php(342): Model->save()
#6 /var/www/MISP/app/Controller/GalaxyClustersController.php(321): GalaxyCluster->saveCluster()
#7 [internal function]: GalaxyClustersController->add()
#8 /var/www/MISP/app/Lib/cakephp/lib/Cake/Controller/Controller.php(500): ReflectionMethod->invokeArgs()
#9 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(193): Controller->invokeAction()
#10 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#11 /var/www/MISP/app/webroot/index.php(107): Dispatcher->dispatch()
#12 {main}

[DeZuko]

I also got similar kind of error but when importing indicators:

[2025-01-15 15:21:26,298] INFO     config  _______ _     _ _______ _______ _     _      _______  _____  __   _ _______ _____  ______
[2025-01-15 15:21:26,298] INFO     config  |       |_____| |______ |       |____/       |       |     | | \  | |______   |   |  ____
[2025-01-15 15:21:26,298] INFO     config  |_____  |     | |______ |_____  |    \_      |_____  |_____| |  \_| |       __|__ |_____|
[2025-01-15 15:21:26,298] INFO     config
[2025-01-15 15:21:26,299] WARNING  config  misp_enable_ssl                             SSL is disabled for MISP API requests
[2025-01-15 15:21:27,024] INFO     config  No configuration errors found (1 warning)
[2025-01-15 15:21:27,024] INFO     config
[2025-01-15 15:21:27,024] INFO     config  ____ _  _ ____ ____ _  _ ____    ___  ____ ____ ____ ____ ___
[2025-01-15 15:21:27,024] INFO     config  |    |__| |___ |    |_/  [__     |__] |__| [__  [__  |___ |  \
[2025-01-15 15:21:27,024] INFO     config  |___ |  | |___ |___ | \_ ___]    |    |  | ___] ___] |___ |__/
[2025-01-15 15:21:27,024] INFO     config
/home/misp/csenv/lib/python3.10/site-packages/pymisp/__init__.py:67: FutureWarning: This class is deprecated, use PyMISP instead
  warnings.warn('This class is deprecated, use PyMISP instead', FutureWarning)
[2025-01-15 15:21:28,066] INFO    processor/main       Retrieving all galaxy cluster values for the Android cluster.
Traceback (most recent call last):
  File "/home/misp/MISP-tools/misp_import.py", line 505, in <module>
    main()
  File "/home/misp/MISP-tools/misp_import.py", line 495, in main
    import_handler = ImportHandler(config, intel_api_client,
  File "/home/misp/MISP-tools/misp_import.py", line 312, in __init__
    self.importer = CrowdstrikeToMISPImporter(
  File "/home/misp/MISP-tools/cs_misp_import/importer.py", line 76, in __init__
    self.all_galaxies = self.get_galaxies()
  File "/home/misp/MISP-tools/cs_misp_import/importer.py", line 270, in get_galaxies
    all_galaxies.append(self.misp_client.search_galaxy_clusters(gal["id"], searchall=""))
  File "/home/misp/csenv/lib/python3.10/site-packages/pymisp/api.py", line 1781, in search_galaxy_clusters
    clusters_j = self._check_json_response(r)
  File "/home/misp/csenv/lib/python3.10/site-packages/pymisp/api.py", line 3978, in _check_json_response
    r = self._check_response(response, expect_json=True)
  File "/home/misp/MISP-tools/cs_misp_import/misp_safe_check_response.py", line 55, in safe_check_response
    raise MISPServerError(fail_msg)
pymisp.exceptions.MISPServerError: Error code 500: An Internal Error Has Occurred.

My MISP error.log shows:

2025-01-15 15:21:28 Error: [PDOException] SQLSTATE[42S22]: Column not found: 1054 Unknown column 'Galaxy.default' in 'where clause'
Request URL: /galaxy_clusters/index/3
Stack Trace:
#0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(502): PDOStatement->execute()
#1 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(468): DboSource->_execute()
#2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(715): DboSource->execute()
#3 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(1226): DboSource->fetchAll()
#4 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(3053): DboSource->read()
#5 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(3025): Model->_readDataSource()
#6 /var/www/MISP/app/Model/AppModel.php(4359): Model->find()
#7 /var/www/MISP/app/Controller/GalaxyClustersController.php(105): AppModel->find()
#8 [internal function]: GalaxyClustersController->index()
#9 /var/www/MISP/app/Lib/cakephp/lib/Cake/Controller/Controller.php(499): ReflectionMethod->invokeArgs()
#10 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(193): Controller->invokeAction()
#11 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#12 /var/www/MISP/app/webroot/index.php(105): Dispatcher->dispatch()
#13 {main}

I realize this happen after I upgraded my MISP version to 2.4.200

[rubentroncon]

I also got similar kind of error but when importing indicators:

I realize this happen after I upgraded my MISP version to 2.4.200

Actually, your specific issue is also reported in the MISP repository: MISP/MISP#10062
I don't know how hard it is to patch the SQL query yourself, but good luck :-)

[rubentroncon]

Fixed my issue by editing line 169 in cs_misp_import/helper.py and setting ta_galaxy_id = gal["Galaxy"]["uuid"] to ta_galaxy_id = gal["Galaxy"]["id"]

The error suggested that the Galaxy ID used to create the GalaxyCluster was too long.

Taking a look at the galaxy_clusters table, a Galaxy ID was max. 11 integers, but when debugging I saw a UUID being used to add the GalaxyCluster.

Created Pull request 192.

[DocArmoryTech]

As described in the following MISP PR, in v2.4.197 to v2.4.204 of MISP, API calls to add a GalaxyCluster to a Galaxy using a Galaxy's uuid would fail (MISP/MISP#10221).

The cs_misp_import/actors.py creates GalaxyClusters to represent Actors/Adversaries and adds those clusters to the Threat Actors Galaxy in MISP using a uuid as a reference to the Galaxy.

Fixed/bypassed by PR: #192