Goresym analyzer, fixes#1451 and fixes executable file support

[g4ze]

closes #1451

Description

Please include a summary of the change and link to the related issue.

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue).
• New feature (non-breaking change which adds functionality).
• Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist

• I have read and understood the rules about how to Contribute to this project
• The pull request is for the branch develop
• A new plugin (analyzer, connector, visualizer, playbook, pivot or ingestor) was added or changed, in which case:
  • I strictly followed the documentation "How to create a Plugin"
  • Usage file was updated.
  • Advanced-Usage was updated (in case the plugin provides additional optional configuration).
  • I have dumped the configuration from Django Admin using the dumpplugin command and added it in the project as a data migration. ("How to share a plugin with the community")
  • If a File analyzer was added and it supports a mimetype which is not already supported, you added a sample of that type inside the archive test_files.zip and you added the default tests for that mimetype in test_classes.py.
  • If you created a new analyzer and it is free (does not require any API key), please add it in the FREE_TO_USE_ANALYZERS playbook by following this guide.
  • Check if it could make sense to add that analyzer/connector to other freely available playbooks.
  • I have provided the resulting raw JSON of a finished analysis and a screenshot of the results.
  • If the plugin interacts with an external service, I have created an attribute called precisely url that contains this information. This is required for Health Checks.
  • If the plugin requires mocked testing, _monkeypatch() was used in its class to apply the necessary decorators.
  • I have added that raw JSON sample to the MockUpResponse of the _monkeypatch() method. This serves us to provide a valid sample for testing.
• If external libraries/packages with restrictive licenses were used, they were added in the Legal Notice section.
• Linters (Black, Flake, Isort) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.
• I have added tests for the feature/bug I solved (see tests folder). All the tests (new and old ones) gave 0 errors.
• If changes were made to an existing model/serializer/view, the docs were updated and regenerated (check CONTRIBUTE.md).
• If the GUI has been modified:
  • I have a provided a screenshot of the result in the PR.
  • I have created new frontend tests for the new component or updated existing ones.
• After you had submitted the PR, if DeepSource, Django Doctors or other third-party linters have triggered any alerts during the CI checks, I have solved those alerts.

Important Rules

• If you miss to compile the Checklist properly, your PR won't be reviewed by the maintainers.
• Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review. After being reviewed and received a "change request", you should explicitly ask for a review again once you have made the requested changes.

[g4ze]

The analyzer works fine when i configure it to support all files. when i configure it to only support binaries and exe , it says incompatible mimetype. Moreover, the FE shows that the file type is application/x-executable , which doesn't seem to be supported anywhere in models.

[g4ze]

@mlodic any suggestions on what files should the analyzer support?

[mlodic]

can you share the file that you used as a test?

how did you get the value application/x-executable? cause that's deprecated and if you use IntelOwl it should classify it as application/vnd.microsoft.portable-executable. On the contrary, if you run it locally, you could have an engine that tells you application/x-executable which is the old one

[g4ze]

main.zip
the file i used is zipped here...

[mlodic]

ok I tried that tool and I think you should use application/vnd.microsoft.portable-executable for that. Can you try with that and lmk?

[g4ze]

I used application/vnd.microsoft.portable-executable only initially but frontend said incompatible mimetype. Does it work done in your local with accepted mime type as application/vnd.microsoft.portable-executable?
If yes then I'll push a commit with that setting only.

[g4ze]

here's a recording of what is exactly happening, in the first run as it can be seen, analyzer is set to accept all files. In the second run it is set to exe only. Moreover, after the first run, file type is shown as application/x-executable.
2024-06-28 21-18-10.zip

[mlodic]

I think you found a bug. When we did this change, we wanted to use the correct mimetype value cause the x- based one are not definitive and the new python library from python 3.11 started to use application/vnd.microsoft.portable-executable instead of application/x-dosexec. But it seems it is not the same for ELF executables and we need to revert this as soon as possible for ELF files, otherwise they are not supported.

Would you like to try to help us in solving this too?
If you check the old migrations, you can find all the analyzers where the x-executable has been removed. We should re-add it.
Plus, we should re-add it as supported mimetype (there are 2 sections in the code where this is done, 1 backend and 1 frontend).
Then, you can use this mimetype for this analyzer too.

[g4ze]

sure, i'll take a look. I did have a look at it before implementing the analyzer, when i was thinking for a way to check if the file was go-compiled as soon as we upload. It seems like the tool we are using is from magic import from_buffer as magic_from_buffer and it stills gives application/x-executable as an output for the go binary.
Here's a code snippet that serves the same implementation and lets us have an insight on how the tool reacts:

from magic import from_buffer as magic_from_buffer

files = ['/home/nilay/Documents/GitHub/byoc/main', "/home/nilay/Downloads/file.exe", "/home/nilay/Downloads/ping.elf"]

for file in files:
    try:
        with open(file, 'rb') as f:
            file_type = magic_from_buffer(f.read(2048), mime=True)
        print(f"{file}: {file_type}")
    except IOError:
        print(f"{file}: File not found or unreadable")

o/p:

/home/nilay/Documents/GitHub/byoc/main: application/x-executable
/home/nilay/Downloads/file.exe: application/vnd.microsoft.portable-executable
/home/nilay/Downloads/ping.elf: application/x-sharedlib

We have made implementations for SHARED_LIB and EXE but not anything like a go-compiled binary. Or anything which stills fals in the category of x-executable

[g4ze]

Plus, we should re-add it as supported mimetype (there are 2 sections in the code where this is done, 1 backend and 1 frontend).

you mean somethings like the recent commit i made.

[g4ze]

This works as expected on my machine.

• changed the migration
  • here I removed the part where x-executable was being filtered and removed
  • appended new file type X-Exe in choices for supported and non supported files
• appended the file type in another migration involving file choices
• updated model and job const [fe and be updates for reflection of file selctions]

[g4ze]

This works as expected on my machine.

i just realised that this fix is very naive and wont help the current users of intelowl who have already migrated their databases in the recent updates.
in the migration where we have removed x-executable, we have replaced two mimetypes with a single one.
application/x-dosexec and application/x-executable are replaced with newer application/vnd.microsoft.portable-executable now the problem is that we want to bring back application/x-executable as a supported type and add it to the old analyzers which supported it. The choke point is that we need to decide which all were the analyzers which supported application/x-executable as one of their mimetypes. Currently, if an analyzer supports application/vnd.microsoft.portable-executable , it is hard to decipher what exact mimetype it used to support previously in our two possible choices.
In my understanding, it leaves us two options:

• Filter all the analyzers that support application/vnd.microsoft.portable-executable and blindly add application/x-executable as one of the supported mimetype.
  OR
• Migrate back, run a query to find all the analyzers that supported application/x-executable in that time, use the resultant list to migrate and update all the specific analyzers.

Do you have any suggestions or add-ons for this? Please correct me if I'm wrong with how I understood the problem.

[g4ze]

I implemented the second option for the time being, here's what the quries returned:

intel_owl_db=# SELECT name
intel_owl_db-# FROM analyzers_manager_analyzerconfig
intel_owl_db-# WHERE 'application/x-executable' = ANY(supported_filetypes);
     name      
---------------
 Capa_Info
 ELF_Info
 Malpedia_Scan
 Intezer_Scan
 Qiling_Linux
(5 rows)

intel_owl_db=# SELECT name
FROM analyzers_manager_analyzerconfig
WHERE 'application/x-executable' = ANY(not_supported_filetypes);
 name 
------
(0 rows)

Migrated the database based on the above o/p

[mlodic]

yep you got everything right. and yes, those are the analyzers that were using that mimetype. You could also have found them just by looking for the string application/x-executable in the old migrations

[mlodic]

here you can also support the new mimetype application/vnd.microsoft.portable-executable so windows executables are correctly analyzed

[mlodic]

maybe I was mistaked, my intention was to add a single bool parameter for each option, and explain the options for each parameter. In that way, the user does not have to go through that tool documentation to understand how to customize it. Does it make more sense now?

[g4ze]

yes, makes sense

[mlodic]

can you provide a sample result with a file that is not go-compiled?

[g4ze]

GoReSym /home/nilay/Downloads/file.exe
{"error": "Failed to parse file: failed to read pclntab: failed to locate pclntab"}

[g4ze]

{'error': 'Failed to parse file: failed to read pclntab: failed to locate pclntab'}
for
file.zip

[mlodic]

if the execution time is fast enough (can you make some tests?) we could add this in the default playbook of Static File Analyzers

[g4ze]

done

[g4ze]

and yes, execution is pretty fast.

[g4ze]

the test fails as application/vnd.microsoft.portable-executable isn't necessarily compatible with the analyzer and our test file is surely not.

[mlodic]

ah ok so I think we should modify the analyzer to avoid that the error that you mentioned {'error': 'Failed to parse file: failed to read pclntab: failed to locate pclntab'} does make the analyzer fail. The result would be "success" and the report would be "this is not a Go-compiled file. Full error...". And we could add a warning log about that.

For all of the other errors triggered by the tool, it would fail the analyzer normally.
In that way, the analyzer should never fail for any portable executables.
Actually there's no better way to handle this and to avoid that this analyzer fails everytime a PE is sent to the platform, which is very common use case

[g4ze]