SignatureMap error

[cleesmith]

From the docs:
http://idstools.readthedocs.org/en/latest/maps.html
and
http://blog.jasonish.org/2013/07/easy-unified2-file-reading-in-python.html
... figured out that MsgMap should be SignatureMap, but running this:

from idstools import maps
sigmap = maps.SignatureMap()
sigmap.load_generator_map(open("/etc/snort/gen-msg.map"))
sigmap.load_signature_map(open("/etc/snort/community-sid-msg.map"))
print(sigmap.get(1, 2495))
print(sigmap.get(1, 1))

gives the following error:

Traceback (most recent call last):
  File "resolve_event_msg_and_class_name.py", line 6, in <module>
    sigmap.load_signature_map(open("/etc/snort/community-sid-msg.map"))
  File "/usr/local/lib/python2.7/dist-packages/idstools/maps.py", line 136, in load_signature_map
    "sid": int(parts[0]),
ValueError: invalid literal for int() with base 10: ''

I tried lots of gids/sids but no luck ... suggestions ?
Otherwise, I like these tools as it's hard to find snort related stuff ... thanks for this :)

[jasonish]

Any chance you can share your gen-msg.map and community-sid-msg.map? I think I see the issue, but just want to be sure.

[cleesmith]

The *.map files are the default ones after a fresh apt-get install snort with no updates (pulled pork).
These are the gists:
https://gist.github.com/cleesmith/f8ffce5e444983d98241
https://gist.github.com/cleesmith/b0f261ac8abd8af32287

Thanks for idstools as I am trying to use it to feed events directly into elasticsearch and bypass
the current very slow process of reading them out of mysql (barnyard2) and indexing into
elasticsearch.

I also had some errors reading old unified2 files, but I'll have to get back to you when I can find
those files again.

[jasonish]

Fixed in git master. It was failing while loading the community-sid-msg.map file. Now you should be able to print(sigmap.get(1, 100000934)) and see some proper output.

[cleesmith]

Thanks :) ... so I need to clone, or just edit maps.py, and wait until later for a pip install ?

[jasonish]

I tossed a 0.4.2 up on PyPi, so you should be able to get it with pip now.

[cleesmith]

Thanks it works now ... one step closer to indexing in elasticsearch.

[jasonish]

Ahh cool. I've been working on the same thing. py-idstools has a bit of a oinkmaster/pulledpork clone in it, and indexing rules into elastic search is something I've been working on to make rules available to my own EveBox tool.

[cleesmith]

I have several projects (ruby/rails) that offer a UI for snort events both in mysql
and elasticsearch. It's just very slow to index into elasticsearch via the 'ancient' snort mysql schema/tables using a trigger, so a more direct way is needed to bypass mysql (or any barnyard2 db). I'm hoping that idstools is fast enough to help with indexing.