Type error: "a bytes-like object is required, not 'str'" #7

faustus25 · 2019-02-12T13:24:33Z

Type error running Pysight.py under Python3:

Id": null, "observationTime": 1549856820}]}':
Unexpected error: %s (<class 'TypeError'>, TypeError("a bytes-like object is required, not 'str'",), <traceback object at 0x7ff411a33ac8>)
ERROR:PySight_settings:Something went wrong while downloading / processing the iSight Request
Traceback (most recent call last):
  File "PySight.py", line 807, in <module>
    misp_process_isight_alert(result)
  File "PySight.py", line 778, in misp_process_isight_alert
    for i in a_result['message']:
TypeError: 'bool' object is not subscriptable

Line 778-791:

for i in a_result['message']:
        PySight_settings.logger.debug("  %s current element %s", len(a_result['message']), i)

        # USING THREADS to proceed with the resulting JSON
        if PySight_settings.use_threading:
            t = threading.Thread(target=isight_process_alert_content_element, args=(i,))
            t.start()
        else:
            # NO THREADING

            isight_process_alert_content_element(i)
            PySight_settings.logger.debug("Sleeping for %s seconds", PySight_settings.time_sleep)
            time.sleep(PySight_settings.time_sleep)
    end = timer()

Line 807:

misp_process_isight_alert(result)

Can you check if you encounter this issue?

The text was updated successfully, but these errors were encountered:

faustus25 · 2019-02-12T13:37:36Z

Some of the code comments point to this:

Line 142:

    new_data = a_query + '2.4' + 'application/json' + time_stamp
    # new_data=''
    # TODO: that is currently broken! TypeError: string argument without an encoding
    message = bytes(new_data, 'utf-8')
    secret = bytes(a_prv, 'utf-8')

github-germ · 2019-02-12T13:42:21Z

Hi, I have a local copy that I've mashed up and get_headers looks like this:

def get_headers(a_prv, a_pub, a_query):
    '''
    :param a_prv:
    :type a_prv:
    :param a_pub:
    :type a_pub:
    :param a_query:
    :type a_query:
    :return: headers for iSight search
    :rtype:
    '''
    time_stamp = email.utils.formatdate(localtime=True)

    new_data = a_query + '2.5' + 'application/json' + time_stamp
    # new_data=''
    # TODO: that is currently broken! TypeError: string argument without an enc\
oding
    message = bytes(new_data, 'utf-8')
    secret  = bytes(a_prv, 'utf-8')

    # hashed = hmac.new(bytearray(a_prv,'utf8'), new_data, hashlib.sha256)
    hashed = hmac.new(secret, message, hashlib.sha256)
    headers = {
        'Accept':         'application/json',
        'Accept-Version': '2.5',
        'X-Auth':         a_pub,
        'X-Auth-Hash':    hashed.hexdigest(),
        'Date':           time_stamp,
    }
    return headers

faustus25 · 2019-02-12T15:30:00Z

Replacing the header function doesn't work either:

rId": null, "observationTime": 1549856820}]}':
Unexpected error: %s (<class 'TypeError'>, TypeError("a bytes-like object is required, not 'str'",), <traceback object at 0x7f54fefc9e48>)
ERROR:PySight_settings:Something went wrong while downloading / processing the iSight Request
Traceback (most recent call last):
  File "PySight.py", line 837, in <module>
    misp_process_isight_alert(result)
  File "PySight.py", line 808, in misp_process_isight_alert
    for i in a_result['message']:
TypeError: 'bool' object is not subscriptable

Does your modified script work? and if so under python3.5?

I set up under virtual environment for python 3.5 but the encoding/decoding issue remains.

Can you share the entire script if it works for you?

github-germ · 2019-02-12T17:06:49Z

i think this is where that exception occurs: https://github.com/deralexxx/PySight2MISP/blob/master/PySight.py#L210

try changing this:

210         return_data_cleaned = r.data.replace('\n', '')

to this:

210         #return_data_cleaned = r.data.replace('\n', '')
211         return_data_cleaned = r.data

faustus25 · 2019-02-12T17:50:52Z

No luck there either:

        #return_data_cleaned = r.data.replace('\n', '')
        return_data_cleaned = r.data
        # return_data_cleaned =

New error for object type:

Unexpected error: %s (<class 'FileNotFoundError'>, FileNotFoundError(2, 'No such file or directory'), <traceback object at 0x7fccf61eb348>)
ERROR:PySight_settings:Something went wrong while downloading / processing the iSight Request
Traceback (most recent call last):
  File "PySight.py", line 838, in <module>
    misp_process_isight_alert(result)
  File "PySight.py", line 809, in misp_process_isight_alert
    for i in a_result['message']:
TypeError: 'bool' object is not subscriptable

github-germ · 2019-02-12T18:38:29Z

i made only that change to that github version, and it works for me. can you run the python debugger on this portion of the code and walk it through and see what's happening?

if that part of the code fails, it can return a False bool -- i suspect you are hitting one of those exception sections.

   208 	       PySight_settings.logger.debug("data %s: ", r.data)
   209
   210	       #commentout#return_data_cleaned = r.data.replace('\n', '')
   211         return_data_cleaned = r.data #addedin#
   212         # return_data_cleaned =
   213
   214 	       json_return_data_cleaned = json.loads(return_data_cleaned.decode\
('utf8'))
   215         PySight_settings.logger.debug(json_return_data_cleaned)
   216
   217	       # print json.dumps(theJson,sort_keys=True,indent = 4, separators\
 = (',', ': '))
   218         PySight_settings.logger.debug("Number of iocs: %s answer is: %s"\
, len(json_return_data_cleaned['message']),
   219                                       json_return_data_cleaned)
   220
   221         if not json_return_data_cleaned['success']:
...

faustus25 · 2019-02-13T00:26:10Z

Bool object is the issue now as you mentioned.

Set the breakpoint for the block of code below:

 def isight_load_data(a_url, a_query, a_headers):
213         """
214
215         :param a_url:
216         :type a_url:
217         :param a_query:
218         :type a_query:
219         :param a_headers:
220         :type a_headers:
221         :return:
222         :rtype:
223         """
224         try:
225             PySight_settings.logger.debug("param headers: %s %s", a_headers, a_url)
226             proxy_request = ProxyManager(str(PySight_settings.proxy_adress))
227             url_to_load = PySight_settings.isight_url + a_query
228             PySight_settings.logger.debug(url_to_load)
229             try:
230
231                 r = proxy_request.request('GET', a_url + a_query, None, headers=a_headers)
232             except urllib.error.HTTPError as e:
233                 print(e.code)
234                 print(e.read())
235
236             PySight_settings.logger.debug("headers %s: ", proxy_request.headers)
237
238             PySight_settings.logger.debug("data %s: ", r.data)
239
240             #return_data_cleaned = r.data.replace('\n', '')
241             return_data_cleaned = r.data
242             # return_data_cleaned =
243
244
245             json_return_data_cleaned = json.loads(return_data_cleaned.decode('utf8'))
246             PySight_settings.logger.debug(json_return_data_cleaned)
247
248             # print json.dumps(theJson,sort_keys=True,indent = 4, separators = (',', ': '))
249             PySight_settings.logger.debug("Number of iocs: %s answer is: %s", len(json_return_data_cleaned['message']),
250                                           json_return_data_cleaned)
251
252             if not json_return_data_cleaned['success']:
253                 PySight_settings.logger.error("Error with iSight connection %s",
254                                               json_return_data_cleaned['message']['description'])
255                 PySight_settings.logger.error(json_return_data_cleaned)
256                 return False
257             else:
258  ->             import time
259                 timestring = time.strftime("%Y%m%d-%H%M%S")
260                 f = open("debug/" + timestring, 'w')
261                 f.write(json.dumps(json_return_data_cleaned, sort_keys=True, indent=6, separators=(',', ': ')))
262                 f.close()
263
264                 return json_return_data_cleaned
265         except:
266             print("Unexpected error: %s", sys.exc_info())
267             return False

Error below:

Unexpected error: %s (<class 'FileNotFoundError'>, FileNotFoundError(2, 'No such file or directory'), <traceback object at 0x7fabd9738d88>)
ERROR:PySight_settings:Something went wrong while downloading / processing the iSight Request
Traceback (most recent call last):
  File "/usr/lib/python3.5/pdb.py", line 1661, in main
    pdb._runscript(mainpyfile)
  File "/usr/lib/python3.5/pdb.py", line 1542, in _runscript
    self.run(statement)
  File "/usr/lib/python3.5/bdb.py", line 431, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/user/python3-virtual-environments/PySight2MISP/PySight.py", line 838, in <module>
    misp_process_isight_alert(result)
  File "/home/user/python3-virtual-environments/PySight2MISP/PySight.py", line 809, in misp_process_isight_alert
    for i in a_result['message']:
TypeError: 'bool' object is not subscriptable
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program

github-germ · 2019-02-13T02:48:04Z

OK, there it is...

Unexpected error: %s (<class 'FileNotFoundError'>, FileNotFoundError(2, 'No such file or directory'), <traceback object at 0x7fabd9738d88>)

f = open("debug/" + timestring, 'w')

You need to mkdir debug in the directory where you are running this script.

faustus25 · 2019-02-13T20:33:31Z

So it progressed further once the debug directory was added:

DEBUG:PySight_settings:1 Inappropriate argument type.
DEBUG:PySight_settings:2 (<class 'TypeError'>, TypeError('exceptions must derive from BaseException',), <traceback object at 0x7ff5303bca48>)
DEBUG:PySight_settings:3 <class 'TypeError'>
DEBUG:PySight_settings:4 exceptions must derive from BaseException
  File "PySight.py", line 285, in isight_process_alert_content_element
DEBUG:PySight_settings:6 None
libgcc_s.so.1 must be installed for pthread_cancel to work
Aborted

installed libgcc next:

sudo apt-get install libgcc1-dbg

looked like it was connecting and downloading the i sight content then it complained about not being able to connect to my MISP instance, swapped the IP address of the instance with the alias version and reviewed the proxy details (same as in MISP instance):

DEBUG:PySight_settings:1 Inappropriate argument type.
DEBUG:PySight_settings:1 Inappropriate argument type.
ERROR:PySight_settings:Unexpected error in MISP init: (<class 'pymisp.exceptions.PyMISPError'>, PyMISPError("Unable to connect to MISP (https://misp.local.com). Please make sure the API key and the URL are correct (http/https is required): HTTPSConnectionPool(host='misp.local.com', port=443): Max retries exceeded with url: /servers/getPyMISPVersion.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6340598588>: Failed to establish a new connection: [Errno 110] Connection timed out',))",), <traceback object at 0x7f6340592fc8>)
ERROR:PySight_settings:Unexpected error in MISP init: (<class 'pymisp.exceptions.PyMISPError'>, PyMISPError("Unable to connect to MISP (https://misp.local.com). Please make sure the API key and the URL are correct (http/https is required): HTTPSConnectionPool(host='misp.local.com', port=443): Max retries exceeded with url: /servers/getPyMISPVersion.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6340591438>: Failed to establish a new connection: [Errno 110] Connection timed out',))",), <traceback object at 0x7f6340588508>)
ERROR:PySight_settings:Unexpected error in MISP init: (<class 'pymisp.exceptions.PyMISPError'>, PyMISPError("Unable to connect to MISP (https://misp.local.com). Please make sure the API key and the URL are correct (http/https is required): HTTPSConnectionPool(host='misp.local.com', port=443): Max retries exceeded with url: /servers/getPyMISPVersion.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6340591cc0>: Failed to establish a new connection: [Errno 110] Connection timed out',))",), <traceback object at 0x7f634058f588>)
DEBUG:PySight_settings:2 (<class 'TypeError'>, TypeError('exceptions must derive from BaseException',), <traceback object at 0x7f63405b9b48>)
DEBUG:PySight_settings:2 (<class 'TypeError'>, TypeError('exceptions must derive from BaseException',), <traceback object at 0x7f634059dfc8>)
DEBUG:PySight_settings:1 Inappropriate argument type.
DEBUG:PySight_settings:1 Inappropriate argument type.
DEBUG:PySight_settings:1 Inappropriate argument type.
DEBUG:PySight_settings:3 <class 'TypeError'>
DEBUG:PySight_settings:3 <class 'TypeError'>
DEBUG:PySight_settings:2 (<class 'TypeError'>, TypeError('exceptions must derive from BaseException',), <traceback object at 0x7f6340592fc8>)
DEBUG:PySight_settings:2 (<class 'TypeError'>, TypeError('exceptions must derive from BaseException',), <traceback object at 0x7f636c0eb048>)
DEBUG:PySight_settings:3 <class 'TypeError'>
DEBUG:PySight_settings:4 exceptions must derive from BaseException
  File "PySight.py", line 285, in isight_process_alert_content_element
    raise "no MISP instance found"
DEBUG:PySight_settings:4 exceptions must derive from BaseException
  File "PySight.py", line 285, in isight_process_alert_content_element
    raise "no MISP instance found"
DEBUG:PySight_settings:6 None
DEBUG:PySight_settings:4 exceptions must derive from BaseException
  File "PySight.py", line 285, in isight_process_alert_content_element
    raise "no MISP instance found"
DEBUG:PySight_settings:3 <class 'TypeError'>
DEBUG:PySight_settings:6 None
DEBUG:PySight_settings:2 (<class 'TypeError'>, TypeError('exceptions must derive from BaseException',), <traceback object at 0x7f6340588508>)
DEBUG:PySight_settings:3 <class 'TypeError'>
DEBUG:PySight_settings:4 exceptions must derive from BaseException
  File "PySight.py", line 285, in isight_process_alert_content_element
    raise "no MISP instance found"
DEBUG:PySight_settings:4 exceptions must derive from BaseException
  File "PySight.py", line 285, in isight_process_alert_content_element
    raise "no MISP instance found"
DEBUG:PySight_settings:6 None
DEBUG:PySight_settings:6 None
DEBUG:PySight_settings:6 None

MISP URL and proxy set as follows: (switched to misp_url=https;//127.0.0.1)

misp_url=https://misp.local.com/
misp_key=key
misp_verifycert=False

port=8080
host=http.proxy.com
protocoll=http
full=http://http.proxy.com:8080

So close to getting it to run smoothly!

github-germ · 2019-02-14T13:55:24Z

The code is not connecting to your MISP server as below. Are you sure you have the keys set correctly in config.cfg under the [MISP] section?

ERROR:PySight_settings:Unexpected error in MISP init: (<class 'pymisp.exceptions.PyMISPError'>, PyMISPError("Unable to connect to MISP (https://misp.local.com). Please make sure the API key and the URL are correct (http/https is required): HTTPSConnectionPool(host='misp.local.com', port=443): Max retries exceeded with url: /servers/getPyMISPVersion.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6340598588>: Failed to establish a new connection: [Errno 110] Connection timed out',))",), <traceback object at 0x7f6340592fc8>)

And regarding this:

DEBUG:PySight_settings:1 Inappropriate argument type.

I would think this is is coming from one of the ConfigParser.get*() calls in Pysight_settings.py which is parsing the content in config.cfg -- can you check what you've put in config.cfg, and if not sure, can you paste that file in here?

faustus25 · 2019-02-14T14:51:33Z

Config.cfg

[MISP]
misp_url=https://misp.localhost.com/
misp_key=key
misp_verifycert=False


[isight]
isight_url=https://api.isightpartners.com
isight_priv_key=key
isight_pub_key=key
last_hours=48


[proxy]
port=8080
host=http.proxy.com
protocoll=https
full=https://http.proxy.com:8080

[general]
use_threading=True
time_sleep=5

github-germ · 2019-02-14T14:57:38Z

OK... You need real values in your config.cfg.

I doubt your misp_url is what you hav in there. Set it to the URL of your MISP host.
You need to set misp_key to your user Authkey found on https://localhost/users/view/me (assuming your MISP is on localhost).
You ned to set your isight_priv_key and isight_pub_key to the keys supplied by your contact at iSIGHT.
Are you using a proxy? If so, I would think host and full are not correct.

faustus25 · 2019-02-14T16:01:30Z

I have swapped those details naturally for the API keys for MISP and i Sight.

No quotes used in those fields,

[MISP]
misp_url=https://10.10.10.10/
misp_key=xxxrandomkeyxxx
misp_verifycert=False


[isight]
isight_url=https://api.isightpartners.com
isight_priv_key=xxxrandomkeyxxx
isight_pub_key=xxxrandomkeyxxx
last_hours=48


[proxy]
port=8080
host=http.proxy.com
protocoll=https
full=https://http.proxy.com:8080

[general]
use_threading=True
time_sleep=5

github-germ · 2019-02-14T16:32:07Z

HA, ok... Gotta see why you are getting those parser inappropriate args. Correct: no quotes in config.cfg. And then why is the connection to MISP failing? Can you write a quick Python script to try connecting to your MISP via PyMISP? If not, I can share some code for that.

faustus25 · 2019-02-14T21:11:15Z

Sure send on your script and I can test it.

How does your config.cfg look?

github-germ · 2019-02-14T22:40:21Z

mispConnectTest.py.gz

github-germ · 2019-02-14T22:41:52Z

My config was greatly enhanced along with my code. Get that test script to connect to MISP, and then carry the settings you edit into the script into your config.cfg and test PySight.py again.

jaegeral · 2019-02-15T11:15:40Z

hm strange issue, but I would be happy to get pull requests, I have to be honest that I have not used the script for a while.

faustus25 · 2019-02-15T11:25:53Z

Run script and it connected:

 echo "xxxMISPuserkeyxxx" | python ./mispConnectTest.py
{'version': '2.4.102', 'perm_sync': False}

Settings in script as follows:

    misp = PyMISP(
        url      = 'https://localhost/',
        key      = key,
        ssl      = False,
        out_type = 'json',
        debug    = None,
        proxies  = None,
        cert     = None,
        asynch   = False)

So need to modify config.cfg to no pass proxy I suspect?:

[MISP]
misp_url=https://localhost/
misp_key=xxxMISPuserkeyxxx
misp_verifycert=False
.....
[proxy]
port=None
host=None
protocoll=None
full=None

github-germ · 2019-02-15T13:37:12Z

So need to modify config.cfg to no pass proxy I suspect?

Yes, give it a try.

faustus25 · 2019-02-15T16:51:28Z

Proxy needs to be set for the i Sight connection but this seems to impact the MISP API connection.
(If set the Proxy details to "None" the script doesn't run.

How should it be set for no proxy instead of 'None'

proxy = None
OR
proxy ='None'

In any case, the script runs on the same server as MISP and is accessible using based your test connection script while when it is set in config.cfg it fails.

ERROR:PySight_settings:Unexpected error in MISP init: (<class 'pymisp.exceptions.PyMISPError'>, PyMISPError("Unable to connect to MISP (https://localhost). Please make sure the API key and the URL are correct (http/https is required): HTTPSConnectionPool(host='localhost', port=443): Max retries exceeded with url: /servers/getPyMISPVersion.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f1e3c1a88d0>: Failed to establish a new connection: [Errno 110] Connection timed out',))",), <traceback object at 0x7f1df8624bc8>)
DEBUG:PySight_settings:1 Inappropriate argument type.
DEBUG:PySight_settings:2 (<class 'TypeError'>, TypeError('exceptions must derive from BaseException',), <traceback object at 0x7f1df8624bc8>)
DEBUG:PySight_settings:3 <class 'TypeError'>
DEBUG:PySight_settings:4 exceptions must derive from BaseException
  File "PySight.py", line 285, in isight_process_alert_content_element
    raise "no MISP instance found"
DEBUG:PySight_settings:6 None
^CException ignored in: <module 'threading' from '/usr/lib/python3.5/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 1288, in _shutdown
    t.join()
  File "/usr/lib/python3.5/threading.py", line 1054, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.5/threading.py", line 1070, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):

Any recommendations to sort this connection issue?

github-germ · 2019-02-15T18:16:39Z

Try...

port=
host=
protocoll=
full=

faustus25 · 2019-02-18T18:11:10Z

No luck leaving the proxy values blank or appending "None".

Same issue:

  File "PySight.py", line 255, in isight_process_alert_content_element
    raise "no MISP instance found"
DEBUG:PySight_settings:2 (<class 'TypeError'>, TypeError('exceptions must derive from BaseException',), <traceback object at 0x7f8ee4667fc8>)
ERROR:PySight_settings:Unexpected error in MISP init: (<class 'pymisp.exceptions.PyMISPError'>, PyMISPError("Unable to connect to MISP (https://10.10.10.10). Please make sure the API key and the URL are correct (http/https is required): HTTPSConnectionPool(host='misp.fmr.com', port=443): Max retries exceeded with url: /servers/getPyMISPVersion.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f8ee4606400>: Failed to establish a new connection: [Errno 110] Connection timed out',))",), <traceback object at 0x7f8ee45fb548>)

Are you make your requests through a proxy and how is it presented in the config?

Interestingly, the initial connection to MISP works once launched:

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): 10.10.10.10:443
DEBUG:urllib3.connectionpool:https://10.10.10.10:443 "GET /attributes/describeTypes.json HTTP/1.1" 200 18520
text_search_sensitive_reports Response:
DEBUG:PySight_settings:param headers: {'Accept': 'application/json', 'Accept-Version': '2.4', 'Date': 'Mon, 18 Feb 2019 ', 'X-Auth-Hash': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', 'X-Auth': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'} https://api.isightpartners.com
DEBUG:PySight_settings:https://api.isightpartners.com/view/indicators?since=1550340460
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.isightpartners.com:443

````

github-germ · 2019-02-18T21:34:00Z

sorry that i cannot share my code, as it was dramatically enhanced. unfortunately, i am not the author of this code (and i rvisdee it a long time ago, and there were some quirky parts). however, you can try modifying the isight_load_data function in PySight.py a bit: (https://github.com/deralexxx/PySight2MISP/blob/master/PySight.py#L182)

from:

  try:
        PySight_settings.logger.debug("param headers: %s %s", a_headers, a_url)
        proxy_request = ProxyManager(str(PySight_settings.proxy_adress))
        url_to_load = PySight_settings.isight_url + a_query
        PySight_settings.logger.debug(url_to_load)
        try:

            r = proxy_request.request('GET', a_url + a_query, None, headers=a_headers)
        except urllib.error.HTTPError as e:
            print(e.code)
            print(e.read())
...

to something like the following. NOTE: if you're not using a proxy, then in config.cfg set full=

  try:
        PySight_settings.logger.debug("param headers: %s %s", a_headers, a_url)
        if PySight_settings.proxy_adress:
           proxy_request = ProxyManager(str(PySight_settings.proxy_adress))
        else:
            proxy_request = PoolManager()

        url_to_load = PySight_settings.isight_url + a_query
        try:
            r = isightApi.request('GET', a_url + a_query, None, headers=a_headers)
       except urllib.error.HTTPError as e:
           print(e.code)
           print(e.read())
...

faustus25 · 2019-02-19T17:59:21Z

The isightAPI variable is unique to your code as not present in config or PySight script after adding your code segment:

Unexpected error: %s (<class 'NameError'>, NameError("name 'isightApi' is not defined",), <traceback object at 0x7f0d151db4c8>)
ERROR:PySight_settings:Something went wrong while downloading / processing the iSight Request

Without digging too much into your code, what have you you set isightAPI as?

Also, I have the proxy already set as global environment variable in /etc/environment on the server but the config script requires it as PySight won't run without setting those proxy values.

github-germ · 2019-02-19T18:12:38Z

Sorry about that... I am trying to take pieces of new code and apply it to you situation. Here's a correction, i.e. replacing isightApi with proxy_request.

  try:
        PySight_settings.logger.debug("param headers: %s %s", a_headers, a_url)
        if PySight_settings.proxy_adress:
            proxy_request = ProxyManager(str(PySight_settings.proxy_adress))
        else:
            proxy_request = PoolManager()

        url_to_load = PySight_settings.isight_url + a_query
        try:
            r = request_request.request('GET', a_url + a_query, None, headers=a_headers)
       except urllib.error.HTTPError as e:
           print(e.code)
           print(e.read())
...

faustus25 · 2019-02-22T17:17:42Z

Interestingly, I can't test it now, getting this error since a terminal crashed and the existing session that was forcefully closed is still locked and any new call to libgcc is still referencing even after a reboot AND it is installed already:

libgcc1-dbg is already the newest version (1:6.0.1-0ubuntu1).

Error:

libgcc_s.so.1 must be installed for pthread_cancel to work

This troubleshooting is neverending :)

faustus25 · 2019-02-25T09:54:13Z

Ignored that libgcc error and inserted your code but further debugging needed; same error message from earlier "TypeError: 'bool' object is not subscriptable" but /debug directory exists:

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): localhost:443
DEBUG:urllib3.connectionpool:https://localhost:443 "GET /servers/getPyMISPVersion.json HTTP/1.1" 200 21
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): localhost:443
DEBUG:urllib3.connectionpool:https://localhost:443 "GET /attributes/describeTypes.json HTTP/1.1" 200 18520
text_search_sensitive_reports Response:
DEBUG:PySight_settings:param headers: {'X-Auth-Hash': 'xxxxxxx', 'Accept': 'application/json', 'Date': 'xxxxx', 'X-Auth': 'xxxxxxx', 'Accept-Version': '2.5'} https://api.isightpartners.com
Unexpected error: %s (<class 'NameError'>, NameError("name 'request_request' is not defined",), <traceback object at 0x7fd7aded9a88>)
ERROR:PySight_settings:Something went wrong while downloading / processing the iSight Request
Traceback (most recent call last):
  File "PySight.py", line 854, in <module>
    misp_process_isight_alert(result)
  File "PySight.py", line 825, in misp_process_isight_alert
    for i in a_result['message']:
TypeError: 'bool' object is not subscriptable

github-germ · 2019-02-25T16:46:21Z

OK, my typo. So sorry. This is what happens when I cannot really test my code :-(
However, this is an easy fix.

I had this line:

            r = request_request.request('GET', a_url + a_query, None, headers=a_headers)

But should've been:

            r = proxy_request.request('GET', a_url + a_query, None, headers=a_headers)

Here it is again in context:

  try:
        PySight_settings.logger.debug("param headers: %s %s", a_headers, a_url)
        if PySight_settings.proxy_adress:
            proxy_request = ProxyManager(str(PySight_settings.proxy_adress))
        else:
            proxy_request = PoolManager()

        url_to_load = PySight_settings.isight_url + a_query
        try:
            r = proxy_request.request('GET', a_url + a_query, None, headers=a_headers)
       except urllib.error.HTTPError as e:
           print(e.code)
           print(e.read())

faustus25 · 2019-02-25T17:26:16Z

That worked, this libgcc error is now persisting and really hampering the debugging:

DEBUG:PySight_settings:1 Inappropriate argument type.
DEBUG:PySight_settings:2 (<class 'TypeError'>, TypeError('exceptions must derive from BaseException',), <traceback object at 0x7fba283b1648>)
DEBUG:PySight_settings:3 <class 'TypeError'>
DEBUG:PySight_settings:4 exceptions must derive from BaseException
  File "PySight.py", line 301, in isight_process_alert_content_element
DEBUG:PySight_settings:6 None
libgcc_s.so.1 must be installed for pthread_cancel to work
Aborted

It is installed:

libgcc1-dbg is already the newest version (1:6.0.1-0ubuntu1).

github-germ · 2019-02-25T17:40:13Z

OK, glad to hear your progress. Sorry, but I don't think I can assist re' thelibgcc issue. I suspect that's related to your OS. I assume you've been searching the web. Good luck with that.

github-germ · 2019-02-25T17:42:24Z

I would guess you are aware... just in case...

You are saying you see this installed:

libgcc1-dbg is already the newest version (1:6.0.1-0ubuntu1).

But the error is reporting a different library name is missing:

libgcc_s.so.1 must be installed for pthread_cancel to work

faustus25 · 2019-02-28T15:49:26Z

Had to install another package which worked (proxy issue still persists):

apt-get install gcc-multilib

Ran the test_pysight script just to see if there was any other issues to help the troubleshooting:

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): misp.local.com:443
DEBUG:urllib3.connectionpool:https://misp.local.com:443 "GET /servers/getPyMISPVersion.json HTTP/1.1" 200 21
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): misp.local.com:443
DEBUG:urllib3.connectionpool:https://misp.local.com:443 "GET /attributes/describeTypes.json HTTP/1.1" 200 18520
DEBUG:model.pySightReport:1469544180
DEBUG:PySight_settings:1 Inappropriate argument type.
DEBUG:PySight_settings:2 (<class 'TypeError'>, TypeError('strptime() argument 1 must be str, not int',), <traceback object at 0x7febe6167248>)
DEBUG:PySight_settings:3 <class 'TypeError'>
DEBUG:PySight_settings:4 strptime() argument 1 must be str, not int
  File "/home/user/python3-virtual-environments/PySight2MISP/PySight.py", line 309, in isight_process_alert_content_element
    isight_report_instance = pySightReport(a_json)
  File "/home/user/python3-virtual-environments/PySight2MISP/model/pySightReport.py", line 99, in __init__
    self._parse_json(a_alert_json)
  File "/home/user/python3-virtual-environments/PySight2MISP/model/pySightReport.py", line 156, in _parse_json
    datetime_object = datetime.strptime(p_alert_json['publishDate'], date_format)
DEBUG:PySight_settings:6 None
Ftest_pysight.py:114: ResourceWarning: unclosed file <_io.TextIOWrapper name='test_data/example_indicator_c2.json' mode='r' encoding='ISO-8859-1'>
  json_data = json.load(open("test_data/example_indicator_c2.json"))

github-germ · 2019-02-28T16:10:02Z

I've never run test_pysight.py; however, just ran it here, and get the same exception. I'd prefer not debug that code (none of this is my code :-)

Can you try running PySight.py directly again?

faustus25 · 2019-02-28T17:34:55Z

I was hoping for test_pysight to reveal something else. In any case PySight script can't find MISP instance despite setting proxy in the config. Does the proxy have to be set in PyMISP keys file too?

DEBUG:urllib3.connectionpool:https://misp.local.com:443 "GET /attributes/describeTypes.json HTTP/1.1" 200 18520
ERROR:PySight_settings:Unexpected error in MISP init: (<class 'pymisp.exceptions.PyMISPError'>, PyMISPError("Unable to connect to MISP (https://misp.local.com/). Please make sure the API key and the URL are correct (http/https is required): HTTPSConnectionPool(host='misp.local.com', port=443): Max retries exceeded with url: /servers/getPyMISPVersion.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f9a6c405978>: Failed to establish a new connection: [Errno 110] Connection timed out',))",), <traceback object at 0x7f9a6c3ff488>)
DEBUG:PySight_settings:1 Inappropriate argument type.
DEBUG:PySight_settings:2 (<class 'TypeError'>, TypeError('exceptions must derive from BaseException',), <traceback object at 0x7f9a6c3ff488>)
DEBUG:PySight_settings:3 <class 'TypeError'>
DEBUG:PySight_settings:4 exceptions must derive from BaseException
  File "PySight.py", line 301, in isight_process_alert_content_element
    raise "no MISP instance found"
DEBUG:PySight_settings:6 None

````

github-germ · 2019-02-28T17:44:59Z

Do you need to connect to your MISP via a proxy or can go direct? The code that maks the connection in PySight.py is below. Notice that proxies=None...

def get_misp_instance():
    """

    :return: MISP Instance
    :rtype: PyMISP
    """
    try:
        if not HAVE_PYMISP:
            PySight_settings.logger.error("Missing dependency, install pymisp (`pip install pymisp`)")
            return False
        else:
            # PyMISP.proxies
            return PyMISP(PySight_settings.misp_url, PySight_settings.misp_key, PySight_settings.misp_verifycert,
                          proxies=None)
    except:
        PySight_settings.logger.error("Unexpected error in MISP init: %s", sys.exc_info())
        return False

faustus25 · 2019-03-01T17:23:12Z

So added proxies settings for MISP via PySight script and PySight settings script;

the first was to PySight Settings

added this:

#### MISP STUFF

misp_url = config.get('MISP', 'misp_url')
misp_key = config.get('MISP', 'misp_key')
misp_verifycert = config.getboolean('MISP', 'misp_verifycert')
PROXY_HOST = config.get('proxy', 'host')
PROXY_PORT = config.get('proxy', 'port')
PROXY_PROTOCOLL = config.get('proxy', 'protocoll')
proxy = config.get('proxy', 'full')
proxy_adress = proxy
**proxies = {
                   'http':'http://proxy.com:xxxx',
                   'https':'https://proxy.com:xxxx}**

Next I added to PySight.py:

def get_misp_instance():
    """

    :return: MISP Instance
    :rtype: PyMISP
    """
    try:
        if not HAVE_PYMISP:
            PySight_settings.logger.error("Missing dependency, install pymisp (`pip install pymisp`)")
            return False
        else:
            # PyMISP.proxies
            return PyMISP(PySight_settings.misp_url, PySight_settings.misp_key, 
            PySight_settings.misp_verifycert,
                          **proxies = PySight_settings.proxies** )
    except:
        PySight_settings.logger.error("Unexpected error in MISP init: %s", sys.exc_info())
        return False

This proceed to run but it just hangs and no error message is specified.

github-germ · 2019-03-01T18:09:12Z

Seems like this is now not a script issue, but rather related to network access. May I suggest you troubleshoot your network path from a shell through the proxy to your MISP host, e.g. using curl. Once you get success there, try the script again with any revisions you discover.

jaegeral · 2019-08-07T11:44:52Z

so seems you guys changed some stuff, do you mind to make a PR? Others might find it useful, I would really appreciate it, and apologise for not updating the script for so long
@github-germ @faustus25

faustus25 · 2019-08-08T00:23:02Z

I haven't tested it without the proxy settings as it was my intention to get it working with it.

I will test without the proxy settings on a separate machine and if it works, will do a PR after.

jaegeral · 2019-08-08T09:15:51Z

ok overall looks like in this issue, several issues are actually discussed and already solved (which is great) maybe we should make an issue for each of the problems and fix them?

faustus25 · 2020-04-06T10:52:13Z

Any need to still do a pull request or has this been updated in the interim?

jaegeral · 2020-04-26T14:34:29Z

Please do a PR.

faustus25 · 2020-05-09T14:06:58Z

Pull Request made if you can review.

jaegeral · 2020-05-09T18:59:49Z

Hey @faustus25 you created the pull requests to your own repository (Fork https://github.com/faustus25/PySight2MISP/pulls and you need to create the pull requests for https://github.com/jaegeral/PySight2MISP

Thx

faustus25 · 2020-05-23T12:59:18Z

Hi @jaegeral, I can only do a pull request of lgtm and not master:

Can you check this?

jaegeral · 2020-05-23T18:51:24Z

Hey, there is nothing I can check, but maybe this helps you:
https://help.github.com/en/enterprise/2.16/user/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request-from-a-fork

Thx

jaegeral · 2020-05-23T19:02:36Z

But tbh, the one PR you made: https://github.com/faustus25/PySight2MISP/pull/2/files does not really change anything.

The second PR you opened to your own repo:
faustus25#3
does only change code in the test script to rename function calls to functions that I can find no where:
e.g. you update: misp_process_isight_indicators to misp_process_isight_alert which are not part of this repository: https://github.com/jaegeral/PySight2MISP/search?q=misp_process_isight_alert&type=Code

So please resolve those things before making a Pull Request. Thank you :-)

faustus25 changed the title ~~Parsing error: "a bytes-like object is required, not 'str'"~~ Type error: "a bytes-like object is required, not 'str'" Feb 12, 2019

Type error: "a bytes-like object is required, not 'str'" #7

Type error: "a bytes-like object is required, not 'str'" #7

Comments

faustus25 commented Feb 12, 2019 • edited Loading

faustus25 commented Feb 12, 2019

github-germ commented Feb 12, 2019

faustus25 commented Feb 12, 2019

github-germ commented Feb 12, 2019

faustus25 commented Feb 12, 2019

github-germ commented Feb 12, 2019

faustus25 commented Feb 13, 2019

github-germ commented Feb 13, 2019 • edited Loading

faustus25 commented Feb 13, 2019 • edited Loading

github-germ commented Feb 14, 2019 • edited Loading

faustus25 commented Feb 14, 2019

github-germ commented Feb 14, 2019

faustus25 commented Feb 14, 2019

github-germ commented Feb 14, 2019

faustus25 commented Feb 14, 2019

github-germ commented Feb 14, 2019

github-germ commented Feb 14, 2019

jaegeral commented Feb 15, 2019

faustus25 commented Feb 15, 2019

github-germ commented Feb 15, 2019

faustus25 commented Feb 15, 2019

github-germ commented Feb 15, 2019

faustus25 commented Feb 18, 2019

github-germ commented Feb 18, 2019

faustus25 commented Feb 19, 2019 • edited Loading

github-germ commented Feb 19, 2019

faustus25 commented Feb 22, 2019

faustus25 commented Feb 25, 2019 • edited Loading

github-germ commented Feb 25, 2019

faustus25 commented Feb 25, 2019

github-germ commented Feb 25, 2019

github-germ commented Feb 25, 2019

faustus25 commented Feb 28, 2019

github-germ commented Feb 28, 2019 • edited Loading

faustus25 commented Feb 28, 2019

github-germ commented Feb 28, 2019

faustus25 commented Mar 1, 2019

github-germ commented Mar 1, 2019 • edited Loading

jaegeral commented Aug 7, 2019

faustus25 commented Aug 8, 2019

jaegeral commented Aug 8, 2019

faustus25 commented Apr 6, 2020

jaegeral commented Apr 26, 2020

faustus25 commented May 9, 2020

jaegeral commented May 9, 2020

faustus25 commented May 23, 2020

jaegeral commented May 23, 2020

jaegeral commented May 23, 2020

faustus25 commented Feb 12, 2019 •

edited

Loading

github-germ commented Feb 13, 2019 •

edited

Loading

faustus25 commented Feb 13, 2019 •

edited

Loading

github-germ commented Feb 14, 2019 •

edited

Loading

faustus25 commented Feb 19, 2019 •

edited

Loading

faustus25 commented Feb 25, 2019 •

edited

Loading

github-germ commented Feb 28, 2019 •

edited

Loading

github-germ commented Mar 1, 2019 •

edited

Loading