Skip to content

KV parsing rework with newline termination in mind #128

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 15, 2016
Merged

KV parsing rework with newline termination in mind #128

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 18 additions & 12 deletions mbed_host_tests/host_tests_conn_proxy/conn_proxy.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,26 +28,32 @@
class KiViBufferWalker():
"""! Simple auxiliary class used to walk through a buffer and search for KV tokens """
def __init__(self):
self.KIVI_REGEX = r"\{\{([\w\d_-]+);([^\}]+)\}\}\n"
self.KIVI_REGEX = r"\{\{([\w\d_-]+);([^\}]+)\}\}"
self.buff = str()
self.buff_idx = 0
self.kvl = []
self.re_kv = re.compile(self.KIVI_REGEX)

def append(self, payload):
"""! Append stream buffer with payload """
"""! Append stream buffer with payload and process"""
self.buff += payload
lines = self.buff.split('\n')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we're now explicitly adding '\r' in ARMmbed/mbed-os#2702, is there a place where this character can be handled and stripped in this PR?

Ideally the next release of htrun should be able to handle cases where the '\r' is or is not present, since the tool may be used with older releases of mbed OS.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd probably fine just to do a rstrip on the returned line before inserting it into the kvl list: https://docs.python.org/2/library/string.html#string.rstrip

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since \n is removed from K,V regex any character outside the K.V pattern does not affect us. I have tested it. It works with mbed-os master and ARMmbed/mbed-os#2702

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we extract key and value using the K,V regex we don't need any extra processing like rstrip.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see @mazimkhan, thanks for explaining. Totally missed that the first time around 😄

self.buff = lines[-1] # remaining
lines.pop(-1)

for line in lines:
m = self.re_kv.search(line)
if m:
(key, value) = m.groups()
self.kvl.append((key, value, time()))

def search(self):
"""! Check if there is a KV value in buffer """
return self.re_kv.search(self.buff[self.buff_idx:])
return len(self.kvl) > 0

def get_kv(self):
m = self.re_kv.search(self.buff[self.buff_idx:])
if m:
(key, value) = m.groups()
kv_str = m.group(0)
self.buff_idx = self.buff.find(kv_str, self.buff_idx) + len(kv_str)
return (key, value, time())
def pop_kv(self):
if len(self.kvl):
return self.kvl.pop(0)
return None, None, time()


def conn_primitive_factory(conn_resource, config, event_queue, logger):
Expand Down Expand Up @@ -208,7 +214,7 @@ def __send_sync(timeout=None):
# Stream data stream KV parsing
kv_buffer.append(data)
while kv_buffer.search():
key, value, timestamp = kv_buffer.get_kv()
key, value, timestamp = kv_buffer.pop_kv()

if sync_uuid_discovered:
event_queue.put((key, value, timestamp))
Expand Down