mbox: allow custom, stable document id #393
Conversation
* via function passed in `id_fn`, eg. `MboxReader(id_fn=lambda msg: md5(msg.encode()).hexdigest())` * overrides UUID-based default from Document
There was a problem hiding this comment.
out of curiosity is msg a useful argument to pass into id_fn? It seems to me that for a lot of use cases you'd need to do a bunch of hacky string parsing to extract out a usable id.
another option is to modify parse_file, extract a more structured dict (not just the formatted message string, but also a dict containing _date, _from, _to_ keys etc.)
llama_hub/file/mbox/base.py
Outdated
| docs.append(Document(text=msg, extra_info=extra_info or {})) | ||
| d = Document(text=msg, extra_info=extra_info or {}) | ||
| if self.id_fn: | ||
| d.id_ = self.id_fn(msg) |
There was a problem hiding this comment.
canonical way to set doc_id is d.doc_id = ....
There was a problem hiding this comment.
Thanks fixed! I think I had this from here
I got good and fast results with But yes I was also considering passing a dict to id_fn, if you like it a lot more I could also do that. |
id_fn, eg.MboxReader(id_fn=lambda msg: md5(msg.encode()).hexdigest())