Skip to content

gguf-py: gguf_writer: Use bytearray to build metadata #4051

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

KerfuffleV2
Copy link
Collaborator

@KerfuffleV2 KerfuffleV2 commented Nov 12, 2023

Repeatedly concatenating a bytes object to build the metadata in the GGUF writer is insanely slow. The part that mainly hurts is the vocab arrays, the non-array values are generally too small for the inefficiency to really be noticeable.

With this change, building the metadata for a model goes from around 5-6 sec to instant.

@KerfuffleV2 KerfuffleV2 added the script Script related label Nov 12, 2023
Copy link
Collaborator

@cebtenzzre cebtenzzre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need a file object here. Why not use bytearray? It has .append and .extend methods.

@KerfuffleV2
Copy link
Collaborator Author

Why not use bytearray?

The stuff I found seemed to indicate BytesIO was faster for this type of use case (appending chunks). This makes sense, since a bytearray has to efficiently support random access to individual elements. I would guess it's not a big difference though and I didn't benchmark it myself.

  1. Poor performance when opening raw data with (wrong) gzip encoding in header mhe/pynrrd#88 (comment)
  2. https://bugs.python.org/issue19801 (pretty old)

Bump gguf-py package version
@KerfuffleV2 KerfuffleV2 changed the title gguf-py: gguf_writer: Use BytesIO to build metadata gguf-py: gguf_writer: Use bytearray to build metadata Nov 12, 2023
@KerfuffleV2
Copy link
Collaborator Author

@cebtenzzre Okay, so I tested it and they're exactly the same speed. I hacked convert.py to add the vocab metadata 10 times for a model with 64,000 vocab entries. It was like 3.020 sec for one and 3.019 for the other. Since using bytearray requires less modifications I switched to that.

@KerfuffleV2 KerfuffleV2 merged commit 21fd874 into ggml-org:master Nov 12, 2023
@KerfuffleV2 KerfuffleV2 deleted the feat-gguf-py-optimize-metadata branch November 17, 2023 03:11
olexiyb pushed a commit to Sanctum-AI/llama.cpp that referenced this pull request Nov 23, 2023
* gguf-py: gguf_writer: Use BytesIO to build metadata

* Use bytearray instead

Bump gguf-py package version
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
script Script related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants