Skip to content

Commit f1b629e

Browse files
umarbutlerumarbutler
committed
Updated links.
1 parent 80d5964 commit f1b629e

File tree

5 files changed

+37
-37
lines changed

5 files changed

+37
-37
lines changed

‎CHANGELOG.md

Lines changed: 28 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,8 @@ All notable changes to `semchunk` will be documented here. This project adheres
1515

1616
## [3.0.0] - 2024-12-31
1717
### Added
18-
- Added an `offsets` argument to `chunk()` and `Chunker.__call__()` that specifies whether to return the start and end offsets of each chunk ([#9](https://github.com/umarbutler/semchunk/issues/9)). The argument defaults to `False`.
19-
- Added an `overlap` argument to `chunk()` and `Chunker.__call__()` that specifies the proportion of the chunk size, or, if >=1, the number of tokens, by which chunks should overlap ([#1](https://github.com/umarbutler/semchunk/issues/1)). The argument defaults to `None`, in which case no overlapping occurs.
18+
- Added an `offsets` argument to `chunk()` and `Chunker.__call__()` that specifies whether to return the start and end offsets of each chunk ([#9](https://github.com/isaacus-dev/semchunk/issues/9)). The argument defaults to `False`.
19+
- Added an `overlap` argument to `chunk()` and `Chunker.__call__()` that specifies the proportion of the chunk size, or, if >=1, the number of tokens, by which chunks should overlap ([#1](https://github.com/isaacus-dev/semchunk/issues/1)). The argument defaults to `None`, in which case no overlapping occurs.
2020
- Added an undocumented, private `_make_chunk_function()` method to the `Chunker` class that constructs chunking functions with call-level arguments passed.
2121
- Added more unit tests for new features as well as for multiple token counters and for ensuring there are no chunks comprised entirely of whitespace characters.
2222

@@ -37,11 +37,11 @@ All notable changes to `semchunk` will be documented here. This project adheres
3737

3838
## [2.2.1] - 2024-12-17
3939
### Changed
40-
- Started benchmarking [`semantic-text-splitter`](https://pypi.org/project/semantic-text-splitter/) in parallel to ensure a fair comparison, courtesy of [@benbrandt](https://github.com/benbrandt) ([#17](https://github.com/umarbutler/semchunk/pull/12)).
40+
- Started benchmarking [`semantic-text-splitter`](https://pypi.org/project/semantic-text-splitter/) in parallel to ensure a fair comparison, courtesy of [@benbrandt](https://github.com/benbrandt) ([#17](https://github.com/isaacus-dev/semchunk/pull/12)).
4141

4242
## [2.2.0] - 2024-07-12
4343
### Changed
44-
- Switched from having `chunkerify()` output a function to having it return an instance of the new `Chunker()` class which should not alter functionality in any way but will allow for the preservation of type hints, fixing [#7](https://github.com/umarbutler/semchunk/pull/7).
44+
- Switched from having `chunkerify()` output a function to having it return an instance of the new `Chunker()` class which should not alter functionality in any way but will allow for the preservation of type hints, fixing [#7](https://github.com/isaacus-dev/semchunk/pull/7).
4545

4646
## [2.1.0] - 2024-06-20
4747
### Fixed
@@ -64,19 +64,19 @@ All notable changes to `semchunk` will be documented here. This project adheres
6464

6565
## [0.3.2] - 2024-06-01
6666
### Fixed
67-
- Fixed a bug where a `DivisionByZeroError` would be raised where a token counter returned zero tokens when called from `merge_splits()`, courtesy of [@jcobol](https://github.com/jcobol) ([#5](https://github.com/umarbutler/semchunk/pull/5)) ([7fd64eb](https://github.com/umarbutler/semchunk/pull/5/commits/7fd64eb8cf51f45702c59f43795be9a00c7d0d17)), fixing [#4](https://github.com/umarbutler/semchunk/issues/4).
67+
- Fixed a bug where a `DivisionByZeroError` would be raised where a token counter returned zero tokens when called from `merge_splits()`, courtesy of [@jcobol](https://github.com/jcobol) ([#5](https://github.com/isaacus-dev/semchunk/pull/5)) ([7fd64eb](https://github.com/isaacus-dev/semchunk/pull/5/commits/7fd64eb8cf51f45702c59f43795be9a00c7d0d17)), fixing [#4](https://github.com/isaacus-dev/semchunk/issues/4).
6868

6969
## [0.3.1] - 2024-05-18
7070
### Fixed
7171
- Fixed typo in error messages in `chunkerify()` where it was referred to as `make_chunker()`.
7272

7373
## [0.3.0] - 2024-05-18
7474
### Added
75-
- Introduced the `chunkerify()` function, which constructs a chunker from a tokenizer or token counter that can be reused and can also chunk multiple texts in a single call. The resulting chunker speeds up chunking by 40.4% thanks, in large part, to a token counter that avoid having to count the number of tokens in a text when the number of characters in the text exceed a certain threshold, courtesy of [@R0bk](https://github.com/R0bk) ([#3](https://github.com/umarbutler/semchunk/pull/3)) ([337a186](https://github.com/umarbutler/semchunk/pull/3/commits/337a18615f991076b076262288b0408cb162b48c)).
75+
- Introduced the `chunkerify()` function, which constructs a chunker from a tokenizer or token counter that can be reused and can also chunk multiple texts in a single call. The resulting chunker speeds up chunking by 40.4% thanks, in large part, to a token counter that avoid having to count the number of tokens in a text when the number of characters in the text exceed a certain threshold, courtesy of [@R0bk](https://github.com/R0bk) ([#3](https://github.com/isaacus-dev/semchunk/pull/3)) ([337a186](https://github.com/isaacus-dev/semchunk/pull/3/commits/337a18615f991076b076262288b0408cb162b48c)).
7676

7777
## [0.2.4] - 2024-05-13
7878
### Changed
79-
- Improved chunking performance with larger chunk sizes by switching from linear to binary search for the identification of optimal chunk boundaries, courtesy of [@R0bk](https://github.com/R0bk) ([#3](https://github.com/umarbutler/semchunk/pull/3)) ([337a186](https://github.com/umarbutler/semchunk/pull/3/commits/337a18615f991076b076262288b0408cb162b48c)).
79+
- Improved chunking performance with larger chunk sizes by switching from linear to binary search for the identification of optimal chunk boundaries, courtesy of [@R0bk](https://github.com/R0bk) ([#3](https://github.com/isaacus-dev/semchunk/pull/3)) ([337a186](https://github.com/isaacus-dev/semchunk/pull/3/commits/337a18615f991076b076262288b0408cb162b48c)).
8080

8181
## [0.2.3] - 2024-03-11
8282
### Fixed
@@ -117,24 +117,24 @@ All notable changes to `semchunk` will be documented here. This project adheres
117117
### Added
118118
- Added the `chunk()` function, which splits text into semantically meaningful chunks of a specified size as determined by a provided token counter.
119119

120-
[3.0.2]: https://github.com/umarbutler/semchunk/compare/v3.0.1...v3.0.2
121-
[3.0.1]: https://github.com/umarbutler/semchunk/compare/v3.0.0...v3.0.1
122-
[3.0.0]: https://github.com/umarbutler/semchunk/compare/v2.2.2...v3.0.0
123-
[2.2.2]: https://github.com/umarbutler/semchunk/compare/v2.2.1...v2.2.2
124-
[2.2.1]: https://github.com/umarbutler/semchunk/compare/v2.2.0...v2.2.1
125-
[2.2.0]: https://github.com/umarbutler/semchunk/compare/v2.1.0...v2.2.0
126-
[2.1.0]: https://github.com/umarbutler/semchunk/compare/v2.0.0...v2.1.0
127-
[2.0.0]: https://github.com/umarbutler/semchunk/compare/v1.0.1...v2.0.0
128-
[1.0.1]: https://github.com/umarbutler/semchunk/compare/v1.0.0...v1.0.1
129-
[1.0.0]: https://github.com/umarbutler/semchunk/compare/v0.3.2...v1.0.0
130-
[0.3.2]: https://github.com/umarbutler/semchunk/compare/v0.3.1...v0.3.2
131-
[0.3.1]: https://github.com/umarbutler/semchunk/compare/v0.3.0...v0.3.1
132-
[0.3.0]: https://github.com/umarbutler/semchunk/compare/v0.2.4...v0.3.0
133-
[0.2.4]: https://github.com/umarbutler/semchunk/compare/v0.2.3...v0.2.4
134-
[0.2.3]: https://github.com/umarbutler/semchunk/compare/v0.2.2...v0.2.3
135-
[0.2.2]: https://github.com/umarbutler/semchunk/compare/v0.2.1...v0.2.2
136-
[0.2.1]: https://github.com/umarbutler/semchunk/compare/v0.2.0...v0.2.1
137-
[0.2.0]: https://github.com/umarbutler/semchunk/compare/v0.1.2...v0.2.0
138-
[0.1.2]: https://github.com/umarbutler/semchunk/compare/v0.1.1...v0.1.2
139-
[0.1.1]: https://github.com/umarbutler/semchunk/compare/v0.1.0...v0.1.1
140-
[0.1.0]: https://github.com/umarbutler/semchunk/releases/tag/v0.1.0
120+
[3.0.2]: https://github.com/isaacus-dev/semchunk/compare/v3.0.1...v3.0.2
121+
[3.0.1]: https://github.com/isaacus-dev/semchunk/compare/v3.0.0...v3.0.1
122+
[3.0.0]: https://github.com/isaacus-dev/semchunk/compare/v2.2.2...v3.0.0
123+
[2.2.2]: https://github.com/isaacus-dev/semchunk/compare/v2.2.1...v2.2.2
124+
[2.2.1]: https://github.com/isaacus-dev/semchunk/compare/v2.2.0...v2.2.1
125+
[2.2.0]: https://github.com/isaacus-dev/semchunk/compare/v2.1.0...v2.2.0
126+
[2.1.0]: https://github.com/isaacus-dev/semchunk/compare/v2.0.0...v2.1.0
127+
[2.0.0]: https://github.com/isaacus-dev/semchunk/compare/v1.0.1...v2.0.0
128+
[1.0.1]: https://github.com/isaacus-dev/semchunk/compare/v1.0.0...v1.0.1
129+
[1.0.0]: https://github.com/isaacus-dev/semchunk/compare/v0.3.2...v1.0.0
130+
[0.3.2]: https://github.com/isaacus-dev/semchunk/compare/v0.3.1...v0.3.2
131+
[0.3.1]: https://github.com/isaacus-dev/semchunk/compare/v0.3.0...v0.3.1
132+
[0.3.0]: https://github.com/isaacus-dev/semchunk/compare/v0.2.4...v0.3.0
133+
[0.2.4]: https://github.com/isaacus-dev/semchunk/compare/v0.2.3...v0.2.4
134+
[0.2.3]: https://github.com/isaacus-dev/semchunk/compare/v0.2.2...v0.2.3
135+
[0.2.2]: https://github.com/isaacus-dev/semchunk/compare/v0.2.1...v0.2.2
136+
[0.2.1]: https://github.com/isaacus-dev/semchunk/compare/v0.2.0...v0.2.1
137+
[0.2.0]: https://github.com/isaacus-dev/semchunk/compare/v0.1.2...v0.2.0
138+
[0.1.2]: https://github.com/isaacus-dev/semchunk/compare/v0.1.1...v0.1.2
139+
[0.1.1]: https://github.com/isaacus-dev/semchunk/compare/v0.1.0...v0.1.1
140+
[0.1.0]: https://github.com/isaacus-dev/semchunk/releases/tag/v0.1.0

‎README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,10 +41,10 @@ text = 'The quick brown fox jumps over the lazy dog.'
4141
# OpenAI `tiktoken` encoding or Hugging Face model, or a custom tokenizer that has an `encode()`
4242
# method (like a `tiktoken`, `transformers` or `tokenizers` tokenizer) or a custom token counting
4343
# function that takes a text and returns the number of tokens in it.
44-
chunker = semchunk.chunkerify('umarbutler/emubert', chunk_size) or \
44+
chunker = semchunk.chunkerify('isaacus-dev/emubert', chunk_size) or \
4545
semchunk.chunkerify('gpt-4', chunk_size) or \
4646
semchunk.chunkerify('cl100k_base', chunk_size) or \
47-
semchunk.chunkerify(AutoTokenizer.from_pretrained('umarbutler/emubert'), chunk_size) or \
47+
semchunk.chunkerify(AutoTokenizer.from_pretrained('isaacus-dev/emubert'), chunk_size) or \
4848
semchunk.chunkerify(tiktoken.encoding_for_model('gpt-4'), chunk_size) or \
4949
semchunk.chunkerify(lambda text: len(text.split()), chunk_size)
5050

‎pyproject.toml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -50,10 +50,10 @@ dependencies = [
5050
]
5151

5252
[project.urls]
53-
Homepage = "https://github.com/umarbutler/semchunk"
54-
Documentation = "https://github.com/umarbutler/semchunk/blob/main/README.md"
55-
Issues = "https://github.com/umarbutler/semchunk/issues"
56-
Source = "https://github.com/umarbutler/semchunk"
53+
Homepage = "https://github.com/isaacus-dev/semchunk"
54+
Documentation = "https://github.com/isaacus-dev/semchunk/blob/main/README.md"
55+
Issues = "https://github.com/isaacus-dev/semchunk/issues"
56+
Source = "https://github.com/isaacus-dev/semchunk"
5757

5858
[tool.hatch.build.targets.sdist]
5959
only-include = ['src/semchunk/__init__.py', 'src/semchunk/py.typed', 'src/semchunk/semchunk.py', 'pyproject.toml', 'README.md', 'LICENCE', 'CHANGELOG.md', 'tests/bench.py', 'tests/test_semchunk.py', '.github/workflows/ci.yml', 'tests/helpers.py']

‎tests/helpers.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ def initialize_test_token_counters() -> dict[str, Callable[[str], int]]:
3737
"""Initialize `tiktoken`, `transformers`, character and word token counters for testing purposes."""
3838

3939
gpt4_tiktoken_tokenizer = tiktoken.encoding_for_model('gpt-4').encode
40-
emubert_transformers_tokenizer = make_transformers_tokenizer(transformers.AutoTokenizer.from_pretrained('umarbutler/emubert'))
40+
emubert_transformers_tokenizer = make_transformers_tokenizer(transformers.AutoTokenizer.from_pretrained('isaacus-dev/emubert'))
4141

4242
def word_tokenizer(text: str) -> list[str]:
4343
"""Tokenize a text into words."""

‎tests/test_semchunk.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -159,7 +159,7 @@ def test_semchunk() -> None:
159159
assert error_raised
160160

161161
# Test using `tiktoken` tokenizers, encodings and a `transformers` tokenizer by name with `chunkerify()`.
162-
for name in ['cl100k_base', 'gpt-4', 'umarbutler/emubert']:
162+
for name in ['cl100k_base', 'gpt-4', 'isaacus-dev/emubert']:
163163
chunker = semchunk.chunkerify(name, 1)
164164
chunker(DETERMINISTIC_TEST_INPUT)
165165
if TEST_OFFSETS: chunker(DETERMINISTIC_TEST_INPUT, offsets = True)
@@ -175,7 +175,7 @@ def test_semchunk() -> None:
175175
assert error_raised
176176

177177
# Test using a `transformers` tokenizer directly.
178-
tokenizer = AutoTokenizer.from_pretrained('umarbutler/emubert')
178+
tokenizer = AutoTokenizer.from_pretrained('isaacus-dev/emubert')
179179
chunker = semchunk.chunkerify(tokenizer, 1)
180180

181181
# Test using a `tiktoken` tokenizer directly.

0 commit comments

Comments
 (0)