Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add RawTask executing docs ad verbatim #395

Merged
merged 65 commits into from
Dec 11, 2023
Merged

Add RawTask executing docs ad verbatim #395

Changes from 1 commit
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
3aba660
Add context length info. Refactor BuiltinTask and models to facilitat…
rmitsch Oct 17, 2023
5699773
Merge branch 'develop' into feat/inf-doc-len
rmitsch Oct 17, 2023
4213372
Add token count estimator plumbing.
rmitsch Oct 17, 2023
f440ca4
Add plumbing for mapper and reducer.
rmitsch Oct 17, 2023
e47f762
Add ShardMapper prototype.
rmitsch Oct 18, 2023
89a5510
Integrating mapping into prompt generation workflow.
rmitsch Oct 19, 2023
086dec9
Update response parsing and component to support sharding (WIP).
rmitsch Oct 20, 2023
23718fc
Fix shard & prompt flow.
rmitsch Oct 27, 2023
7ce670d
Fix shard & prompt flow.
rmitsch Oct 27, 2023
0d75ea8
Remove todo comments.
rmitsch Oct 27, 2023
9da7098
Fix Anthropic, Cohere, NoOp model tests.
rmitsch Oct 27, 2023
0cb9afd
Merge branch 'develop' into feat/inf-doc-len
rmitsch Oct 30, 2023
f368412
Fix test_llm_pipe().
rmitsch Oct 31, 2023
b1f111d
Fix type checking test.
rmitsch Nov 3, 2023
44a2787
Fix span parsing tests.
rmitsch Nov 3, 2023
6d8cdc7
Fix internal tests.
rmitsch Nov 3, 2023
e712f41
Fix _CountTask.
rmitsch Nov 3, 2023
985fd68
Fix sentiment and summarization tasks and tests.
rmitsch Nov 3, 2023
98842a2
Fix Azure connection URL. Fix Model test pings.
rmitsch Nov 3, 2023
b54a3d9
Fix Lemma parsing.
rmitsch Nov 3, 2023
9bf365d
Start work on doc-to-shard property copying.
rmitsch Nov 3, 2023
dddfaab
Fix REL doc preprocessing.
rmitsch Nov 6, 2023
3af21b5
Remove comment on doc attribute handling during sharding, as this is …
rmitsch Nov 6, 2023
fee9ca7
Add reducer implementations.
rmitsch Nov 8, 2023
e508499
Implement outstanding task reducers.
rmitsch Nov 14, 2023
3218541
Resolve merge conflicts.
rmitsch Nov 14, 2023
c104387
Add shardable/non-shardable LLM task typing distinction. Add support …
rmitsch Nov 20, 2023
2c6d899
Merge branch 'develop' into feat/inf-doc-len
rmitsch Nov 21, 2023
2502c4d
Fix EL task.
rmitsch Nov 23, 2023
03055c5
Fix EL tokenization and highlighting partially.
rmitsch Nov 23, 2023
4e4a2cd
Fix tokenization and whitespaces for EL task.
rmitsch Nov 24, 2023
865acec
Fix merge conflicts.
rmitsch Nov 24, 2023
694d5da
Add new registry handlers (with context length and arbitrary model na…
rmitsch Nov 24, 2023
5295400
Add sharding test with simple count task.
rmitsch Nov 24, 2023
70e3643
Fix sharding algorithm.
rmitsch Nov 24, 2023
4321483
Add test with simple count task.
rmitsch Nov 27, 2023
ef6e738
Add context length as init arg in HF models.
rmitsch Nov 27, 2023
e3ff37d
Fix tests. Don't stringify IO lists if sharded.
rmitsch Nov 28, 2023
056730a
Fix tests.
rmitsch Nov 29, 2023
196c235
Add NER sharding test.
rmitsch Nov 29, 2023
1f51a4a
Add REL and sentiment sharding tests.
rmitsch Nov 29, 2023
e18b302
Add summary sharding tests.
rmitsch Nov 29, 2023
7c092ca
Add EL sharding task. Fix bug in shard mapper.
rmitsch Nov 29, 2023
358ba72
Fix REL error with RELExample parsing.
rmitsch Nov 29, 2023
0c96fb6
Use regex for punctuation in REL conversion.
rmitsch Nov 29, 2023
dc926bd
Maintain custom doc attributes, incl. test.
rmitsch Dec 1, 2023
5585174
Filter merge warnings in textcat reduction.
rmitsch Dec 1, 2023
1ae710c
Fix custom doc data merging.
rmitsch Dec 4, 2023
dc5efee
Add RawTask.
rmitsch Dec 5, 2023
dac8ae3
Fix task version.
rmitsch Dec 5, 2023
01cccdf
Add sharding test.
rmitsch Dec 5, 2023
e94b356
Update spacy_llm/models/langchain/model.py
rmitsch Dec 7, 2023
e68f5d3
Update spacy_llm/pipeline/llm.py
rmitsch Dec 7, 2023
f40bc88
Incorporate feedback.
rmitsch Dec 7, 2023
ac0559d
Move sharding compatibility warning to component constructor.
rmitsch Dec 7, 2023
1763821
Update spacy_llm/tasks/entity_linker/util.py
rmitsch Dec 7, 2023
ae2e837
Update spacy_llm/models/hf/base.py
rmitsch Dec 7, 2023
63367fa
Incorporate feedback.
rmitsch Dec 7, 2023
795f675
Update spacy_llm/tasks/raw/registry.py
rmitsch Dec 8, 2023
99fe286
Apply suggestions from code review
rmitsch Dec 8, 2023
e728e2c
Merge branch 'feat/inf-doc-len' into feat/raw-task
rmitsch Dec 8, 2023
4070efa
Fix tests.
rmitsch Dec 8, 2023
281998b
Remove boilerplate text in raw template.
rmitsch Dec 8, 2023
87776d3
Fix sharding test.
rmitsch Dec 8, 2023
191e42b
Merge branch 'develop' into feat/raw-task
rmitsch Dec 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update spacy_llm/tasks/entity_linker/util.py 
Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>
  • Loading branch information
@rmitsch @svlandeg
rmitsch and svlandeg authored Dec 7, 2023
commit 1763821b2101746a4b85f832d850ace3bbdb6cdd
2 changes: 1 addition & 1 deletion spacy_llm/tasks/entity_linker/util.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ def __call__(self, vocab: Vocab) -> Tuple[InMemoryLookupKB, DescFormat]:


def reduce_shards_to_doc(task: EntityLinkerTask, shards: Iterable[Doc]) -> Doc:
"""Reduces shards to docs for LemmaTask.
"""Reduces shards to docs for EntityLinkerTask.
task (EntityLinkerTask): Task.
shards (Iterable[Doc]): Shards to reduce to single doc instance.
RETURNS (Doc): Fused doc instance.
Expand Down
Loading