Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

15275 feature request make all events accessible like mappings #15310

Merged
merged 4 commits into from
Aug 13, 2024
Merged

15275 feature request make all events accessible like mappings #15310

merged 4 commits into from
Aug 13, 2024

Conversation

nerdai
Copy link
Contributor

@nerdai nerdai commented Aug 12, 2024 •
edited
Loading

Description

  • Fixing dict interface for StartEvent (maximum recursion error with __getattr__ and __setattr__).
  • Updates Event type to take on the dict-like interface that was priorly only set for StartEvent -- StartEvent is now a light subclass of Event.
  • Also, adds DictLikeEvent (naming?) as an event type that can be used generally with a dict-like interface.

Closes #15275 #15348

New Package?

Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

  • No

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

  • NA

Type of Change

Please delete options that are not relevant.

  • New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • Added new unit/integration tests
  • Added new notebook (that tests end-to-end)
  • I stared at the code and made sure it makes sense

@nerdai nerdai linked an issue Aug 12, 2024 that may be closed by this pull request
[Feature Request]: Make all events accessible like mappings #15275
Closed
@dosubot dosubot bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Aug 12, 2024
@nerdai nerdai requested a review from masci August 12, 2024 19:47
masci
masci requested changes Aug 13, 2024
View reviewed changes
Copy link
Member

@masci masci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noice! Left a couple of comments

llama-index-core/llama_index/core/workflow/events.py Outdated
Comment on lines 31 to 69
private_attrs = {
k: v for k, v in params.items() if k in self.__private_attributes__
}
for private_attr, value in private_attrs.items():
super().__setattr__(private_attr, value)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think we can merge these two for loops into one:

for k,v in params.items():
    if k in self.__private__attributes__:
        super().__setattr__(k, v)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call. I have to modify this slightly, since we need to ensure that we call super().__init__() first before setting any private attributes (order didn't matter I think for Pydantic V1 but it does for Pydantic V2).

llama-index-core/llama_index/core/workflow/events.py Outdated
@@ -10,28 +10,55 @@ class Config:
arbitrary_types_allowed = True


class StartEvent(Event):
"""StartEvent is implicitly sent when a workflow runs. Mimics the interface of a dict."""
class DictLikeEvent(Event):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we just call this Event? I think assuming that every event is dict-like lowers the cognitive load for the users.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah good call!

@masci masci self-assigned this Aug 13, 2024
@nerdai nerdai force-pushed the 15275-feature-request-make-all-events-accessible-like-mappings branch from 97277b2 to 9b0efc4 Compare August 13, 2024 15:35
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Aug 13, 2024
@nerdai
Copy link
Contributor Author

nerdai commented Aug 13, 2024

@masci: made the changes, and also added a Class docstring with examples + unit test for Event

masci
masci requested changes Aug 13, 2024
View reviewed changes
Copy link
Member

@masci masci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One small comment and we're good to go!

llama-index-core/tests/workflow/test_event.py Outdated
from typing import Any


class TestEvent(Event):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: pytest will try to run any class named TestSomething it finds like a test case and emit a warning. I usually prefix an underscore like _TestEvent

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ohhhh my bad, I should've known -- merci!

@nerdai
Copy link
Contributor Author

nerdai commented Aug 13, 2024

@masci -- all done!

masci
masci approved these changes Aug 13, 2024
View reviewed changes
Copy link
Member

@masci masci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✨

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Aug 13, 2024
@nerdai nerdai merged commit 627d32a into main Aug 13, 2024
8 checks passed
@nerdai nerdai deleted the 15275-feature-request-make-all-events-accessible-like-mappings branch August 13, 2024 16:19
nerdai added a commit that referenced this pull request Aug 20, 2024
@nerdai @Tonyrj3268
@rucinsk1 @detain @masci @dependabot @ravi03071991 @chetanchoudhary @afalf @openvino-dev-samples @pradeep-suresh2002 @logan-markewich @iansrobinson @bechbd @jerryjliu @AliAbbasi @IANTHEREAL @AshishSardana @mvaquero-atlasti @sagarjadhavcalsoft @Anirudh31415926535 @nightosong @jonathanhliu21 @nirga @KirillKukharev @preprocess-co @pitchdarkdata @HollowMan6
Pydantic V2 (llama-index-core) (#15282)
6efeee0 
* update to v2

* update deps, fix __modify_schema__

* model_validator instead of root_validator

* field_validator over validator

* TypeAdapter over parse_obj_as

* wip model_dump_json over json

* wip

* wip

* move private vars after super init

* move private vars after super init

* populate_by_name over allow_population_by_field_name

* add str(v) in legacy_json_to_doc of docstore/utils module

* add Annotated types

* use annotation

* transformations passing

* pydantic annotations module

* more callback fixes

* update lock

* dict shouldn't use by_alias

* add default callables

* add default callables

* wip llms

* annotations for LLMs to add class name in model_dump

* add model_validator to LLM

* update __pydantic_private__

* make callback manager optional for llms

* add field_serialization

* add field_serialization

* in favor of definitions

* fix pydantic validation errors

* fix dict interface

* pants

* try typing_extensions

* fix optimum

* fix optimum

* 15275 feature request make all events accessible like mappings (#15310)

* add DictLikeEvent

* remove test event

* cr

* cr

* feat: make send_event assign the target step (#15259)

* feat: make send_event assign the target step

* fix: change exception type and add else block

* use __private_attributes__

* bug: Semantic double merging splitter creates chunks larger thank chunk size (#15188)

fix bug with semantic double merging splitter creating chunks larger than max_chunk_size

Co-authored-by: Konrad Rucińsk <konrad.rucinski@bitpeak.pl>

* Update code_hierarchy.py adding php support (#15145)

* Update code_hierarchy.py adding php support

* Update code_hierarchy.py fixing linting error

* Update pyproject.toml bumping version

* test core only

* revert back integration changes

* fix annotations

* impl __get_pydantic_core_schema__ and __get_pydantic_json_schema__ for BaseOutputParser

* impl __get_pydantic_core_schema__ and __get_pydantic_json_schema__ for CallbackManager

* use CallbackManager instead of Annotation

* cleanup

* remove pydantic_annotations module

* remove deprecated BaseConfig from pydantic bridge

* chore: bump deprecated actions (#15331)

bump deprecated actions

* chore: let dependabot monitor Github actions (#15360)

let dependabot monitor Github actions

* chore(deps): bump github/codeql-action from 2 to 3 (#15369)

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2 to 3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@v2...v3)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump JRubics/poetry-publish from 1.17 to 2.0 (#15370)

* Add citation query engine with workflows (#15372)

* Add citation query engine with workflows

* Add descriotion, cookbook link, google colab

* decouple ingest from workflows

* Update description

* Add docstring for create_citations_nodes function

* GCSReader: Implementing ResourcesReaderMixin and FileSystemReaderMixin (#15365)

* fix bug when caling llama-index-postprocessor-dashscope-rerank (#15358)

* update HF's completion_to_prompt (#15354)

* Feat: Wrapper for SambaNova (Sambaverse and SambaStudio) with Llama-index (#15220)

* v0.10.66 (#15403)

* Add JSONAlyze Query Engine using workflows cookbook (#15408)

* Add JSONAlyze Query Engine

* Update with review

* Use backticks to escape label (#15324)

* update pydantic dep

* Fixed issue #15414 and added ability to do partial matchfor Neptune Analytics (#15415)

* escape json in structured llm (#15404)

* update completion_to_prompt of HF pipeline (#15437)

* Fix missing 'self' in example notebook. (#15429)

* Add workaround for Handling MySQL Protocol Limitations in TiDB Vector with insert_batch_size (#15428)

* Cleanlab's cookbook (#15352)

* When building a vector store with use_async=True it doesn't set a TTL on records (#15333)

* Vertexai embedding enhancement (#15367)

* feat: add tool calling for cohere (#15144)

* Add get/set API to the Context and make it coroutine-safe (#15152)

* make the Context coroutine-safe

* remove parent property

* change API

* docs

* use context manager and add unit tests

* Fix LangChainDeprecationWarning (#15397)

* Update langchain.py

fix LangChainDeprecationWarning of ChatMessageHistory

* Fix LangChainDeprecationWarning 

Handling different versions

* Fix LangChainDeprecationWarning 

fix changes

* fix linting

* Ports over LongRAGPack, Corrective RAG Pack, and Self-Discover Pack to Workflows (#15160)

* settings, secret, and start using SerializeAsAny

* wip

* serialize as any T

* wip, SerializeAsAny

* wip, SerializeAsAny

* wip, SerializeAsAny

* chore: remove leftover occurences of `GradientFinetuneEngine` (#15456)

* Exposed NeptuneQueryException and added additional debug information (#15448)

* docs: promote openllmetry to one-click integration (#15443)

* feat: Integrating Azure OpenAI Finetuning (#15297)

* avoid nltk 3.9 since its broken (#15473)

* Add GigaChat embedding (#15278)

* feat: support to configure the base url of jina.ai (#15431)

* Enhance PreprocessReader (#15302)

* v0.10.67 (#15474)

* wip, SerializeAsAny

* GenericModel, ValidationError import, model_rebuild

* handle warning of protected  prefix on fields

* wip ConfigDict

* use ollama client for embeddings (#15478)

* wip ConfigDict

* gte, lte

* Bug-15158: Fixing the issues in loading file paths (#15311)

* Bug-15158: Fixing the issues in loading file paths

* Bump up the version in pyproject.toml

* Add llama-index-graph-stores-neo4j dependency in GraphRAG v2 (#15479)

Signed-off-by: Hollow Man <hollowman@opensuse.org>

* wip model_dump over dict

* model_dump over dict

* llm_rerank conflict

* model_json_dump over dict and model_copy over copy

* model_validate over parse_obj

* model_validate_json over parse_raw

* model_dump_json over json

* wip

* revert to see if breaking

* add back previous changes with fixed BUILD

* make default_id_func serializable

* remove type param from Field

* make CallbackManager not optional

* implement model_serializer to override model_dump for nested classes

* implement model_serializer to override model_dump for nested classes

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Hollow Man <hollowman@opensuse.org>
Co-authored-by: Lin Chia Cheng <74391293+Tonyrj3268@users.noreply.github.com>
Co-authored-by: Konrad Ruciński <46794180+rucinsk1@users.noreply.github.com>
Co-authored-by: Konrad Rucińsk <konrad.rucinski@bitpeak.pl>
Co-authored-by: Joe Huss <detain@interserver.net>
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ravi Theja <ravi03071991@gmail.com>
Co-authored-by: Chetan Choudhary <chetanchoudhary975@gmail.com>
Co-authored-by: afalf <101537385+afalf@users.noreply.github.com>
Co-authored-by: Ethan Yang <ethan.yang@intel.com>
Co-authored-by: pradeep-suresh2002 <124234409+pradeep-suresh2002@users.noreply.github.com>
Co-authored-by: Logan <logan.markewich@live.com>
Co-authored-by: Ian Robinson <iansrobinson@gmail.com>
Co-authored-by: bechbd <bechbd@users.noreply.github.com>
Co-authored-by: Jerry Liu <jerryjliu98@gmail.com>
Co-authored-by: Ali Abbasi <abbasi.ali.tab@gmail.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>
Co-authored-by: Ashish Sardana <ashishsardana21@gmail.com>
Co-authored-by: mvaquero-atlasti <147490466+mvaquero-atlasti@users.noreply.github.com>
Co-authored-by: Sagar Jadhav <108457149+sagarjadhavcalsoft@users.noreply.github.com>
Co-authored-by: Anirudh31415926535 <anirudh@cohere.com>
Co-authored-by: guodong <songoodong@163.com>
Co-authored-by: Jonathan Liu <81734282+jonathanhliu21@users.noreply.github.com>
Co-authored-by: Nir Gazit <nirga@users.noreply.github.com>
Co-authored-by: Dongwoo Jeong <dongwoo.jeong@lge.com>
Co-authored-by: Kirill <58888049+KirillKukharev@users.noreply.github.com>
Co-authored-by: Cheese <qizhi.wang@pingcap.com>
Co-authored-by: preprocess-co <137915090+preprocess-co@users.noreply.github.com>
Co-authored-by: ReviewBuddy <91291176+pitchdarkdata@users.noreply.github.com>
Co-authored-by: ℍ𝕠𝕝𝕝𝕠𝕨 𝕄𝕒𝕟 <hollowman@opensuse.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@masci masci masci approved these changes

Assignees

@masci masci

Labels
lgtm This PR has been approved by a maintainer size:L This PR changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature Request]: Make all events accessible like mappings
2 participants
@nerdai @masci