Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added support for automatic_label_demo.py without chatgpt #142

Merged
merged 3 commits into from
Apr 18, 2023

Conversation

lsch0lz
Copy link
Contributor

@lsch0lz lsch0lz commented Apr 16, 2023

Hi! Since it's only possible to use automatic_label_demo.py with a paid OpenAI Account I implemented a method to use the functionality without ChatGPT.
To achieve this I used NLTK to extract nouns from the caption. I also used a Lemmatizer to improve the produced tags.

Your implementation with ChatGPT:
automatic_label_output_demo3

My implementation without ChatGPT:
automatic_label_output

I implemented the feature in way to let the user choose if they want to use ChatGPT or not. If the script get startet without the openai_key by default NLTK will be used. If the user provides a valid key ChatGPT will be used.

Let me know what you think about this feature.

@Andy1621
Copy link
Contributor

Andy1621 commented Apr 16, 2023

@lsch0lz Thanks for your commit! I also try to use nltk in the begining, but in some cases, it's hard to extract the nouns directly.
image
where the group is not a current num, but it is also hard for our ChatGPT prompt now hhh.
However, it often works. By the way, in my environment, it needs to download the packages manually.
Maybe you can add more instructions in README. And then I accept the change.

wget https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/packages/tokenizers/punkt.zip
# unzip in '~/nltk_data/tokenizers'
wget https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/packages/taggers/averaged_perceptron_tagger.zip
# unzip in '~/nltk_data/taggers'
wget https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/packages/corpora/wordnet.zip
# unzip in 'nltk_data/corpora'

@lsch0lz
Copy link
Contributor Author

lsch0lz commented Apr 16, 2023

@Andy1621 Thanks for the Feedback!
I totally forgot that I already downloaded the NLTK packages. Now all the necessary packages are downloaded when NLTK is used. I think it's more elegenat, than let the user download all the packages manually.
Let me know what you think about that.

@lsch0lz
Copy link
Contributor Author

lsch0lz commented Apr 16, 2023

Also what do you mean by:

where the group is not a current num

If I understand the intention correct group is a noun and should be detected as a tag in that case, right?

Copy link

@pierizvi pierizvi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks Good To Test 🛠

@Andy1621
Copy link
Contributor

@lsch0lz Thanks! For group, unfortunately, I found that some caption models generate unrelated nouns, such as a group of, a family of and so on. These nouns are used as adjectives and should not be detected. Like in Tag2Text, it generates the caption as follows,
image
We should only detect the bear, shore and water.

However, using nltk can handle most of cases, while ChatGPT can handle some potentially difficult captions with fine-grained prompts, for example ignore those phrases like 'a group (or other noun) of'.
@rentainhe I approve to accept it!

@rentainhe
Copy link
Collaborator

Thanks for this PR, I'm going to merge it!

@rentainhe rentainhe merged commit bc526e6 into IDEA-Research:main Apr 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants