Skip to content
This repository has been archived by the owner on Mar 1, 2024. It is now read-only.

Add args of trafilaturas function extract to function load_data #872

Merged
merged 2 commits into from
Jan 19, 2024

Conversation

marcelreschke
Copy link
Contributor

@marcelreschke marcelreschke commented Jan 17, 2024

Description

Add args of trafilaturas function extract to function load_data

I added the following args:

  • include_comments (bool, optional): Include comments in the output. Defaults to True.
  • output_format (str, optional): Output format. Defaults to 'txt'.
  • include_tables (bool, optional): Include tables in the output. Defaults to True.
  • include_images (bool, optional): Include images in the output. Defaults to False.
  • include_formatting (bool, optional): Include formatting in the output. Defaults to False.
  • include_links (bool, optional): Include links in the output. Defaults to False.

Type of Change

Please delete options that are not relevant.

  • Bug fix / Smaller change

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • Added new unit/integration tests
  • Added new notebook (that tests end-to-end)
  • I stared at the code and made sure it makes sense

Suggested Checklist:

  • I have added a library.json file if a new loader/tool was added
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I ran make format; make lint to appease the lint gods

Added args:
- include_comments (bool, optional): Include comments in the output. Defaults to True.
- output_format (str, optional): Output format. Defaults to 'txt'.
- include_tables (bool, optional): Include tables in the output. Defaults to True.
- include_images (bool, optional): Include images in the output. Defaults to False.
- include_formatting (bool, optional): Include formatting in the output. Defaults to False.
- include_links (bool, optional): Include links in the output. Defaults to False.
@anoopshrma
Copy link
Collaborator

Thank you for the contribution! All looks good only linting failed.
Could you take a look into that.
Thanks!

@marcelreschke
Copy link
Contributor Author

I have reformatted the failing file using black.

@nerdai
Copy link
Contributor

nerdai commented Jan 19, 2024

looks like everything passes now - merging! Thanks @marcelreschke!

@nerdai nerdai merged commit 0db48a5 into run-llama:main Jan 19, 2024
3 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants