Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 18 additions & 8 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,24 +15,34 @@
</div>
<br>


The following guide will walk you through contributing new sources or changes to existing sources
and their demo pipelines and contains a troubleshooting section. Please also read
[DISTRIBUTION.md](docs/DISTRIBUTION.md) to understand how our sources are distributed to the users.
Refer to [BUILDING-BLOCKS.md](docs/BUILDING-BLOCKS.md) to learn about the basic building blocks of a
dlt source.

What do you can do here:
## Before you start
* 🚀 We are happy to accept **bugfixes and improvements** to existing sources!
* 📣 **We accept new sources but only those that cannot be easily implemented** via `REST API`, `sql_database` or `filesystem` or with **vibe coding**.
- see issues for source requests!
- queues and brokers like google pub sub or rabbitMQ
- SAP/ERP/HEALTH
- HR platforms
- Graph QL



## What you can do here

- Contribute a change to an existing verified source or its demo pipeline: Go to the
["Walktrough: Fix, improve, customize, document an existing source"](#walktrough-fix-improve-customize-document-an-existing-pipeline)
["Walkthrough: Fix, improve, customize, document an existing source"](#Walkthrough-fix-improve-customize-document-an-existing-pipeline)
section.
- Contribute a new verified source: Go to the
["Walktrough: Create and contribute a new source"](#walktrough-create-and-contribute-a-new-source)
["Walkthrough: Create and contribute a new source"](#Walkthrough-create-and-contribute-a-new-source)
section.
- Join our slack to get support from us by following the
[invitation link](https://dlthub.com/community).

## Walktrough: Fix, improve, customize, document an existing pipeline
## Walkthrough: Fix, improve, customize, document an existing pipeline

In this section you will learn how to contribute changes to an existing pipeline.

Expand All @@ -50,7 +60,7 @@ In this section you will learn how to contribute changes to an existing pipeline
4. Run the linter and formatter to check for any problems: `make lint-code`.
3. Proceed to the pull request section to [create a pull request to the main repo](#making-a-pull-request-to-the-main-repo-from-fork).

## Walktrough: Create and contribute a new source
## Walkthrough: Create and contribute a new source

In this section you will learn how to contribute a new source including tests and a demo pipeline
for that source. It is helpful to also read through the above section to see all the steps that are
Expand Down Expand Up @@ -342,7 +352,7 @@ In essence **DLT_SECRETS_TOML** is just your `secrets.toml` file and will be use


### Sharing and obtaining source credentials, test accounts, destination access
Typically we created a common test account for your source [before you started coding](#walktrough-create-and-contribute-a-new-source). This is an ideal situation - we can reuse your tests directly and can merge your work quickly.
Typically we create a common test account for your source [before you started coding](#Walkthrough-create-and-contribute-a-new-source). This is an ideal situation - we can reuse your tests directly and can merge your work quickly.

If you contributed a source and created own credentials, test accounts or test datasets please
include them in the tests or share them with `dlt` team so we can configure the CI job. If
Expand Down
22 changes: 14 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,23 +19,29 @@ This repository contains verified dlt sources and demo pipelines for each source

## How to add a verified source to your dlt project
`dlt` offers an `init` command that will clone and inject any source with an example pipeline from this repository into your project, setup the credentials and python dependencies. Please follow the step by step instructions in our [docs](https://dlthub.com/docs/walkthroughs/add-a-verified-source).

### We encourage you to hack the code
Verified sources code is added to your project with the expectation that you will hack or customize code yourself - as you need it.


## How to contact us and get help
Join our slack by following the [invitation link](https://dlthub.com/community)

- If you have added a source and something does not work, post in the `technical-help` channel
- If you want to contribute sources, customization or a fix, post in the `dlt-contributors` channel

## Reporting a source bug
Follow this link: [bug report template](https://github.com/dlt-hub/verified-sources/issues/new?template=bug-report.md)

## Requesting a source implementation
Follow this link: [source request template](https://github.com/dlt-hub/verified-sources/issues/new?template=request-new-source.md)

## How to contribute
* 🚀 We are happy to accept **bugfixes and improvements** to existing sources!
* 📣 **We accept new sources but only those that cannot be easily implemented** via `REST API`, `sql_database` or `filesystem` or with **vibe coding**.
- see issues for source requests!
- queues and brokers like google pub sub or rabbitMQ
- SAP/ERP/HEALTH
- HR platforms
- Graph QL
* 📣 **Before starting, announce your PR** with [source request template](https://github.com/dlt-hub/verified-sources/issues/new?template=bug-report.md)

Find step by step instruction as well as troubleshooting help in [CONTRIBUTING.md](CONTRIBUTING.md).

## Building blocks
Find a list of the building blocks of dlt with links to our official documentation in [BUILDING-BLOCKS.md](docs/BUILDING-BLOCKS.md).

## Building with LLMs
See README [`ai`](ai/README.md) folder for details.
35 changes: 0 additions & 35 deletions docs/BUILDING-BLOCKS.md

This file was deleted.

3 changes: 1 addition & 2 deletions sources/pipedrive/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,6 @@ def pipedrive_source(
Args:
pipedrive_api_key: https://pipedrive.readme.io/docs/how-to-find-the-api-token
since_timestamp: Starting timestamp for incremental loading. By default complete history is loaded on first run.
incremental: Enable or disable incremental loading.

Returns resources:
custom_fields_mapping
Expand Down Expand Up @@ -175,7 +174,7 @@ def parsed_mapping(


@dlt.resource(primary_key="id", write_disposition="merge")
def leads(
def leads(
pipedrive_api_key: str = dlt.secrets.value,
update_time: dlt.sources.incremental[str] = dlt.sources.incremental(
"update_time", "1970-01-01 00:00:00"
Expand Down
2 changes: 1 addition & 1 deletion tests/inbox/test_inbox.py
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ def test_parse_pdf() -> None:
def pdf_to_text(file_items: Sequence[FileItemDict]) -> Iterator[Dict[str, Any]]:
# extract data from PDF page by page
for file_item in file_items:
with file_item.open() as file:
with file_item.open(compression="disable") as file:
reader = PdfReader(file)
for page_no in range(len(reader.pages)):
# add page content to file item
Expand Down
Loading
Loading