RC: RDI in the cloud #570

kaitlynmichael · 2024-08-19T21:02:19Z

EDIT: Main staging links:
https://redis.io/docs/staging/DOC-4137/operate/rc/databases/rdi/
https://redis.io/docs/staging/DOC-4137/operate/rc/databases/rdi/define
https://redis.io/docs/staging/DOC-4137/operate/rc/databases/rdi/setup
https://redis.io/docs/staging/DOC-4137/operate/rc/databases/rdi/view-edit

github-actions · 2024-08-19T21:02:36Z

Staging links:
https://redis.io/docs/staging/DOC-4137/operate/rc/databases/rdi/
https://redis.io/docs/staging/DOC-4137/operate/rc/databases/rdi/define
https://redis.io/docs/staging/DOC-4137/operate/rc/databases/rdi/setup
https://redis.io/docs/staging/DOC-4137/operate/rc/databases/rdi/view-edit

andy-stark-redis

Couple of possible suggestions, but there are also some gltiches that need attention. Otherwise, looking great so far :-)

content/operate/rc/databases/rdi/_index.md

joniredis · 2024-08-28T17:05:10Z

content/operate/rc/databases/rdi/_index.md

+
+Only the following are supported at this time:
+
+- One source database connected to one target database


I would go with these limitations in this order:

Redis Cloud Pro tier

Ingest use case

AWS platform (source and target database)

One source database connected to one target database

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

CLAassistant · 2024-10-07T14:25:00Z

All committers have signed the CLA.

yaronp68 · 2024-10-09T14:37:02Z

content/operate/rc/databases/rdi/_index.md

+
+Using RDI with Redis Cloud simplifies managing your data integration pipeline. No need to worry about hardware or underlying infrastructure, as Redis Cloud manages that for you. Creating the data flow from source to target is much easier, and there are validations in place to reduce errors.
+
+## Data pipeline architecture


typically in cloud service, you don't explain about the architecture too much but more about the functionality. I think in this case what matters to the user:

The fact that it can start with "backfill" process taking a baseline snapshot of the desired data

then move automatically to track the changes for this dataset

guaranteed to deliver at least once each record unless the RDI database crushed (and then can go back to baseline)

can transform relational database rows to Hash or JSON in Redis

I'm pretty sure I cover the "backfill" process by using the term "ingest" and the "tracking changes" by talking about "change streaming" - are these meaningfully different terms?

Added Hash/JSON in 129d6af.

content/operate/rc/databases/rdi/_index.md

yaronp68 · 2024-10-09T14:39:31Z

content/operate/rc/databases/rdi/_index.md

+
+To create a new data pipeline, you need to:
+
+1. [Prepare your source database]({{<relref "/operate/rc/databases/rdi/setup">}}) and any associated credentials.


except for preparing we need to create the private link and the secret

Yes, that page also includes PrivateLink and creating the credentials secret.

content/operate/rc/databases/rdi/create.md

yaronp68 · 2024-10-09T14:45:33Z

content/operate/rc/databases/rdi/create.md

+At this point, Redis Cloud will attempt to create your new pipeline. 
+
+Pipelines are created in the background. While they're being created, you aren't allowed to make changes. This process will take some time.
+


Missing all the data set selection phases
Missing selecting the Redis data type

That's all covered in the Define data pipeline page. If we move it here, this page risks getting too long.

content/operate/rc/databases/rdi/define.md

yaronp68 · 2024-10-09T14:46:47Z

content/operate/rc/databases/rdi/setup.md

+
+See the [RDI architecture overview]({{< relref "/integrate/redis-data-integration/architecture#overview" >}}) for more information about CDC.
+
+## Share source database credentials


credentials and TLS certificates

That's covered in this section. I thought "credentials" in the header would include certificates without making the header too long.

content/operate/rc/databases/rdi/view-edit.md

andy-stark-redis

A few minor things (broken link does need fixing, though) but otherwise looking really good :-)

content/operate/rc/databases/rdi/define.md

andy-stark-redis · 2024-10-22T08:57:01Z

content/operate/rc/databases/rdi/define.md

+    Select **Add schema** to add additional database schemas. 
+
+    Select **Delete** to delete a schema. You must have at least one schema to continue.
+
+    After you've selected the schemas and tables you want to sync, select **Continue**.


Is this intended to be a sublist (bulletted)? I definitely think it should be because the formatting currently looks a bit odd without it.

Suggested change

Select **Add schema** to add additional database schemas.

Select **Delete** to delete a schema. You must have at least one schema to continue.

After you've selected the schemas and tables you want to sync, select **Continue**.

- Select **Add schema** to add additional database schemas.

- Select **Delete** to delete a schema. You must have at least one schema to continue.

- After you've selected the schemas and tables you want to sync, select **Continue**.

I think this will be a lot easier to parse once I add screenshots.

content/operate/rc/databases/rdi/provision.md

andy-stark-redis · 2024-10-22T09:01:25Z

content/operate/rc/databases/rdi/provision.md

+
+## Pipeline provisioning errors {#errors}
+
+Add errors here.


I'm guessing you haven't been provided with the list of errors yet :-)

Basically, yes. I need to go through the mockups again.

andy-stark-redis · 2024-10-22T09:06:29Z

content/operate/rc/databases/rdi/setup.md

+
+1. In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab.
+1. Select **Create data pipeline**.
+1. Select your source database type. The following database types are supported:


You also mention MariaDB and Aurora in the intro page. AFAIK, MariaDB is basically the same as MySQL, so I generally write MySQL/MariaDB together. Is Aurora specific to cloud RDI, with it being an Amazon product? Also, if we do support Aurora for this then do we need some extra instructions for it in the Prepare Source Databases section?

@yaronp68 - Can you elaborate more on the Aurora inclusion? Is it the same as mySQL/mariaDB, or do we need to add more information as Andy said?

andy-stark-redis · 2024-10-22T09:12:03Z

content/operate/rc/databases/rdi/view-edit.md

+    Select **Add schema** to add additional database schema.
+
+    Select **Delete** to delete a schema. You must have at least one schema to continue.
+
+    After you've selected the schemas and tables you want to sync, select **Continue**.


Sublist again, maybe?

Suggested change

Select **Add schema** to add additional database schema.

Select **Delete** to delete a schema. You must have at least one schema to continue.

After you've selected the schemas and tables you want to sync, select **Continue**.

- Select **Add schema** to add additional database schema.

- Select **Delete** to delete a schema. You must have at least one schema to continue.

- After you've selected the schemas and tables you want to sync, select **Continue**.

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

andy-stark-redis

One place where a link might not be right but otherwise looks good. I think there are still screenshots and other stuff still to add, so possibly too early to give final approval for the PR.

content/operate/rc/databases/rdi/define.md

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

begin index and setup drafts

c4d39e6

kaitlynmichael added do not merge yet rc Redis Cloud docs labels Aug 19, 2024

kaitlynmichael self-assigned this Aug 19, 2024

kaitlynmichael added 4 commits August 22, 2024 12:01

overview draft

d586971

edit desc

b5beb6a

fix relref

10d5de8

get started, doc link

c9e5161

kaitlynmichael requested a review from andy-stark-redis August 22, 2024 17:08

kaitlynmichael added 4 commits August 22, 2024 12:13

prepare source database with link

42fcff9

setup connectivity

17270d8

connectivity steps

9e0de8e

pane > box

efc8b0b

andy-stark-redis requested changes Aug 23, 2024

View reviewed changes

content/operate/rc/databases/rdi/_index.md Outdated Show resolved Hide resolved

content/operate/rc/databases/rdi/_index.md Outdated Show resolved Hide resolved

content/operate/rc/databases/rdi/_index.md Outdated Show resolved Hide resolved

Limitations draft

20c3050

joniredis reviewed Aug 28, 2024

View reviewed changes

cmilesb self-assigned this Aug 29, 2024

cmilesb and others added 3 commits August 30, 2024 14:51

Apply suggestions from code review

c3b7bca

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

Merge branch 'main' into DOC-4137

85c6d08

Edits and DB credentials changes

e938762

cmilesb requested review from joniredis and andy-stark-redis September 3, 2024 20:56

andy-stark-redis approved these changes Sep 4, 2024

View reviewed changes

cmilesb added 4 commits September 6, 2024 16:29

Add setup step and more edits

a5977a2

Move last step to provision

289f0e4

Merge branch 'main' into DOC-4137

1d6fce4

Add some of Yaron's suggestions to intro

272e518

cmilesb requested a review from yaronp68 September 13, 2024 20:46

cmilesb unassigned kaitlynmichael Sep 13, 2024

cmilesb added 8 commits September 20, 2024 17:04

Add define steps

a32204e

Merge branch 'main' into DOC-4137

07bc249

Merge branch 'main' into DOC-4137

bc41625

Fix relrefs

71f431c

stash commit

e8ea310

Add security information

860944a

Merge branch 'main' into DOC-4137

639fc50

Merge branch 'main' into DOC-4137

697d665

Add View/Edit

fa87f86

yaronp68 requested changes Oct 9, 2024

View reviewed changes

cmilesb added 5 commits October 11, 2024 12:46

Incorporate Yaron's feedback, part 1

49d1f54

Merge branch 'main' into DOC-4137

4abac23

stash commit

4d69274

More suggestions from Yaron

129d6af

Merge branch 'main' into DOC-4137

99229b1

cmilesb requested review from yaronp68 and andy-stark-redis October 21, 2024 19:08

andy-stark-redis requested changes Oct 22, 2024

View reviewed changes

cmilesb and others added 3 commits October 22, 2024 10:54

Apply suggestions from code review

c3e501e

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

Merge branch 'main' into DOC-4137

01907d8

Combine define and provision and remove errors section

c9469e0

cmilesb requested review from andy-stark-redis and mich-elle-luna October 25, 2024 19:51

andy-stark-redis reviewed Oct 28, 2024

View reviewed changes

content/operate/rc/databases/rdi/define.md Outdated Show resolved Hide resolved

cmilesb and others added 5 commits November 5, 2024 12:01

Merge branch 'main' into DOC-4137

d56188a

Add secret permissions and keys

f817848

Update content/operate/rc/databases/rdi/define.md

8f274f5

Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com>

Fix note

2d2820d

replace account ID in resource permissions

46603a6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RC: RDI in the cloud #570

RC: RDI in the cloud #570

kaitlynmichael commented Aug 19, 2024 •

edited by cmilesb

Loading

github-actions bot commented Aug 19, 2024 •

edited

Loading

andy-stark-redis left a comment

joniredis Aug 28, 2024

CLAassistant commented Oct 7, 2024 •

edited

Loading

yaronp68 Oct 9, 2024

cmilesb Oct 21, 2024

cmilesb Oct 21, 2024

yaronp68 Oct 9, 2024

cmilesb Oct 9, 2024

yaronp68 Oct 9, 2024

cmilesb Oct 9, 2024

yaronp68 Oct 9, 2024

cmilesb Oct 9, 2024

andy-stark-redis left a comment

andy-stark-redis Oct 22, 2024

cmilesb Oct 24, 2024

andy-stark-redis Oct 22, 2024

cmilesb Oct 22, 2024

andy-stark-redis Oct 22, 2024

cmilesb Oct 24, 2024

andy-stark-redis Oct 22, 2024

andy-stark-redis left a comment


		Only the following are supported at this time:

		- One source database connected to one target database


		Using RDI with Redis Cloud simplifies managing your data integration pipeline. No need to worry about hardware or underlying infrastructure, as Redis Cloud manages that for you. Creating the data flow from source to target is much easier, and there are validations in place to reduce errors.

		## Data pipeline architecture


		To create a new data pipeline, you need to:

		1. [Prepare your source database]({{<relref "/operate/rc/databases/rdi/setup">}}) and any associated credentials.

		At this point, Redis Cloud will attempt to create your new pipeline.

		Pipelines are created in the background. While they're being created, you aren't allowed to make changes. This process will take some time.


		See the [RDI architecture overview]({{< relref "/integrate/redis-data-integration/architecture#overview" >}}) for more information about CDC.

		## Share source database credentials

RC: RDI in the cloud #570

Are you sure you want to change the base?

RC: RDI in the cloud #570

Conversation

kaitlynmichael commented Aug 19, 2024 • edited by cmilesb Loading

github-actions bot commented Aug 19, 2024 • edited Loading

andy-stark-redis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CLAassistant commented Oct 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andy-stark-redis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andy-stark-redis left a comment

Choose a reason for hiding this comment

kaitlynmichael commented Aug 19, 2024 •

edited by cmilesb

Loading

github-actions bot commented Aug 19, 2024 •

edited

Loading

CLAassistant commented Oct 7, 2024 •

edited

Loading