Skip to content

chore: Move chef-cli docs to fern #186

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 28 commits into from
Apr 1, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
314e918
Fix bash code
tilenkavcic Mar 29, 2025
7b9dc82
Make diagram look better
tilenkavcic Mar 29, 2025
6364bd4
Adjust colors
tilenkavcic Mar 29, 2025
7472f76
Merge step-by-step with existing
tilenkavcic Mar 29, 2025
cd659e9
Add missing things to metadata extraction
tilenkavcic Mar 29, 2025
8db455d
Add schema and validation info
tilenkavcic Mar 29, 2025
4d4331a
Make faq accordion group
tilenkavcic Mar 29, 2025
03e97fd
Improve metadata extraction
tilenkavcic Mar 29, 2025
6b87ad4
Add IDM and the rest from chef
tilenkavcic Mar 29, 2025
0f4f7fa
Fix path
tilenkavcic Mar 29, 2025
0e035dd
Revert diagram change
tilenkavcic Mar 29, 2025
5c63e01
Update faq
tilenkavcic Mar 29, 2025
f1066ea
Proofread IDM
tilenkavcic Mar 29, 2025
b8dd7b5
Proofread fax and metadata extraction
tilenkavcic Mar 29, 2025
0a4c788
Apply suggestions from code review
tilenkavcic Mar 29, 2025
3c1705a
Apply suggestions from EkLine
tilenkavcic Mar 29, 2025
5280d82
Fix wrapping
tilenkavcic Mar 29, 2025
4880a26
Fix title
tilenkavcic Mar 30, 2025
bdf4944
Refine styling
tilenkavcic Mar 30, 2025
7ba56de
Add code highlight
tilenkavcic Mar 30, 2025
5300b32
Small visual change
tilenkavcic Mar 30, 2025
9496ebd
Refine rich text docs
tilenkavcic Mar 30, 2025
91bae49
Don't use ellipsis
tilenkavcic Mar 31, 2025
5520847
Address comments
tilenkavcic Mar 31, 2025
178d6ae
Fix getting started overview
tilenkavcic Mar 31, 2025
4ff66d9
Apply EkLine suggestions
tilenkavcic Mar 31, 2025
f3ffd4e
Address comments
tilenkavcic Apr 1, 2025
0b97d10
Adjust code indent, address comments
tilenkavcic Apr 1, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 25 additions & 8 deletions fern/docs/pages/airdrop/data-extraction.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@

In the data extraction phase, the extractor is expected to call the external system's APIs
to retrieve all the items that were updated since the start of the last extraction.
If there was no previous extraction (the current run is an initial import),
Expand Down Expand Up @@ -43,12 +42,11 @@ Each artifact is submitted with an `item_type`, defining a separate domain objec
external system and matching the `record_type` in the provided metadata.
Item types defined when uploading extracted data must validate the declarations in the metadata file.

Extracted data must be normalized.

Extracted data must be normalized:
- Null values: All fields without a value should either be omitted or set to null.
For example, if an external system provides values such as "", -1 for missing values,
For example, if an external system provides values such as "", 1 for missing values,
those must be set to null.
- Timestamps: Full-precision timestamps should be formatted as RFC3999 (`1972-03-29T22:04:47+01:00`),
- Timestamps: Full-precision timestamps should be formatted as RFC3339 (`1972-03-29T22:04:47+01:00`),
and dates should be just `2020-12-31`.
- References: references must be strings, not numbers or objects.
- Number fields must be valid JSON numbers (not strings).
Expand All @@ -58,7 +56,7 @@ Each line of the file contains an `id` and the optional `created_date` and `modi
in the beginning of the record.
All other fields are contained within the `data` attribute.

```json
```json {2-4}
{
"id": "2102e01F",
"created_date": "1972-03-29T22:04:47+01:00",
Expand All @@ -74,9 +72,28 @@ All other fields are contained within the `data` attribute.
}
```

## Validating extracted data

Extracted artifacts can be validated with the chef-cli using the following command:

```bash
chef-cli validate-data -m external_domain_metadata.json -r issue < extractor_issues_2.json
```

Extracted artifacts can be validated with the `chef-cli` using the following command:
You can also generate example data to show the format the data has to be normalized to, using:

```bash
$ chef-cli validate-metadata -m external_domain_metadata.json -r issue < extractor_issues_2.json
echo '{}' | chef-cli fuzz-extracted -r issue -m external_domain_metadata.json > example_issues.json
```

## Deploying and testing the snap-in

Once you have implemented data extraction, you should deploy your snap-in to your test organization and run an import.

To deploy the snap-in, run `make auth` and `make deploy` in the snap-in repository.
Then, activate the snap-in by running `devrev snap_in activate`.

After activation, you can create an import in the DevRev UI, which will initially reach the 'waiting for user input' stage.
During this phase, you can verify your data extraction implementation is working correctly.

Relevant documentation can be found in the [Snap-in development](/snapin-development/locally-testing-snap-ins) section.
59 changes: 59 additions & 0 deletions fern/docs/pages/airdrop/data-model/mapping-reasons.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
Airdrop uses transformation methods to map data from one data model to another.
These methods operate at the individual field level, meaning they transform data field-by-field rather than transforming entire records at once.
Each transformation method has specific requirements regarding the field types and data formats it can work with.
If the source data field does not meet these requirements for a particular transformation method, that mapping option is not available for selection during the mapping configuration process.

There are several reasons why some mappings might be unavailable.

1. A common reason is mismatch of types. For example, if a DevRev field is expected to be `rich_text`,
but the field is set as `text` mapping to some fields is unavailable.
Refer to the [supported types](../metadata-extraction.mdx#declare-fields-for-each-record-type)
section and the general DevRev documentation for more information.
2. Only references can be mapped to references. Ensure that source system fields are correctly
mapped to reference fields in DevRev.
3. Support for the `struct` type is limited. Marking a field as a struct in the metadata schema will
make it unavailable for mapping outside of using the custom jq transformation method.
Refer to the [metadata tips](../faq.mdx#metadata-extraction) for more information.
4. Links are supported only on works and conversations.

## Transformation methods
The following tables outline the available transformation methods categorized by their applicable field types:

### Custom fields transformation methods

| Method | Description | Requirements |
|--------|-------------|-------------|
| `make_constrained_simple_value` | Propagates validation constraints from the external system and enforces those in DevRev | - |
| `make_enum` | Produces an enum field | External field must be of type enum |
| `make_uenum` | Produces an enum field | External field must be of type enum |
| `reference_custom_field` | Produces a reference field | External field must be of type reference |

### Metadata extraction transformation methods

| Method | Description | Requirements |
|--------|-------------|-------------|
| `make_custom_stages` | Makes custom stages | Requires stage diagram data in the domain metadata |
| `map_enum` | Produces an enum field | External field must be of type enum |
| `map_roles` | Maps permission roles from external to DevRev format | - |
| `use_as_array_value` | Produces an array field | External field must be a scalar (single-value) |
| `use_devrev_record` | Enables use of a fixed reference to something in DevRev | DON should be of the right type |
| `use_directly` | Identity operator that returns exactly the input | External field must be of the same type as the DevRev field; if external field is an array, internal field must also be an array |
| `use_fixed_value` | Produces a fixed value in DevRev | Only available for boolean or enum DevRev fields |

### Custom and stock fields transformation methods

| Method | Description | Requirements |
|--------|-------------|-------------|
| `use_rich_text` | Produces a rich text field | External field must be of type `rich_text` |

### Constructed custom fields transformation methods

| Method | Description | Requirements |
|--------|-------------|-------------|
| `construct_text_field` | Produces a text field | External field must be of type text |

### Universal transformation method

| Method | Description | Requirements |
|--------|-------------|-------------|
| `use_raw_jq` | Enables the use of `jq` to transform data | Can be used on all fields, but should be used sparingly, only if no other transformation method is available |
22 changes: 22 additions & 0 deletions fern/docs/pages/airdrop/data-model/permissions.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
DevRev uses the concept of permissions to control access to records.
Permissions are associated with users or groups and define the level of access they have to leaf types and their fields.

## Article permissions

{/* TODO: more on article permissions */}
Articles can be shared with users or groups.

The `shared_with` field allows to specify the permission level for each user or group using the `permission` type.
The `permission` type is a special structure associating a reference to an user-like record type (the field `member_id`)
with an `enum` value that provides the user with the role or permission level.
## Platform groups

Platform groups are groups automatically created by the DevRev platform for every organization.
For example, DevRev provides *Dev users* and *Rev users* groups that contain all the Dev and Rev users respectively.

We allow developers to map their external default groups to platform groups.
This functionality allows for flexible integration with the platform's permission system while allowing for re-mapping by the end user.

To implement this mapping, developers need to define a new object in external metadata with one enum field containing the possible values of default groups available in the external system. During the initial domain mapping, developers can map this object to the *platform groups* object in DevRev. The platform groups object can then be referenced in other objects, such as in the `shared_with` field of articles.

{/* ### Authorization policy */}
160 changes: 160 additions & 0 deletions fern/docs/pages/airdrop/data-model/rich-text.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
Rich text is used in DevRev to represent text fields that can be formatted and can contain mentions, for example, description of an issue or body of a conversation.

A simple rich text looks like one markdown string wrapped in an array: `["Hello **world**!"]`.
Markdown must conform to [CommonMark Spec v0.30](https://spec.commonmark.org/0.30).

## Rich text mentions

To support mentions, `rich_text` can be formatted as an array of strings and mention objects like so:
```json
[
"Hello ",
{"ref_type":"external_user_type", "id":"1...", "fallback_record_name": "John Doe"},
"how are you?"
]
```


Mention represents any mention (user, issue, etc.) in rich text and is defined as:
| Field | Type | Required | Description |
| ---------------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `id` | String | Yes | Identifier of the item being mentioned. This could be a user ID or any other identifier, in the format used by the source system. |
| `ref_type` | String | Yes | Type of the item being mentioned. Examples include "issue", "comment", etc. The recipe converts this according to user mappings. |
| `fallback_record_name` | String | No | The text to display if the mention cannot be resolved. This could be a user's display name or a ticket title, for instance. |

In reverse, the loader should expect the following structure:
```json
{
"type": "rich_text",
"content": [
"Hello ",
{
"ref_type": "external_user_type",
"id": "don:identity:dvrv-us-1:devo/xyz:devu/1",
"fallback_record_name": "John Smith"
},
"how are you?"
]
}
```

## Importing articles

An article is a document containing essential information about the company's products, services, and processes.
Articles support Markdown as well as HTML.

Articles support mentions to artifacts and articles. An inline attachment must be mapped to an artifact.
A link to another article must be mapped to an article.

### Managing permissions

You can manage article permissions in the `shared_with` field. Permissions can reference users, groups, and [platform groups](./platform_groups.md).

### Inline attachments
If an inline attachment is hosted in the source system, it must be created as an artifact in DevRev.
The same link cannot be used as the attachment is deleted in the source system when our customers deactivate the account.
However, creating an artifact is not enough. The artifact must be linked in the appropriate place in the article content.

The following HTML example shows an inline attachment:
```html
<img src="don:core:dvrv-us-1:devo/0:artifact/1" alt="Alt Text"/>
```

The following Markdown example shows an inline attachment:
```markdown
![Alt Text](don:core:dvrv-us-1:devo/0:artifact/1)
```

Let's say the content of your external system looks like this:
```html
<p>
This is an article with one image.
</p>
<p>
<img src="https://devrev.zendesk.com/hc/article_attachments/29908544740244" alt="download.jpeg">
</p>
```

The content in DevRev should look like this:
```html
<p>
This is an article with one image.
</p>
<p>
<img src="don:core:dvrv-us-1:devo/0:artifact/1" alt="download.jpeg">
</p>
```

`don:core:dvrv-us-1:devo/0:artifact/1` is the ID of the artifact created in DevRev corresponding to the attachment with ID `29908544740244` in the external source system.
To achieve this, you need to transform the content of the article to the following JSON:
```json
[
"<p>This is an article with one image.</p><p><img src=\"",
{
"ref_type": "artifact",
"id": "29908544740244",
"fallback_record_name": "<fallback link>"
},
"\" alt=\"download.jpeg\"></p>"
]
```

The `ref_type` should be set to artifact and the ID should be the ID of the attachment in the external source system.
The platform replaces the mention block with the ID of the corresponding artifact.
The resolved value is not wrapped in double quotes.

### Links to other articles
If there is a link to another article in the content of the article, you need to create a mention to the article.
The link must be to an article that was either created in previous syncs or is created in the current sync.
At the extractor stage, it is impossible to predict the ID of the article that is created in DevRev.
This is why the platform handles this. This feature is only available for the HTML format.
However, since Markdown can contain HTML, you can use the same approach for Markdown as well.

The following HTML example shows a link to another article:
```html
<a data-article-id="don:core:dvrv-us-1:devo/0:article/10" href="/ART-10" target="_self">
Contact our Support Team
</a>
```

Let's say the content of your external system looks like this:
```html
<p>
You can create an account and log-in
<a href="https://devrev.zendesk.com/hc/en-us/articles/360059607772" target="_self">
only
</a>
with the company email.
</p>
```

The content in DevRev should look like this:
```html
<p>
You can create an account and log-in
<a data-article-id="don:core:dvrv-us-1:devo/0:article/10" href="/ART-10" target="_self">
only
</a>
with the company email.
</p>
```

`don:core:dvrv-us-1:devo/0:article/10` is the ID of the article created in DevRev corresponding to the article with ID `360059607772` in the external source system.
To achieve this, you need to transform the content of the article to the following JSON:
```json
[
"You can create an account and log-in <a data-article-id=\"",
{
"ref_type": "article",
"id": "360059607772",
"fallback_record_name": "<fallback article ID>"
},
"\" target=\"_self\"> only</a> with the company email."
]
```

The `ref_type` should be set to the item type in the external system that is being mapped to articles.
For example, if you're importing documents from the external system as articles,
the `ref_type` should be set to documents. The ID should be the ID of the item in the external source system.

The platform replaces the mention block with the ID of the corresponding article in DevRev as well as adds the href attribute with the appropriate value.
Loading
Loading