Skip to content

feat(minio): log MinIO actions with requester #972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Feb 11, 2025
Merged

feat(minio): log MinIO actions with requester #972

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6e9a322
chore(component): regenerate component docs
jvallesm Feb 11, 2025
24bb847
feat(minio): log MinIO actions with requester
jvallesm Feb 11, 2025
7539afd
chore(worker): simplify MinIO activity params
jvallesm Feb 11, 2025
2a345d1
fix(ci): remove duplicated env variable
jvallesm Feb 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/integration-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,6 @@ jobs:
EDITION=local-ce:test \
RAY_LATEST_TAG=latest \
RAY_RELEASE_TAG=${RAY_SERVER_VERSION} \
COMPONENT_ENV=.env.component-test \
docker compose -f docker-compose.yml -f docker-compose-latest.yml up -d --quiet-pull
COMPOSE_PROFILES=all \
EDITION=local-ce:test \
Expand Down
4 changes: 2 additions & 2 deletions cmd/main/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -512,8 +512,8 @@ func main() {
lw.RegisterActivity(cw.UpdatePipelineRunActivity)
lw.RegisterActivity(cw.UpsertComponentRunActivity)

mw.RegisterActivity(cw.UploadOutputsToMinioActivity)
mw.RegisterActivity(cw.UploadRecipeToMinioActivity)
mw.RegisterActivity(cw.UploadOutputsToMinIOActivity)
mw.RegisterActivity(cw.UploadRecipeToMinIOActivity)
mw.RegisterActivity(cw.UploadComponentInputsActivity)
mw.RegisterActivity(cw.UploadComponentOutputsActivity)

Expand Down
5 changes: 3 additions & 2 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ require (
github.com/influxdata/influxdb-client-go/v2 v2.12.3
github.com/instill-ai/protogen-go v0.3.3-alpha.0.20250203091356-2b4937e1c3a2
github.com/instill-ai/usage-client v0.2.4-alpha.0.20240123081026-6c78d9a5197a
github.com/instill-ai/x v0.6.0-alpha
github.com/instill-ai/x v0.6.0-alpha.0.20250211123925-1b1559c35b51
github.com/itchyny/gojq v0.12.14
github.com/jackc/pgconn v1.14.3
github.com/jackc/pgx/v5 v5.5.5
Expand All @@ -62,7 +62,6 @@ require (
github.com/lestrrat-go/structinfo v0.0.0-20210312050401-7f8bd69d6acb
github.com/lib/pq v1.10.9
github.com/mennanov/fieldmask-utils v1.0.0
github.com/minio/minio-go/v7 v7.0.76
github.com/nakagami/firebirdsql v0.9.10
github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646
github.com/openfga/api/proto v0.0.0-20240318145204-66b9e5cb403c
Expand Down Expand Up @@ -113,6 +112,8 @@ require (
gorm.io/plugin/dbresolver v1.5.3
)

require github.com/minio/minio-go/v7 v7.0.76 // indirect

require (
cloud.google.com/go v0.115.0 // indirect
cloud.google.com/go/auth v0.7.2 // indirect
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1288,8 +1288,8 @@ github.com/instill-ai/protogen-go v0.3.3-alpha.0.20250203091356-2b4937e1c3a2 h1:
github.com/instill-ai/protogen-go v0.3.3-alpha.0.20250203091356-2b4937e1c3a2/go.mod h1:fusT92ceR5+GVn1LT5mT4XcOq1DlemBjpb6JpodLLdc=
github.com/instill-ai/usage-client v0.2.4-alpha.0.20240123081026-6c78d9a5197a h1:gmy8BcCFDZQan40c/D3f62DwTYtlCwi0VrSax+pKffw=
github.com/instill-ai/usage-client v0.2.4-alpha.0.20240123081026-6c78d9a5197a/go.mod h1:EpX3Yr661uWULtZf5UnJHfr5rw2PDyX8ku4Kx0UtYFw=
github.com/instill-ai/x v0.6.0-alpha h1:HaKCrRgQb4rY3K0c8A+EGp4oT5oxaoP760XtjPOrZPw=
github.com/instill-ai/x v0.6.0-alpha/go.mod h1:qztbVw9eW69byjdQINhYc7dm9tmhR0zuP46GjV3hwlQ=
github.com/instill-ai/x v0.6.0-alpha.0.20250211123925-1b1559c35b51 h1:BwyxAMRKkD5DDZqUYKTQi7PQ7HKRIPnNF81D1JgLbjo=
github.com/instill-ai/x v0.6.0-alpha.0.20250211123925-1b1559c35b51/go.mod h1:OOTTVkmYw/4bOrDS2nfunh4M6hvdqZsFZrl4mW9PztA=
github.com/intel/goresctrl v0.2.0/go.mod h1:+CZdzouYFn5EsxgqAQTEzMfwKwuc0fVdMrT9FCCAVRQ=
github.com/itchyny/gojq v0.12.14 h1:6k8vVtsrhQSYgSGg827AD+PVVaB1NLXEdX+dda2oZCc=
github.com/itchyny/gojq v0.12.14/go.mod h1:y1G7oO7XkcR1LPZO59KyoCRy08T3j9vDYRV0GgYSS+s=
Expand Down
220 changes: 10 additions & 210 deletions pkg/component/ai/instill/v0/README.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,8 @@ Classify images into predefined categories.

| Output | Field ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Category | `category` | string | The predicted category of the input. |
| Score | `score` | number | The confidence score of the predicted category of the input. |
| Category | `category` | any | The predicted category of the input. |
| Score | `score` | any | The confidence score of the predicted category of the input. |
</div>


Expand All @@ -89,37 +89,9 @@ Detect, localize and delineate multiple objects in images.

| Output | Field ID | Type | Description |
| :--- | :--- | :--- | :--- |
| [Objects](#instance-segmentation-objects) | `objects` | array[object] | A list of detected instance bounding boxes. |
| Objects | `objects` | any | A list of detected instance bounding boxes. |
</div>

<details>
<summary> Output Objects in Instance Segmentation</summary>

<h4 id="instance-segmentation-objects">Objects</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Bounding Box](#instance-segmentation-bounding-box) | `bounding-box` | object | The detected bounding box in (left, top, width, height) format. |
| Category | `category` | string | The predicted category of the bounding box. |
| RLE | `rle` | string | Run Length Encoding (RLE) of instance mask within the bounding box. |
| Score | `score` | number | The confidence score of the predicted instance object. |
</div>

<h4 id="instance-segmentation-bounding-box">Bounding Box</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| Height | `height` | number | Bounding box height value |
| Left | `left` | number | Bounding box left x-axis value |
| Top | `top` | number | Bounding box top y-axis value |
| Width | `width` | number | Bounding box width value |
</div>
</details>


### Keypoint

Expand All @@ -143,46 +115,8 @@ Detect and localize multiple keypoints of objects in images.

| Output | Field ID | Type | Description |
| :--- | :--- | :--- | :--- |
| [Objects](#keypoint-objects) | `objects` | array[object] | A list of keypoint objects, a keypoint object includes all the pre-defined keypoints of a detected object. |
</div>

<details>
<summary> Output Objects in Keypoint</summary>

<h4 id="keypoint-objects">Objects</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Bounding Box](#keypoint-bounding-box) | `bounding-box` | object | The detected bounding box in (left, top, width, height) format. |
| [Keypoints](#keypoint-keypoints) | `keypoints` | array | A keypoint group is composed of a list of pre-defined keypoints of a detected object. |
| Score | `score` | number | The confidence score of the predicted object. |
</div>

<h4 id="keypoint-keypoints">Keypoints</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| Visibility Score | `v` | number | visibility score of the keypoint. |
| X Coordinate | `x` | number | x coordinate of the keypoint. |
| Y Coordinate | `y` | number | y coordinate of the keypoint. |
</div>

<h4 id="keypoint-bounding-box">Bounding Box</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| Height | `height` | number | Bounding box height value |
| Left | `left` | number | Bounding box left x-axis value |
| Top | `top` | number | Bounding box top y-axis value |
| Width | `width` | number | Bounding box width value |
| Objects | `objects` | any | A list of keypoint objects, a keypoint object includes all the pre-defined keypoints of a detected object. |
</div>
</details>


### Detection
Expand All @@ -207,35 +141,8 @@ Detect and localize multiple objects in images.

| Output | Field ID | Type | Description |
| :--- | :--- | :--- | :--- |
| [Objects](#detection-objects) | `objects` | array[object] | A list of detected objects. |
</div>

<details>
<summary> Output Objects in Detection</summary>

<h4 id="detection-objects">Objects</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Bounding box](#detection-bounding-box) | `bounding-box` | object | The detected bounding box in (left, top, width, height) format. |
| Category | `category` | string | The predicted category of the bounding box. |
| Score | `score` | number | The confidence score of the predicted category of the bounding box. |
</div>

<h4 id="detection-bounding-box">Bounding Box</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| Height | `height` | number | Bounding box height value |
| Left | `left` | number | Bounding box left x-axis value |
| Top | `top` | number | Bounding box top y-axis value |
| Width | `width` | number | Bounding box width value |
| Objects | `objects` | any | A list of detected objects. |
</div>
</details>


### OCR
Expand All @@ -260,36 +167,9 @@ Detect and recognize text in images.

| Output | Field ID | Type | Description |
| :--- | :--- | :--- | :--- |
| [Objects](#ocr-objects) | `objects` | array[object] | A list of detected bounding boxes. |
| Objects | `objects` | any | A list of detected bounding boxes. |
</div>

<details>
<summary> Output Objects in OCR</summary>

<h4 id="ocr-objects">Objects</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Bounding Box](#ocr-bounding-box) | `bounding-box` | object | The detected bounding box in (left, top, width, height) format. |
| Score | `score` | number | The confidence score of the predicted object. |
| Text | `text` | string | Text string recognised per bounding box. |
</div>

<h4 id="ocr-bounding-box">Bounding Box</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| Height | `height` | number | Bounding box height value |
| Left | `left` | number | Bounding box left x-axis value |
| Top | `top` | number | Bounding box top y-axis value |
| Width | `width` | number | Bounding box width value |
</div>
</details>


### Semantic Segmentation

Expand All @@ -313,22 +193,8 @@ Classify image pixels into predefined categories.

| Output | Field ID | Type | Description |
| :--- | :--- | :--- | :--- |
| [Stuffs](#semantic-segmentation-stuffs) | `stuffs` | array[object] | A list of RLE binary masks. |
</div>

<details>
<summary> Output Objects in Semantic Segmentation</summary>

<h4 id="semantic-segmentation-stuffs">Stuffs</h4>

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| Category | `category` | string | Category text string corresponding to each stuff mask. |
| RLE | `rle` | string | Run Length Encoding (RLE) of each stuff mask within the image. |
| Stuffs | `stuffs` | any | A list of RLE binary masks. |
</div>
</details>


### Text Generation
Expand Down Expand Up @@ -392,31 +258,9 @@ Incorporate external chat history, specifically previous messages within the con

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Content](#text-generation-chat-content) | `content` | array | The message content. |
| Content | `content` | | The message content. |
| Role | `role` | string | The message role, i.e. 'system', 'user' or 'assistant'. |
</div>
<h4 id="text-generation-chat-content">Content</h4>

The message content.

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Image URL](#text-generation-chat-image-url) | `image-url` | object | The image URL |
| Text | `text` | string | The text content. |
| Type | `type` | string | The type of the content part. <br/><details><summary><strong>Enum values</strong></summary><ul><li>`text`</li><li>`image-url`</li></ul></details> |
</div>
<h4 id="text-generation-chat-image-url">Image URL</h4>

The image URL

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| URL | `url` | string | Either a URL of the image or the base64 encoded image data. |
</div>
</details>


Expand Down Expand Up @@ -490,31 +334,9 @@ Incorporate external chat history, specifically previous messages within the con

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Content](#visual-question-answering-content) | `content` | array | The message content. |
| Content | `content` | | The message content. |
| Role | `role` | string | The message role, i.e. 'system', 'user' or 'assistant'. |
</div>
<h4 id="visual-question-answering-content">Content</h4>

The message content.

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Image URL](#visual-question-answering-image-url) | `image-url` | object | The image URL |
| Text | `text` | string | The text content. |
| Type | `type` | string | The type of the content part. <br/><details><summary><strong>Enum values</strong></summary><ul><li>`text`</li><li>`image-url`</li></ul></details> |
</div>
<h4 id="visual-question-answering-image-url">Image URL</h4>

The image URL

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| URL | `url` | string | Either a URL of the image or the base64 encoded image data. |
</div>
</details>


Expand Down Expand Up @@ -558,31 +380,9 @@ Incorporate external chat history, specifically previous messages within the con

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Content](#chat-content) | `content` | array | The message content. |
| Content | `content` | | The message content. |
| Role | `role` | string | The message role, i.e. 'system', 'user' or 'assistant'. |
</div>
<h4 id="chat-content">Content</h4>

The message content.

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Image URL](#chat-image-url) | `image-url` | object | The image URL |
| Text | `text` | string | The text content. |
| Type | `type` | string | The type of the content part. <br/><details><summary><strong>Enum values</strong></summary><ul><li>`text`</li><li>`image-url`</li></ul></details> |
</div>
<h4 id="chat-image-url">Image URL</h4>

The image URL

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| URL | `url` | string | Either a URL of the image or the base64 encoded image data. |
</div>
</details>


Expand Down
26 changes: 2 additions & 24 deletions pkg/component/ai/openai/v0/README.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,31 +91,9 @@ Incorporate external chat history, specifically previous messages within the con

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Content](#text-generation-content) | `content` | array | The message content. |
| Content | `content` | | The message content. |
| Role | `role` | string | The message role, i.e. 'system', 'user' or 'assistant'. |
</div>
<h4 id="text-generation-content">Content</h4>

The message content.

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Image URL](#text-generation-image-url) | `image-url` | object | The image URL |
| Text | `text` | string | The text content. |
| Type | `type` | string | The type of the content part. <br/><details><summary><strong>Enum values</strong></summary><ul><li>`text`</li><li>`image-url`</li></ul></details> |
</div>
<h4 id="text-generation-image-url">Image URL</h4>

The image URL

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| URL | `url` | string | Either a URL of the image or the base64 encoded image data. |
</div>
<h4 id="text-generation-prediction">Prediction</h4>

Configuration for a Predicted Output, which can greatly improve response times when large parts of the model response are known ahead of time. This is most common when you are regenerating a file with only minor changes to most of the content.
Expand Down Expand Up @@ -279,7 +257,7 @@ Turn text into numbers, unlocking use cases like search.

| Output | Field ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Embedding | `embedding` | array[number] | Embedding of the input text. |
| Embedding | `embedding` | any | Embedding of the input text. |
</div>


Expand Down
Loading
Loading