Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .cspell-wordlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -116,3 +116,5 @@ antonov
rfdetr
basemodule
IMAGENET
lraspp
LRASPP
13 changes: 9 additions & 4 deletions docs/docs/03-hooks/02-computer-vision/useSemanticSegmentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,12 @@ function App() {

## Supported models

| Model | Number of classes | Class list |
| ------------------------------------------------------------------------------------------------ | ----------------- | ----------------------------------------------------------------------------------------- |
| [deeplabv3_resnet50](https://huggingface.co/software-mansion/react-native-executorch-deeplab-v3) | 21 | [DeeplabLabel](../../06-api-reference/enumerations/DeeplabLabel.md) |
| selfie-segmentation | 2 | [SelfieSegmentationLabel](../../06-api-reference/enumerations/SelfieSegmentationLabel.md) |
| Model | Number of classes | Class list | Quantized |
| ----------------------------------------------------------------------------------------------------------- | ----------------- | ----------------------------------------------------------------------------------------- | :-------: |
| [deeplab-v3-resnet50](https://huggingface.co/software-mansion/react-native-executorch-deeplab-v3) | 21 | [DeeplabLabel](../../06-api-reference/enumerations/DeeplabLabel.md) | Yes |
| [deeplab-v3-resnet101](https://huggingface.co/software-mansion/react-native-executorch-deeplab-v3) | 21 | [DeeplabLabel](../../06-api-reference/enumerations/DeeplabLabel.md) | Yes |
| [deeplab-v3-mobilenet-v3-large](https://huggingface.co/software-mansion/react-native-executorch-deeplab-v3) | 21 | [DeeplabLabel](../../06-api-reference/enumerations/DeeplabLabel.md) | Yes |
| [lraspp-mobilenet-v3-large](https://huggingface.co/software-mansion/react-native-executorch-lraspp) | 21 | [DeeplabLabel](../../06-api-reference/enumerations/DeeplabLabel.md) | Yes |
| [fcn-resnet50](https://huggingface.co/software-mansion/react-native-executorch-fcn) | 21 | [DeeplabLabel](../../06-api-reference/enumerations/DeeplabLabel.md) | Yes |
| [fcn-resnet101](https://huggingface.co/software-mansion/react-native-executorch-fcn) | 21 | [DeeplabLabel](../../06-api-reference/enumerations/DeeplabLabel.md) | Yes |
| [selfie-segmentation](https://huggingface.co/software-mansion/react-native-executorch-selfie-segmentation) | 2 | [SelfieSegmentationLabel](../../06-api-reference/enumerations/SelfieSegmentationLabel.md) | No |
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ const segmentation = await SemanticSegmentationModule.fromModelName(
);
```

The `config` parameter is a discriminated union — TypeScript ensures you provide the correct fields for each model name. Available built-in models: `'deeplab-v3'`, `'selfie-segmentation'`.
The `config` parameter is a discriminated union — TypeScript ensures you provide the correct fields for each model name. Available built-in models: `'deeplab-v3-resnet50'`, `'deeplab-v3-resnet50-quantized'`, `'deeplab-v3-resnet101'`, `'deeplab-v3-resnet101-quantized'`, `'deeplab-v3-mobilenet-v3-large'`, `'deeplab-v3-mobilenet-v3-large-quantized'`, `'lraspp-mobilenet-v3-large'`, `'lraspp-mobilenet-v3-large-quantized'`, `'fcn-resnet50'`, `'fcn-resnet50-quantized'`, `'fcn-resnet101'`, `'fcn-resnet101-quantized'`, and `'selfie-segmentation'`.

### Custom models — `fromCustomConfig`

Expand Down
21 changes: 13 additions & 8 deletions docs/docs/06-api-reference/classes/SemanticSegmentationModule.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Class: SemanticSegmentationModule\<T\>

Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:60](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L60)
Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:81](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L81)

Generic semantic segmentation module with type-safe label maps.
Use a model name (e.g. `'deeplab-v3'`) as the generic parameter for built-in models,
Use a model name (e.g. `'deeplab-v3-resnet50'`) as the generic parameter for built-in models,
or a custom label enum for custom configs.

## Extends
Expand All @@ -16,8 +16,13 @@ or a custom label enum for custom configs.

`T` _extends_ [`SemanticSegmentationModelName`](../type-aliases/SemanticSegmentationModelName.md) \| [`LabelEnum`](../type-aliases/LabelEnum.md)

Either a built-in model name (`'deeplab-v3'`, `'selfie-segmentation'`)
or a custom [LabelEnum](../type-aliases/LabelEnum.md) label map.
Either a built-in model name (`'deeplab-v3-resnet50'`,
`'deeplab-v3-resnet50-quantized'`, `'deeplab-v3-resnet101'`,
`'deeplab-v3-resnet101-quantized'`, `'deeplab-v3-mobilenet-v3-large'`,
`'deeplab-v3-mobilenet-v3-large-quantized'`, `'lraspp-mobilenet-v3-large'`,
`'lraspp-mobilenet-v3-large-quantized'`, `'fcn-resnet50'`,
`'fcn-resnet50-quantized'`, `'fcn-resnet101'`, `'fcn-resnet101-quantized'`,
`'selfie-segmentation'`) or a custom [LabelEnum](../type-aliases/LabelEnum.md) label map.

## Properties

Expand Down Expand Up @@ -57,7 +62,7 @@ Unloads the model from memory.

> **forward**\<`K`\>(`imageSource`, `classesOfInterest?`, `resizeToInput?`): `Promise`\<`Record`\<`"ARGMAX"`, `Int32Array`\<`ArrayBufferLike`\>\> & `Record`\<`K`, `Float32Array`\<`ArrayBufferLike`\>\>\>

Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:176](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L176)
Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:197](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L197)

Executes the model's forward pass to perform semantic segmentation on the provided image.

Expand Down Expand Up @@ -166,7 +171,7 @@ The input shape as an array of numbers.

> **load**(): `Promise`\<`void`\>

Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:76](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L76)
Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:97](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L97)

#### Returns

Expand All @@ -182,7 +187,7 @@ Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:76](https://g

> `static` **fromCustomConfig**\<`L`\>(`modelSource`, `config`, `onDownloadProgress?`): `Promise`\<`SemanticSegmentationModule`\<`L`\>\>

Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:142](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L142)
Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:163](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L163)

Creates a segmentation instance with a user-provided label map and custom config.
Use this when working with a custom-exported segmentation model that is not one of the built-in models.
Expand Down Expand Up @@ -235,7 +240,7 @@ const segmentation = await SemanticSegmentationModule.fromCustomConfig(

> `static` **fromModelName**\<`C`\>(`config`, `onDownloadProgress?`): `Promise`\<`SemanticSegmentationModule`\<[`ModelNameOf`](../type-aliases/ModelNameOf.md)\<`C`\>\>\>

Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:95](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L95)
Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:116](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L116)

Creates a segmentation instance for a built-in model.
The config object is discriminated by `modelName` — each model can require different fields.
Expand Down
44 changes: 22 additions & 22 deletions docs/docs/06-api-reference/enumerations/DeeplabLabel.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Enumeration: DeeplabLabel

Defined in: [types/semanticSegmentation.ts:53](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L53)
Defined in: [types/semanticSegmentation.ts:70](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L70)

Labels used in the DeepLab semantic segmentation model.

Expand All @@ -10,164 +10,164 @@ Labels used in the DeepLab semantic segmentation model.

> **AEROPLANE**: `1`

Defined in: [types/semanticSegmentation.ts:55](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L55)
Defined in: [types/semanticSegmentation.ts:72](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L72)

---

### BACKGROUND

> **BACKGROUND**: `0`

Defined in: [types/semanticSegmentation.ts:54](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L54)
Defined in: [types/semanticSegmentation.ts:71](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L71)

---

### BICYCLE

> **BICYCLE**: `2`

Defined in: [types/semanticSegmentation.ts:56](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L56)
Defined in: [types/semanticSegmentation.ts:73](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L73)

---

### BIRD

> **BIRD**: `3`

Defined in: [types/semanticSegmentation.ts:57](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L57)
Defined in: [types/semanticSegmentation.ts:74](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L74)

---

### BOAT

> **BOAT**: `4`

Defined in: [types/semanticSegmentation.ts:58](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L58)
Defined in: [types/semanticSegmentation.ts:75](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L75)

---

### BOTTLE

> **BOTTLE**: `5`

Defined in: [types/semanticSegmentation.ts:59](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L59)
Defined in: [types/semanticSegmentation.ts:76](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L76)

---

### BUS

> **BUS**: `6`

Defined in: [types/semanticSegmentation.ts:60](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L60)
Defined in: [types/semanticSegmentation.ts:77](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L77)

---

### CAR

> **CAR**: `7`

Defined in: [types/semanticSegmentation.ts:61](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L61)
Defined in: [types/semanticSegmentation.ts:78](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L78)

---

### CAT

> **CAT**: `8`

Defined in: [types/semanticSegmentation.ts:62](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L62)
Defined in: [types/semanticSegmentation.ts:79](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L79)

---

### CHAIR

> **CHAIR**: `9`

Defined in: [types/semanticSegmentation.ts:63](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L63)
Defined in: [types/semanticSegmentation.ts:80](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L80)

---

### COW

> **COW**: `10`

Defined in: [types/semanticSegmentation.ts:64](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L64)
Defined in: [types/semanticSegmentation.ts:81](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L81)

---

### DININGTABLE

> **DININGTABLE**: `11`

Defined in: [types/semanticSegmentation.ts:65](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L65)
Defined in: [types/semanticSegmentation.ts:82](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L82)

---

### DOG

> **DOG**: `12`

Defined in: [types/semanticSegmentation.ts:66](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L66)
Defined in: [types/semanticSegmentation.ts:83](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L83)

---

### HORSE

> **HORSE**: `13`

Defined in: [types/semanticSegmentation.ts:67](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L67)
Defined in: [types/semanticSegmentation.ts:84](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L84)

---

### MOTORBIKE

> **MOTORBIKE**: `14`

Defined in: [types/semanticSegmentation.ts:68](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L68)
Defined in: [types/semanticSegmentation.ts:85](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L85)

---

### PERSON

> **PERSON**: `15`

Defined in: [types/semanticSegmentation.ts:69](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L69)
Defined in: [types/semanticSegmentation.ts:86](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L86)

---

### POTTEDPLANT

> **POTTEDPLANT**: `16`

Defined in: [types/semanticSegmentation.ts:70](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L70)
Defined in: [types/semanticSegmentation.ts:87](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L87)

---

### SHEEP

> **SHEEP**: `17`

Defined in: [types/semanticSegmentation.ts:71](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L71)
Defined in: [types/semanticSegmentation.ts:88](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L88)

---

### SOFA

> **SOFA**: `18`

Defined in: [types/semanticSegmentation.ts:72](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L72)
Defined in: [types/semanticSegmentation.ts:89](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L89)

---

### TRAIN

> **TRAIN**: `19`

Defined in: [types/semanticSegmentation.ts:73](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L73)
Defined in: [types/semanticSegmentation.ts:90](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L90)

---

### TVMONITOR

> **TVMONITOR**: `20`

Defined in: [types/semanticSegmentation.ts:74](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L74)
Defined in: [types/semanticSegmentation.ts:91](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L91)
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Enumeration: SelfieSegmentationLabel

Defined in: [types/semanticSegmentation.ts:82](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L82)
Defined in: [types/semanticSegmentation.ts:99](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L99)

Labels used in the selfie semantic segmentation model.

Expand All @@ -10,12 +10,12 @@ Labels used in the selfie semantic segmentation model.

> **BACKGROUND**: `1`

Defined in: [types/semanticSegmentation.ts:84](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L84)
Defined in: [types/semanticSegmentation.ts:101](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L101)

---

### SELFIE

> **SELFIE**: `0`

Defined in: [types/semanticSegmentation.ts:83](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L83)
Defined in: [types/semanticSegmentation.ts:100](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L100)
Loading
Loading