-
Notifications
You must be signed in to change notification settings - Fork 464
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Model] add Paddle.js web demo (#392)
* add application include paddle.js web demo and xcx * cp PR #5 * add readme * fix comments and link * fix xcx readme * fix Task 1 * fix bugs * refine readme * delete ocrxcx readme * refine readme * fix bugs * delete old readme * 200px to 300px * revert 200px to 300px Co-authored-by: Jason <jiangjiajun@baidu.com>
- Loading branch information
1 parent
587ffd4
commit 0a22979
Showing
302 changed files
with
63,930 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
|
||
# 前端AI应用 | ||
|
||
人工智能技术的快速发展带动了计算机视觉、自然语言处理领域的产业升级。另外,随着PC和移动设备上算力的稳步增强、模型压缩技术迭代更新以及各种创新需求的不断催生,在浏览器中部署AI模型实现前端智能已经具备了良好的基础条件。 | ||
针对前端部署AI深度学习模型困难的问题,百度开源了Paddle.js前端深度学习模型部署框架,可以很容易的将深度学习模型部署到前端项目中。 | ||
|
||
## Paddle.js简介 | ||
|
||
[Paddle.js](https://github.com/PaddlePaddle/Paddle.js)是百度`PaddlePaddle`的web方向子项目,是一个运行在浏览器中的开源深度学习框架。`Paddle.js`可以加载`PaddlePaddle`动转静的模型,经过`Paddle.js`的模型转换工具`paddlejs-converter`转换成浏览器友好的模型,易于在线推理预测使用。`Paddle.js`支持`WebGL/WebGPU/WebAssembly`的浏览器中运行,也可以在百度小程序和微信小程序环境下运行。 | ||
|
||
简言之,利用Paddle.js,我们可以在浏览器、小程序等前端应用场景上线AI功能,包括但不限于目标检测,图像分割,OCR,物品分类等AI能力。 | ||
|
||
## Web Demo使用 | ||
|
||
在浏览器中直接运行官方demo参考[文档](./web_demo/README.md) | ||
|
||
|demo名称|web demo目录|可视化| | ||
|-|-|-| | ||
|目标检测|[ScrewDetection/FaceDetection](./web_demo/demo/src/pages/cv/detection/)| <img src="https://user-images.githubusercontent.com/26592129/196874536-b7fa2c0a-d71f-4271-8c40-f9088bfad3c9.png" height="200px">| | ||
|人像分割背景替换|[HumanSeg](./web_demo//demo/src/pages/cv/segmentation/HumanSeg)|<img src="https://user-images.githubusercontent.com/26592129/196874452-4ef2e770-fbb3-4a35-954b-f871716d6669.png" height="200px">| | ||
|物体识别|[GestureRecognition/ItemIdentification](./web_demo//demo/src/pages/cv/recognition/)|<img src="https://user-images.githubusercontent.com/26592129/196874416-454e6bb0-4ebd-4b51-a88a-8c40614290ae.png" height="200px">| | ||
|OCR|[TextDetection/TextRecognition](./web_demo//demo/src/pages/cv/ocr/)|<img src="https://user-images.githubusercontent.com/26592129/196874354-1b5eecb0-f273-403c-aa6c-4463bf6d78db.png" height="200px">| | ||
|
||
|
||
## 微信小程序Demo使用 | ||
|
||
在微信小程序运行官方demo参考[文档](./mini_program/README.md) | ||
|
||
|名称|目录| | ||
|-|-| | ||
|OCR文本检测| [ocrdetecXcx](./mini_program/ocrdetectXcx/) | | ||
|OCR文本识别| [ocrXcx](./mini_program/ocrXcx/) | | ||
|目标检测| coming soon | | ||
|图像分割| coming soon | | ||
|物品分类| coming soon | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,126 @@ | ||
|
||
# Paddle.js微信小程序Demo | ||
|
||
- [1.简介](#1) | ||
- [2. 项目启动](#2) | ||
* [2.1 准备工作](#21) | ||
* [2.2 启动步骤](#22) | ||
* [2.3 效果展示](#23) | ||
- [3. 模型推理pipeline](#3) | ||
- [4. 常见问题](#4) | ||
|
||
<a name="1"></a> | ||
## 1.简介 | ||
|
||
|
||
本目录下包含文本检测、文本识别小程序demo,通过使用 [Paddle.js](https://github.com/PaddlePaddle/Paddle.js) 以及 [Paddle.js微信小程序插件](https://mp.weixin.qq.com/wxopen/plugindevdoc?appid=wx7138a7bb793608c3&token=956931339&lang=zh_CN) 完成在小程序上利用用户终端算力实现文本检测框选效果。 | ||
|
||
<a name="2"></a> | ||
## 2. 项目启动 | ||
|
||
<a name="21"></a> | ||
### 2.1 准备工作 | ||
* [申请微信小程序账号](https://mp.weixin.qq.com/) | ||
* [微信小程序开发者工具](https://developers.weixin.qq.com/miniprogram/dev/devtools/download.html) | ||
* 前端开发环境准备:node、npm | ||
* 小程序管理后台配置服务器域名,或打开开发者工具【不校验合法域名】 | ||
|
||
详情参考:https://mp.weixin.qq.com/wxamp/devprofile/get_profile?token=1132303404&lang=zh_CN) | ||
|
||
<a name="22"></a> | ||
### 2.2 启动步骤 | ||
|
||
#### **1. 克隆Demo代码** | ||
```sh | ||
git clone https://github.com/PaddlePaddle/FastDeploy | ||
cd FastDeploy/examples/application/js/mini_program | ||
``` | ||
|
||
#### **2. 进入小程序目录,安装依赖** | ||
|
||
```sh | ||
# 运行文本识别demo,进入到ocrXcx目录 | ||
cd ./ocrXcx && npm install | ||
# 运行文本检测demo,进入到ocrdetectXcx目录 | ||
# cd ./ocrdetectXcx && npm install | ||
``` | ||
|
||
#### **3. 微信小程序导入代码** | ||
打开微信开发者工具 --> 导入 --> 选定目录,输入相关信息 | ||
|
||
#### **4. 添加 Paddle.js微信小程序插件** | ||
小程序管理界面 --> 设置 --> 第三方设置 --> 插件管理 --> 添加插件 --> 搜索 `wx7138a7bb793608c3` 并添加 | ||
[参考文档](https://developers.weixin.qq.com/miniprogram/dev/framework/plugin/using.html) | ||
|
||
#### **5. 构建依赖** | ||
点击开发者工具中的菜单栏:工具 --> 构建 npm | ||
|
||
原因:node_modules 目录不会参与编译、上传和打包中,小程序想要使用 npm 包必须走一遍“构建 npm”的过程,构建完成会生成一个 miniprogram_npm 目录,里面会存放构建打包后的 npm 包,也就是小程序真正使用的 npm 包。* | ||
[参考文档](https://developers.weixin.qq.com/miniprogram/dev/devtools/npm.html) | ||
|
||
<a name="23"></a> | ||
### 2.3 效果展示 | ||
|
||
<img src="https://user-images.githubusercontent.com/43414102/157648579-cdbbee61-9866-4364-9edd-a97ac0eda0c1.png" width="300px"> | ||
|
||
<a name="3"></a> | ||
## 3. 模型推理pipeline | ||
|
||
```typescript | ||
// 引入 paddlejs 和 paddlejs-plugin,注册小程序环境变量和合适的 backend | ||
import * as paddlejs from '@paddlejs/paddlejs-core'; | ||
import '@paddlejs/paddlejs-backend-webgl'; | ||
const plugin = requirePlugin('paddlejs-plugin'); | ||
plugin.register(paddlejs, wx); | ||
|
||
// 初始化推理引擎 | ||
const runner = new paddlejs.Runner({modelPath, feedShape, mean, std}); | ||
await runner.init(); | ||
|
||
// 获取图像信息 | ||
wx.canvasGetImageData({ | ||
canvasId: canvasId, | ||
x: 0, | ||
y: 0, | ||
width: canvas.width, | ||
height: canvas.height, | ||
success(res) { | ||
// 推理预测 | ||
runner.predict({ | ||
data: res.data, | ||
width: canvas.width, | ||
height: canvas.height, | ||
}, function (data) { | ||
// 获取推理结果 | ||
console.log(data) | ||
}); | ||
} | ||
}); | ||
``` | ||
|
||
<a name="4"></a> | ||
## 4. 常见问题 | ||
### 4.1 出现报错 `Invalid context type [webgl2] for Canvas#getContext` | ||
|
||
可以不管,不影响正常代码运行和demo功能 | ||
|
||
### 4.2 预览看不到结果 | ||
|
||
建议尝试真机调试 | ||
|
||
### 4.3 微信开发者工具出现黑屏,然后出现超多报错 | ||
|
||
重启微信开发者工具 | ||
|
||
### 4.4 模拟和真机调试结果不一致;模拟检测不到文本等 | ||
|
||
可以以真机为准; | ||
|
||
模拟检测不到文本等可以尝试随意改动下代码(增删换行等)再点击编译 | ||
|
||
|
||
### 4.5 手机调试或运行时出现 长时间无反应等提示 | ||
|
||
请继续等待,模型推理需要一定时间 | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
/* global wx, App */ | ||
import * as paddlejs from '@paddlejs/paddlejs-core'; | ||
import '@paddlejs/paddlejs-backend-webgl'; | ||
// eslint-disable-next-line no-undef | ||
const plugin = requirePlugin('paddlejs-plugin'); | ||
plugin.register(paddlejs, wx); | ||
|
||
App({ | ||
globalData: { | ||
Paddlejs: paddlejs.Runner | ||
} | ||
}); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
{ | ||
"pages": [ | ||
"pages/index/index" | ||
], | ||
"plugins": { | ||
"paddlejs-plugin": { | ||
"version": "2.0.1", | ||
"provider": "wx7138a7bb793608c3" | ||
} | ||
}, | ||
"sitemapLocation": "sitemap.json" | ||
} |
72 changes: 72 additions & 0 deletions
72
examples/application/js/mini_program/ocrXcx/package-lock.json
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
{ | ||
"name": "paddlejs-demo", | ||
"version": "0.0.1", | ||
"description": "", | ||
"main": "app.js", | ||
"dependencies": { | ||
"@paddlejs/paddlejs-backend-webgl": "^1.2.0", | ||
"@paddlejs/paddlejs-core": "^2.1.18", | ||
"d3-polygon": "2.0.0", | ||
"js-clipper": "1.0.1", | ||
"number-precision": "1.5.2" | ||
}, | ||
"scripts": { | ||
"test": "echo \"Error: no test specified\" && exit 1" | ||
}, | ||
"keywords": [], | ||
"author": "", | ||
"license": "ISC" | ||
} |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.