This is the open source code of our paper.
We build a VQA application platform from dataset generation to model application.
We upload the recorded video, and users can learn how to use the system according to the videos.
The demo has the following modules.
Upload the ZIP file of the medical record and build the dataset.
Label by patient id and picture id.
Medical VQA model is provided, and users can select parameter annotation. (For the time being, there are only three models: NLM, VGG-Seq2Seq, MMbert, and models will be introduced in the future.)
After the training, the model score report is provided and the model effect can be tested on the AI Robot interface.
Ubuntu 20.04
mysql 8.0.25
jdk 1.8
Vue
How to run frontend?
The front files are in the fronted folder.
npm install
npm run dev
How to run backend? The back-end code is in the demo folder.
Configuration
The sql file is in the resources. We only provide several pieces of records for illustration. We've created three new Python environments. Python environments are provided by requirements.