Skip to content

Commit 3eca45d

Browse files
jsl-modelsjsl-models
authored
2023-02-25-ner_vop_slim_wip_en (#13566)
1 parent 046b044 commit 3eca45d

File tree

1 file changed

+217
-0
lines changed

1 file changed

+217
-0
lines changed

‎docs/_posts/Meryem1425/2023-02-25-ner_vop_slim_wip_en.md

Lines changed: 217 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,217 @@
1+
---
2+
layout: model
3+
title: Voice of the Patients
4+
author: John Snow Labs
5+
name: ner_vop_slim_wip
6+
date: 2023-02-25
7+
tags: [ner, clinical, en, licensed, vop, voice, patient]
8+
task: Named Entity Recognition
9+
language: en
10+
edition: Healthcare NLP 4.3.1
11+
spark_version: 3.0
12+
supported: true
13+
annotator: MedicalNerModel
14+
article_header:
15+
type: cover
16+
use_language_switcher: "Python-Scala-Java"
17+
---
18+
19+
## Description
20+
21+
This model extracts healthcare-related terms from the documents transferred from the patient's own sentences.
22+
23+
Note: 'wip' suffix indicates that the model development is work-in-progress and will be finalised and the model performance will improved in the upcoming releases.
24+
25+
## Predicted Entities
26+
27+
`AdmissionDischarge`, `Age`, `BodyPart`, `ClinicalDept`, `DateTime`, `Disease`, `Dosage_Strength`, `Drug`, `Duration`, `Employment`, `Form`, `Frequency`, `Gender`, `Laterality`, `Procedure`, `PsychologicalCondition`, `RelationshipStatus`, `Route`, `Symptom`, `Test`, `Vaccine`, `VitalTest`
28+
29+
{:.btn-box}
30+
<button class="button button-orange" disabled>Live Demo</button>
31+
<button class="button button-orange" disabled>Open in Colab</button>
32+
[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/models/ner_vop_slim_wip_en_4.3.1_3.0_1677342424243.zip){:.button.button-orange}
33+
[Copy S3 URI](s3://auxdata.johnsnowlabs.com/clinical/models/ner_vop_slim_wip_en_4.3.1_3.0_1677342424243.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3}
34+
35+
## How to use
36+
37+
38+
39+
<div class="tabs-box" markdown="1">
40+
{% include programmingLanguageSelectScalaPythonNLU.html %}
41+
```python
42+
document_assembler = DocumentAssembler()\
43+
.setInputCol("text")\
44+
.setOutputCol("document")
45+
46+
sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl", "en")\
47+
.setInputCols(["document"])\
48+
.setOutputCol("sentence")
49+
50+
tokenizer = Tokenizer()\
51+
.setInputCols(["sentence"])\
52+
.setOutputCol("token")
53+
54+
clinical_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")\
55+
.setInputCols(["sentence", "token"])\
56+
.setOutputCol("embeddings")
57+
58+
ner_model = MedicalNerModel.pretrained("ner_vop_slim_wip", "en", "clinical/models")\
59+
.setInputCols(["sentence", "token","embeddings"])\
60+
.setOutputCol("ner")
61+
62+
ner_converter = NerConverterInternal()\
63+
.setInputCols(["sentence", "token", "ner"])\
64+
.setOutputCol("ner_chunk")
65+
66+
pipeline = Pipeline(stages=[
67+
document_assembler,
68+
sentence_detector,
69+
tokenizer,
70+
clinical_embeddings,
71+
ner_model,
72+
ner_converter
73+
])
74+
75+
sample_texts = ["Hello,I'm 20 year old girl. I'm diagnosed with hyperthyroid 1 month ago. I was feeling weak, light headed,poor digestion, panic attacks, depression, left chest pain, increased heart rate, rapidly weight loss, from 4 months. Because of this, I stayed in the hospital and just discharged from hospital. I had many other blood tests, brain mri, ultrasound scan, endoscopy because of some dumb doctors bcs they were not able to diagnose actual problem. Finally I got an appointment with a homeopathy doctor finally he find that i was suffering from hyperthyroid and my TSH was 0.15 T3 and T4 is normal . Also i have b12 deficiency and vitamin D deficiency so I'm taking weekly supplement of vitamin D and 1000 mcg b12 daily. I'm taking homeopathy medicine for 40 days and took 2nd test after 30 days. My TSH is 0.5 now. I feel a little bit relief from weakness and depression but I'm facing with 2 new problem from last week that is breathtaking problem and very rapid heartrate. I just want to know if i should start allopathy medicine or homeopathy is okay? Bcs i heard that thyroid take time to start recover. So please let me know if both of medicines take same time. Because some of my friends advising me to start allopathy and never take a chance as i can develop some serious problems.Sorry for my poor english😐Thank you."]
76+
77+
78+
data = spark.createDataFrame(sample_texts, StringType()).toDF("text")
79+
80+
result = pipeline.fit(data).transform(data)
81+
```
82+
```scala
83+
val document_assembler = new DocumentAssembler()
84+
.setInputCol("text")
85+
.setOutputCol("document")
86+
87+
val sentence_detector = SentenceDetectorDLModel.pretrained("sentence_detector_dl", "en")
88+
.setInputCols("document")
89+
.setOutputCol("sentence")
90+
91+
val tokenizer = new Tokenizer()
92+
.setInputCols("sentence")
93+
.setOutputCol("token")
94+
95+
val clinical_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")
96+
.setInputCols(Array("sentence", "token"))
97+
.setOutputCol("embeddings")
98+
99+
val ner_model = MedicalNerModel.pretrained("ner_vop_slim_wip", "en", "clinical/models")
100+
.setInputCols(Array("sentence", "token","embeddings"))
101+
.setOutputCol("ner")
102+
103+
val ner_converter = new NerConverterInternal()
104+
.setInputCols(Array("sentence", "token", "ner"))
105+
.setOutputCol("ner_chunk")
106+
107+
val pipeline = new Pipeline().setStages(Array(
108+
document_assembler,
109+
sentence_detector,
110+
tokenizer,
111+
clinical_embeddings,
112+
ner_model,
113+
ner_converter
114+
))
115+
116+
val data = Seq("Hello,I'm 20 year old girl. I'm diagnosed with hyperthyroid 1 month ago. I was feeling weak, light headed,poor digestion, panic attacks, depression, left chest pain, increased heart rate, rapidly weight loss, from 4 months. Because of this, I stayed in the hospital and just discharged from hospital. I had many other blood tests, brain mri, ultrasound scan, endoscopy because of some dumb doctors bcs they were not able to diagnose actual problem. Finally I got an appointment with a homeopathy doctor finally he find that i was suffering from hyperthyroid and my TSH was 0.15 T3 and T4 is normal . Also i have b12 deficiency and vitamin D deficiency so I'm taking weekly supplement of vitamin D and 1000 mcg b12 daily. I'm taking homeopathy medicine for 40 days and took 2nd test after 30 days. My TSH is 0.5 now. I feel a little bit relief from weakness and depression but I'm facing with 2 new problem from last week that is breathtaking problem and very rapid heartrate. I just want to know if i should start allopathy medicine or homeopathy is okay? Bcs i heard that thyroid take time to start recover. So please let me know if both of medicines take same time. Because some of my friends advising me to start allopathy and never take a chance as i can develop some serious problems.Sorry for my poor english😐Thank you.").toDS.toDF("text")
117+
118+
val result = pipeline.fit(data).transform(data)
119+
```
120+
</div>
121+
122+
## Results
123+
124+
```bash
125+
+--------------------+-----+----+----------------------+
126+
|chunk |begin|end |ner_label |
127+
+--------------------+-----+----+----------------------+
128+
|20 year old |10 |20 |Age |
129+
|girl |22 |25 |Gender |
130+
|hyperthyroid |47 |58 |Disease |
131+
|1 month ago |60 |70 |DateTime |
132+
|weak |87 |90 |Symptom |
133+
|panic attacks |122 |134 |PsychologicalCondition|
134+
|depression |137 |146 |PsychologicalCondition|
135+
|left |149 |152 |Laterality |
136+
|chest |154 |158 |BodyPart |
137+
|pain |160 |163 |Symptom |
138+
|heart rate |176 |185 |VitalTest |
139+
|weight loss |196 |206 |Symptom |
140+
|4 months |215 |222 |Duration |
141+
|hospital |258 |265 |ClinicalDept |
142+
|discharged |276 |285 |AdmissionDischarge |
143+
|hospital |292 |299 |ClinicalDept |
144+
|blood tests |319 |329 |Test |
145+
|brain |332 |336 |BodyPart |
146+
|mri |338 |340 |Test |
147+
|ultrasound scan |343 |357 |Test |
148+
|endoscopy |360 |368 |Procedure |
149+
|doctors |391 |397 |Employment |
150+
|homeopathy doctor |486 |502 |Employment |
151+
|he |512 |513 |Gender |
152+
|hyperthyroid |546 |557 |Disease |
153+
|TSH |566 |568 |Test |
154+
|T3 |579 |580 |Test |
155+
|T4 |586 |587 |Test |
156+
|b12 deficiency |613 |626 |Disease |
157+
|vitamin D deficiency|632 |651 |Disease |
158+
|weekly |667 |672 |Frequency |
159+
|supplement |674 |683 |Drug |
160+
|vitamin D |688 |696 |Drug |
161+
|1000 mcg |702 |709 |Dosage_Strength |
162+
|b12 |711 |713 |Drug |
163+
|daily |715 |719 |Frequency |
164+
|homeopathy medicine |733 |751 |Drug |
165+
|40 days |757 |763 |Duration |
166+
|after 30 days |783 |795 |DateTime |
167+
|TSH |801 |803 |Test |
168+
|now |812 |814 |DateTime |
169+
|weakness |849 |856 |Symptom |
170+
|depression |862 |871 |PsychologicalCondition|
171+
|last week |912 |920 |DateTime |
172+
|rapid heartrate |960 |974 |Symptom |
173+
|thyroid |1074 |1080|BodyPart |
174+
+--------------------+-----+----+----------------------+
175+
```
176+
177+
{:.model-param}
178+
## Model Information
179+
180+
{:.table-model}
181+
|---|---|
182+
|Model Name:|ner_vop_slim_wip|
183+
|Compatibility:|Healthcare NLP 4.3.1+|
184+
|License:|Licensed|
185+
|Edition:|Official|
186+
|Input Labels:|[sentence, token, embeddings]|
187+
|Output Labels:|[ner]|
188+
|Language:|en|
189+
|Size:|2.4 MB|
190+
191+
## Benchmarking
192+
193+
```bash
194+
label tp fp fn total precision recall f1
195+
Route 25.0 4.0 15.0 40.0 0.8621 0.625 0.7246
196+
Procedure 161.0 48.0 70.0 231.0 0.7703 0.697 0.7318
197+
Vaccine 22.0 7.0 8.0 30.0 0.7586 0.7333 0.7458
198+
RelationshipStatus 6.0 2.0 2.0 8.0 0.75 0.75 0.75
199+
Disease 884.0 201.0 285.0 1169.0 0.8147 0.7562 0.7844
200+
Frequency 342.0 61.0 113.0 455.0 0.8486 0.7516 0.7972
201+
Duration 720.0 188.0 146.0 866.0 0.793 0.8314 0.8117
202+
Test 478.0 106.0 103.0 581.0 0.8185 0.8227 0.8206
203+
Symptom 1569.0 337.0 340.0 1909.0 0.8232 0.8219 0.8225
204+
DateTime 1558.0 277.0 296.0 1854.0 0.849 0.8403 0.8447
205+
ClinicalDept 157.0 9.0 48.0 205.0 0.9458 0.7659 0.8464
206+
Form 110.0 28.0 11.0 121.0 0.7971 0.9091 0.8494
207+
Dosage_Strength 184.0 25.0 33.0 217.0 0.8804 0.8479 0.8638
208+
Drug 672.0 109.0 103.0 775.0 0.8604 0.8671 0.8638
209+
VitalTest 73.0 7.0 16.0 89.0 0.9125 0.8202 0.8639
210+
Laterality 262.0 43.0 38.0 300.0 0.859 0.8733 0.8661
211+
Age 236.0 42.0 14.0 250.0 0.8489 0.944 0.8939
212+
PsychologicalCond... 144.0 20.0 14.0 158.0 0.878 0.9114 0.8944
213+
BodyPart 1319.0 139.0 160.0 1479.0 0.9047 0.8918 0.8982
214+
Employment 541.0 25.0 77.0 618.0 0.9558 0.8754 0.9139
215+
AdmissionDischarge 13.0 0.0 1.0 14.0 1.0 0.9286 0.963
216+
Gender 548.0 26.0 12.0 560.0 0.9547 0.9786 0.9665
217+
```

0 commit comments

Comments
 (0)