Skip to content

Doc fix scala and open source #13008

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 3, 2022
Merged

Doc fix scala and open source #13008

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/_posts/josejuanmartinez/2021-11-01-roberta_base_biomedical_es.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,15 +57,15 @@ tokenizer,
roberta_embeddings])
```
```scala
val documentAssembler = new nlp.DocumentAssembler()
val documentAssembler = new DocumentAssembler()
.setInputCol("term")
.setOutputCol("document")

val tokenizer = new nlp.Tokenizer()
val tokenizer = new Tokenizer()
.setInputCols("document")
.setOutputCol("token")

val roberta_embeddings = nlp.RoBertaEmbeddings.pretrained("roberta_base_biomedical", "es")
val roberta_embeddings = RoBertaEmbeddings.pretrained("roberta_base_biomedical", "es")
.setInputCols(Array("document", "token"))
.setOutputCol("roberta_embeddings")

Expand Down
12 changes: 6 additions & 6 deletions docs/_posts/josejuanmartinez/2021-11-03-robertaresolve_snomed_es.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ chunk_word_embeddings = nlp.RoBertaEmbeddings.pretrained("roberta_base_biomedica
.setInputCols(["sentence", "token"])\
.setOutputCol("ner_chunk_word_embeddings")

chunk_embeddings = SentenceEmbeddings() \
chunk_embeddings = nlp.SentenceEmbeddings() \
.setInputCols(["sentence", "ner_chunk_word_embeddings"]) \
.setOutputCol("ner_chunk_embeddings") \
.setPoolingStrategy("AVERAGE")
Expand Down Expand Up @@ -77,15 +77,15 @@ result = p_model.transform(spark.createDataFrame(pd.DataFrame({'text': [test_sen

```scala
...
val c2doc = new nlp.Chunk2Doc()
val c2doc = new Chunk2Doc()
.setInputCols(Array("ner_chunk"))
.setOutputCol("sentence")

val chunk_tokenizer = new nlp.Tokenizer()
val chunk_tokenizer = new Tokenizer()
.setInputCols("sentence")
.setOutputCol("token")

val chunk_word_embeddings = nlp.RoBertaEmbeddings.pretrained("roberta_base_biomedical", "es")
val chunk_word_embeddings = RoBertaEmbeddings.pretrained("roberta_base_biomedical", "es")
.setInputCols(Array("sentence", "token"))
.setOutputCol("ner_chunk_word_embeddings")

Expand All @@ -94,12 +94,12 @@ val chunk_embeddings = new SentenceEmbeddings()
.setOutputCol("ner_chunk_embeddings")
.setPoolingStrategy("AVERAGE")

val er = medical.SentenceEntityResolverModel.pretrained("robertaresolve_snomed", "es", "clinical/models")
val er = SentenceEntityResolverModel.pretrained("robertaresolve_snomed", "es", "clinical/models")
.setInputCols(Array("sentence", "ner_chunk_embeddings"))
.setOutputCol("snomed_code")
.setDistanceFunction("EUCLIDEAN")

val snomed_pipeline = new PipelineModel().setStages(Array(
val snomed_pipeline = new Pipeline().setStages(Array(
c2doc,
chunk_tokenizer,
chunk_word_embeddings,
Expand Down
16 changes: 8 additions & 8 deletions ...s/josejuanmartinez/2021-11-04-bert_sequence_classifier_question_statement_en.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,19 +37,19 @@ This model was imported from Hugging Face (https://huggingface.co/shahrukhx01/qu
<div class="tabs-box" markdown="1">
{% include programmingLanguageSelectScalaPythonNLU.html %}
```python
documentAssembler = nlp.DocumentAssembler()\
documentAssembler = DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")

sentenceDetector = nlp.SentenceDetectorDLModel.pretrained() \
sentenceDetector = SentenceDetectorDLModel.pretrained() \
.setInputCols(["document"]) \
.setOutputCol("sentence")

tokenizer = nlp.Tokenizer()\
tokenizer = Tokenizer()\
.setInputCols("sentence")\
.setOutputCol("token")

seq = nlp.BertForSequenceClassification.pretrained('bert_sequence_classifier_question_statement', 'en')\
seq = BertForSequenceClassification.pretrained('bert_sequence_classifier_question_statement', 'en')\
.setInputCols(["token", "sentence"])\
.setOutputCol("label")\
.setCaseSensitive(True)
Expand All @@ -75,19 +75,19 @@ data=spark.createDataFrame(pd.DataFrame({'text': test_sentences}))
res = pipeline.fit(data).transform(data)
```
```scala
val documentAssembler = new nlp.DocumentAssembler()
val documentAssembler = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")

val sentenceDetector = nlp.SentenceDetectorDLModel.pretrained()
val sentenceDetector = SentenceDetectorDLModel.pretrained()
.setInputCols(Array("document"))
.setOutputCol("sentence")

val tokenizer = new nlp.Tokenizer()
val tokenizer = new Tokenizer()
.setInputCols("sentence")
.setOutputCol("token")

val seq = nlp.BertForSequenceClassification.pretrained("bert_sequence_classifier_question_statement", "en")
val seq = BertForSequenceClassification.pretrained("bert_sequence_classifier_question_statement", "en")
.setInputCols(Array("token", "sentence"))
.setOutputCol("label")
.setCaseSensitive(True)
Expand Down
12 changes: 6 additions & 6 deletions docs/_posts/josejuanmartinez/2021-11-04-roberta_ner_diag_proc_es.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,27 +94,27 @@ import pandas as pd
res = p_model.transform(spark.createDataFrame(pd.DataFrame({'text': [test_sentence]})))
```
```scala
val documentAssembler = new nlp.DocumentAssembler()
val documentAssembler = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")

val sentenceDetector = nlp.SentenceDetectorDLModel.pretrained()
val sentenceDetector = SentenceDetectorDLModel.pretrained()
.setInputCols(Array("document"))
.setOutputCol("sentence")

val tokenizer = new nlp.Tokenizer()
val tokenizer = new Tokenizer()
.setInputCols("sentence")
.setOutputCol("token")

val embeddings = nlp.RoBertaEmbeddings.pretrained("roberta_base_biomedical", "es")
val embeddings = RoBertaEmbeddings.pretrained("roberta_base_biomedical", "es")
.setInputCols(Array("sentence", "token"))
.setOutputCol("embeddings")

val ner = medical.NerModel.pretrained("roberta_ner_diag_proc", "es", "clinical/models")
val ner = MedicalNerModel.pretrained("roberta_ner_diag_proc", "es", "clinical/models")
.setInputCols(Array("sentence", "token", "embeddings"))
.setOutputCol("ner")

val ner_converter = new nlp.NerConverter()
val ner_converter = new NerConverter()
.setInputCols(Array("sentence", "token", "ner"))
.setOutputCol("ner_chunk")

Expand Down
8 changes: 4 additions & 4 deletions ...nmartinez/2021-11-05-bert_sequence_classifier_question_statement_clinical_en.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,19 +79,19 @@ data = spark.createDataFrame(test_sentences).toDF("text")
res = pipeline.fit(data).transform(data)
```
```scala
val documentAssembler = new nlp.DocumentAssembler()
val documentAssembler = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")

val sentenceDetector = nlp.SentenceDetectorDLModel.pretrained()
val sentenceDetector = SentenceDetectorDLModel.pretrained()
.setInputCols(Array("document"))
.setOutputCol("sentence")

val tokenizer = new nlp.Tokenizer()
val tokenizer = new Tokenizer()
.setInputCols("sentence")
.setOutputCol("token")

val seq = nlp.BertForSequenceClassification.pretrained("bert_sequence_classifier_question_statement_clinical", "en", "clinical/models")
val seq = BertForSequenceClassification.pretrained("bert_sequence_classifier_question_statement_clinical", "en", "clinical/models")
.setInputCols(Array("token", "sentence"))
.setOutputCol("label")
.setCaseSensitive(True)
Expand Down
16 changes: 8 additions & 8 deletions docs/_posts/josejuanmartinez/2022-01-04-electra_medal_acronym_en.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,38 +35,38 @@ Electra model fine tuned on MeDAL, a large dataset on abbreviation disambiguatio
<div class="tabs-box" markdown="1">
{% include programmingLanguageSelectScalaPythonNLU.html %}
```python
documentAssembler= nlp.DocumentAssembler()\
documentAssembler= DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")

sentenceDetector = nlp.SentenceDetector()\
sentenceDetector = SentenceDetector()\
.setInputCols(["document"])\
.setOutputCol("sentence")

tokenizer= nlp.Tokenizer()\
tokenizer= Tokenizer()\
.setInputCols(["sentence"])\
.setOutputCol("token")

embeddings = nlp.BertEmbeddings.pretrained("electra_medal_acronym", "en") \
embeddings = BertEmbeddings.pretrained("electra_medal_acronym", "en") \
.setInputCols("sentence", "token") \
.setOutputCol("embeddings")

nlpPipeline= Pipeline(stages=[documentAssembler, sentenceDetector, tokenizer, embeddings])
```
```scala
val documentAssembler = new nlp.DocumentAssembler()
val documentAssembler = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")

val sentenceDetector = new nlp.SentenceDetector()
val sentenceDetector = new SentenceDetector()
.setInputCols(Array("document"))
.setOutputCol("sentence")

val tokenizer = new nlp.Tokenizer()
val tokenizer = new Tokenizer()
.setInputCols(Array("sentence"))
.setOutputCol("token")

val embeddings = nlp.BertEmbeddings.pretrained("electra_medal_acronym", "en")
val embeddings = BertEmbeddings.pretrained("electra_medal_acronym", "en")
.setInputCols("sentence", "token")
.setOutputCol("embeddings")

Expand Down
12 changes: 6 additions & 6 deletions docs/_posts/josejuanmartinez/2022-01-17-ner_deid_generic_roberta_es.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,23 +85,23 @@ df = spark.createDataFrame([text]).toDF("text")
results = nlpPipeline.fit(df).transform(df)
```
```scala
val documentAssembler = new nlp.DocumentAssembler()
val documentAssembler = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")

val sentenceDetector = nlp.SentenceDetectorDLModel.pretrained("sentence_detector_dl","xx")
val sentenceDetector = SentenceDetectorDLModel.pretrained("sentence_detector_dl","xx")
.setInputCols(Array("document"))
.setOutputCol("sentence")

val tokenizer = new nlp.Tokenizer()
val tokenizer = new Tokenizer()
.setInputCols(Array("sentence"))
.setOutputCol("token")

val roberta_embeddings = nlp.RoBertaEmbeddings.pretrained("roberta_base_biomedical", "es")
val roberta_embeddings = RoBertaEmbeddings.pretrained("roberta_base_biomedical", "es")
.setInputCols(Array("sentence", "token"))
.setOutputCol("embeddings")

val clinical_ner = medical.NerModel.pretrained("ner_deid_generic_roberta", "es", "clinical/models")
val clinical_ner = MedicalNerModel.pretrained("ner_deid_generic_roberta", "es", "clinical/models")
.setInputCols(Array("sentence","token","embeddings"))
.setOutputCol("ner")

Expand All @@ -111,7 +111,7 @@ val text = """Antonio Pérez Juan, nacido en Cadiz, España. Aún no estaba vacu

val df = Seq(text).toDS.toDF("text")

val results = pipeline.fit(data).transform(data)
val results = pipeline.fit(df).transform(df)
```
</div>

Expand Down
12 changes: 6 additions & 6 deletions docs/_posts/josejuanmartinez/2022-01-17-ner_deid_subentity_roberta_es.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,23 +85,23 @@ df = spark.createDataFrame([text]).toDF("text")
results = nlpPipeline.fit(df).transform(df)
```
```scala
val documentAssembler = new nlp.DocumentAssembler()
val documentAssembler = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")

val sentenceDetector = nlp.SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","xx")
val sentenceDetector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","xx")
.setInputCols(Array("document"))
.setOutputCol("sentence")

val tokenizer = new nlp.Tokenizer()
val tokenizer = new Tokenizer()
.setInputCols(Array("sentence"))
.setOutputCol("token")

val roberta_embeddings = nlp.RoBertaEmbeddings.pretrained("roberta_base_biomedical", "es")
val roberta_embeddings = RoBertaEmbeddings.pretrained("roberta_base_biomedical", "es")
.setInputCols(Array("sentence", "token"))
.setOutputCol("embeddings")

val clinical_ner = medical.NerModel.pretrained("ner_deid_subentity_roberta", "es", "clinical/models")
val clinical_ner = MedicalNerModel.pretrained("ner_deid_subentity_roberta", "es", "clinical/models")
.setInputCols(Array("sentence","token","embeddings"))
.setOutputCol("ner")

Expand All @@ -111,7 +111,7 @@ val text = """Antonio Pérez Juan, nacido en Cadiz, España. Aún no estaba vacu

val df = Seq(text).toDS.toDF("text")

val results = pipeline.fit(data).transform(data)
val results = pipeline.fit(df).transform(df)
```
</div>

Expand Down
12 changes: 6 additions & 6 deletions docs/_posts/josejuanmartinez/2022-01-18-ner_deid_generic_es.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,23 +85,23 @@ df = spark.createDataFrame([text]).toDF("text")
results = nlpPipeline.fit(df).transform(df)
```
```scala
val documentAssembler = new nlp.DocumentAssembler()
val documentAssembler = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")

val sentenceDetector = nlp.SentenceDetectorDLModel.pretrained("sentence_detector_dl","xx")
val sentenceDetector = SentenceDetectorDLModel.pretrained("sentence_detector_dl","xx")
.setInputCols(Array("document"))
.setOutputCol("sentence")

val tokenizer = new nlp.Tokenizer()
val tokenizer = new Tokenizer()
.setInputCols(Array("sentence"))
.setOutputCol("token")

embeddings = nlp.WordEmbeddingsModel.pretrained("embeddings_sciwiki_300d","es","clinical/models")
embeddings = WordEmbeddingsModel.pretrained("embeddings_sciwiki_300d","es","clinical/models")
.setInputCols(Array("sentence","token"))
.setOutputCol("word_embeddings")

clinical_ner = medical.NerModel.pretrained("ner_deid_generic", "es", "clinical/models")
clinical_ner = MedicalNerModel.pretrained("ner_deid_generic", "es", "clinical/models")
.setInputCols(Array("sentence","token","word_embeddings"))
.setOutputCol("ner")

Expand All @@ -111,7 +111,7 @@ val text = """Antonio Pérez Juan, nacido en Cadiz, España. Aún no estaba vacu

val df = Seq(text).toDS.toDF("text")

val results = pipeline.fit(data).transform(data)
val results = pipeline.fit(df).transform(df)
```


Expand Down
10 changes: 5 additions & 5 deletions docs/_posts/josejuanmartinez/2022-01-18-ner_deid_subentity_es.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,23 +84,23 @@ data = spark.createDataFrame([text]).toDF("text")
results = nlpPipeline.fit(data).transform(data)
```
```scala
val documentAssembler = new nlp.DocumentAssembler()
val documentAssembler = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")

val sentenceDetector = nlp.SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","xx")
val sentenceDetector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","xx")
.setInputCols(Array("document"))
.setOutputCol("sentence")

val tokenizer = new nlp.Tokenizer()
val tokenizer = new Tokenizer()
.setInputCols(Array("sentence"))
.setOutputCol("token")

val embeddings = nlp.WordEmbeddingsModel.pretrained("embeddings_sciwiki_300d","es","clinical/models")
val embeddings = WordEmbeddingsModel.pretrained("embeddings_sciwiki_300d","es","clinical/models")
.setInputCols(Array("sentence", "token"))
.setOutputCol("embeddings")

val clinical_ner = medical.NerModel.pretrained("ner_deid_subentity", "es", "clinical/models")
val clinical_ner = MedicalNerModel.pretrained("ner_deid_subentity", "es", "clinical/models")
.setInputCols(Array("sentence","token","embeddings"))
.setOutputCol("ner")

Expand Down
2 changes: 1 addition & 1 deletion docs/_posts/josejuanmartinez/2022-01-24-clinical_deidentification_es.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ This pipeline is trained with sciwiki_300d embeddings and can be used to deident
<div class="tabs-box" markdown="1">
{% include programmingLanguageSelectScalaPythonNLU.html %}
```python
from sparknlp.pretrained import PretrainedPipeline
from johnsnowlabs import *

deid_pipeline = PretrainedPipeline("clinical_deidentification", "es", "clinical/models")

Expand Down
2 changes: 1 addition & 1 deletion docs/_posts/josejuanmartinez/2022-02-11-clinical_deidentification_es.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ This pipeline is trained with sciwiki_300d embeddings and can be used to deident
<div class="tabs-box" markdown="1">
{% include programmingLanguageSelectScalaPythonNLU.html %}
```python
from sparknlp.pretrained import PretrainedPipeline
from johnsnowlabs import *

deid_pipeline = PretrainedPipeline("clinical_deidentification", "es", "clinical/models")

Expand Down
2 changes: 1 addition & 1 deletion docs/_posts/josejuanmartinez/2022-02-14-clinical_deidentification_es.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ This pipeline is trained with sciwiki_300d embeddings and can be used to deident
<div class="tabs-box" markdown="1">
{% include programmingLanguageSelectScalaPythonNLU.html %}
```python
from sparknlp.pretrained import PretrainedPipeline
from johnsnowlabs import *

deid_pipeline = PretrainedPipeline("clinical_deidentification", "es", "clinical/models")

Expand Down
Loading