PVQ-4317 Implement detect_language_txt_to_txt subtype action

- added option for desktop to access this feature
- added test for this feature

Author: peterkral-pdfix

config.json

@@ -57,6 +57,35 @@
                     "value": ""
                 }
             ]
+        },
+        {
+            "title": "Detect TXT Language to Text (LangDetect)",
+            "name": "pdfix_detect_pdf_language_to_text_langdetect",
+            "desc": "Automatically detects the language of a TXT and saves the detected language code to a TXT file [Local]",
+            "version": "v0.0.0",
+            "icon": "language_txt",
+            "category": "Metadata",
+            "subtype": "detect_language_txt_to_txt",
+            "local": "True",
+            "program": "docker run -v ${working_directory}:/data -w /data --rm pdfix/detect-language:latest lang-detect --name \"${license_name}\" --key \"${license_key}\" -i \"/data/${input_txt}\" -o \"/data/${output_txt}\"",
+            "args": [
+                {
+                    "name": "input_txt",
+                    "desc": "Input text file",
+                    "flags": 2,
+                    "type": "file_path",
+                    "ext": "txt",
+                    "value": ""
+                },
+                {
+                    "name": "output_txt",
+                    "desc": "Output text file containing the detected language code",
+                    "flags": 4,
+                    "type": "file_path",
+                    "ext": "txt",
+                    "value": ""
+                }
+            ]
         }
     ]
 }

example/language_identification_wikipedia.txt

@@ -0,0 +1,24 @@
+Language identification
+---
+
+From Wikipedia, the free encyclopedia
+In natural language processing, language identification or language guessing is the problem of determining which natural language given content is in. Computational approaches to this problem view it as a special case of text categorization, solved with various statistical methods.
+
+Overview
+---
+There are several statistical approaches to language identification using different techniques to classify the data. One technique is to compare the compressibility of the text to the compressibility of texts in a set of known languages. This approach is known as mutual information based distance measure. The same technique can also be used to empirically construct family trees of languages which closely correspond to the trees constructed using historical methods.[citation needed] Mutual information based distance measure is essentially equivalent to more conventional model-based methods and is not generally considered to be either novel or better than simpler techniques.
+
+Another technique, as described by Cavnar and Trenkle (1994) and Dunning (1994) is to create a language n-gram model from a "training text" for each of the languages. These models can be based on characters (Cavnar and Trenkle) or encoded bytes (Dunning); in the latter, language identification and character encoding detection are integrated. Then, for any piece of text needing to be identified, a similar model is made, and that model is compared to each stored language model. The most likely language is the one with the model that is most similar to the model from the text needing to be identified. This approach can be problematic when the input text is in a language for which there is no model. In that case, the method may return another, "most similar" language as its result. Also problematic for any approach are pieces of input text that are composed of several languages, as is common on the Web.
+
+For a more recent method, see Řehůřek and Kolkus (2009). This method can detect multiple languages in an unstructured piece of text and works robustly on short texts of only a few words: something that the n-gram approaches struggle with.
+
+An older statistical method by Grefenstette was based on the prevalence of certain function words (e.g., "the" in English).
+
+A common non-statistical intuitive approach (though highly uncertain) is to look for common letter combinations, or distinctive diacritics or punctuation.[1][2]
+
+Identifying similar languages
+---
+One of the great bottlenecks of language identification systems is to distinguish between closely related languages. Similar languages like Bulgarian and Macedonian or Indonesian and Malay present significant lexical and structural overlap, making it challenging for systems to discriminate between them.
+
+In 2014 the DSL shared task[3] has been organized providing a dataset (Tan et al., 2014) containing 13 different languages (and language varieties) in six language groups: Group A (Bosnian, Croatian, Serbian), Group B (Indonesian, Malaysian), Group C (Czech, Slovak), Group D (Brazilian Portuguese, European Portuguese), Group E (Peninsular Spanish, Argentine Spanish), Group F (American English, British English). The best system reached performance of over 95% results (Goutte et al., 2014). Results of the DSL shared task are described in Zampieri et al. 2014.
+

test.sh

@@ -63,7 +63,7 @@ else
     EXIT_STATUS=1
 fi
 
-info "Test #04: Run language detection to txt"
+info "Test #04: Run language detection pdf to txt"
 docker run --rm $PLATFORM -v $(pwd):/data -w /data $DOCKER_IMAGE lang-detect -i example/air_quality.pdf -o $TEMPORARY_DIRECTORY/air_quality.txt > /dev/null
 if [ -f "$(pwd)/$TEMPORARY_DIRECTORY/air_quality.txt" ]; then
     success "passed"
@@ -72,9 +72,18 @@ else
     EXIT_STATUS=1
 fi
 
+info "Test #05: Run language detection txt to txt"
+docker run --rm $PLATFORM -v $(pwd):/data -w /data $DOCKER_IMAGE lang-detect -i example/language_identification_wikipedia.txt -o $TEMPORARY_DIRECTORY/language_identification_wikipedia_lang.txt > /dev/null
+if [ -f "$(pwd)/$TEMPORARY_DIRECTORY/language_identification_wikipedia_lang.txt" ]; then
+    success "passed"
+else
+    error "language detection to txt failed on example/language_identification_wikipedia.txt"
+    EXIT_STATUS=1
+fi
+
 # Move these tests to functional tests
 
-# info "Test #05: Run lang-detect on pdf with empty page"
+# info "Test #06: Run lang-detect on pdf with empty page"
 # docker run --rm $PLATFORM -v $(pwd):/data -w /data $DOCKER_IMAGE lang-detect -i example/empty_page.pdf -o $TEMPORARY_DIRECTORY/empty_page.txt > /dev/null
 # if [ -f "$(pwd)/$TEMPORARY_DIRECTORY/empty_page.txt" ]; then
 #     success "passed"
@@ -83,7 +92,7 @@ fi
 #     EXIT_STATUS=1
 # fi
 
-# info "Test #06: Run lang-detect on pdf with numbers"
+# info "Test #07: Run lang-detect on pdf with numbers"
 # docker run --rm $PLATFORM -v $(pwd):/data -w /data $DOCKER_IMAGE lang-detect -i example/pdfix_6_0_0_0053.pdf -o $TEMPORARY_DIRECTORY/num.txt > /dev/null
 # if [ -f "$(pwd)/$TEMPORARY_DIRECTORY/empty_page.txt" ]; then
 #     success "passed"
@@ -96,6 +105,7 @@ info "Cleaning up temporary files from tests"
 rm -f $TEMPORARY_DIRECTORY/config.json
 rm -f $TEMPORARY_DIRECTORY/air_quality.pdf
 rm -f $TEMPORARY_DIRECTORY/air_quality.txt
+rm -f $TEMPORARY_DIRECTORY/language_identification_wikipedia_lang.txt
 rmdir $(pwd)/$TEMPORARY_DIRECTORY
 
 info "Removing testing docker image"