[formrecognizer] adds samples for labeled tables and addresses sample…

…/doc feedback (#18409)

* add snippet about where to find data in RecognizedForm docstring

* add tables output to the recognize custom form samples

* create labeled tables samples and add training docs

* catalina's feedback

* mari's feedback

sdk/formrecognizer/azure-ai-formrecognizer/azure/ai/formrecognizer/_models.py

@@ -186,6 +186,9 @@ def __init__(self, **kwargs):
 
 class RecognizedForm(object):
     """Represents a form that has been recognized by a trained or prebuilt model.
+    The `fields` property contains the form fields that were extracted from the
+    form. Tables, text lines/words, and selection marks are extracted per page
+    and found in the `pages` property.
 
     :ivar str form_type:
         The type of form the model identified the submitted form to be.

sdk/formrecognizer/azure-ai-formrecognizer/samples/README.md

@@ -66,6 +66,7 @@ what you can do with the Azure Form Recognizer client library.
 |[sample_strongly_typing_recognized_form.py][sample_strongly_typing_recognized_form] and [sample_strongly_typing_recognized_form_async.py][sample_strongly_typing_recognized_form_async]|Use the fields in your recognized forms to create an object with strongly-typed fields|
 |[sample_get_bounding_boxes.py][sample_get_bounding_boxes] and [sample_get_bounding_boxes_async.py][sample_get_bounding_boxes_async]|Get info to visualize the outlines of form content and fields, which can be used for manual validation|
 |[sample_differentiate_output_models_trained_with_and_without_labels.py][sample_differentiate_output_models_trained_with_and_without_labels] and [sample_differentiate_output_models_trained_with_and_without_labels_async.py][sample_differentiate_output_models_trained_with_and_without_labels_async]|See the differences in output when using a custom model trained with labeled data and one trained with unlabeled data|
+|[sample_differentiate_output_labeled_tables.py][sample_differentiate_output_labeled_tables] and [sample_differentiate_output_labeled_tables_async.py][sample_differentiate_output_labeled_tables_async]|See the differences in output when using a custom model trained with fixed vs. dynamic table tags|
 
 [azure_identity]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/identity/azure-identity
 
@@ -109,3 +110,5 @@ what you can do with the Azure Form Recognizer client library.
 [sample_strongly_typing_recognized_form_async]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/formrecognizer/azure-ai-formrecognizer/samples/async_samples/sample_strongly_typing_recognized_form_async.py
 [sample_create_composed_model]: https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/formrecognizer/azure-ai-formrecognizer/samples/sample_create_composed_model.py
 [sample_create_composed_model_async]: https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/formrecognizer/azure-ai-formrecognizer/samples/async_samples/sample_create_composed_model_async.py
+[sample_differentiate_output_labeled_tables]: https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/formrecognizer/azure-ai-formrecognizer/samples/sample_differentiate_output_labeled_tables.py
+[sample_differentiate_output_labeled_tables_async]: https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/formrecognizer/azure-ai-formrecognizer/samples/async_samples/sample_differentiate_output_labeled_tables_async.py

sdk/formrecognizer/azure-ai-formrecognizer/samples/async_samples/sample_differentiate_output_labeled_tables_async.py

@@ -0,0 +1,137 @@
+# coding: utf-8
+
+# -------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for
+# license information.
+# --------------------------------------------------------------------------
+
+"""
+FILE: sample_differentiate_output_labeled_tables_async.py
+
+DESCRIPTION:
+    This sample demonstrates the differences in output that arise when begin_recognize_custom_forms
+    is called with custom models trained with fixed vs. dynamic table tags.
+    The models used in this sample can be created in the sample_train_model_with_labels.py using the
+    training files in
+    https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/formrecognizer/azure-ai-formrecognizer/samples/sample_forms/labeled_tables
+
+    Note that Form Recognizer automatically finds and extracts all tables in your documents whether the tables
+    are tagged/labeled or not. Tables extracted automatically by Form Recognizer will be included in the
+    `tables` property under `RecognizedForm.pages`.
+
+    A conceptual explanation of using table tags to train your custom form model can be found in the
+    service documentation: https://docs.microsoft.com/azure/cognitive-services/form-recognizer/supervised-table-tags
+
+USAGE:
+    python sample_differentiate_output_labeled_tables_async.py
+
+    Set the environment variables with your own values before running the sample:
+    1) AZURE_FORM_RECOGNIZER_ENDPOINT - the endpoint to your Cognitive Services resource.
+    2) AZURE_FORM_RECOGNIZER_KEY - your Form Recognizer API key
+    3) MODEL_ID_FIXED_ROW_TABLES - the ID of your custom model trained with labels on fixed row tables
+    4) MODEL_ID_DYNAMIC_ROW_TABLES - the ID of your custom model trained with labels on dynamic row tables
+"""
+
+import os
+import asyncio
+
+
+class TestDifferentiateOutputLabeledTablesAsync(object):
+
+    async def test_recognize_tables_fixed_rows_async(self):
+        from azure.core.credentials import AzureKeyCredential
+        from azure.ai.formrecognizer.aio import FormRecognizerClient
+
+        endpoint = os.environ["AZURE_FORM_RECOGNIZER_ENDPOINT"]
+        key = os.environ["AZURE_FORM_RECOGNIZER_KEY"]
+        model_id_fixed_rows_table = os.environ["MODEL_ID_FIXED_ROW_TABLES"]
+
+        form_recognizer_client = FormRecognizerClient(
+            endpoint=endpoint, credential=AzureKeyCredential(key)
+        )
+
+        path_to_sample_forms = os.path.abspath(os.path.join(os.path.abspath(__file__),
+                                                            "..", "..", "./sample_forms/forms/label_table_fixed_rows1.pdf"))
+        with open(path_to_sample_forms, "rb") as f:
+            form = f.read()
+
+        async with form_recognizer_client:
+            poller = await form_recognizer_client.begin_recognize_custom_forms(
+                model_id=model_id_fixed_rows_table, form=form
+            )
+
+            result = await poller.result()
+
+        print("\n--------Recognizing labeled table with fixed rows--------\n")
+        for form in result:
+            for name, field in form.fields.items():
+                # substitute "table" for the label given to the table tag during training
+                # (if different than sample training docs)
+                if name == "table":
+                    for row_name, column in field.value.items():
+                        print("Row '{}' has columns:".format(row_name))
+                        for column_name, column_value in column.value.items():
+                            print("...Column '{}' with value '{}' and a confidence score of {}".format(
+                                column_name, column_value.value, column_value.confidence
+                            ))
+                else:  # non-table tagged FormField
+                    print("...Field '{}' has value '{}' with a confidence score of {}".format(
+                        name,
+                        field.value,
+                        field.confidence
+                    ))
+
+    async def test_recognize_tables_dynamic_rows_async(self):
+        from azure.core.credentials import AzureKeyCredential
+        from azure.ai.formrecognizer.aio import FormRecognizerClient
+
+        endpoint = os.environ["AZURE_FORM_RECOGNIZER_ENDPOINT"]
+        key = os.environ["AZURE_FORM_RECOGNIZER_KEY"]
+        model_id_dynamic_rows_table = os.environ["MODEL_ID_DYNAMIC_ROW_TABLES"]
+
+        form_recognizer_client = FormRecognizerClient(
+            endpoint=endpoint, credential=AzureKeyCredential(key)
+        )
+
+        path_to_sample_forms = os.path.abspath(os.path.join(os.path.abspath(__file__),
+                                                            "..", "..", "./sample_forms/forms/label_table_dynamic_rows1.pdf"))
+        with open(path_to_sample_forms, "rb") as f:
+            form = f.read()
+
+        async with form_recognizer_client:
+            poller = await form_recognizer_client.begin_recognize_custom_forms(
+                model_id=model_id_dynamic_rows_table, form=form
+            )
+
+            result = await poller.result()
+
+        print("\n\n--------Recognizing labeled table with dynamic rows--------\n")
+        for form in result:
+            for name, field in form.fields.items():
+                # substitute "table" for the label given to the table tag during training
+                # (if different than sample training docs)
+                if name == "table":
+                    for idx, row in enumerate(field.value):
+                        print("Row {}".format(idx+1))
+                        for column_name, row_value in row.value.items():
+                            print("...Column '{}' with value '{}' and a confidence score of {}".format(
+                                column_name, row_value.value, row_value.confidence
+                            ))
+                else:  # non-table tagged FormField
+                    print("...Field '{}' has value '{}' with a confidence score of {}".format(
+                        name,
+                        field.value,
+                        field.confidence
+                    ))
+
+
+async def main():
+    sample = TestDifferentiateOutputLabeledTablesAsync()
+    await sample.test_recognize_tables_fixed_rows_async()
+    await sample.test_recognize_tables_dynamic_rows_async()
+
+
+if __name__ == '__main__':
+    loop = asyncio.get_event_loop()
+    loop.run_until_complete(main())

sdk/formrecognizer/azure-ai-formrecognizer/samples/async_samples/sample_recognize_custom_forms_async.py

@@ -48,7 +48,7 @@ async def recognize_custom_forms(self):
             # Make sure your form's type is included in the list of form types the custom model can recognize
             with open(path_to_sample_forms, "rb") as f:
                 poller = await form_recognizer_client.begin_recognize_custom_forms(
-                    model_id=model_id, form=f
+                    model_id=model_id, form=f, include_field_elements=True
                 )
             forms = await poller.result()
 
@@ -72,6 +72,30 @@ async def recognize_custom_forms(self):
                         field.label_data.text if field.label_data else name, field.value, field.confidence
                     ))
 
+                # iterate over tables, lines, and selection marks on each page
+                for page in form.pages:
+                    for i, table in enumerate(page.tables):
+                        print("\nTable {} on page {}".format(i + 1, table.page_number))
+                        for cell in table.cells:
+                            print("...Cell[{}][{}] has text '{}' with confidence {}".format(
+                                cell.row_index, cell.column_index, cell.text, cell.confidence
+                            ))
+                    print("\nLines found on page {}".format(page.page_number))
+                    for line in page.lines:
+                        print("...Line '{}' is made up of the following words: ".format(line.text))
+                        for word in line.words:
+                            print("......Word '{}' has a confidence of {}".format(
+                                word.text,
+                                word.confidence
+                            ))
+                    if page.selection_marks:
+                        print("\nSelection marks found on page {}".format(page.page_number))
+                        for selection_mark in page.selection_marks:
+                            print("......Selection mark is '{}' and has a confidence of {}".format(
+                                selection_mark.state,
+                                selection_mark.confidence
+                            ))
+
                 print("-----------------------------------")
         # [END recognize_custom_forms_async]
 

sdk/formrecognizer/azure-ai-formrecognizer/samples/sample_differentiate_output_labeled_tables.py

@@ -0,0 +1,129 @@
+# coding: utf-8
+
+# -------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for
+# license information.
+# --------------------------------------------------------------------------
+
+"""
+FILE: sample_differentiate_output_labeled_tables.py
+
+DESCRIPTION:
+    This sample demonstrates the differences in output that arise when begin_recognize_custom_forms
+    is called with custom models trained with fixed vs. dynamic table tags.
+    The models used in this sample can be created in the sample_train_model_with_labels.py using the
+    training files in
+    https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/formrecognizer/azure-ai-formrecognizer/samples/sample_forms/labeled_tables
+
+    Note that Form Recognizer automatically finds and extracts all tables in your documents whether the tables
+    are tagged/labeled or not. Tables extracted automatically by Form Recognizer will be included in the
+    `tables` property under `RecognizedForm.pages`.
+
+    A conceptual explanation of using table tags to train your custom form model can be found in the
+    service documentation: https://docs.microsoft.com/azure/cognitive-services/form-recognizer/supervised-table-tags
+
+USAGE:
+    python sample_differentiate_output_labeled_tables.py
+
+    Set the environment variables with your own values before running the sample:
+    1) AZURE_FORM_RECOGNIZER_ENDPOINT - the endpoint to your Cognitive Services resource.
+    2) AZURE_FORM_RECOGNIZER_KEY - your Form Recognizer API key
+    3) MODEL_ID_FIXED_ROW_TABLES - the ID of your custom model trained with labels on fixed row tables
+    4) MODEL_ID_DYNAMIC_ROW_TABLES - the ID of your custom model trained with labels on dynamic row tables
+"""
+
+import os
+
+
+class TestDifferentiateOutputLabeledTables(object):
+
+    def test_recognize_tables_fixed_rows(self):
+        from azure.core.credentials import AzureKeyCredential
+        from azure.ai.formrecognizer import FormRecognizerClient
+
+        endpoint = os.environ["AZURE_FORM_RECOGNIZER_ENDPOINT"]
+        key = os.environ["AZURE_FORM_RECOGNIZER_KEY"]
+        model_id_fixed_rows_table = os.environ["MODEL_ID_FIXED_ROW_TABLES"]
+
+        form_recognizer_client = FormRecognizerClient(
+            endpoint=endpoint, credential=AzureKeyCredential(key)
+        )
+
+        path_to_sample_forms = os.path.abspath(os.path.join(os.path.abspath(__file__),
+                                                            "..", "./sample_forms/forms/label_table_fixed_rows1.pdf"))
+
+        with open(path_to_sample_forms, "rb") as f:
+            form = f.read()
+        poller = form_recognizer_client.begin_recognize_custom_forms(
+            model_id=model_id_fixed_rows_table, form=form
+        )
+
+        result = poller.result()
+
+        print("\n--------Recognizing labeled table with fixed rows--------\n")
+        for form in result:
+            for name, field in form.fields.items():
+                # substitute "table" for the label given to the table tag during training
+                # (if different than sample training docs)
+                if name == "table":
+                    for row_name, column in field.value.items():
+                        print("Row '{}' has columns:".format(row_name))
+                        for column_name, column_value in column.value.items():
+                            print("...Column '{}' with value '{}' and a confidence score of {}".format(
+                                column_name, column_value.value, column_value.confidence
+                            ))
+                else:  # non-table tagged FormField
+                    print("...Field '{}' has value '{}' with a confidence score of {}".format(
+                        name,
+                        field.value,
+                        field.confidence
+                    ))
+
+    def test_recognize_tables_dynamic_rows(self):
+        from azure.core.credentials import AzureKeyCredential
+        from azure.ai.formrecognizer import FormRecognizerClient
+
+        endpoint = os.environ["AZURE_FORM_RECOGNIZER_ENDPOINT"]
+        key = os.environ["AZURE_FORM_RECOGNIZER_KEY"]
+        model_id_dynamic_rows_table = os.environ["MODEL_ID_DYNAMIC_ROW_TABLES"]
+
+        form_recognizer_client = FormRecognizerClient(
+            endpoint=endpoint, credential=AzureKeyCredential(key)
+        )
+
+        path_to_sample_forms = os.path.abspath(os.path.join(os.path.abspath(__file__),
+                                                            "..", "./sample_forms/forms/label_table_dynamic_rows1.pdf"))
+
+        with open(path_to_sample_forms, "rb") as f:
+            form = f.read()
+        poller = form_recognizer_client.begin_recognize_custom_forms(
+            model_id=model_id_dynamic_rows_table, form=form
+        )
+
+        result = poller.result()
+
+        print("\n\n--------Recognizing labeled table with dynamic rows--------\n")
+        for form in result:
+            for name, field in form.fields.items():
+                # substitute "table" for the label given to the table tag during training
+                # (if different than sample training docs)
+                if name == "table":
+                    for idx, row in enumerate(field.value):
+                        print("Row {}".format(idx+1))
+                        for column_name, row_value in row.value.items():
+                            print("...Column '{}' with value '{}' and a confidence score of {}".format(
+                                column_name, row_value.value, row_value.confidence
+                            ))
+                else:  # non-table tagged FormField
+                    print("...Field '{}' has value '{}' with a confidence score of {}".format(
+                        name,
+                        field.value,
+                        field.confidence
+                    ))
+
+
+if __name__ == '__main__':
+    sample = TestDifferentiateOutputLabeledTables()
+    sample.test_recognize_tables_fixed_rows()
+    sample.test_recognize_tables_dynamic_rows()

sdk/formrecognizer/azure-ai-formrecognizer/samples/sample_forms/labeled_tables/dynamic/fields.json

@@ -0,0 +1,50 @@
+{
+    "$schema": "https://schema.cognitiveservices.azure.com/formrecognizer/2021-03-01/fields.json",
+    "fields": [
+        {
+            "fieldKey": "table",
+            "fieldType": "array",
+            "fieldFormat": "not-specified",
+            "itemType": "table_object",
+            "fields": null
+        }
+    ],
+    "definitions": {
+        "table_object": {
+            "fieldKey": "table_object",
+            "fieldType": "object",
+            "fieldFormat": "not-specified",
+            "itemType": null,
+            "fields": [
+                {
+                    "fieldKey": "Item",
+                    "fieldType": "string",
+                    "fieldFormat": "not-specified",
+                    "itemType": null,
+                    "fields": null
+                },
+                {
+                    "fieldKey": "Price",
+                    "fieldType": "string",
+                    "fieldFormat": "not-specified",
+                    "itemType": null,
+                    "fields": null
+                },
+                {
+                    "fieldKey": "Tax",
+                    "fieldType": "string",
+                    "fieldFormat": "not-specified",
+                    "itemType": null,
+                    "fields": null
+                },
+                {
+                    "fieldKey": "Total",
+                    "fieldType": "string",
+                    "fieldFormat": "not-specified",
+                    "itemType": null,
+                    "fields": null
+                }
+            ]
+        }
+    }
+}