Skip to content

Commit

Permalink
[translation] simplify input to translation (Azure#19060)
Browse files Browse the repository at this point in the history
* add polling implementation

* add polling generation transforms and regenerate

* add tests and recordings

* samples, readme, changelog

* update changelog

* pylint

* some docs updates

* missed an await

* simplify input to translation

* small doc fixes

* pylint

* fix wording

* review feedback
  • Loading branch information
kristapratico authored Jun 3, 2021
1 parent 05ed2d1 commit 338c6a9
Show file tree
Hide file tree
Showing 20 changed files with 1,818 additions and 141 deletions.
3 changes: 3 additions & 0 deletions sdk/translation/azure-ai-translation-document/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ translation has completed.
- Authentication using `azure-identity` credentials now supported.
- see the [Azure Identity documentation](https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/identity/azure-identity/README.md) for more information.
- Added paging and filtering options to `list_all_document_statuses` and `list_submitted_jobs`.
- The input to `begin_translation` now accepts either the parameter `inputs` as a `List[DocumentTranslationInput]` to
perform multiple translations, or the parameters `source_url`, `target_url`, and `target_language_code` to perform a
single translation of your documents.

**Dependency updates**

Expand Down
84 changes: 62 additions & 22 deletions sdk/translation/azure-ai-translation-document/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,26 +141,21 @@ To begin translating your documents, pass a list of `DocumentTranslationInput` i
Constructing a `DocumentTranslationInput` requires that you pass the SAS URLs to your source and target containers (or files)
and the target language(s) for translation.

A single source container with documents can be translated to many different languages:
A single source container with documents can be translated to a different language:

```python
from azure.ai.translation.document import DocumentTranslationInput, TranslationTarget
from azure.core.credentials import AzureKeyCredential
from azure.ai.translation.document import DocumentTranslationClient

my_input = [
DocumentTranslationInput(
source_url="<sas_url_to_source>",
targets=[
TranslationTarget(target_url="<sas_url_to_target_fr>", language_code="fr"),
TranslationTarget(target_url="<sas_url_to_target_de>", language_code="de")
]
)
]
document_translation_client = DocumentTranslationClient("<endpoint>", AzureKeyCredential("<api_key>"))
poller = document_translation_client.begin_translation("<sas_url_to_source>", "<sas_url_to_target>", "<target_language_code>")
```

Or multiple different sources can be provided each with their own targets.

```python
from azure.ai.translation.document import DocumentTranslationInput, TranslationTarget
from azure.core.credentials import AzureKeyCredential
from azure.ai.translation.document import DocumentTranslationClient, DocumentTranslationInput, TranslationTarget

my_input = [
DocumentTranslationInput(
Expand All @@ -185,6 +180,9 @@ my_input = [
]
)
]

document_translation_client = DocumentTranslationClient("<endpoint>", AzureKeyCredential("<api_key>"))
poller = document_translation_client.begin_translation(my_input)
```

> Note: the target_url for each target language must be unique.
Expand All @@ -206,20 +204,61 @@ Sample code snippets are provided to illustrate using long-running operations [b
The following section provides several code snippets covering some of the most common Document Translation tasks, including:

* [Translate your documents](#translate-your-documents "Translate Your Documents")
* [Translate multiple inputs](#translate-multiple-inputs "Translate Multiple Inputs")
* [List translation operations](#list-translation-operations "List Translation Operations")

### Translate your documents
Translate the documents in your source container to the target containers.
Translate the documents in your source container to the target container.

```python
from azure.core.credentials import AzureKeyCredential
from azure.ai.translation.document import DocumentTranslationClient

endpoint = "https://<resource-name>.cognitiveservices.azure.com/"
credential = AzureKeyCredential("<api_key>")
source_container_sas_url_en = "<sas-url-en>"
target_container_sas_url_es = "<sas-url-es>"

document_translation_client = DocumentTranslationClient(endpoint, credential)

poller = document_translation_client.begin_translation(source_container_sas_url_en, target_container_sas_url_es, "es")

result = poller.result()

print("Status: {}".format(poller.status()))
print("Created on: {}".format(poller.details.created_on))
print("Last updated on: {}".format(poller.details.last_updated_on))
print("Total number of translations on documents: {}".format(poller.details.documents_total_count))

print("\nOf total documents...")
print("{} failed".format(poller.details.documents_failed_count))
print("{} succeeded".format(poller.details.documents_succeeded_count))

for document in result:
print("Document ID: {}".format(document.id))
print("Document status: {}".format(document.status))
if document.status == "Succeeded":
print("Source document location: {}".format(document.source_document_url))
print("Translated document location: {}".format(document.translated_document_url))
print("Translated to language: {}\n".format(document.translate_to))
else:
print("Error Code: {}, Message: {}\n".format(document.error.code, document.error.message))
```

### Translate multiple inputs
Begin translating with documents in multiple source containers to multiple target containers in different languages.

```python
from azure.core.credentials import AzureKeyCredential
from azure.ai.translation.document import DocumentTranslationClient, DocumentTranslationInput, TranslationTarget

endpoint = "https://<resource-name>.cognitiveservices.azure.com/"
credential = AzureKeyCredential("<api_key>")
source_container_sas_url_de = "<sas-url-de>"
source_container_sas_url_en = "<sas-url-en>"
target_container_sas_url_es = "<sas-url-es>"
target_container_sas_url_fr = "<sas-url-fr>"
target_container_sas_url_ar = "<sas-url-ar>"

document_translation_client = DocumentTranslationClient(endpoint, credential)

Expand All @@ -231,21 +270,18 @@ poller = document_translation_client.begin_translation(
TranslationTarget(target_url=target_container_sas_url_es, language_code="es"),
TranslationTarget(target_url=target_container_sas_url_fr, language_code="fr"),
],
),
DocumentTranslationInput(
source_url=source_container_sas_url_de,
targets=[
TranslationTarget(target_url=target_container_sas_url_ar, language_code="ar"),
],
)
]
)

result = poller.result()

print("Status: {}".format(poller.status()))
print("Created on: {}".format(poller.details.created_on))
print("Last updated on: {}".format(poller.details.last_updated_on))
print("Total number of translations on documents: {}".format(poller.details.documents_total_count))

print("\nOf total documents...")
print("{} failed".format(poller.details.documents_failed_count))
print("{} succeeded".format(poller.details.documents_succeeded_count))

for document in result:
print("Document ID: {}".format(document.id))
print("Document status: {}".format(document.status))
Expand Down Expand Up @@ -321,6 +357,7 @@ These code samples show common scenario operations with the Azure Document Trans

* Client authentication: [sample_authentication.py][sample_authentication]
* Begin translating documents: [sample_begin_translation.py][sample_begin_translation]
* Translate with multiple inputs: [sample_translate_multiple_inputs.py][sample_translate_multiple_inputs]
* Check the status of documents: [sample_check_document_statuses.py][sample_check_document_statuses]
* List all submitted translation jobs: [sample_list_all_submitted_jobs.py][sample_list_all_submitted_jobs]
* Apply a custom glossary to translation: [sample_translation_with_glossaries.py][sample_translation_with_glossaries]
Expand All @@ -334,6 +371,7 @@ are found under the `azure.ai.translation.document.aio` namespace.

* Client authentication: [sample_authentication_async.py][sample_authentication_async]
* Begin translating documents: [sample_begin_translation_async.py][sample_begin_translation_async]
* Translate with multiple inputs: [sample_translate_multiple_inputs_async.py][sample_translate_multiple_inputs_async]
* Check the status of documents: [sample_check_document_statuses_async.py][sample_check_document_statuses_async]
* List all submitted translation jobs: [sample_list_all_submitted_jobs_async.py][sample_list_all_submitted_jobs_async]
* Apply a custom glossary to translation: [sample_translation_with_glossaries_async.py][sample_translation_with_glossaries_async]
Expand Down Expand Up @@ -390,6 +428,8 @@ This project has adopted the [Microsoft Open Source Code of Conduct][code_of_con
[sample_authentication_async]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/translation/azure-ai-translation-document/samples/async_samples/sample_authentication_async.py
[sample_begin_translation]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/translation/azure-ai-translation-document/samples/sample_begin_translation.py
[sample_begin_translation_async]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/translation/azure-ai-translation-document/samples/async_samples/sample_begin_translation_async.py
[sample_translate_multiple_inputs]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/translation/azure-ai-translation-document/samples/sample_translate_multiple_inputs.py
[sample_translate_multiple_inputs_async]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/translation/azure-ai-translation-document/samples/async_samples/sample_translate_multiple_inputs_async.py
[sample_check_document_statuses]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/translation/azure-ai-translation-document/samples/sample_check_document_statuses.py
[sample_check_document_statuses_async]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/translation/azure-ai-translation-document/samples/async_samples/sample_check_document_statuses_async.py
[sample_list_all_submitted_jobs]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/translation/azure-ai-translation-document/samples/sample_list_all_submitted_jobs.py
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
# ------------------------------------

import json
from typing import Any, TYPE_CHECKING, List, Union
from typing import Any, TYPE_CHECKING, List, Union, overload
from azure.core.tracing.decorator import distributed_trace
from ._generated import BatchDocumentTranslationClient as _BatchDocumentTranslationClient
from ._models import (
Expand All @@ -16,7 +16,7 @@
)
from ._user_agent import USER_AGENT
from ._polling import TranslationPolling, DocumentTranslationLROPollingMethod
from ._helpers import get_http_logging_policy, convert_datetime, get_authentication_policy
from ._helpers import get_http_logging_policy, convert_datetime, get_authentication_policy, get_translation_input
if TYPE_CHECKING:
from azure.core.paging import ItemPaged
from azure.core.credentials import TokenCredential, AzureKeyCredential
Expand Down Expand Up @@ -89,15 +89,32 @@ def close(self):
"""Close the :class:`~azure.ai.translation.document.DocumentTranslationClient` session."""
return self._client.close()

@distributed_trace
@overload
def begin_translation(self, source_url, target_url, target_language_code, **kwargs):
# type: (str, str, str, **Any) -> DocumentTranslationPoller[ItemPaged[DocumentStatusResult]]
pass

@overload
def begin_translation(self, inputs, **kwargs):
# type: (List[DocumentTranslationInput], **Any) -> DocumentTranslationPoller[ItemPaged[DocumentStatusResult]]
"""Begin translating the document(s) in your source container to your TranslationTarget(s)
in the given language.
pass

def begin_translation(self, *args, **kwargs): # pylint: disable=client-method-missing-type-annotations
"""Begin translating the document(s) in your source container to your target container
in the given language. To perform a single translation from source to target, pass the `source_url`,
`target_url`, and `target_language_code` parameters. To pass multiple inputs for translation, including
other translation options, pass the `inputs` parameter as a list of DocumentTranslationInput.
For supported languages and document formats, see the service documentation:
https://docs.microsoft.com/azure/cognitive-services/translator/document-translation/overview
:param str source_url: The source SAS URL to the Azure Blob container containing the documents
to be translated. Requires read and list permissions at the minimum.
:param str target_url: The target SAS URL to the Azure Blob container where the translated documents
should be written. Requires write and list permissions at the minimum.
:param str target_language_code: This is the language you want your documents to be translated to.
See supported language codes here:
https://docs.microsoft.com/azure/cognitive-services/translator/language-support#translate
:param inputs: A list of translation inputs. Each individual input has a single
source URL to documents and can contain multiple TranslationTargets (one for each language)
for the destination to write translated documents.
Expand All @@ -118,6 +135,10 @@ def begin_translation(self, inputs, **kwargs):
:caption: Translate the documents in your storage container.
"""

continuation_token = kwargs.pop("continuation_token", None)

inputs = get_translation_input(args, kwargs, continuation_token)

def deserialization_callback(
raw_response, _, headers
): # pylint: disable=unused-argument
Expand All @@ -128,7 +149,6 @@ def deserialization_callback(
"polling_interval", self._client._config.polling_interval # pylint: disable=protected-access
)

continuation_token = kwargs.pop("continuation_token", None)
pipeline_response = None
if continuation_token:
pipeline_response = self._client.document_translation.get_translation_status(
Expand All @@ -138,8 +158,7 @@ def deserialization_callback(

callback = kwargs.pop("cls", deserialization_callback)
return self._client.document_translation.begin_start_translation(
inputs=DocumentTranslationInput._to_generated_list(inputs) # pylint: disable=protected-access
if not continuation_token else None,
inputs=inputs if not continuation_token else None,
polling=DocumentTranslationLROPollingMethod(
timeout=polling_interval,
lro_algorithms=[
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,51 @@
from azure.core.credentials import AzureKeyCredential
from azure.core.pipeline.policies import AzureKeyCredentialPolicy
from azure.core.pipeline.policies import HttpLoggingPolicy
from ._generated.models import (
BatchRequest as _BatchRequest,
SourceInput as _SourceInput,
TargetInput as _TargetInput,
)
from ._models import DocumentTranslationInput
COGNITIVE_KEY_HEADER = "Ocp-Apim-Subscription-Key"


def get_translation_input(args, kwargs, continuation_token):
try:
inputs = kwargs.pop('inputs', None)
if not inputs:
inputs = args[0]
request = DocumentTranslationInput._to_generated_list(inputs) \
if not continuation_token else None # pylint: disable=protected-access
except (AttributeError, TypeError, IndexError):
try:
source_url = kwargs.pop('source_url', None)
if not source_url:
source_url = args[0]
target_url = kwargs.pop("target_url", None)
if not target_url:
target_url = args[1]
target_language_code = kwargs.pop("target_language_code", None)
if not target_language_code:
target_language_code = args[2]
request = [
_BatchRequest(
source=_SourceInput(
source_url=source_url
),
targets=[_TargetInput(
target_url=target_url,
language=target_language_code
)]
)
]
except (AttributeError, TypeError, IndexError):
raise ValueError("Pass 'inputs' for multiple inputs or 'source_url', 'target_url', "
"and 'target_language_code' for a single input.")

return request


def get_authentication_policy(credential):
authentication_policy = None
if credential is None:
Expand Down
Loading

0 comments on commit 338c6a9

Please sign in to comment.