Ocr actions

[zahin178]

PR Type

Feature

PR Checklist

• Tests for the changes have been added / updated.
• Documentation comments have been added / updated.
• A changelog entry has been made.
• Version number has been updated.
• Required modules have been added to respective "requirements*.txt" files.
• Relevant Test Cases added to this description (below).
• (Team) Label with affected action categories and semver status.

Overview

Added 4 new OCR-based actions for clicking and extracting value

Made changes in the Framework\Built_In_Automation\Sequential_Actions\action_declarations\desktop.py file. Linked the newly created functions to their action names

{ "name": "ocr click",                  "function": "ocr_click",                    "screenshot": "desktop" },
{ "name": "ocr get value with coordinates",   "function": "ocr_get_value_with_coordinates",      "screenshot": "desktop" },
{ "name": "ocr get value with image",   "function": "ocr_get_value_with_image",     "screenshot": "desktop" },
{ "name": "ocr get value with text",    "function": "ocr_get_value_with_text",      "screenshot": "desktop" },

Made changes in the Framework\Built_In_Automation\Desktop\CrossPlatform\BuiltInFunctions.py file. Added the functions

1. def ocr_click(data_set)
   Action structure can be seen using the following link
   https://docs.google.com/document/d/1WdX4hmWTjO_07JImZNQeNlzzzB1b7mRk-klaPXrsw3g/edit#bookmark=id.i70nik1sh80p
   Description:

   • This action lets you find a text through OCR and click on it.
   • Using EasyOCR to find the texts.
   • Using Threading to read the EasyOCR English model while taking Screenshot of the screen to reduce time of the action

     ocr_thread = threading.Thread(target=get_easyocr_reader)
     ocr_thread.start()
     path = ocr_screenshot()
     ocr_thread.join()
     

   • Getting all the texts present in the Screenshot.
   • After that finding the desired text using the thefuzz library
   • Checking if the match score passes the threshold
   • If it passes the threshold then we get the coordinates of the text.
   • Then using the pyautogui we click on the center of that element

2. ocr_get_value_with_coordinates(data_set)
   Action structure can be seen using the following link
   https://docs.google.com/document/d/1WdX4hmWTjO_07JImZNQeNlzzzB1b7mRk-klaPXrsw3g/edit#bookmark=id.p4iehys78507
   Description:

   • This action lets you crop a portion of the image of the screen and extract the text shown in that
     particular portion of the image

3. def ocr_get_value_with_image(data_set)
   Action structure can be seen using the following link
   https://docs.google.com/document/d/1WdX4hmWTjO_07JImZNQeNlzzzB1b7mRk-klaPXrsw3g/edit#bookmark=id.u2fev3710p9c
   Description:

   • This action lets you extract a text based on the position of the text with respect to the image you provide.
   • The image has to be attached in the attachment section.
   • First, we will get the bounding box coordinates of the image that you provided.
   • Then extract the coordinates of all the texts present on the screenshot.
   • Compare the distance of coordinates of the provided image with the coordinates of the texts present on the screenshot on a given direction
   • Return the text that has the least distance and also satisfies a threshold.

4. ocr_get_value_with_text(data_set)
   Action structure can be seen using the following link
   https://docs.google.com/document/d/1WdX4hmWTjO_07JImZNQeNlzzzB1b7mRk-klaPXrsw3g/edit#bookmark=id.7whztgsmjxqt
   Description:

   • This action lets you extract a text based on the position of the text with respect to the text you provide.
   • First, we will get the bounding box coordinates of the text that you provided.
   • Then extract the coordinates of all the texts present on the screenshot.
   • Compare the distance of coordinates of the provided image with the coordinates of the texts present on the screenshot on a given direction
   • Return the text that has the least distance and also satisfies a threshold.

Test Cases

• TEST-7704