Automation Tool Layer
Principle:
The responsibility of the automation tool layer is to expose an API to manipulate GUI elements of the SUT.
The tool would, for instance, be able to find and interact with objects like buttons, tables, textboxes, ...
Things to keep in mind when choosing a GUI automation tool:
The tool should be compatible with the underlying GUI framework used by the SUT.
Possible GUI rendering subsystems include but aren't limited to: Win32, WinForms, WPF, Silverlight, SWT,...
The ideal automation tool has full support for the target GUI framework, maximizing the manipulation and interaction possibilities with that framework.
Some automation tools are framework agnostic and they often implement an image recognition algorithm in combination with user generated images or image maps to find and identify GUI elements.
At first, this might seem like an elegant and universal solution.
However, in my personal experience, these tools often generate false positives and false negatives when detecting GUI elements.
Furthermore, generating screenshots of all possible GUI elements in all of their possible states for every release of the SUT is no small feat.
Even in the case that both of the above problem would be mitigated, framework specific tools still have an edge over these general tools since they (should) support all manipulation and discovery possibilities that the GUI framework offers.
We could always homebrew a functional layer on top of one of these generalist automation tools to properly support all features a GUI framework offers (on an as-needed basis), but this could very well become a project of its own.
Final nail in the coffin for this type of generalist tools is that image recognition is heavy on resources and typically a specialized automation tool's performance is vastly superior.
These generalist automation tools are thus a last resort for our automation tool layer, only to be used if there's no appropriate framework-specific tool up to the job.
A less obvious aspect is the available documentation for your tool of choice.
What use is an elaborate API if we don't know its expected behavior?
Not only the sunny-side scenario's should be documented, we also need to know what exceptions/return values/errors to expect when, for instance, a mapped GUI element couldn't be located or when we try to interact with a disabled/unavailable element.
It's been my experience that, regretfully, proper documentation is often the weakest aspect of automation tools.
Which brings us to the next point to consider...
At first sight, it might seem unimportant whether your automation tool is open or closed source.
However, as mentioned above, documentation of these tools is often sub par or almost non-existent.
It is therefor a big advantage if the source code is available for us to understand the low-level details of the implemented behavior.
Software contains bugs and these automation tools are no exception.
Of course it is preferable if the tool you're using in your framework is still being actively developed.
In the case of open source software, you could even write your own bug fixes or feature extensions and issue a pull request!