Cloud machine learning (ML) APIs example robot
Supporting advanced data analytics or decision logic beyond simple deterministic rules often requires machine learning (ML) solutions. One way to provide the needed intelligence is to use commercial cloud services that can be applied for a number of typical use cases. Such services include, for example, image analysis, natural language processing, document analysis, and optical character recognition (OCR).
These examples demonstrate how to use cloud-based ML APIs with RPA Framework. Currently, a selected set of services from AWS (RPA.Cloud.AWS), Microsoft Azure (RPA.Cloud.Azure) and Google Cloud (RPA.Cloud.Google) are available.
You can access cloud ML services that are not supported by RPA Framework by writing custom Robot Framework or Python libraries to connect them via REST APIs or service-specific SDKs. If your automation team has in-house machine learning expertise, the possibilities for self-developed ML services from the cloud are almost limitless. A separate article on this topic will be published later.
Prerequisites
To use cloud ML APIs, you need:
- An account for the cloud provider of your choice.
- The required ML service(s) activated and accessible from the public internet.
- Credentials to the service(s) - the format depends on the cloud provider.
Setup the access keys
The authentication method varies depending on the cloud service:
- AWS: AWS key and AWS key ID (
AWS_KEY
andAWS_KEY_ID
) - Azure: Azure subscription key (
AZURE_SUBSCRIPTION_KEY
) - Google Cloud: Service credentials file
There are three different ways to set the access credentials - See the documentation of the RPA.Cloud
libraries. Here we present an approach that works both in a local and Control Room setting.
For local robot runs, we use a vault.json
file placed outside the repository, e.g., in /Users/<username>/vault.json
. For AWS and Azure, we can save the key values directly in the file. For Google Cloud, the content of the service credentials file must be mapped to a string, with special characters escaped. This can be done, e.g., with the following Python script.
This will result in the following JSON file:
If the robot is run in Control Room, store the above credential information in your workspace vault with the names aws
, azure
and gcp
.
Configure file vault support
To run in your local or development environment, edit the path to the vault file in devdata/env.json
.
Sample data
In this example, we will use images of an invoice and a sphygmomanometer to showcase the image processing functionalities. For natural language processing, we will be working on a piece of text given as a string.
The sample invoice and picture are shown below.
Using ML services with RPA Framework
The robot code resides in the tasks.robot
file. In the *** Settings ***
section, we define which libraries are used and configure the cloud libraries.
Match the
region
argument ofRPA.Cloud.AWS
andRPA.Cloud.Azure
to your region in AWS or Azure.
The basic functionality of RPA.Cloud
libraries is to send a request to the API and return the response as JSON without any modification to the content. The JSON data can then be processed further based on the needs of the automated process. Such operations are not currently fully supported by RPA Framework since each service has a different response syntax.
AWS
To analyze a scanned invoice, we use the Textract service from AWS.
After configuring the client and getting the answer from AWS, we use the Get Tables
keyword to collect tables from the response. In this example, table data is correctly found.
QTY | DESCRIPTION | UNIT PRICE | AMOUNT |
---|---|---|---|
1 | Front and rear brake cables | 100.00 | 100.00 |
2 | New set of pedal arms | 15.00 | 30.00 |
3 | Labor 3hrs | 5.00 | 15.00 |
Subtotal | 145.00 | ||
Sales Tax 6.25% | 9.06 | ||
TOTAL | $154.06 |
The full response from AWS is stored in a JSON file (here output/textract.json
). Parsing the other data, like key-value pairs, from the JSON response needs to be done by separate functions. The syntax of the Textract response and code examples are available on the Textract developer guide.
Azure
The usage of (RPA.Cloud.Azure) is demonstrated by Text analytics service. In the following task, the robot recognizes the language, key phrases, and sentiment of a sample text.
The outcome of the analysis is stored in JSON files (here output/text_lang.json
, output/text_phrases.json
, and output/text_sentiment.json
). As a summary, we find the following information:
detectedLanguage | confidenceScore |
---|---|
Swedish | 1.0 |
keyPhrases |
---|
'teams of software robot developers' 'external developers', 'test automation', 'robotic process automation', 'RPA industry', 'digital agents', 'application monitoring', 'Tens of thousands', 'new jobs', 'companies', 'demand', 'service', 'use', 'robotics', 'organizations', 'unattended', 'model' |
sentiment | confidenceScores |
---|---|
neutral | { 'positive': 0.03, 'neutral': 0.96, 'negative': 0.01 } |
Google Cloud
From RPA.Cloud.Google we try Google Vision AI which can be used, for example, to label an image or detect the text.
The full responses are stored in JSON files (here output/vision_labels.json
and output/vision_text.json
). The summary of the outcome is shown below. Note that in order to get "sphygmomanometer" among the labels, one needs to train a model since such a term is not included in Google Vision AI. The recognized text blocks cover almost all the text appearing in the image. The full response data also contains the bounding box information.
Label | Score |
---|---|
Electronic device | 0.8969995 |
Technology | 0.89112115 |
Measuring instrument | 0.827221 |
Arm | 0.75604373 |
Cyclocomputer | 0.53043586 |
Pedometer | 0.5304172 |
Text |
---|
"beurer", "102", "63", "LINKER ARM", "LEFT ARM", "WHO", "SYS", "mmHg", "mmHg", "DIA", "mmHg", "2-3 cm", "Arterie", "artery", "LATEX", "12 20M" "9.05.", "FREE", "PULSE", "/min", "22-35 cm", "Item No.: 162.972", "urer GmbH, Söflinger Strasse 218, 89077 Ulm, Germany", "M+C"; "Set" |
Other examples
- Process PDF invoices with Amazon Textract
- Process purchase invoices automatically using Aito.ai and Google Sheets
- IMDB Movie Review Sentiment Analysis With Amazon Comprehend