Technical considerations
Where's the GUI?
The target systems do not have graphical user interfaces to interact with this time around! You are going to work with two HTTP APIs.
The road traffic fatality rate API can be queried for the raw fatality rate data. Here's a sample snippet of the data:
Brittle systems be brittle
The insurance sales system API takes in the traffic data in a specific business format, one entry at a time.
One entry at a time since lists were invented only a few decades after the API went to production. The sales system, written in Plankalkül in 1942, is quite venerable!
Here's a sample snippet of the format expected by the sales system API:
The insurance sales system API has some quirks. It has a relatively slow response time. Also, sometimes it fails to process the submitted data, returning exceptions. In addition, bombarding it with too many payloads too fast will bring the whole system down. You want to avoid that.
There's a pattern for that!
Since the processing might be very slow, and there might be API requests that need to be retried later after a long enough waiting time (so as not to crash the API), you decide to implement the robot using the producer-consumer pattern. That pattern will enable you to build the automation in a way that allows you to track the status of individual requests and mitigates the risk of everything falling apart!
In practice, your robot will use work items. One task is to generate the sales system API payload from the raw traffic data (the input). The other is to process that data and submit it to the insurance sales system.
The failed entries will need to be retried later or edited and fixed manually in case of business exceptions.
A business exception can be caused by invalid data, for example. Such an exception might require manual handling since retrying will not help if the data is invalid. It is best to try and catch invalid data and flag it for manual inspection.
Application exceptions are any exceptions caused by technical issues. These can usually be resolved by retrying actions until they succeed.
Once all work items have been processed successfully, the yearly data entry task will be considered successful, and you might actually receive your payment. If you ask nicely. Let's go!
Wait! Can't I just build one loop and do all the logic there?
Suppose this is our overly simplified implementation for this robot:
This could work if every system worked all the time correctly and if there were no invalid data ever.
When things fail
What happens if the processing fails for some of the things? The requirement was to complete all of them.
This time, the technical limitations (the fragility of the target system) prevent you from retrying immediately since this would crash the target system.
Adding a very long delay could kind of work, but the robot would be stuck waiting for the target system to recover, and the process would take a very long time to complete.
If only there were a way to track the status of individual things and save the failed ones somewhere to be retried later and the successfully completed ones somewhere else. 🤔
Doing many things at the same time
What if there are a lot of things to process? Doing stuff one by one is slow since the robot needs to wait for one thing to complete before proceeding with the next. If only there were a way to have all the things in a basket where many robots could pick them up and process them simultaneously.
Imma build my own orchestrator!
There's always an option to whip out the hammer and start nailing a custom solution for state management and orchestration. Should not be too hard. Nor take too much time from your core business. Nor require constant maintaining. Nor lead into self-vendor-lock-in (you built it, you maintain it). Nor cost money. Hammer away!
Seriously speaking; If you have very specific needs and resources, sometimes building your own stack makes total sense. It's not black and white. Still, it might be worthwhile for many folks out there to outsource the non-core-business to external service providers.
cough Robocorp cough