Make integrations with other services more resilient using a Queue

One downside of automating things is dealing with outages. Sometimes services go down, and are not available for some period of time. When this happens, how does it impact your automation?

If you don't design your automation carefully, you might lose the data entirely. Or get spammed by hundreds of submissions when it comes back up. Handling this correctly does end up needing some understanding of how the automation works, as well as any quirks of the remote system.

One great tool in the Drupal arsenal is the Queue API, and the Events, Conditions, and Actions module makes it super easy to ueue into your model. I tend to reach for a queue anytime I'm doing an integration with an external service.

For one client, we made a custom action plugin to post data to a Salesforce case. We wired it all together using ECA. There was already a webform in use for collecting the information. The client wanted certain destinations to go to SalesForce, and others to go to different email addresses.

Using ECA allowed us to use a queue to get around any connectivity issues with SalesForce. Once the data was collected, we sent it to an action to "Enqueue a task." And then used an event "ECA processing queued task" to do the actual SalesForce post.

Our custom plugin throws a "requeue" exception if it can't reach Salesforce, and then the job just goes back on the queue to be tried again later.

In this advent calender, on the 17th, the server for our Mastodon account stopped accepting image uploads. We added a queue to the model we built for Day 15 - Automatically post to Mastodon and retriggered it. For Mastodon's API, we had to explicitly look at the web response to determine whether it was successful, and if it was not, then we had to re-add it to the queue.

When the API started working again, our post for the 17th automatically published!