Taskhawk is an async task execution framework (à la celery) that works on AWS and GCP, while keeping things pretty simple and straight forward. Any unbound function can be converted into a Taskhawk task.
Taskhawk works on cloud-native backends such as AWS and GCP. Before you can publish/consume messages, you need to provision the required infra. This may be done manually, or, preferably, using Terraform. Taskhawk provides tools to make infra configuration easier: see Terraform and Terraform Google for further details.
Taskhawk utilizes SQS and Pub/Sub for cloud hosted queues. These systems are very highly reliable and lets you focus on code rather than infra.
At a high level, Taskhawk requires one topic and one subscription with dead letter queue per priority per application.
In GCP, queues are called “subscriptions”.
$ gcloud auth application-default login
$ APP=<myapp>
$ gcloud config set project <GCP_PROJECT_ID>
$ gcloud pubsub topics create taskhawk-dev-myapp-dlq
$ gcloud pubsub subscriptions create taskhawk-dev-myapp-dlq --topic taskhawk-dev-myapp-dlq
$ gcloud pubsub topics create taskhawk-dev-myapp
$ gcloud pubsub subscriptions create taskhawk-dev-myapp --topic taskhawk-dev-myapp --dead-letter-topic taskhawk-dev-myapp-dlq
$ aws configure
$ APP=<myapp>
$ AWS_REGION=$(aws configure get region)
$ AWS_ACCOUNT_ID=$(aws sts get-caller-identity | jq -r '.Account')
$ aws sqs create-queue --queue-name TASKHAWK-DEV-MYAPP
$ aws sqs create-queue --queue-name TASKHAWK-DEV-MYAPP-DLQ
$ aws sqs set-queue-attributes --queue-url https://$AWS_REGION.queue.amazonaws.com/$AWS_ACCOUNT_ID/TASKHAWK-DEV-MYAPP --attributes "{\"Policy\":\"{\\\"Version\\\":\\\"2012-10-17\\\",\\\"Statement\\\":[{\\\"Action\\\":[\\\"sqs:SendMessage\\\",\\\"sqs:SendMessageBatch\\\"],\\\"Effect\\\":\\\"Allow\\\",\\\"Resource\\\":\\\"arn:aws:sqs:$AWS_REGION:$AWS_ACCOUNT_ID:TASKHAWK-DEV-MYAPP\\\",\\\"Principal\\\":{\\\"Service\\\":[\\\"sns.amazonaws.com\\\"]}}]}\",\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"arn:aws:sqs:$AWS_REGION:$AWS_ACCOUNT_ID:TASKHAWK-DEV-MYAPP-DLQ\\\",\\\"maxReceiveCount\\\":\\\"5\\\"}\"}"