taskhawk

Taskhawk is an async task execution framework (à la celery) that works on AWS and GCP, while keeping things pretty simple and straight forward. Any unbound function can be converted into a Taskhawk task.

View the Project on GitHub cloudchacho/taskhawk

Backends

Taskhawk works on cloud-native backends such as AWS and GCP. Before you can publish/consume messages, you need to provision the required infra. This may be done manually, or, preferably, using Terraform. Taskhawk provides tools to make infra configuration easier: see Terraform and Terraform Google for further details.

Queue

Taskhawk utilizes SQS and Pub/Sub for cloud hosted queues. These systems are very highly reliable and lets you focus on code rather than infra.

Manual provisioning

At a high level, Taskhawk requires one topic and one subscription with dead letter queue per priority per application.

GCP

In GCP, queues are called “subscriptions”.

  1. Install gcloud
  2. Authenticate with gcloud:
$ gcloud auth application-default login
  1. Configure project (and repeat as appropriate for topics / apps):
$ APP=<myapp>

$ gcloud config set project <GCP_PROJECT_ID>
$ gcloud pubsub topics create taskhawk-dev-myapp-dlq
$ gcloud pubsub subscriptions create taskhawk-dev-myapp-dlq --topic taskhawk-dev-myapp-dlq
$ gcloud pubsub topics create taskhawk-dev-myapp
$ gcloud pubsub subscriptions create taskhawk-dev-myapp --topic taskhawk-dev-myapp --dead-letter-topic taskhawk-dev-myapp-dlq

AWS

  1. Install awscli
  2. Authenticate with AWS:
$ aws configure
  1. Configure project:
$ APP=<myapp>

$ AWS_REGION=$(aws configure get region)
$ AWS_ACCOUNT_ID=$(aws sts get-caller-identity | jq -r '.Account')
$ aws sqs create-queue --queue-name TASKHAWK-DEV-MYAPP
$ aws sqs create-queue --queue-name TASKHAWK-DEV-MYAPP-DLQ
$ aws sqs set-queue-attributes --queue-url https://$AWS_REGION.queue.amazonaws.com/$AWS_ACCOUNT_ID/TASKHAWK-DEV-MYAPP --attributes "{\"Policy\":\"{\\\"Version\\\":\\\"2012-10-17\\\",\\\"Statement\\\":[{\\\"Action\\\":[\\\"sqs:SendMessage\\\",\\\"sqs:SendMessageBatch\\\"],\\\"Effect\\\":\\\"Allow\\\",\\\"Resource\\\":\\\"arn:aws:sqs:$AWS_REGION:$AWS_ACCOUNT_ID:TASKHAWK-DEV-MYAPP\\\",\\\"Principal\\\":{\\\"Service\\\":[\\\"sns.amazonaws.com\\\"]}}]}\",\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"arn:aws:sqs:$AWS_REGION:$AWS_ACCOUNT_ID:TASKHAWK-DEV-MYAPP-DLQ\\\",\\\"maxReceiveCount\\\":\\\"5\\\"}\"}"