Architecture
Last updated
Last updated
AllQueue consists of 2 pieces. A local client app (AllQ Client) and server(AllQ Server). These are easily deployed via docker containers.
AllQueue is a Docker only platform. There is no "installable" version besides Docker image downloads. AllQueue requires Docker on both the server, and the clients talking to the server.
The client container knows about the available servers via config, and provides a local socket/REST API for applications to talk to on localhost. The client then manages all encryption and transfer to the server, handling failure and retries automatically. The client will then decide which servers to send data to, and act as a liaison between the server and your application.
By having a local REST API, you can use one of the many available REST API libraries to communicate with AllQueue.
WARNING: Since the client handles all communications, it is IMPORTANT that you DO NOT put AllQueue server behind a load balancer. It is designed to be federated ONLY and that cannot be load balanced
Also of interest is that the REST API is to the local client only, the client-to-server communications is through a low-level encrypted TCP call.
AllQueue requires encryption between the server and clients. There is no way for AllQueue to work without the required server and client encryption keys.
When you start the AllQueue server without keys, it will output a suggested set of keys for you to use for both the client and server.
AllQueue was modeled after beanstalkd and follows most of the same principles. It is not intended to "mimic" beanstalkd, but the general workflow and terminology are the same.
A job is an atomic unit of work put on the queue. You "reserve" a job from the queue when you intend to work on it. When you are done you mark a job as "done" (or it's alias "delete"). A job has a specific amount of time to complete ( TTL ) in seconds. If a job takes longer than that amount of time it will "expire", and be "released" back to the queue. You can "touch" a job to reset the TTL timer if you know a job will take a long time. If a job "expires" more that (default 3) times, it will but put into the "buried" (dead letter) queue, it is then up to you to decide what to do with these. You can delete them or review them, or even "kick" them back into the queue again. Jobs will stay in the "buried" (dead letter) queue indefinitely.
Every job has a number of optional settings you can set at runtime such as TTL, expired count limits, priorities, and other jobs to run "after" the current job ("parent_jobs").
An important design decision about AllQueue is that it is a pull-only paradigm. The client needs to ask the server for a job, the server will NOT push jobs to clients. This means at some layer, you will need a loop to do polling from AllQueue.
This design decision was made because if there are a thousand clients trying to use the server, we don't want to make each one maintain a permanent tcp connection, we allow the natural networking algorithms handle tcp caching and maintenance and try to keep a relatively stateless tcp connection, and this means we have no permanent connection to make push events.
