What is AllQueue?

And why do I need it...

Meet AllQueue, a queue platform that meets all your needs. Most platforms meets 'some' of your needs, AllQueue tries to fill all the needs of a high performance scalable queue

Introduction

Designed from modern needs and standards

AllQueue is a modern job queue platform which incorporates many of the best features from other popular queues into a single full featured system.

For large applications, we rely on queues to keep things running. Often times we rely on different queues to provide different functionality to support our needs

AllQueue is a task/job queue, NOT a messaging queue. There is no sense of broadcasting, subscriptions, or streaming.

With AllQueue, you can lean on a single, reliable, performant system.

Features

AllQueue is the proven large scale architecture behind Blitline.com

• Federated for high availability

• Encrypted by default

• Workflows

• Scheduling

• Prioritization

• Persistence (optional)

• Automated Dead Letter Queues (autobury)

• Named Queues (grouping)

• Fair Queuing

• Throttles

• Simple Docker Deployment

• Fast

Scaling is hard, especially when dealing with millions of jobsand scaling to thousands of machines. There can be no 'single point of failure, nor should there be 1 machine talking to thousands of others.

Architecture

AllQueue is a scalable, federated, sharded queue that isn't missing 'important' features like prioritization, or persistence. It's self contained, with CLI tools for monitoring.

AllQueue consists of a server component and a client component, which we refer to as AllQServer and AllQClient respectively. The client/server architecture allow better control of job flow and also provides easier omni-language support.

AllQServers run as standalone services, and AllQClient services generally run on machines you want to communicate with AllQServers. All communication to AllQServers go through AllQClient only.

The AllQClient provides a TCP, UNIX socket, or REST HTTP interface for your application to use to send/receive messages from AllQServer. The simplest mechanism is to use the REST HTTP interface which can use any number of Swagger Client languages components.

Complexity is the Enemy

AllQueue does not rely on external tools and platforms for its functionality. There is no required DB to maintain, no 3rd party data store. Those ALL require maintenance outside the scope of your queue itself. AllQueue is self contained, and requires only release updates to AllQ itself, which is as simple as downloading the latest docker image.

AllQueue communicates entirely is JSON, which makes debugging easier.