Skip to content

Commit

Permalink
[ci skip] Update readme
Browse files Browse the repository at this point in the history
  • Loading branch information
hibiken committed Jan 25, 2020
1 parent 58d2ed9 commit 3ed155b
Show file tree
Hide file tree
Showing 4 changed files with 137 additions and 44 deletions.
99 changes: 67 additions & 32 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,39 +6,37 @@
[![GoDoc](https://godoc.org/github.com/hibiken/asynq?status.svg)](https://godoc.org/github.com/hibiken/asynq)
[![Gitter chat](https://badges.gitter.im/go-asynq/gitter.svg)](https://gitter.im/go-asynq/community)

Simple and efficent asynchronous task processing library in Go.
Simple and efficient asynchronous task processing library in Go.

**Important Note**: Current major version is zero (v0.x.x) to accomodate rapid development and fast iteration while getting early feedback from users. The public API could change without a major version update before the release of verson 1.0.0.
**Important Note**: Current major version is zero (v0.x.x) to accomodate rapid development and fast iteration while getting early feedback from users. The public API could change without a major version update before v1.0.0 release.

## Table of Contents

- [Overview](#overview)
- [Requirements](#requirements)
- [Installation](#installation)
- [Getting Started](#getting-started)
- [Monitoring CLI](#monitoring-cli)
- [Command Line Tool](#command-line-tool)
- [Acknowledgements](#acknowledgements)
- [License](#license)

## Overview

![Gif](/docs/assets/asynqmon_stats.gif)

Asynq provides a simple interface to asynchronous task processing.

It also ships with a tool to monitor the queues and take manual actions if needed.
Package asynq provides a framework for asynchronous task processing.

Asynq provides:

- Clear separation of task producer and consumer
- Ability to process multiple tasks concurrently
- Ability to schedule task processing in the future
- Automatic retry of failed tasks with exponential backoff
- [Automatic failover](https://github.com/hibiken/asynq/wiki/Automatic-Failover) using Redis sentinels
- [Ability to configure](https://github.com/hibiken/asynq/wiki/Task-Retry) max retry count per task
- Ability to configure max number of worker goroutines to process tasks
- [Ability to configure](https://github.com/hibiken/asynq/wiki/Task-Retry) task retry count and retry delay
- Support for [priority queues](https://github.com/hibiken/asynq/wiki/Priority-Queues)
- [Unix signal handling](https://github.com/hibiken/asynq/wiki/Signals) to gracefully shutdown background processing
- [CLI tool](/tools/asynqmon/README.md) to query and mutate queues state for mointoring and administrative purposes
- [Automatic failover](https://github.com/hibiken/asynq/wiki/Automatic-Failover) using Redis sentinels
- [Command line tool](/tools/asynqmon/README.md) to query tasks for monitoring and troubleshooting purposes

## Requirements

Expand All @@ -49,7 +47,7 @@ Asynq provides:

## Installation

To install both `asynq` library and `asynqmon` CLI tool, run the following command:
To install both `asynq` library and `asynqmon` command line tool, run the following command:

```
go get -u github.com/hibiken/asynq
Expand All @@ -66,15 +64,22 @@ In this quick tour of `asynq`, we are going to create two programs.
**This guide assumes that you are running a Redis server at `localhost:6379`**.
Before we start, make sure you have Redis installed and running.

1. Import `asynq` in both files.
The first thing we need to do is create two main files.

```sh
mkdir producer consumer
touch producer/producer.go consumer/consumer.go
```

Import `asynq` in both files.

```go
import "github.com/hibiken/asynq"
```

2. Asynq uses Redis as a message broker.
Use one of `RedisConnOpt` types to specify how to connect to Redis.
We are going to use `RedisClientOpt` here.
Asynq uses Redis as a message broker.
Use one of `RedisConnOpt` types to specify how to connect to Redis.
We are going to use `RedisClientOpt` here.

```go
// both in producer.go and consumer.go
Expand All @@ -88,7 +93,25 @@ var redis = &asynq.RedisClientOpt{
}
```

3. In `producer.go`, create a `Client` instance to create and schedule tasks.
In `producer.go`, we are going to create a `Client` instance to create and schedule tasks.

In `asynq`, a unit of work to be performed is encapsluated in a struct called `Task`.
Which has two fields: `Type` and `Payload`.

```go
// Task represents a task to be performed.
type Task struct {
// Type indicates the type of task to be performed.
Type string

// Payload holds data needed to perform the task.
Payload Payload
}
```

To create a task, use `NewTask` function and pass type and payload for the task.

You schedule a task by calling `Client.Schedule` passing in the task and the timethe task neeeds to be processed.

```go
// producer.go
Expand Down Expand Up @@ -118,7 +141,13 @@ func main() {
}
```

4. In `consumer.go`, create a `Background` instance to process tasks.
In `consumer.go`, create a `Background` instance to process the tasks.

`NewBackground` function takes `RedisConnOpt` and `Config`.

You can take a look at documentation on `Config` to see the available options.

We are only going to specify the concurrency in this example.

```go
// consumer.go
Expand Down Expand Up @@ -240,42 +269,48 @@ func sendReminderEmail(t *asynq.Task) error {
Now that we have both task producer and consumer, we can run both programs.

```sh
go run consumer.go
go run producer.go
```

**Note**: This will not exit until you send a signal to terminate the program. See [Signal Wiki page](https://github.com/hibiken/asynq/wiki/Signals) for best practice on how to safely terminate background processing.
This will create two tasks: One that should processed immediately and another to be processed 24 hours later.

With our consumer running, also run
Let's use `asynqmon` tool to inspect the tasks.

```sh
go run producer.go
asynqmon stats
```

This will create a task and the first task will get processed immediately by the consumer. The second task will be processed 24 hours later.
You should able to see that there's one task in **Enqueued** state and another in **Scheduled** state.

Let's use `asynqmon` tool to inspect the tasks.
Note: To understand the meaning of each state, see [Life of a Task](https://github.com/hibiken/asynq/wiki/Life-of-a-Task) on our Wiki page.

Let's run `asynqmon` with `watch` command so that we can continuously run the command to see the changes.

```sh
asynqmon stats
watch -n 3 asynqmon stats # Runs `asynqmon stats` every 3 seconds
```

This command will show the number of tasks in each state and stats for the current date as well as redis information.
And finally, let's start the consumer program to process scheduled tasks.

```sh
go run consumer.go
```

To understand the meaning of each state, see [Life of a Task Wiki page](https://github.com/hibiken/asynq/wiki/Life-of-a-Task).
**Note**: This will not exit until you send a signal to terminate the program. See [Signal Wiki page](https://github.com/hibiken/asynq/wiki/Signals) for best practice on how to safely terminate background processing.

For in-depth guide on `asynqmon` tool, see the [README](/tools/asynqmon/README.md) for the CLI.
You should be able to see text printed in your terminal indicating that the task was processed successfully.

This was a quick tour of `asynq` basics. To see all of its features such as **[priority queues](https://github.com/hibiken/asynq/wiki/Priority-Queues)** and **[custom retry](https://github.com/hibiken/asynq/wiki/Task-Retry)**, see [the Wiki page](https://github.com/hibiken/asynq/wiki).
This was a whirlwind tour of `asynq` basics. To learn more about all of its features such as **[priority queues](https://github.com/hibiken/asynq/wiki/Priority-Queues)** and **[custom retry](https://github.com/hibiken/asynq/wiki/Task-Retry)**, see our [Wiki page](https://github.com/hibiken/asynq/wiki).

## Monitoring CLI
## Command Line Tool

Asynq ships with a CLI tool to inspect the state of queues and tasks.
Asynq ships with a command line tool to inspect the state of queues and tasks.

To install the CLI, run the following command:
To install, run the following command:

go get github.com/hibiken/asynq/tools/asynqmon

For details on how to use the tool, see the [README](/tools/asynqmon/README.md) for the asynqmon CLI.
For details on how to use the tool, refer to the tool's [README](/tools/asynqmon/README.md).

## Acknowledgements

Expand Down
2 changes: 1 addition & 1 deletion doc.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
// that can be found in the LICENSE file.

/*
Package asynq provides a framework for background task processing.
Package asynq provides a framework for asynchronous task processing.
Asynq uses Redis as a message broker. To connect to redis server,
specify the options using one of RedisConnOpt types.
Expand Down
Binary file added docs/assets/asynqmon_history.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
80 changes: 69 additions & 11 deletions tools/asynqmon/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Asynqmon

Asynqmon is a CLI tool to monitor the queues managed by `asynq` package.
Asynqmon is a command line tool to monitor the tasks managed by `asynq` package.

## Table of Contents

Expand All @@ -24,7 +24,7 @@ This will create the asynqmon executable under your `$GOPATH/bin` directory.

## Quick Start

Asynqmon tool has a few commands to inspect the state of tasks and queues.
The tool has a few commands to inspect the state of tasks and queues.

Run `asynqmon help` to see all the available commands.

Expand All @@ -34,7 +34,7 @@ By default, Asynqmon will try to connect to a redis server running at `localhost

### Stats

Stats command gives the overview of the current state of tasks and queues. Run it in conjunction with `watch` command to repeatedly run `stats`.
Stats command gives the overview of the current state of tasks and queues. You can run it in conjunction with `watch` command to repeatedly run `stats`.

Example:

Expand All @@ -46,35 +46,93 @@ This will run `asynqmon stats` command every 3 seconds.

### History

TODO: Add discription
History command shows the number of processed and failed tasks from the last x days.

By default, it shows the stats from the last 10 days. Use `--days` to specify the number of days.

Example:

asynqmon history --days=30

![Gif](/docs/assets/asynqmon_history.gif)

### List

TODO: Add discription
List command shows all tasks in the specified state in a table format

Example:

asynqmon ls retry
asynqmon ls scheduled
asynqmon ls dead
asynqmon ls enqueued
asynqmon ls inprogress

### Enqueue

TODO: Add discription
There are two commands to enqueue tasks.

Command `enq` takes a task ID and moves the task to **Enqueued** state. You can obtain the task ID by running `ls` command.

Example:

asynqmon enq d:1575732274:bnogo8gt6toe23vhef0g

Command `enqall` moves all tasks to **Enqueued** state from the specified state.

Example:

asynqmon enqall retry

Running the above command will move all **Retry** tasks to **Enqueued** state.

### Delete

TODO: Add discription
There are two commands for task deletion.

Command `del` takes a task ID and deletes the task. You can obtain the task ID by running `ls` command.

Example:

asynqmon del r:1575732274:bnogo8gt6toe23vhef0g

Command `delall` deletes all tasks which are in the specified state.

Example:

asynqmon delall retry

Running the above command will delete all **Retry** tasks.

### Kill

TODO: Add discription
There are two commands to kill (i.e. move to dead state) tasks.

Command `kill` takes a task ID and kills the task. You can obtain the task ID by running `ls` command.

Example:

asynqmon kill r:1575732274:bnogo8gt6toe23vhef0g

Command `killall` kills all tasks which are in the specified state.

Example:

asynqmon killall retry

Running the above command will move all **Retry** tasks to **Dead** state.

## Config File

You can use a config file to set default values for flags.
You can use a config file to set default values for the flags.
This is useful, for example when you have to connect to a remote redis server.

By default, `asynqmon` will try to read config file located in
`$HOME/.asynqmon.(yml|json)`. You can specify the file location via `--config` flag.
`$HOME/.asynqmon.(yaml|json)`. You can specify the file location via `--config` flag.

Config file example:

```yml
```yaml
uri: 127.0.0.1:6379
db: 2
password: mypassword
Expand Down

0 comments on commit 3ed155b

Please sign in to comment.