Running environment:

• Win10
• celery 3.1.18
• RabbitMQ

1. Demand

We have many different tasks with different priorities, such as video upload and compression tasks, photo compression upload tasks and other unimportant tasks.These tasks take different time and require different worker s to process them.Just using celery's default queue won't meet our needs.
This requires us to route different tasks to different queues and have different worker s handle different types of tasks.

2. Create queues and switches

You can look at switches and queues first http://rabbitmq.mr-ping.com/

default_exchange = Exchange('dedfault', type='direct')

# Define a media switch of type Direct Switch
media_exchange = Exchange('media', type='direct')

# Create three queues, one default, one video, one image
CELERY_QUEUES = (
    Queue('default', default_exchange, routing_key='default'),
    Queue('videos', media_exchange, routing_key='media.video'),
    Queue('images', media_exchange, routing_key='media.image')
)
# Define default queue and switch routing_key
CELERY_DEFAULT_QUEUE = 'default'
CELERY_DEFAULT_EXCHANGE = 'default'
CELERY_DEFAULT_ROUTING_KEY = 'default'

3. Create task

from celery import Celery
import time

app = Celery()
app.config_from_object('celeryconfig')

# video compression
@app.task
def video_compress(video_name):
    time.sleep(10)
    print 'Compressing the:', video_name
    return 'success'

@app.task
def video_upload(video_name):
    time.sleep(5)
    print u'Uploading video'
    return 'success'

# Compressed photos
@app.task
def image_compress(image_name):
    time.sleep(10)
    print 'Compressing the:', image_name
    return 'success'

# Other Tasks
@app.task
def other(str):
    time.sleep(10)
    print 'Do other things'
    return 'success'

Now that we have defined three queues, we want to route task for action videos and action photos to specific queues.

4. Specify Routes

CELERY_ROUTES = ({'tasks.image_compress': {
                        'queue': 'images',
                        'routing_key': 'media.image'
                 }},{'tasks.video_upload': {
                        'queue': 'videos',
                        'routing_key': 'media.video'
                 }},{'tasks.video_compress': {
                        'queue': 'videos',
                        'routing_key': 'media.video'
                 }}, )

Each task is queued by CELERY_ROUTES, and if a task arrives, the worker is given the name of the task to handle.

5. Tak Registration

See this document for the name of the task http://docs.jinkan.org/docs/celery/userguide/tasks.html
celery can automatically generate names and errors will occur if the task is not registered.After searching, someone uses the following method to solve the problem.

CELERY_IMPORTS = ("tasks",)

tasks is the module name that I save the task so that when I create the worker, I can register the task with the worker, as follows:

The tasks below in the picture are those in my tasks file.

6. Complete Code

tasks.py

from celery import Celery
import time


app = Celery()
app.config_from_object('celeryconfig')

# video compression
@app.task
def video_compress(video_name):
    time.sleep(10)
    print 'Compressing the:', video_name
    return 'success'

@app.task
def video_upload(video_name):
    time.sleep(5)
    print u'Uploading video'
    return 'success'

# Compressed photos
@app.task
def image_compress(image_name):
    time.sleep(10)
    print 'Compressing the:', image_name
    return 'success'

# Other Tasks
@app.task
def other(str):
    time.sleep(10)
    print 'Do other things'
    return 'success'

celeryconfig.py

from kombu import Exchange, Queue
from routers import MyRouter

# Configure Urban Areas
CELERY_TIMEZONE = 'Asia/Shanghai'
CELERY_BROKER = 'amqp://localhost'
# Define a default switch
default_exchange = Exchange('dedfault', type='direct')

# Define a media switch
media_exchange = Exchange('media', type='direct')

# Create three queues, one default, one video, one image
CELERY_QUEUES = (
    Queue('default', default_exchange, routing_key='default'),
    Queue('videos', media_exchange, routing_key='media.video'),
    Queue('images', media_exchange, routing_key='media.image')
)

CELERY_DEFAULT_QUEUE = 'default'
CELERY_DEFAULT_EXCHANGE = 'default'
CELERY_DEFAULT_ROUTING_KEY = 'default'
#
CELERY_ROUTES = ({'tasks.image_compress': {
                        'queue': 'images',
                        'routing_key': 'media.image'
                 }},{'tasks.video_upload': {
                        'queue': 'videos',
                        'routing_key': 'media.video'
                 }},{'tasks.video_compress': {
                        'queue': 'videos',
                        'routing_key': 'media.video'
                 }}, )

# Use the following sentence to resolve unregistered message s received by the worker
CELERY_IMPORTS = ("tasks",)

Note that you need to set up a queue when starting the worker and open the terminal startup worker in the directory where you save the task. I don't know yet!

# Start worker1
celery worker -Q default --loglevel=info

celery worker -Q videos --loglevel=info

Start a worker that processes pictures

celery worker -Q images --loglevel=info

This allows us to route different types of tasks to different worker s.

Pits encountered

When you modify the configuration file and need to restart the worker, remember to turn off the python shell as well.
If the task is invoked but the worker has such an error

unregistered task of type

That means your task is not registered and needs to be added to the configuration file with CERY_IMPORTS = ("tasks"), as I mentioned above.
It is recommended that you do your own debugging and look at the official celery documentation, which will make us feel more relaxed.