[ANN] beanstalkd 0.5

[Keith Rarick]

On Dec 11, 2007 2:35 PM, mike <mike503 at gmail.com> wrote:
> When I design a queue system (like MySQL queue table and PHP running
> via cron to process jobs) I have a status of scheduled/pending,
> processing (currently processing), error, completed, start time, end
> time, any message if there is an error, or output in general even with
> success (it depends, I suppose)
>
> I would hope it has most of that functionality.

It has a great deal of that functionality, but also bear in mind that
this is just the server. Some of those things are better taken care of
by the worker processes that run jobs. For example, recording start
and stop time, errors, and other output can all be done by having each
worker write to a log file.

Each job in beanstalk starts out in the "ready" state, and when a
worker wants to run the job it gets marked as "reserved" by that
worker. When the worker finishes running the job it deletes it from
the server. If there is a transitory error, the worker can release the
job (optionally with a delay) back into the ready queue so it can run
again later. If the job takes too long (say, the worker process gets
wedged), it times out (currently the timeout is fixed at 120 seconds)
and goes back into the ready queue.

If a job has timed out or been released, those facts are marked in the
job stats. So it's easy to tell if something has ever gone wrong for a
given job.

One feature that we're thinking of adding is a flag to mark jobs as
idempotent, so the server knows it's safe to rerun them automatically.

> It is cool but I would
> not feel comfortable sending jobs into a void that might process it so
> fast I can't tell but at the same time I can't tell also that it
> processed the job and was successful or errored :)

That's understandable. The best way to tell that your jobs are
actually running is to watch what the workers are doing, rather than
sampling the server stats.

We've been using the beantsalk system (including a rails plugin that
we're about to release) for almost two months now. The server has been
extremely reliable. By far the worst aspect of the system has been the
workers. We've had a handful of mishaps during the development and
early deployment of this system, and all of them were caused by
mistakes in the worker code.

At this point, I'd call the server semi-mature. There are a few
details that still need to be changed, but the overall design is
pretty solid.

kr