Introduction to gearman

Matt Sergeant <matt@sergeant.org>

Unreliable Perl Job Server

The Problem

• Spare "job" machines that can do tasks
• Need to load balance between them
• Need to bring in and out machines as available
• Want to support async calls

Gearmand

• A daemon for synchronising workers and clients
• Also does load balancing and failover
• A worker side which registers itself with the daemon
• A client side which requests jobs to be performed
• And a simple network protocol between them

Example

• Qpsmtpd needs to do "slow" (e.g. spam scanning)
• One central gearmand server marshalls requests
• Call is asynchronous
• Multiple spam scanners can be brought online as required

Synchronous Client Code

my $gearman = Gearman::Client::Async->new(
    job_servers => ['hp-monitor.messagelabs.net']
    );
my $result_ref = $gearman->do_task("scan", \$mail, {});

Async Client Code

my $gearman = Gearman::Client::Async->new(
    job_servers => ['hp-monitor.messagelabs.net']
    );
my %opts;
$opts{on_complete} = sub { ... };
$opts{timeout}     = 10; # seconds
$opts{on_fail}     = sub { ... };
my $task = Gearman::Task->new("scan", \$mail, \%opts);
$gearman->add_task($task);

Worker code

sub start_worker {
    my $worker = Gearman::Worker->new();
    $worker->job_servers('hp-monitor.messagelabs.net');
    $worker->register_function(scan => \&scan);
    $worker->work while 1;
}

• Start a bunch of fork()s to do this.