memcache: poor man's IPC

[Clint Webb]

Yes, I use memcached for several things like that.

I use it for logging site activity.  I also use it for gathering
statistics.

The sitelog functionality is especially beneficial.   Basically all the
web-nodes connect to the sitelog memcache instance.  When it needs to log an
activity, it does an 'incr' on a Counter key.  Then 'add's the log entry
adding the counter to the key (eg, 'logentry-450').  An external script runs
every 10 seconds or so and pulls out all the log entries and writes them to
a file.  It does this by doing a get for the Counter and a Process key.  If
counter is greater than Process, then it does an 'incr' on the Process key,
and then retrieves that key (eg, 'logentry-450'), and repeats that until it
catches up to counter.  Add in some logic to handle the case where the
logentry hasnt been completed (ie, sleep 1 second and try again), and to
handle when the Counter rolls over, and thats basically it.

The reason for using memcache to log this data is that if we logged straight
to a file, we have some complications with multiple web-nodes accessing the
file, and ensuring the file is locked so that only one can write to it at a
time.  Locking the file also put a bottle-neck on processing the connections
that come in.  We could have used a database to store the data, but then we
have a lot of write activity using up some CPU time on our database server
that can be better used actually serving real data.

I could post the code we use if anyone wants it.  Its in Perl though.

In addition to that, we use it for statistical purposes, incrementing
counters when certain activities occur (such as 'new account created',
etc).  An external script pulls out (and if applicable, resets) the
statistical data and puts it in a seperate sqlite database for
accessibility, and also to some daily summary reports (such, number of new
accounts, number of new posts, etc).

Clint Webb
rhokz.com


On 9/11/07, Cal Heldenbrand <cal at fbsdata.com> wrote:
>
> Hey everyone,
>
> We're working on a new feature that has ended up being pretty cool, and
> just thought I'd bring it up for discussion.  We have a long running process
> to print reports with a simple client/server architecture.  The server
> churns through the database stuff and populates the page (along with a
> memcache status variable) and a client AJAX call reads the memcache variable
> and displays a progress indicator.
>
> I'm sure this can be done with a few server side languages out there, but
> adding in a load balanced web farm complicates any RPC stuff since the load
> balancer will redistribute each HTTP request to a different machine.  Using
> memcache for a primitive IPC takes all of the guesswork on communicating
> between any given two machines.
>
> I think it's kind of neat, is anyone doing anything similar?
>
> --Cal
>
>
> --
> Cal Heldenbrand
>    FBS Data Systems
>    E-mail:  cal at fbsdata.com




-- 
"Be excellent to each other"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.danga.com/pipermail/memcached/attachments/20070912/b7a89be6/attachment.htm