memcache: poor man's IPC

[Clint Webb]

sure.

On 9/12/07, dormando <dormando at rydia.net> wrote:
> Would you like to add a page to the wiki describing this in more detail?
> :) It's an unusal use case I'd love to get up on there.
>
> -Dormando
>
> Clint Webb wrote:
> > Yes, I use memcached for several things like that.
> >
> > I use it for logging site activity.  I also use it for gathering
> > statistics.
> >
> > The sitelog functionality is especially beneficial.   Basically all the
> > web-nodes connect to the sitelog memcache instance.  When it needs to
> > log an activity, it does an 'incr' on a Counter key.  Then 'add's the
> > log entry adding the counter to the key (eg, 'logentry-450').  An
> > external script runs every 10 seconds or so and pulls out all the log
> > entries and writes them to a file.  It does this by doing a get for the
> > Counter and a Process key.  If counter is greater than Process, then it
> > does an 'incr' on the Process key, and then retrieves that key (eg,
> > 'logentry-450'), and repeats that until it catches up to counter.  Add
> > in some logic to handle the case where the logentry hasnt been completed
> > (ie, sleep 1 second and try again), and to handle when the Counter rolls
> > over, and thats basically it.
> >
> > The reason for using memcache to log this data is that if we logged
> > straight to a file, we have some complications with multiple web-nodes
> > accessing the file, and ensuring the file is locked so that only one can
> > write to it at a time.  Locking the file also put a bottle-neck on
> > processing the connections that come in.  We could have used a database
> > to store the data, but then we have a lot of write activity using up
> > some CPU time on our database server that can be better used actually
> > serving real data.
> >
> > I could post the code we use if anyone wants it.  Its in Perl though.
> >
> > In addition to that, we use it for statistical purposes, incrementing
> > counters when certain activities occur (such as 'new account created',
> > etc).  An external script pulls out (and if applicable, resets) the
> > statistical data and puts it in a seperate sqlite database for
> > accessibility, and also to some daily summary reports (such, number of
> > new accounts, number of new posts, etc).
> >
> > Clint Webb
> > rhokz.com <http://rhokz.com>
> >
> >
> > On 9/11/07, *Cal Heldenbrand* <cal at fbsdata.com <mailto:cal at fbsdata.com>
> >  > wrote:
> >
> >     Hey everyone,
> >
> >     We're working on a new feature that has ended up being pretty cool,
> >     and just thought I'd bring it up for discussion.  We have a long
> >     running process to print reports with a simple client/server
> >     architecture.  The server churns through the database stuff and
> >     populates the page (along with a memcache status variable) and a
> >     client AJAX call reads the memcache variable and displays a progress
> >     indicator.
> >
> >     I'm sure this can be done with a few server side languages out
> >     there, but adding in a load balanced web farm complicates any RPC
> >     stuff since the load balancer will redistribute each HTTP request to
> >     a different machine.  Using memcache for a primitive IPC takes all
> >     of the guesswork on communicating between any given two machines.
> >
> >     I think it's kind of neat, is anyone doing anything similar?
> >
> >     --Cal
> >
> >
> >     --
> >     Cal Heldenbrand
> >        FBS Data Systems
> >        E-mail:   cal at fbsdata.com <mailto:cal at fbsdata.com>
> >
> >
> >
> >
> > --
> > "Be excellent to each other"
>
>


-- 
"Be excellent to each other"