memcache: poor man's IPC
webb.clint at gmail.com
Thu Sep 13 00:58:05 UTC 2007
On 9/12/07, dormando <dormando at rydia.net> wrote:
> Would you like to add a page to the wiki describing this in more detail?
> :) It's an unusal use case I'd love to get up on there.
> Clint Webb wrote:
> > Yes, I use memcached for several things like that.
> > I use it for logging site activity. I also use it for gathering
> > statistics.
> > The sitelog functionality is especially beneficial. Basically all the
> > web-nodes connect to the sitelog memcache instance. When it needs to
> > log an activity, it does an 'incr' on a Counter key. Then 'add's the
> > log entry adding the counter to the key (eg, 'logentry-450'). An
> > external script runs every 10 seconds or so and pulls out all the log
> > entries and writes them to a file. It does this by doing a get for the
> > Counter and a Process key. If counter is greater than Process, then it
> > does an 'incr' on the Process key, and then retrieves that key (eg,
> > 'logentry-450'), and repeats that until it catches up to counter. Add
> > in some logic to handle the case where the logentry hasnt been completed
> > (ie, sleep 1 second and try again), and to handle when the Counter rolls
> > over, and thats basically it.
> > The reason for using memcache to log this data is that if we logged
> > straight to a file, we have some complications with multiple web-nodes
> > accessing the file, and ensuring the file is locked so that only one can
> > write to it at a time. Locking the file also put a bottle-neck on
> > processing the connections that come in. We could have used a database
> > to store the data, but then we have a lot of write activity using up
> > some CPU time on our database server that can be better used actually
> > serving real data.
> > I could post the code we use if anyone wants it. Its in Perl though.
> > In addition to that, we use it for statistical purposes, incrementing
> > counters when certain activities occur (such as 'new account created',
> > etc). An external script pulls out (and if applicable, resets) the
> > statistical data and puts it in a seperate sqlite database for
> > accessibility, and also to some daily summary reports (such, number of
> > new accounts, number of new posts, etc).
> > Clint Webb
> > rhokz.com <http://rhokz.com>
> > On 9/11/07, *Cal Heldenbrand* <cal at fbsdata.com <mailto:cal at fbsdata.com>
> > > wrote:
> > Hey everyone,
> > We're working on a new feature that has ended up being pretty cool,
> > and just thought I'd bring it up for discussion. We have a long
> > running process to print reports with a simple client/server
> > architecture. The server churns through the database stuff and
> > populates the page (along with a memcache status variable) and a
> > client AJAX call reads the memcache variable and displays a progress
> > indicator.
> > I'm sure this can be done with a few server side languages out
> > there, but adding in a load balanced web farm complicates any RPC
> > stuff since the load balancer will redistribute each HTTP request to
> > a different machine. Using memcache for a primitive IPC takes all
> > of the guesswork on communicating between any given two machines.
> > I think it's kind of neat, is anyone doing anything similar?
> > --Cal
> > --
> > Cal Heldenbrand
> > FBS Data Systems
> > E-mail: cal at fbsdata.com <mailto:cal at fbsdata.com>
> > --
> > "Be excellent to each other"
"Be excellent to each other"
More information about the memcached