Logging

Timo Ewalds timo at tzc.com
Tue Dec 7 00:01:59 PST 2004


Sean Chittenden wrote:

>> I run a large site that is using memcached to a small degree at the 
>> moment. Really it only caches things that are used on almost every 
>> page, not things that are specific to a single page or to a single 
>> user. As the database becomes the bottleneck instead of the 
>> webservers as it is right now, I'll need to drop SELECTs and UPDATEs. 
>> Caching things that are specific to individual pages will help drop 
>> the selects, but won't drop the updates. I do alot of updates of the 
>> form UPDATE ... SET count = count + 1 WHERE ... for stats and other 
>> logging, that could be combined into single queries over time. The 
>> various hit counters, banner view counters, etc, could all be done in 
>> the form of incr some memcached entry, then once a minute zero them 
>> and do a single query of the form UPDATE ... SET count = count + (sum 
>> of all from the memcached entry) WHERE ..., greatly reducing the 
>> UPDATE query count. How hard would it be to create a command similar 
>> to incr/decr that would just reset that key to 0 and return the old 
>> value?
>>
>> Does anyone have a better solution to this problem, memcached related 
>> or not?
>
>
> Right now it's not possible to have memcached output anything when 
> data expires, but it's an addition that could be added.  The best way 
> would be via syslog(3), but therein lies a huge potential for a local 
> DoS if lots of data gets expired.  A better approach would be to set a 
> "log me on expiration" bit in the new protocol, that way there is some 
> selectivity for logging.  Brad, would you mind if I added that?  -sc
>
I was thinking just along the lines of a zero($key) type function that 
returns the old value and sets the value of that key to 0 as an atomic 
operation. It would be used from a script activated once every minute or 
so from cron to keep the database at least relatively up to date. You'd 
then use incr($key) on each page to update the results. I realized 
afterwards though, that a zero() function seems a bit pointless. Why not 
just keep the full version instead of an incremental version then 
instead of count = count + (incremental as tracked by memcached), just 
use count = (total as tracked by memcached) from the cron page.

That can't possibly be the optimimum solution though, as memcached is 
meant to be a cache, not a statistics daemon. Does anyone know of a 
better solution?

Timo


More information about the memcached mailing list