Memcached Database Use

[Dustin Sallings]

On Jun 22, 2007, at 13:56 , Chris Miller wrote:

> I see how that by storing database results in memcached would be  
> very helpful, but how does memcached know when the result set in  
> cache has changed?

	Long answer:

	I wrote an app called diggwatch[0] that uses the digg API as my  
primary data store, and stores all the useful information in  
memcached locally.  Cache misses for me are really expensive, and the  
digg API makes certain operations I want to perform somewhat difficult.

	For example, the primary thing I wanted this app to do for me is  
tell me when anyone responds to any comment I make on a digg  
article.  Basically, that looks like this:

	1) Ask for any recent comments by username.
	2) Ask for all of the stories to which any of these comments belong  
so I can put useful titles on things.
	3) Ask for any children comments of #1 (or children of the comments'  
parent as defined by the old system).

	As this is primarily used (at least by me) as an RSS provider, that  
request occurs several times throughout the day and I'd like it to be  
cached.  However, I'd *also* like it to be fresh, and I don't get  
notifications from digg.

	I cache the result of #1 for about a minute -- fairly insignificant  
amount of time, but I don't consider that request version expensive.

	I cache the results from #2 for about five minutes.  It's a single  
request for up to something like 100 stories, and I can optimize some  
of it out if I have some of the stories in my cache already.

	#3 is the most expensive query, because I need to run it almost once  
per comment (result of #1).  I cache these for about a day, *but* the  
key includes the number of comments on a given story (which I get in  
the result of #2).  If nobody's commented on a story at all, I can be  
guaranteed that nobody's commented on a thread I'm involved in within  
the story.

	It's not perfect, but it's quite effective and greatly reduces the  
number of trips to digg without having my latency drop below ~5 minutes.


	Short answer:

	Depends on your application, but don't think of it as working with  
result sets as much as objects.  I cache collections of pre-build  
objects, and mash them together in my application code.

	A neat benefit of doing things this way (going back to the long  
answer above), is that understanding my data at this level allows me  
to generate smarter etags such that the typical response sent to an  
RSS reader from my app is 0 bytes (after headers).


[0] http://bleu.west.spy.net/diggwatch/

-- 
Dustin Sallings