Large lists

[Brian Beuning]

I like this solution.  It is simple and uses memcached
as a cache and not a data store.

Note that it handles both positive and negative cases.
If a query returns no rows, store that result in memcached.

It does introduce the issue of updating the cache when
the database is changed.

Brian Beuning


-----Original Message-----
From: John Kramlich [mailto:john.kramlich at gigoit.org]
Sent: Monday, November 05, 2007 3:41 AM
To: J A
Cc: memcached
Subject: Re: Large lists


J A wrote:
> I have a fairly large members list that I want to keep in memcache.  
> What I do with this list is query it against particular user IDs to 
> see if they are a member of that list or not.  If they are they get 
> certain priviledges.
>  
> The problem is, this list has gotten to the point of saturating the 
> PHP's memory when fetching the MySQL query the first time.
>  
> Is there a way to do this more effectively, for instance, partitioning 
> the list into separate smaller lists, grouped by time of login?  I'm 
> thinking of this, as users who have logged in in the past 3 months are 
> more likely to be in the list anyway. 

Would it be possible to do the check as a SQL query and cache the result 
of that query?  You would of course need to maintain this list of 
privileged members in MySQL.  Then, when you want to check the status of 
a particular member query MySQL to see if they have the right privledges 
and then store that information in Memcache.  The next time you need to 
check that particular member the result should pull from the cache and 
bypass MySQL .

This also would allow the application to keep working even if Memcache 
went down.

You would be storing more keys in Memcache but each one would be small 
and could be set to expire.  Users who log in frequently would always be 
in the cache, users who don't log in very often would be automatically 
removed from the cache.

- John Kramlich
------------------------------
http://www.gigoit.org  - Give and get free items within your community