Large lists

[Dustin Sallings]

On Nov 4, 2007, at 23:32, J A wrote:

> I have a fairly large members list that I want to keep in memcache.   
> What I do with this list is query it against particular user IDs to  
> see if they are a member of that list or not.  If they are they get  
> certain priviledges.
>
> The problem is, this list has gotten to the point of saturating the  
> PHP's memory when fetching the MySQL query the first time.
>
> Is there a way to do this more effectively, for instance,  
> partitioning the list into separate smaller lists, grouped by time  
> of login?  I'm thinking of this, as users who have logged in in the  
> past 3 months are more likely to be in the list anyway.


	It'd be easier to not think of it as a list if you're just testing  
for membership.  All you want to know is if a particular object is an  
element of a particular set.  You could do this by key convention if  
you batch populate the records.

	However, memcached semantics don't quite give you what you want.   
Depending on whether you can reasonably get a configuration to do what  
you want, it might be easier to think of memcached as a bloom filter  
than as a set in this case.  That is, if you negatively cache things  
that *aren't* part of your list, then the presence of a key will tell  
you for certain that a particular key is not a member, but the absence  
of a key would mean that you don't know (or perhaps memcached *did*  
know, but had since forgotten).

	You could, of course, record the status either way so as to tell the  
difference between not knowing and knowing whether it's a member or  
not.  This is probably best suited to your needs.

	You could optionally preload objects that are likely to be used if  
you think the natural population wouldn't do it effectively (you can  
measure this with stats).

-- 
Dustin Sallings