Caching in general. Re: Listing all keys on a server: why?

[Paul T]

1. Are the "general" caching conversations allowed on
this list?

2. If not - could somebody please point me to any
mailing list that allows those?

Thanks.

( Below are some explanations )

> Hi everyone,
> 
> what I do not understand about these discussions
> that CONSTANTLY go on 
> in this list (it is one of the most common questions
> my oppinion, or at 
> least a version of it) is that amount of time spent
> discussing this back 
> and forth has taken 5 times longer by now than
> simply adding the feature 
> to memcached. Its not a HUGE deal to crawl the tree
> for all key names 
> and just dump them, no need to sort them or
> anything, just dump them.

The side effect of this is that entire memcached
instance will be locked (not responding to any other
requests) until the 'dump' is completed. On a hash of
1Gb you would begin to notice this kind of stuff -
very annoying. If you store plenty of small keys you
would get such a trouble on hashes less than 1Gb.

Which can be called 'huge', no?

> If someone thinks they need this, let them use it
> and if you think it 
> would never be usefull then let them fail at it, it
> would remove 50% of 
> the traffic this list sees day in and out though...

Because I am implementing a 'smart' cache, I am *very*
 interested in exploring as many real-life complex
caching (or else) usecases as possible.

Unfortunately, understanding the complicated caching
problem implies asking many questions.

Unfortunately many of those questions would *not* be
related to memcached, but mostly investigating the
usecase per se.

For example, in my universe, because I implement the
'lazy invalidation' trick, one can execute commands
like 'invalidate key=key_regexpr, value=value_regexpr'
assuming that such an operation has no preformance
impact (no locking). 

Now, discussing the caching strategies based on lazy
invalidation on memcached mailing list seems wrong
(because memcached does not implement lazy
invalidation and there is basically no 'massive'
operations that are in general allowed by memcached -
by design I think). 

Dropping the discussions of the *possible* caching
strategies only because they are not implemented by
memcached - also seems wrong.

As to the usecase described below, instead of
datamining the memcached for a list of logged in users
it can be accomplished by maintaining a variable with
the name "list_of_users" inside memcached, modifying
it on session creation / session expiration. 

Long time ago I suggested how to implement this kind
of 'list' variables in memcached by twisting the
'incr' command, but you can implement it with standard
memcached 'get' and 'set' - should be no big deal 

And I believe that it will still be (much) better than
datamining the memcached hash.

Rgds.Paul.

> In certain cases such a list is usefull.
> 
> For example:
> session management via memcached.
> 
> Lets pretend you use memcached stored sessions in
> php, you know your 
> memcached has 64MB of data and you know you have
> session load at any 
> given time of about 5000 sessions each day. Each
> session takes 1000 
> bytes of data, which alrady is a good amount for a
> session, making it 
> 5MB total, PLENTY of room left in memcached that you
> know that the 
> session data would not get removed due to no more
> room TOO soon, it 
> MIGHT be removed sooner, but not after a minute or
> even an hour, it will 
> take long to happen IF it even happends, something
> you can live with 
> basically..
> 
> Now you have a problem though, you want to build a
> "logged in user 
> list", best way? get all sessions in memcached, look
> through each one 
> and filter out the username.
> 
> You could theoretically even add something which
> would limit the keys 
> listed depending on the timestamps they have (might
> not understand 
> memcached enough though, so not sure if this
> definately works) which 
> would limit the sessions you need to heck since you
> only want the ones 
> used in the past 10 minutes.
> 
> Fabian
> 
> 
> Brian Moon wrote:
> > Jamie McCarthy wrote:
> >> ...the list you would get would be obsolete as
> soon as you got it.  So
> >> my guess would be that, if you think you need
> that list, you may be
> >> wanting to do something with memcached that you
> can't really do.  But my
> >> guesses are frequently wrong and if there's
> something I'm forgetting
> >> please email me and let me know.  Thanks.
> >
> > I see no reason to take this off list.  Lets hash
> it out here.
> >
> > At the BoF last night, I talked with some guys
> that need/want that. 
> > They had valid reasons.  Not sure if they are on
> the list or not, but 
> > they had some valid arguments for their business. 
> Now, you could say 
> > that they are using memcached for something it was
> not made for, but 
> > the problem is that there is not anything better
> to use in its place.
> >
> > Brian Aiken spoke last night at the BoF about NDB
> (the backend to 
> > mysql cluster) and using it directly, skipping the
> mysql part.  That 
> > could be a good solution for some people that need
> more control and 
> > redundancy. However, there is no PHP/Perl/Python,
> etc. interface for 
> > it at this time.
> >
> 
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com