Memcached now used on HowardStern.com
brad at danga.com
Fri Nov 5 15:05:34 PST 2004
Be aware that memcached is mostly limited to O(1) operations. While we
have some things like, "List recently updated keys", they're just there
for our debugging. We don't have any "show me all keys" operations.
What you want to do, deleting all things starting with "frontpage:" is
what we intend namespaces to fix. Then you just put "foo" into namespace
"frontpage" instead of "frontpage:foo" and then you can flush_all on the
Currently the Perl API implements namespaces by just prefixing the keys
with your namespace, but we intend for the Perl API (and others) in the
future to discover the capabilities of the remote memcached server and use
true server-supported namespaces when possible.
For now, here's an alternative solution to your problem: make a new
"frontpage_dirty_at" key which stores a time you want to wipe the
frontpage. Then each key you use for the frontpage is stored as an array
of: [ $time, $data ], and your app can do a get_multi fetch of both
frontpage_dirty_at and the other key, and ignore the other key if its time
is too old.
Lame, but it'd work.
Then again, if you clear that infrequently, you could just do flush_all
and take the hit for a bit.
Or maintain a list of keys you use for the frontpage and make a script to
iterate over them all and delete them.
On Fri, 5 Nov 2004, Alex Firmani wrote:
> Hi Brad,
> I've finished implementing memcached on HowardStern.com and the
> forums. I'm thrilled to report that memcached is now serving 70% of
> the *dynamic* forum pages from it's cache. It was tricky work keeping
> user functionality while hitting the cache, but it was worth it. Our
> mySQL database server load has dropped from 5 to 0.5. Not only is it
> serving 70% of the pages but it has removed probably 95% of the
> redundant queries that the forums used to do.
> One question -- I'm using the PHP API and it is missing two functions
> that I would like to have. The first is stats and the second, rather
> important, is flush_all. I've coded them now myself and added them to
> the memcached PHP include, but they are sloppy. It seems the PHP
> socket requires a valid $key to connect. It doesn't make sense to me
> that I'd need to specify a $key to create a socket when all I want to
> do is flush all of them.
> I have a few more specific questions; like how do I display all the
> keys that memcached has stored? I'd like to get a list or array of
> all the key names and then be able to delete certain ones that start
> with a certain prefix. Like delete all caches that start with
> "frontpage" .. so it would clear only a certain set of caches.
> Thanks for your time if you have a moment to answer,
> Brad Fitzpatrick wrote:
> BF> Great to hear!
> BF> I listen to Howard Stern most days on my way to work. I just heard him
> BF> talking about the forums the other day, and him wanting to shut it down
> BF> because people are fighting on it. Hopefully memcached makes it faster
> BF> for people to fight with each other. :-)
> BF> - Brad
> BF> On Sun, 24 Oct 2004, Alex Firmani wrote:
> >> Hello to the team @ Danga,
> >> I want to say thank you and let you know that memcached is now being
> >> used on the official HowardStern.com website.
> >> Howard Stern, with over 12 million radio listeners each week,
> >> presented a huge problem when he wanted to open up a message board. On
> >> the first day, over 40,000 people tried to log in when he announced on
> >> the air that it was open. At the moment, traffic has calmed down from
> >> that, but it is now in the top-5 busiest message boards on the web,
> >> getting almost a half a million posts per week. It's not unusual to
> >> register 10,000 new users in one day.
> >> We've been struggling to keep the hardware matched to the traffic
> >> level and ever-growing database size. I've now implemented memcached
> >> across major sections of the site and the vBulletin-based forum. We're
> >> using three memcached servers to support the one large mySQL server.
> >> The performance improvement has been tremendous and it allows me to
> >> have an elegant memory caching solution always available instead of
> >> my own cache on the webservers' filesystems, as I had been doing.
> >> Thanks again for an excellent product and feel free to add us
> >> to your Users page!
> >> Alex Firmani
> >> HowardStern.com
More information about the memcached