list_keys listing all keys

[Brad Fitzpatrick]

Heh.  I love it when that happens.

So I guess all we need is a smarter client so we can do something like:

    $mogclient->foreach_key(domain => "foodomain",
                            prefix => "pic:", sub {
        my $key = shift;
    })

And then have the MogileFS::Client library do the multiple calls into the
server as necessary, hiding it all for the caller and just running their
callback for each key returned over multiple round-trips to server.

And yes, we can remove the need for a prefix in the server now.


On Mon, 20 Nov 2006, Daniel Burke wrote:

> Hi Brad,
>
> I would like to write a patch - but it looks like you've already
> implemented this since the earliest revision of Query.pm in the log:
> r249 (When you split mogilefsd into multiple files)
>
> Is there any difference between this design and the current
> specification:
>
> # used to get a list of keys matching a certain prefix.
> # expected arguments:
> #   ( $prefix, $after, $limit )
> # prefix specifies what you want to get a list of.
> # after is the item specified
> # as a return value from this function last time you called it.
> # limit is optional and defaults to 1000 keys returned.
> #
> # if you expect an array of return values, returns:
> #   ($after, $keys)
> # but if you expect only a single value,
> # you just get the arrayref of keys.  the
> # value $after is to be used as $after when you
> # call this function again.
> #
> # when there are no more keys in the list, you will get back undef(s).
>
> Daniel
>
>
> On Mon, 2006-11-20 at 02:46 +0000, Brad Fitzpatrick wrote:
> > Really what we need is a:
> >
> >    list_keys after=&limit=1000
> >
> > Which means "return up to 1000 keys with key after ''".
> >
> > Then we get to "foo" as the 1000th item in the response, and client can
> > re-issue:
> >
> >    list_keys after=foo&limit=1000
> >
> > And so on.
> >
> > And prefix can still live as an additional filter, in addition to after=.
> >
> > Care to write a patch?
> >
> >
> > On Mon, 20 Nov 2006, Brad Fitzpatrick wrote:
> >
> > > The whole server method is flawed because there's no way to iteratively
> > > return stuff via the protocol.  It's single-line one-response one-answer,
> > > so if your request is "All keys please!" the answer might be a 2GB single
> > > line, which would probably break dozens of layers as it traverses stuff.
> > >
> > > So the list_keys limitation in the server is really a "I bet this isn't
> > > what you want." type of thing.
> > >
> > >
> > > On Mon, 20 Nov 2006, Daniel Burke wrote:
> > >
> > > >
> > > > Hi MogileFSers,
> > > >
> > > > If list_keys is passed an empty prefix, it will return an error:
> > > >
> > > >   $mogc->list_keys("", undef);
> > > >
> > > > I had expected all keys to be returned but is this an intentional
> > > > design decision?
> > > >
> > > > If I apply the attached patch to Worker/Query.pm::cmd_list_keys:476
> > > >
> > > >   return $self->err_line("no_key") unless $prefix;
> > > >
> > > > To become:
> > > >
> > > >   return $self->err_line("no_key") unless defined $prefix;
> > > >
> > > > list_keys in the Client API will now list all keys if given an empty
> > > > prefix.
> > > > Will anything bad happen? I ask because I think there must be a good
> > > > reason for this design decision and I want to be careful.
> > > >
> > > > Thanks!
> > > > Daniel
> > > >
> > > >
> > > >
> > >
> > >
>
>