Python client for MogileFS

[Justin Azoff]

Gaelen Hadlett wrote:
> Hi Justin,
> 
> I've been testing out our mogile setup using your python client.
> Admittedly, I have no experience using the perl tools, but I have hit
> a few errors.
> 
> My tests writes 5000 files between 1kb and 3mb and then reads them
> back.  I am using mfs_client_conn.send_file(mfid, filepath) to send
> files to our server and read_data =
> mfs_client_conn.get_file_data(mfid) to read the files.
> 
> Occasionally, (1-5 out of 5000 times) a read throws the exception
> message "MogileFSError: MogileFS::Backend: LASTERR: no_temp_file
> no_temp_file".  The write for this file id did not give any indication
> that it failed, but the file is not in the file system.  Have you ever
> come across this error?  Is there a way to verify a write executed
> correctly?
> 
> Thanks,
> Gaelen

That is odd..  The first thing I would do is
export MOGFS_DEBUG=1
and re-run your test program.  That will hopefully shed some light on
what is happening.

you can set verify_data=True in the main Client() call which will cause
it to verify for you as soon as you call .close() [in some cases.. it
won't if you were streaming the file, which send_file does, since it
doesn't have anything to verify against].  It would be nice if you could
tell the back-end what the checksum for the file should be, that would
let you do a lot of neat stuff.

If you are writing really important files you may also want to set
verify_repcount=True, this will block on .close() for up to 20 seconds
until the file is replicated to whatever the mindevcount is. If the file
doesn't get replicated in the 20 seconds an exception will be raised..
It probably would be a good idea to make that a tunable parameter, but I
never had any problems... :)

This way if you were doing something like:

for file in legacy_system_data:
  add_to_mogilefs(file)
  legacy_delete(file)

you wouldn't have to worry about a device dying right after you saved a
file to it, but before the file had a chance to be replicated.



-- 
-- Justin Azoff
-- Network Performance Analyst