Mogilefs - many db connections causing performance problem
dormando at rydia.net
Thu Apr 10 23:48:05 UTC 2008
Not sure I follow what you're doing here?
Are those extra connections running queries? Why is it killing your
Pick through 'show processlist' and find the source ip:port of the
Log into the host by that IP, and use netstat -np and map the 'port'
from that entry in show processlist to a specific process. Find out what
that process is, what it's doing, if it's actually a tracker, and what
type of worker. (which you can find in ps).
> this matches with the number of DB connections I see on the tracker, but
> in the DB server there are many more connections coming from the tracker
> server. The total number of connections I saw directly on MySQL is equal
> to the number of connections I saw with netstat on the DB server. All
> the connections are in state ESTABLISHED. Do you have any idea on what
> can be happening?
> Thanks again!
> On Thu, 2008-04-10 at 16:04 -0700, dormando wrote:
>> telnet to the tracker's management port and run !jobs.
>> you should have one DB connection per job type that exists. I figure
>> you're probably sending most of your traffic to one of the trackers?
>> If you have fewer workers than connections something else might be up.
>> Are those connections busy with queries? Can you tell what kind of
>> queries? etc.
>> Fernando Gomes wrote:
>>> I'm using MogileFS with mysql DB, with two trackers and two storage
>>> nodes. There are around 65000 files stored, and as they are replicated,
>>> about 130000 total, distributed by the different devices.
>>> All was working as expected until recently I noticed two problems
>>> (perhaps related). One is that from time to time the files I got from
>>> the client (java) wasn't the requested file (I suppose that might be
>>> something like reported here:
>>> http://davidrasch.com/2008/01/29/mogilefs-and-race-condition). I'll try
>>> to look to the client java code in order to see what can be done.
>>> The other problem is that today the trackers started creating a lot of
>>> database connections (one tracker more than the other), almost taking
>>> the database down. After finding that the database performance problem
>>> was caused by one of the trackers I restarted it and things got a bit
>>> better, but I still see many connections on the database from the
>>> When trying to diagnose the problem also found something that seems
>>> strange to me (perhaps caused by the hours spent on this problem) - the
>>> number of connections from one tracker to the database, evaluated by
>>> netstat -n |grep 3306, is not the same if I execute netstat on the
>>> tracker server or on the database server (I am now having 8 connections
>>> from tracker 2 to the database server if I run netstat on the tracker,
>>> but I see 68 connections from tracker2 to the database if i run it on
>>> the database server.
>>> If you can give me some tip about what might be the problem it will be
>>> very useful to me!
More information about the mogilefs