pbstat

Adam Jacob adam at hjksolutions.com
Wed May 9 21:55:06 UTC 2007


Hello all. I've had occasion to start using Perlbal to front some Rails
applications of late. Since it has fabulous statistics, we wanted to be able
to see what's up with our Perlbal instances in a vmstat-like way. Attached is
the (admittedly quick) first pass at a script that does just that, at the
moment for Client Socket states and URI listings. Language snobs beware, it's
written in Ruby, as that's what the client is most familiar with. :)

For example:

./pbstat perlbal1 perlbal2
hostname        tc      r_h     w_b     b_s     w_r     x_r     d_r     w_s     w_o     x_d
perlbal1        0       0       0       0       0       0       0       0       0       0
perlbal2        18      0       0       0       17      1       0       0       0       0
perlbal1        0       0       0       0       0       0       0       0       0       0
perlbal2        18      0       0       0       16      2       0       0       0       0
perlbal1        0       0       0       0       0       0       0       0       0       0
perlbal2        20      0       0       0       19      1       0       0       0       0
perlbal1        0       0       0       0       0       0       0       0       0       0
perlbal2        19      0       0       0       17      2       0       0       0       0

The columns are:

  total connections: tc
  reading_headers: r_h
  wait_backend: w_b
  backend_req_sent: b_s
  wait_res: w_r
  xfer_res: x_r
  draining_res: d_r
  wait_stat: w_s
  wait_open: w_o
  xfer_disk: x_d

A more entertaining mode (but perhaps less useful) will show all the URIs that
the client proxy is serving:

./pbstat -d 0.1 -u perlbal1 perlbal2
perlbal1       http://127.0.0.1:4999/  1
perlbal2       http://127.0.0.1:4999/search    1

(The number on the end is a count.)

Finally, you can just issue raw commands to the Perlbal backends:

./pbstat -c 1 -r "proc" amos1test amos2test
perlbal1       time: 1178747525
perlbal1       pid: 21240
perlbal1       utime: 10.752672 (+9.732609)
perlbal1       stime: 2.756172 (+2.576161)
perlbal1       reqs: 3590 (+0)
perlbal2       time: 1178747525
perlbal2       pid: 25966
perlbal2       utime: 2629.856355 (+2629.856355)
perlbal2       stime: 519.304454 (+519.304454)
perlbal2       reqs: 1041492 (+0)

Like I said above, it's pretty rough, but not a bad first start. Are there
other statistics that would be useful to display? How should they be
displayed?

If people have suggestions, I'll be happy to refactor this and make it a more
useful tool.

Otherwise, fun toy. :)

It's attached to this message, or you can get it from:

svn://cabo.hjksolutions.com/pbstat/trunk

Adam

-- 
HJK Solutions - We Launch Startups - http://www.hjksolutions.com
Adam Jacob, Senior Partner
T: (206) 508-4759 E: adam at hjksolutions.com
-------------- next part --------------
#!/usr/bin/ruby
#
# A Simple Perlbal Statistics Parser
#
# Adam Jacob <adam at hjksolutions.com>
# Written for Avvo, Inc. (http://www.avvo.com)
#
# $Id: pbstat 16 2007-05-09 21:25:13Z adam $
#

require 'socket'
require 'timeout'
require 'optparse'

module Perlbal
  
  class Connection 
    
    attr_reader :server
   
    def initialize(server="127.0.0.1", port="9999", timeout=3)
      @server = server
      @port = port
      @timeout = timeout
      @socket = nil
    end
    
    def run_command(cmd, &block)
      @socket.print "#{cmd}\n"
      fetch = true
      while fetch
        response = @socket.gets
        fetch = false if response =~ /^\./
        fetch = false if response =~ /^ERROR/
        block.call(response) if fetch
      end
    end
    
    def connect
      return @socket if @socket
      begin
        timeout(@timeout) do
          @socket = TCPSocket.new(@server, @port) 
        end
      rescue
        raise "connect error: #{$!}"
      end
    end
    
  end
  
  class Socks 
    
    CLIENT_STATES = [
      :reading_headers,
      :wait_backend,
      :backend_req_sent,
      :wait_res,
      :xfer_res,
      :draining_res,
      :wait_stat,
      :wait_open,
      :xfer_disk,
    ]
    
    def initialize(pbconn)
      @client_proxy = {
        :total_connects => 0,
      }
      CLIENT_STATES.each { |s| @client_proxy[s] = 0 }
      @client_uri = Hash.new
      @raw_output = String.new
      @pbconn = pbconn
    end
    
    def run
       cmd = "socks"

       @pbconn.run_command(cmd) do |data|
         data.chomp!
         data =~ /^\s+\d+\s+(.+?) (Perlbal::.+)\(R\): (.+): (.+)$/
         sock_secs = $1
         sock_type = $2
         sock_status = $3
         sock_extended = $4

         if sock_type == "Perlbal::ClientProxy"
           port, reqs, state, uri, backend = sock_extended.split('; ')
           @client_proxy[state.to_sym] = @client_proxy[state.to_sym] + 1
           @client_proxy[:total_connects] = @client_proxy[:total_connects] + 1
           if @client_uri.has_key?(uri.to_sym)
             @client_uri[uri.to_sym] = @client_uri[uri.to_sym] + 1
           else
             @client_uri[uri.to_sym] = 1
           end if uri
         elsif sock_type == "Perlbal::BackendHTTP"
         end
       end
    end
    
    def run_raw(cmd)
      @pbconn.run_command(cmd) do |data|
        @raw_output << data
      end
    end
    
    def Socks.shorten_state(s)
      s.to_s.sub(/^(.).+_(.).+$/, '\1_\2')
    end
  
    
    def Socks.list_states
      CLIENT_STATES.each do |s|
        yield(s, Socks.shorten_state(s))
      end
    end
    
    def Socks.header
      state_strings = ["hostname","tc"]
      CLIENT_STATES.each do |s|
        state_strings << shorten_state(s)
      end
      puts("#{state_strings.join("\t")}")
    end
    
    def output_raw
      @raw_output.each { |line| puts("#{@pbconn.server}\t#{line}") }
    end
    
    def output_uri
      @client_uri.each do |uri, value|
        puts("#{@pbconn.server}\t#{uri}\t#{value}")
      end
    end

    def output_socket
      states = Array.new
      CLIENT_STATES.each do |s|
        states << @client_proxy[s].to_s
      end
        puts("#{@pbconn.server}\t#{@client_proxy[:total_connects]}\t" + 
            "#{states.join("\t")}")
    end 
  end
end

Signal.trap("SIGINT") { exit 130 }
Signal.trap("SIGKILL") { exit 120 }

options = {
  :do => :socket,
  :count => 0,
  :delay => 1,
  :port => 9999
}
OptionParser.new do |opts|
  opts.banner = "Usage: #{$0} (-p port) (-d delay) (-c count) (-r cmd) (-u) (-s) hosts.."

  opts.on("-p", "--port [PORT]", Float, 
    "Port number of Perlbal Management Interface") do |p|
    options[:port] = p
  end
  
  opts.on("-d", "--delay [FLOAT]", Float,
    "Interval in seconds between statistics.  Default 1") do |d|
    options[:delay] = d
  end
  
  opts.on("-c", "--count [INTEGER]", Integer, "Number of runs to make.  Default infinity.") do |c|
    options[:count] = c
  end
  
  opts.on("-r", "--raw [STRING]", String, "Run a raw command") do |r|
    options[:raw] = r
    options[:do] = :raw
  end
  
  opts.on("-u", "--uri", "Show current URIs, with count.") do |u|
    options[:do] = :uri
  end
  
  opts.on("-s", "--socket", "Show socket states") do |s|
    options[:do] = :socket
  end
  opts.separator "\tStates:"
  Perlbal::Socks.list_states do |long, short|
    opts.separator "\t #{long}: #{short}"
  end
  opts.separator ""
  
  opts.on_tail("-h", "--help", "Show this message") do
    puts opts
    exit
  end

end.parse!

unless ARGV.length >= 1
  puts "You must supply at least one host to connect to!"
  exit 1
end

connections = []

ARGV.each do |host|
  pbconn = Perlbal::Connection.new(host, options[:port])
  pbconn.connect
  connections << pbconn
end

count = 0
output_rows, output_cols = %x{stty size}.split.collect { |x| x.to_i }

while 1
  threads = []
  connections.each do |pbconn|
    if options[:do] == :socket
      Perlbal::Socks.header if count == 0 || count % output_rows == 0
    end
    threads << Thread.new(pbconn) do |pbconn|
      this_socks = Perlbal::Socks.new(pbconn)
      if options[:do] == :socket
        this_socks.run
        this_socks.output_socket
      elsif options[:do] == :uri
        this_socks.run
        this_socks.output_uri
      elsif options[:do] == :raw
        this_socks.run_raw(options[:raw])
        this_socks.output_raw
      end
    end
    count = count + 1
  end
  threads.each { |t| t.join }
  if options[:count] != 0
    exit 0 if options[:count] * connections.length == count
  end
  sleep options[:delay]
end



More information about the perlbal mailing list