percona-toolkit/docs/user/pt-tcp-model.rst

############
pt-tcp-model
############

.. highlight:: perl


****
NAME
****


pt-tcp-model - Transform tcpdump into metrics that permit performance and scalability modeling.


********
SYNOPSIS
********


Usage: pt-tcp-model [OPTION...] [FILE]

pt-tcp-model parses and analyzes tcpdump files.  With no FILE, or when
FILE is -, it read standard input.

Dump TCP requests and responses to a file, capturing only the packet headers to
avoid dropped packets, and ignoring any packets without a payload (such as
ack-only packets).  Capture port 3306 (MySQL database traffic).  Note that to
avoid line breaking in terminals and man pages, the TCP filtering expression
that follows has a line break at the end of the second line; you should omit
this from your tcpdump command.


.. code-block:: perl

  tcpdump -s 384 -i any -nnq -tttt \
 	'tcp port 3306 and (((ip[2:2] - ((ip[0]&0xf)<<2)) 
       - ((tcp[12]&0xf0)>>2)) != 0)' \
    > /path/to/tcp-file.txt


Extract individual response times, sorted by end time:


.. code-block:: perl

  pt-tcp-model /path/to/tcp-file.txt > requests.txt


Sort the result by arrival time, for input to the next step:


.. code-block:: perl

  sort -n -k1,1 requests.txt > sorted.txt


Slice the result into 10-second intervals and emit throughput, concurrency, and
response time metrics for each interval:


.. code-block:: perl

  pt-tcp-model --type=requests --run-time=10 sorted.txt > sliced.txt


Transform the result for modeling with Aspersa's usl tool, discarding the first
and last line of each file if you specify multiple files (the first and last
line are normally incomplete observation periods and are aberrant):


.. code-block:: perl

  for f in sliced.txt; do
     tail -n +2 "$f" | head -n -1 | awk '{print $2, $3, $7/$4}'
  done > usl-input.txt



*****
RISKS
*****


The following section is included to inform users about the potential risks,
whether known or unknown, of using this tool.  The two main categories of risks
are those created by the nature of the tool (e.g. read-only tools vs. read-write
tools) and those created by bugs.

pt-tcp-model merely reads and transforms its input, printing it to the output.
It should be very low risk.

At the time of this release, we know of no bugs that could cause serious harm
to users.

The authoritative source for updated information is always the online issue
tracking system.  Issues that affect this tool will be marked as such.  You can
see a list of such issues at the following URL:
`http://www.percona.com/bugs/pt-tcp-model <http://www.percona.com/bugs/pt-tcp-model>`_.

See also "BUGS" for more information on filing bugs and getting help.


***********
DESCRIPTION
***********


This tool recognizes requests and responses in a TCP stream, and extracts the
"conversations".  You can use it to capture the response times of individual
queries to a database, for example.  It expects the TCP input to be in the
following format, which should result from the sample shown in the SYNOPSIS:


.. code-block:: perl

  <date> <time.microseconds> IP <IP.port> > <IP.port>: <junk>


The tool watches for "incoming" packets to the port you specify with the
"--watch-server" option.  This begins a request.  If multiple inbound packets
follow each other, then by default the last inbound packet seen determines the
time at which the request is assumed to begin.  This is logical if one assumes
that a server must receive the whole SQL statement before beginning execution,
for example.

When the first outbound packet is seen, the server is considered to have
responded to the request.  The tool might see an inbound packet, but never see a
response.  This can happen when the kernel drops packets, for example.  As a
result, the tool never prints a request unless it sees the response to it.
However, the tool actually does not print any request until it sees the "last"
outbound packet.  It determines this by waiting for either another inbound
packet, or EOF, and then considers the previous inbound/outbound pair to be
complete.  As a result, the tool prints requests in a relatively random order.
Most types of analysis require processing in either arrival or completion order.
Therefore, the second type of processing this tool can do requires that you sort
the output from the first stage and supply it as input.

The second type of processing is selected with the "--type" option set to
"requests".  In this mode, the tool reads a group of requests and aggregates
them, then emits the aggregated metrics.


******
OUTPUT
******


In the default mode (parsing tcpdump output), requests are printed out one per
line, in the following format:


.. code-block:: perl

  <id> <start> <end> <elapsed> <IP:port>


The ID is an incrementing number, assigned in arrival order in the original TCP
traffic.  The start and end timestamps, and the elapsed time, can be customized
with the "--start-end" option.

In "--type=requests" mode, the tool prints out one line per time interval as
defined by "--run-time", with the following columns: ts, concurrency,
throughput, arrivals, completions, busy_time, weighted_time, sum_time,
variance_mean, quantile_time, obs_time.  A detailed explanation follows:


ts
 
 The timestamp that defines the beginning of the interval.
 


concurrency
 
 The average number of requests resident in the server during the interval.
 


throughput
 
 The number of arrivals per second during the interval.
 


arrivals
 
 The number of arrivals during the interval.
 


completions
 
 The number of completions during the interval.
 


busy_time
 
 The total amount of time during which at least one request was resident in
 the server during the interval.
 


weighted_time
 
 The total response time of all the requests resident in the server during the
 interval, including requests that neither arrived nor completed during the
 interval.
 


sum_time
 
 The total response time of all the requests that arrived in the interval.
 


variance_mean
 
 The variance-to-mean ratio (index of dispersion) of the response times of the
 requests that arrived in the interval.
 


quantile_time
 
 The Nth percentile response time for all the requests that arrived in the
 interval.  See also "--quantile".
 


obs_time
 
 The length of the observation time window.  This will usually be the same as the
 interval length, except for the first and last intervals in a file, which might
 have a shorter observation time.
 



*******
OPTIONS
*******


This tool accepts additional command-line arguments.  Refer to the
"SYNOPSIS" and usage information for details.


--config
 
 type: Array
 
 Read this comma-separated list of config files; if specified, this must be the
 first option on the command line.
 


--help
 
 Show help and exit.
 


--progress
 
 type: array; default: time,30
 
 Print progress reports to STDERR.  The value is a comma-separated list with two
 parts.  The first part can be percentage, time, or iterations; the second part
 specifies how often an update should be printed, in percentage, seconds, or
 number of iterations.
 


--quantile
 
 type: float
 
 The percentile for the last column when "--type" is "requests" (default .99).
 


--run-time
 
 type: float
 
 The size of the aggregation interval in seconds when "--type" is "requests"
 (default 1).  Fractional values are permitted.
 


--start-end
 
 type: Array; default: ts,end
 
 Define how the arrival and completion timestamps of a query, and thus its
 response time (elapsed time) are computed.  Recall that there may be multiple
 inbound and outbound packets per request and response, and refer to the
 following ASCII diagram.  Suppose that a client sends a series of three inbound
 (I) packets to the server, whch computes the result and then sends two outbound
 (O) packets back:
 
 
 .. code-block:: perl
 
    I I    I ..................... O    O
    |<---->|<---response time----->|<-->|
    ts0    ts                      end  end1
 
 
 By default, the query is considered to arrive at time ts, and complete at time
 end.  However, this might not be what you want.  Perhaps you do not want to
 consider the query to have completed until time end1.  You can accomplish this
 by setting this option to \ ``ts,end1``\ .
 


--type
 
 type: string
 
 The type of input to parse (default tcpdump).  The permitted types are
 
 
 tcpdump
  
  The parser expects the input to be formatted with the following options: \ ``-x -n
  -q -tttt``\ .  For example, if you want to capture output from your local machine,
  you can do something like the following (the port must come last on FreeBSD):
  
  
  .. code-block:: perl
  
     tcpdump -s 65535 -x -nn -q -tttt -i any -c 1000 port 3306 \
       > mysql.tcp.txt
     pt-query-digest --type tcpdump mysql.tcp.txt
  
  
  The other tcpdump parameters, such as -s, -c, and -i, are up to you.  Just make
  sure the output looks like this (there is a line break in the first line to
  avoid man-page problems):
  
  
  .. code-block:: perl
  
     2009-04-12 09:50:16.804849 IP 127.0.0.1.42167
            > 127.0.0.1.3306: tcp 37
  
  
  All MySQL servers running on port 3306 are automatically detected in the
  tcpdump output.  Therefore, if the tcpdump out contains packets from
  multiple servers on port 3306 (for example, 10.0.0.1:3306, 10.0.0.2:3306,
  etc.), all packets/queries from all these servers will be analyzed
  together as if they were one server.
  
  If you're analyzing traffic for a protocol that is not running on port
  3306, see "--watch-server".
  
 
 


--version
 
 Show version and exit.
 


--watch-server
 
 type: string; default: 10.10.10.10:3306
 
 This option tells pt-tcp-model which server IP address and port (such as
 "10.0.0.1:3306") to watch when parsing tcpdump for "--type" tcpdump.  If you
 don't specify it, the tool watches all servers by looking for any IP address
 using port 3306.  If you're watching a server with a non-standard port, this
 won't work, so you must specify the IP address and port to watch.
 
 Currently, IP address filtering isn't implemented; so even though you must
 specify the option in IP:port form, it ignores the IP and only looks at the port
 number.
 



***********
ENVIRONMENT
***********


The environment variable \ ``PTDEBUG``\  enables verbose debugging output to STDERR.
To enable debugging and capture all output to a file, run the tool like:


.. code-block:: perl

    PTDEBUG=1 pt-tcp-model ... > FILE 2>&1


Be careful: debugging output is voluminous and can generate several megabytes
of output.


*******************
SYSTEM REQUIREMENTS
*******************


You need Perl, DBI, DBD::mysql, and some core packages that ought to be
installed in any reasonably new version of Perl.


****
BUGS
****


For a list of known bugs, see `http://www.percona.com/bugs/pt-tcp-model <http://www.percona.com/bugs/pt-tcp-model>`_.

Please report bugs at `https://bugs.launchpad.net/percona-toolkit <https://bugs.launchpad.net/percona-toolkit>`_.
Include the following information in your bug report:


\* Complete command-line used to run the tool



\* Tool "--version"



\* MySQL version of all servers involved



\* Output from the tool including STDERR



\* Input files (log/dump/config files, etc.)



If possible, include debugging output by running the tool with \ ``PTDEBUG``\ ;
see "ENVIRONMENT".


***********
DOWNLOADING
***********


Visit `http://www.percona.com/software/percona-toolkit/ <http://www.percona.com/software/percona-toolkit/>`_ to download the
latest release of Percona Toolkit.  Or, get the latest release from the
command line:


.. code-block:: perl

    wget percona.com/get/percona-toolkit.tar.gz
 
    wget percona.com/get/percona-toolkit.rpm
 
    wget percona.com/get/percona-toolkit.deb


You can also get individual tools from the latest release:


.. code-block:: perl

    wget percona.com/get/TOOL


Replace \ ``TOOL``\  with the name of any tool.


*******
AUTHORS
*******


Baron Schwartz


*********************
ABOUT PERCONA TOOLKIT
*********************


This tool is part of Percona Toolkit, a collection of advanced command-line
tools developed by Percona for MySQL support and consulting.  Percona Toolkit
was forked from two projects in June, 2011: Maatkit and Aspersa.  Those
projects were created by Baron Schwartz and developed primarily by him and
Daniel Nichter, both of whom are employed by Percona.  Visit
`http://www.percona.com/software/ <http://www.percona.com/software/>`_ for more software developed by Percona.


********************************
COPYRIGHT, LICENSE, AND WARRANTY
********************************


This program is copyright 2011 Baron Schwartz, 2011 Percona Inc.
Feedback and improvements are welcome.

THIS PROGRAM IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

This program is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation, version 2; OR the Perl Artistic License.  On UNIX and similar
systems, you can issue \`man perlgpl' or \`man perlartistic' to read these
licenses.

You should have received a copy of the GNU General Public License along with
this program; if not, write to the Free Software Foundation, Inc., 59 Temple
Place, Suite 330, Boston, MA  02111-1307  USA.


*******
VERSION
*******


pt-tcp-model 1.0.1