Add standard pkg files (COPYING, README, etc.), percona-toolkit.pod, and user docs. Remove dev/docs/html.

2025-09-19 02:05:23 +00:00 · 2011-07-14 13:08:47 -06:00
parent ffcb47a0d2
commit 767f8b3211
285 changed files with 27046 additions and 13387 deletions
--- a/docs/user/pt-tcp-model.rst
+++ b/docs/user/pt-tcp-model.rst
@@ -0,0 +1,528 @@
+
+############
+pt-tcp-model
+############
+
+.. highlight:: perl
+
+
+****
+NAME
+****
+
+
+pt-tcp-model - Transform tcpdump into metrics that permit performance and
+scalability modeling.
+
+
+********
+SYNOPSIS
+********
+
+
+Usage: pt-tcp-model [OPTION...] [FILE]
+
+pt-tcp-model parses and analyzes tcpdump files.  With no FILE, or when
+FILE is -, it read standard input.
+
+Dump TCP requests and responses to a file, capturing only the packet headers to
+avoid dropped packets, and ignoring any packets without a payload (such as
+ack-only packets).  Capture port 3306 (MySQL database traffic).  Note that to
+avoid line breaking in terminals and man pages, the TCP filtering expression
+that follows has a line break at the end of the second line; you should omit
+this from your tcpdump command.
+
+
+.. code-block:: perl
+
+  tcpdump -s 384 -i any -nnq -tttt \
+ 	'tcp port 3306 and (((ip[2:2] - ((ip[0]&0xf)<<2)) 
+       - ((tcp[12]&0xf0)>>2)) != 0)' \
+    > /path/to/tcp-file.txt
+
+
+Extract individual response times, sorted by end time:
+
+
+.. code-block:: perl
+
+  pt-tcp-model /path/to/tcp-file.txt > requests.txt
+
+
+Sort the result by arrival time, for input to the next step:
+
+
+.. code-block:: perl
+
+  sort -n -k1,1 requests.txt > sorted.txt
+
+
+Slice the result into 10-second intervals and emit throughput, concurrency, and
+response time metrics for each interval:
+
+
+.. code-block:: perl
+
+  pt-tcp-model --type=requests --run-time=10 sorted.txt > sliced.txt
+
+
+Transform the result for modeling with Aspersa's usl tool, discarding the first
+and last line of each file if you specify multiple files (the first and last
+line are normally incomplete observation periods and are aberrant):
+
+
+.. code-block:: perl
+
+  for f in sliced.txt; do
+     tail -n +2 "$f" | head -n -1 | awk '{print $2, $3, $7/$4}'
+  done > usl-input.txt
+
+
+
+*****
+RISKS
+*****
+
+
+The following section is included to inform users about the potential risks,
+whether known or unknown, of using this tool.  The two main categories of risks
+are those created by the nature of the tool (e.g. read-only tools vs. read-write
+tools) and those created by bugs.
+
+pt-tcp-model merely reads and transforms its input, printing it to the output.
+It should be very low risk.
+
+At the time of this release, we know of no bugs that could cause serious harm
+to users.
+
+The authoritative source for updated information is always the online issue
+tracking system.  Issues that affect this tool will be marked as such.  You can
+see a list of such issues at the following URL:
+`http://www.percona.com/bugs/pt-tcp-model <http://www.percona.com/bugs/pt-tcp-model>`_.
+
+See also "BUGS" for more information on filing bugs and getting help.
+
+
+***********
+DESCRIPTION
+***********
+
+
+This tool recognizes requests and responses in a TCP stream, and extracts the
+"conversations".  You can use it to capture the response times of individual
+queries to a database, for example.  It expects the TCP input to be in the
+following format, which should result from the sample shown in the SYNOPSIS:
+
+
+.. code-block:: perl
+
+  <date> <time.microseconds> IP <IP.port> > <IP.port>: <junk>
+
+
+The tool watches for "incoming" packets to the port you specify with the
+"--watch-server" option.  This begins a request.  If multiple inbound packets
+follow each other, then by default the last inbound packet seen determines the
+time at which the request is assumed to begin.  This is logical if one assumes
+that a server must receive the whole SQL statement before beginning execution,
+for example.
+
+When the first outbound packet is seen, the server is considered to have
+responded to the request.  The tool might see an inbound packet, but never see a
+response.  This can happen when the kernel drops packets, for example.  As a
+result, the tool never prints a request unless it sees the response to it.
+However, the tool actually does not print any request until it sees the "last"
+outbound packet.  It determines this by waiting for either another inbound
+packet, or EOF, and then considers the previous inbound/outbound pair to be
+complete.  As a result, the tool prints requests in a relatively random order.
+Most types of analysis require processing in either arrival or completion order.
+Therefore, the second type of processing this tool can do requires that you sort
+the output from the first stage and supply it as input.
+
+The second type of processing is selected with the "--type" option set to
+"requests".  In this mode, the tool reads a group of requests and aggregates
+them, then emits the aggregated metrics.
+
+
+******
+OUTPUT
+******
+
+
+In the default mode (parsing tcpdump output), requests are printed out one per
+line, in the following format:
+
+
+.. code-block:: perl
+
+  <id> <start> <end> <elapsed> <IP:port>
+
+
+The ID is an incrementing number, assigned in arrival order in the original TCP
+traffic.  The start and end timestamps, and the elapsed time, can be customized
+with the "--start-end" option.
+
+In "--type=requests" mode, the tool prints out one line per time interval as
+defined by "--run-time", with the following columns: ts, concurrency,
+throughput, arrivals, completions, busy_time, weighted_time, sum_time,
+variance_mean, quantile_time, obs_time.  A detailed explanation follows:
+
+
+ts
+ 
+ The timestamp that defines the beginning of the interval.
+ 
+
+
+concurrency
+ 
+ The average number of requests resident in the server during the interval.
+ 
+
+
+throughput
+ 
+ The number of arrivals per second during the interval.
+ 
+
+
+arrivals
+ 
+ The number of arrivals during the interval.
+ 
+
+
+completions
+ 
+ The number of completions during the interval.
+ 
+
+
+busy_time
+ 
+ The total amount of time during which at least one request was resident in
+ the server during the interval.
+ 
+
+
+weighted_time
+ 
+ The total response time of all the requests resident in the server during the
+ interval, including requests that neither arrived nor completed during the
+ interval.
+ 
+
+
+sum_time
+ 
+ The total response time of all the requests that arrived in the interval.
+ 
+
+
+variance_mean
+ 
+ The variance-to-mean ratio (index of dispersion) of the response times of the
+ requests that arrived in the interval.
+ 
+
+
+quantile_time
+ 
+ The Nth percentile response time for all the requests that arrived in the
+ interval.  See also "--quantile".
+ 
+
+
+obs_time
+ 
+ The length of the observation time window.  This will usually be the same as the
+ interval length, except for the first and last intervals in a file, which might
+ have a shorter observation time.
+ 
+
+
+
+*******
+OPTIONS
+*******
+
+
+This tool accepts additional command-line arguments.  Refer to the
+"SYNOPSIS" and usage information for details.
+
+
+--config
+ 
+ type: Array
+ 
+ Read this comma-separated list of config files; if specified, this must be the
+ first option on the command line.
+ 
+
+
+--help
+ 
+ Show help and exit.
+ 
+
+
+--progress
+ 
+ type: array; default: time,30
+ 
+ Print progress reports to STDERR.  The value is a comma-separated list with two
+ parts.  The first part can be percentage, time, or iterations; the second part
+ specifies how often an update should be printed, in percentage, seconds, or
+ number of iterations.
+ 
+
+
+--quantile
+ 
+ type: float
+ 
+ The percentile for the last column when "--type" is "requests" (default .99).
+ 
+
+
+--run-time
+ 
+ type: float
+ 
+ The size of the aggregation interval in seconds when "--type" is "requests"
+ (default 1).  Fractional values are permitted.
+ 
+
+
+--start-end
+ 
+ type: Array; default: ts,end
+ 
+ Define how the arrival and completion timestamps of a query, and thus its
+ response time (elapsed time) are computed.  Recall that there may be multiple
+ inbound and outbound packets per request and response, and refer to the
+ following ASCII diagram.  Suppose that a client sends a series of three inbound
+ (I) packets to the server, whch computes the result and then sends two outbound
+ (O) packets back:
+ 
+ 
+ .. code-block:: perl
+ 
+    I I    I ..................... O    O
+    |<---->|<---response time----->|<-->|
+    ts0    ts                      end  end1
+ 
+ 
+ By default, the query is considered to arrive at time ts, and complete at time
+ end.  However, this might not be what you want.  Perhaps you do not want to
+ consider the query to have completed until time end1.  You can accomplish this
+ by setting this option to \ ``ts,end1``\ .
+ 
+
+
+--type
+ 
+ type: string
+ 
+ The type of input to parse (default tcpdump).  The permitted types are
+ 
+ 
+ tcpdump
+  
+  The parser expects the input to be formatted with the following options: \ ``-x -n
+  -q -tttt``\ .  For example, if you want to capture output from your local machine,
+  you can do something like the following (the port must come last on FreeBSD):
+  
+  
+  .. code-block:: perl
+  
+     tcpdump -s 65535 -x -nn -q -tttt -i any -c 1000 port 3306 \
+       > mysql.tcp.txt
+     pt-query-digest --type tcpdump mysql.tcp.txt
+  
+  
+  The other tcpdump parameters, such as -s, -c, and -i, are up to you.  Just make
+  sure the output looks like this (there is a line break in the first line to
+  avoid man-page problems):
+  
+  
+  .. code-block:: perl
+  
+     2009-04-12 09:50:16.804849 IP 127.0.0.1.42167
+            > 127.0.0.1.3306: tcp 37
+  
+  
+  All MySQL servers running on port 3306 are automatically detected in the
+  tcpdump output.  Therefore, if the tcpdump out contains packets from
+  multiple servers on port 3306 (for example, 10.0.0.1:3306, 10.0.0.2:3306,
+  etc.), all packets/queries from all these servers will be analyzed
+  together as if they were one server.
+  
+  If you're analyzing traffic for a protocol that is not running on port
+  3306, see "--watch-server".
+  
+ 
+ 
+
+
+--version
+ 
+ Show version and exit.
+ 
+
+
+--watch-server
+ 
+ type: string; default: 10.10.10.10:3306
+ 
+ This option tells pt-tcp-model which server IP address and port (such as
+ "10.0.0.1:3306") to watch when parsing tcpdump for "--type" tcpdump.  If you
+ don't specify it, the tool watches all servers by looking for any IP address
+ using port 3306.  If you're watching a server with a non-standard port, this
+ won't work, so you must specify the IP address and port to watch.
+ 
+ Currently, IP address filtering isn't implemented; so even though you must
+ specify the option in IP:port form, it ignores the IP and only looks at the port
+ number.
+ 
+
+
+
+***********
+DOWNLOADING
+***********
+
+
+Visit `http://www.percona.com/software/ <http://www.percona.com/software/>`_ to download the latest release of
+Percona Toolkit.  Or, to get the latest release from the command line:
+
+
+.. code-block:: perl
+
+    wget percona.com/latest/percona-toolkit/PKG
+
+
+Replace \ ``PKG``\  with \ ``tar``\ , \ ``rpm``\ , or \ ``deb``\  to download the package in that
+format.  You can also get individual tools from the latest release:
+
+
+.. code-block:: perl
+
+    wget percona.com/latest/percona-toolkit/TOOL
+
+
+Replace \ ``TOOL``\  with the name of any tool.
+
+
+***********
+ENVIRONMENT
+***********
+
+
+The environment variable \ ``PTDEBUG``\  enables verbose debugging output to STDERR.
+To enable debugging and capture all output to a file, run the tool like:
+
+
+.. code-block:: perl
+
+    PTDEBUG=1 pt-tcp-model ... > FILE 2>&1
+
+
+Be careful: debugging output is voluminous and can generate several megabytes
+of output.
+
+
+*******************
+SYSTEM REQUIREMENTS
+*******************
+
+
+You need Perl, DBI, DBD::mysql, and some core packages that ought to be
+installed in any reasonably new version of Perl.
+
+
+****
+BUGS
+****
+
+
+For a list of known bugs, see `http://www.percona.com/bugs/pt-tcp-model <http://www.percona.com/bugs/pt-tcp-model>`_.
+
+Please report bugs at `https://bugs.launchpad.net/percona-toolkit <https://bugs.launchpad.net/percona-toolkit>`_.
+Include the following information in your bug report:
+
+
+\* Complete command-line used to run the tool
+
+
+
+\* Tool "--version"
+
+
+
+\* MySQL version of all servers involved
+
+
+
+\* Output from the tool including STDERR
+
+
+
+\* Input files (log/dump/config files, etc.)
+
+
+
+If possible, include debugging output by running the tool with \ ``PTDEBUG``\ ;
+see "ENVIRONMENT".
+
+
+*******
+AUTHORS
+*******
+
+
+Baron Schwartz
+
+
+*********************
+ABOUT PERCONA TOOLKIT
+*********************
+
+
+This tool is part of Percona Toolkit, a collection of advanced command-line
+tools developed by Percona for MySQL support and consulting.  Percona Toolkit
+was forked from two projects in June, 2011: Maatkit and Aspersa.  Those
+projects were created by Baron Schwartz and developed primarily by him and
+Daniel Nichter, both of whom are employed by Percona.  Visit
+`http://www.percona.com/software/ <http://www.percona.com/software/>`_ for more software developed by Percona.
+
+
+********************************
+COPYRIGHT, LICENSE, AND WARRANTY
+********************************
+
+
+This program is copyright 2011 Baron Schwartz, 2011 Percona Inc.
+Feedback and improvements are welcome.
+
+THIS PROGRAM IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
+WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+
+This program is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free Software
+Foundation, version 2; OR the Perl Artistic License.  On UNIX and similar
+systems, you can issue \`man perlgpl' or \`man perlartistic' to read these
+licenses.
+
+You should have received a copy of the GNU General Public License along with
+this program; if not, write to the Free Software Foundation, Inc., 59 Temple
+Place, Suite 330, Boston, MA  02111-1307  USA.
+
+
+*******
+VERSION
+*******
+
+
+Percona Toolkit v1.0.0 released 2011-08-01
+