percona-toolkit/docs/user/pt-log-player.rst

#############
pt-log-player
#############

.. highlight:: perl


****
NAME
****


pt-log-player - Replay MySQL query logs.


********
SYNOPSIS
********


Usage: pt-log-player [OPTION...] [DSN]

pt-log-player splits and plays slow log files.

Split slow.log on Thread_id into 16 session files, save in ./sessions:


.. code-block:: perl

   pt-log-player --split Thread_id --session-files 16 --base-dir ./sessions slow.log


Play all those sessions on host1, save results in ./results:


.. code-block:: perl

   pt-log-player --play ./sessions --base-dir ./results h=host1


Use pt-query-digest to summarize the results:


.. code-block:: perl

   pt-query-digest ./results/*



*****
RISKS
*****


The following section is included to inform users about the potential risks,
whether known or unknown, of using this tool.  The two main categories of risks
are those created by the nature of the tool (e.g. read-only tools vs. read-write
tools) and those created by bugs.

This tool is meant to load a server as much as possible, for stress-testing
purposes.  It is not designed to be used on production servers.

At the time of this release there is a bug which causes pt-log-player to
exceed max open files during "--split".

The authoritative source for updated information is always the online issue
tracking system.  Issues that affect this tool will be marked as such.  You can
see a list of such issues at the following URL:
`http://www.percona.com/bugs/pt-log-player <http://www.percona.com/bugs/pt-log-player>`_.

See also "BUGS" for more information on filing bugs and getting help.


***********
DESCRIPTION
***********


pt-log-player does two things: it splits MySQL query logs into session files
and it plays (executes) queries in session files on a MySQL server.  Only
session files can be played; slow logs cannot be played directly without
being split.

A session is a group of queries from the slow log that all share a common
attribute, usually Thread_id.  The common attribute is specified with
"--split".  Multiple sessions are saved into a single session file.
See "--session-files", "--max-sessions", "--base-file-name" and
"--base-dir".  These session files are played with "--play".

pt-log-player will "--play" session files in parallel using N number of
"--threads".  (They're not technically threads, but we call them that
anyway.)  Each thread will play all the sessions in its given session files.
The sessions are played as fast as possible--there are no delays--because the
goal is to stress-test and load-test the server.  So be careful using this
script on a production server!

Each "--play" thread writes its results to a separate file.  These result
files are in slow log format so they can be aggregated and summarized with
pt-query-digest.  See "OUTPUT".


******
OUTPUT
******


Both "--split" and "--play" have two outputs: status messages printed to
STDOUT to let you know what the script is doing, and session or result files
written to separate files saved in "--base-dir".  You can suppress all
output to STDOUT for each with "--quiet", or increase output with
"--verbose".

The session files written by "--split" are simple text files containing
queries grouped into sessions.  For example:


.. code-block:: perl

   -- START SESSION 10
 
   use foo
 
   SELECT col FROM foo_tbl


The format of these session files is important: each query must be a single
line separated by a single blank line.  And the "-- START SESSION" comment
tells pt-log-player where individual sessions begin and end so that "--play"
can correctly fake Thread_id in its result files.

The result files written by "--play" are in slow log format with a minimal
header: the only attributes printed are Thread_id, Query_time and Schema.


*******
OPTIONS
*******


Specify at least one of "--play", "--split" or "--split-random".

"--play" and "--split" are mutually exclusive.

This tool accepts additional command-line arguments.  Refer to the
"SYNOPSIS" and usage information for details.


--ask-pass
 
 group: Play
 
 Prompt for a password when connecting to MySQL.
 


--base-dir
 
 type: string; default: ./
 
 Base directory for "--split" session files and "--play" result file.
 


--base-file-name
 
 type: string; default: session
 
 Base file name for "--split" session files and "--play" result file.
 
 Each "--split" session file will be saved as <base-file-name>-N.txt, where
 N is a four digit, zero-padded session ID.  For example: session-0003.txt.
 
 Each "--play" result file will be saved as <base-file-name>-results-PID.txt,
 where PID is the process ID of the executing thread.
 
 All files are saved in "--base-dir".
 


--charset
 
 short form: -A; type: string; group: Play
 
 Default character set.  If the value is utf8, sets Perl's binmode on STDOUT to
 utf8, passes the mysql_enable_utf8 option to DBD::mysql, and runs SET NAMES UTF8
 after connecting to MySQL.  Any other value sets binmode on STDOUT without the
 utf8 layer, and runs SET NAMES after connecting to MySQL.
 


--config
 
 type: Array
 
 Read this comma-separated list of config files; if specified, this must be the
 first option on the command line.
 


--defaults-file
 
 short form: -F; type: string
 
 Only read mysql options from the given file.
 


--dry-run
 
 Print which processes play which session files then exit.
 


--filter
 
 type: string; group: Split
 
 Discard "--split" events for which this Perl code doesn't return true.
 
 This option only works with "--split".
 
 This option allows you to inject Perl code into the tool to affect how the
 tool runs.  Usually your code should examine \ ``$event``\  to decided whether
 or not to allow the event.  \ ``$event``\  is a hashref of attributes and values of
 the event being filtered.  Or, your code could add new attribute-value pairs
 to \ ``$event``\  for use by other options that accept event attributes as their
 value.  You can find an explanation of the structure of \ ``$event``\  at
 `http://code.google.com/p/maatkit/wiki/EventAttributes <http://code.google.com/p/maatkit/wiki/EventAttributes>`_.
 
 There are two ways to supply your code: on the command line or in a file.
 If you supply your code on the command line, it is injected into the following
 subroutine where \ ``$filter``\  is your code:
 
 
 .. code-block:: perl
 
     sub {
        MKDEBUG && _d('callback: filter');
        my( $event ) = shift;
        ( $filter ) && return $event;
     }
 
 
 Therefore you must ensure two things: first, that you correctly escape any
 special characters that need to be escaped on the command line for your
 shell, and two, that your code is syntactically valid when injected into
 the subroutine above.
 
 Here's an example filter supplied on the command line that discards
 events that are not SELECT statements:
 
 
 .. code-block:: perl
 
    --filter '$event->{arg} =~ m/^select/i'
 
 
 The second way to supply your code is in a file.  If your code is too complex
 to be expressed on the command line that results in valid syntax in the
 subroutine above, then you need to put the code in a file and give the file
 name as the value to "--filter".  The file should not contain a shebang
 (\ ``#!/usr/bin/perl``\ ) line.  The entire contents of the file is injected into
 the following subroutine:
 
 
 .. code-block:: perl
 
     sub {
        MKDEBUG && _d('callback: filter');
        my( $event ) = shift;
        $filter && return $event;
     }
 
 
 That subroutine is almost identical to the one above except your code is
 not wrapped in parentheses.  This allows you to write multi-line code like:
 
 
 .. code-block:: perl
 
     my $event_ok;
     if (...) {
        $event_ok = 1;
     }
     else {
        $event_ok = 0;
     }
     $event_ok
 
 
 Notice that the last line is not syntactically valid by itself, but it
 becomes syntactically valid when injected into the subroutine because it
 becomes:
 
 
 .. code-block:: perl
 
     $event_ok && return $event;
 
 
 If your code doesn't compile, the tool will die with an error.  Even if your
 code compiles, it may crash to tool during runtime if, for example, it tries
 a pattern match an undefined value.  No safeguards of any kind of provided so
 code carefully!
 


--help
 
 Show help and exit.
 


--host
 
 short form: -h; type: string; group: Play
 
 Connect to host.
 


--iterations
 
 type: int; default: 1; group: Play
 
 How many times each thread should play all its session files.
 


--max-sessions
 
 type: int; default: 5000000; group: Split
 
 Maximum number of sessions to "--split".
 
 By default, \ ``pt-log-player``\  tries to split every session from the log file.
 For huge logs, however, this can result in millions of sessions.  This
 option causes only the first N number of sessions to be saved.  All sessions
 after this number are ignored, but sessions split before this number will
 continue to have their queries split even if those queries appear near the end
 of the log and after this number has been reached.
 


--only-select
 
 group: Play
 
 Play only SELECT and USE queries; ignore all others.
 


--password
 
 short form: -p; type: string; group: Play
 
 Password to use when connecting.
 


--pid
 
 type: string
 
 Create the given PID file.  The file contains the process ID of the script.
 The PID file is removed when the script exits.  Before starting, the script
 checks if the PID file already exists.  If it does not, then the script creates
 and writes its own PID to it.  If it does, then the script checks the following:
 if the file contains a PID and a process is running with that PID, then
 the script dies; or, if there is no process running with that PID, then the
 script overwrites the file with its own PID and starts; else, if the file
 contains no PID, then the script dies.
 


--play
 
 type: string; group: Play
 
 Play (execute) session files created by "--split".
 
 The argument to play must be a comma-separated list of session files
 created by "--split" or a directory.  If the argument is a directory,
 ALL files in that directory will be played.
 


--port
 
 short form: -P; type: int; group: Play
 
 Port number to use for connection.
 


--print
 
 group: Play
 
 Print queries instead of playing them; requires "--play".
 
 You must also specify "--play" with "--print".  Although the queries
 will not be executed, "--play" is required to specify which session files to
 read.
 


--quiet
 
 short form: -q
 
 Do not print anything; disables "--verbose".
 


--[no]results
 
 default: yes
 
 Print "--play" results to files in "--base-dir".
 


--session-files
 
 type: int; default: 8; group: Split
 
 Number of session files to create with "--split".
 
 The number of session files should either be equal to the number of
 "--threads" you intend to "--play" or be an even multiple of
 "--threads".  This number is important for maximum performance because it:
 
 
 .. code-block:: perl
 
    * allows each thread to have roughly the same amount of sessions to play
    * avoids having to open/close many session files
    * avoids disk IO overhead by doing large sequential reads
 
 
 You may want to increase this number beyond "--threads" if each session
 file becomes too large.  For example, splitting a 20G log into 8 sessions
 files may yield roughly eight 2G session files.
 
 See also "--max-sessions".
 


--set-vars
 
 type: string; group: Play; default: wait_timeout=10000
 
 Set these MySQL variables.  Immediately after connecting to MySQL, this string
 will be appended to SET and executed.
 


--socket
 
 short form: -S; type: string; group: Play
 
 Socket file to use for connection.
 


--split
 
 type: string; group: Split
 
 Split log by given attribute to create session files.
 
 Valid attributes are any which appear in the log: Thread_id, Schema,
 etc.
 


--split-random
 
 group: Split
 
 Split log without an attribute, write queries round-robin to session files.
 
 This option, if specified, overrides "--split" and causes the log to be
 split query-by-query, writing each query to the next session file in round-robin
 style.  If you don't care about "sessions" and just want to split a lot into
 N many session files and the relation or order of the queries does not matter,
 then use this option.
 


--threads
 
 type: int; default: 2; group: Play
 
 Number of threads used to play sessions concurrently.
 
 Specifies the number of parallel processes to run.  The default is 2.  On
 GNU/Linux machines, the default is the number of times 'processor' appears in
 \ */proc/cpuinfo*\ .  On Windows, the default is read from the environment.
 In any case, the default is at least 2, even when there's only a single
 processor.
 
 See also "--session-files".
 


--type
 
 type: string; group: Split
 
 The type of log to "--split" (default slowlog).  The permitted types are
 
 
 binlog
  
  Split the output of running \ ``mysqlbinlog``\  against a binary log file.
  Currently, splitting binary logs does not always work well depending
  on what the binary logs contain.  Be sure to check the session files
  after splitting to ensure proper "OUTPUT".
  
  If the binary log contains row-based replication data, you need to run
  \ ``mysqlbinlog``\  with options \ ``--base64-output=decode-rows --verbose``\ ,
  else invalid statements will be written to the session files.
  
 
 
 genlog
  
  Split a general log file.
  
 
 
 slowlog
  
  Split a log file in any variation of MySQL slow-log format.
  
 
 


--user
 
 short form: -u; type: string; group: Play
 
 User for login if not current user.
 


--verbose
 
 short form: -v; cumulative: yes; default: 0
 
 Increase verbosity; can be specified multiple times.
 
 This option is disabled by "--quiet".
 


--version
 
 Show version and exit.
 


--[no]warnings
 
 default: no; group: Play
 
 Print warnings about SQL errors such as invalid queries to STDERR.
 



***********
DSN OPTIONS
***********


These DSN options are used to create a DSN.  Each option is given like
\ ``option=value``\ .  The options are case-sensitive, so P and p are not the
same option.  There cannot be whitespace before or after the \ ``=``\  and
if the value contains whitespace it must be quoted.  DSN options are
comma-separated.  See the percona-toolkit manpage for full details.


\* A
 
 dsn: charset; copy: yes
 
 Default character set.
 


\* D
 
 dsn: database; copy: yes
 
 Default database.
 


\* F
 
 dsn: mysql_read_default_file; copy: yes
 
 Only read default options from the given file
 


\* h
 
 dsn: host; copy: yes
 
 Connect to host.
 


\* p
 
 dsn: password; copy: yes
 
 Password to use when connecting.
 


\* P
 
 dsn: port; copy: yes
 
 Port number to use for connection.
 


\* S
 
 dsn: mysql_socket; copy: yes
 
 Socket file to use for connection.
 


\* u
 
 dsn: user; copy: yes
 
 User for login if not current user.
 



***********
ENVIRONMENT
***********


The environment variable \ ``PTDEBUG``\  enables verbose debugging output to STDERR.
To enable debugging and capture all output to a file, run the tool like:


.. code-block:: perl

    PTDEBUG=1 pt-log-player ... > FILE 2>&1


Be careful: debugging output is voluminous and can generate several megabytes
of output.


*******************
SYSTEM REQUIREMENTS
*******************


You need Perl, DBI, DBD::mysql, and some core packages that ought to be
installed in any reasonably new version of Perl.


****
BUGS
****


For a list of known bugs, see `http://www.percona.com/bugs/pt-log-player <http://www.percona.com/bugs/pt-log-player>`_.

Please report bugs at `https://bugs.launchpad.net/percona-toolkit <https://bugs.launchpad.net/percona-toolkit>`_.
Include the following information in your bug report:


\* Complete command-line used to run the tool



\* Tool "--version"



\* MySQL version of all servers involved



\* Output from the tool including STDERR



\* Input files (log/dump/config files, etc.)



If possible, include debugging output by running the tool with \ ``PTDEBUG``\ ;
see "ENVIRONMENT".


***********
DOWNLOADING
***********


Visit `http://www.percona.com/software/percona-toolkit/ <http://www.percona.com/software/percona-toolkit/>`_ to download the
latest release of Percona Toolkit.  Or, get the latest release from the
command line:


.. code-block:: perl

    wget percona.com/get/percona-toolkit.tar.gz
 
    wget percona.com/get/percona-toolkit.rpm
 
    wget percona.com/get/percona-toolkit.deb


You can also get individual tools from the latest release:


.. code-block:: perl

    wget percona.com/get/TOOL


Replace \ ``TOOL``\  with the name of any tool.


*******
AUTHORS
*******


Daniel Nichter


*********************
ABOUT PERCONA TOOLKIT
*********************


This tool is part of Percona Toolkit, a collection of advanced command-line
tools developed by Percona for MySQL support and consulting.  Percona Toolkit
was forked from two projects in June, 2011: Maatkit and Aspersa.  Those
projects were created by Baron Schwartz and developed primarily by him and
Daniel Nichter, both of whom are employed by Percona.  Visit
`http://www.percona.com/software/ <http://www.percona.com/software/>`_ for more software developed by Percona.


********************************
COPYRIGHT, LICENSE, AND WARRANTY
********************************


This program is copyright 2008-2011 Percona Inc.
Feedback and improvements are welcome.

THIS PROGRAM IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

This program is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation, version 2; OR the Perl Artistic License.  On UNIX and similar
systems, you can issue \`man perlgpl' or \`man perlartistic' to read these
licenses.

You should have received a copy of the GNU General Public License along with
this program; if not, write to the Free Software Foundation, Inc., 59 Temple
Place, Suite 330, Boston, MA  02111-1307  USA.


*******
VERSION
*******


pt-log-player 1.0.1