percona/percona-toolkit
1
0
Fork 0
mirror of https://github.com/percona/percona-toolkit.git synced 2025-09-21 11:30:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update documentation

Browse Source
This commit is contained in:
baron@percona.com
2012-03-30 20:47:56 -04:00
parent a18c08f16a
commit dacc36994e
1 changed files with 101 additions and 198 deletions
Download Patch File Download Diff File Expand all files Collapse all files

299
bin/pt-table-usage
View File

@@ -5864,12 +5864,12 @@ sub _d {
# ########################################################################### # ###########################################################################
# ########################################################################### # ###########################################################################
# MysqldumpParser package 7500 # MysqldumpParser package
# This package is a copy without comments from the original. The original # This package is a copy without comments from the original. The original
# with comments and its test file can be found in the SVN repository at, # with comments and its test file can be found in the Bazaar repository at,
# trunk/common/MysqldumpParser.pm # lib/MysqldumpParser.pm
# trunk/common/t/MysqldumpParser.t # t/lib/MysqldumpParser.t
# See http://code.google.com/p/maatkit/wiki/Developers for more information. # See https://launchpad.net/percona-toolkit for more information.
# ########################################################################### # ###########################################################################
package MysqldumpParser; package MysqldumpParser;
@@ -5968,12 +5968,12 @@ sub _d {
# ########################################################################### # ###########################################################################
# ########################################################################### # ###########################################################################
# SchemaQualifier package 7499 # SchemaQualifier package
# This package is a copy without comments from the original. The original # This package is a copy without comments from the original. The original
# with comments and its test file can be found in the SVN repository at, # with comments and its test file can be found in the SVN repository at,
# trunk/common/SchemaQualifier.pm # lib/SchemaQualifier.pm
# trunk/common/t/SchemaQualifier.t # t/lib/SchemaQualifier.t
# See http://code.google.com/p/maatkit/wiki/Developers for more information. # See https://launchpad.net/percona-toolkit for more information.
# ########################################################################### # ###########################################################################
package SchemaQualifier; package SchemaQualifier;
@@ -6680,24 +6680,22 @@ if ( !caller ) { exit main(@ARGV); }
=head1 NAME =head1 NAME
pt-table-usage - Read queries from a log and analyze how they use tables. pt-table-usage - Analyze how queries use tables.
=head1 SYNOPSIS =head1 SYNOPSIS
Usage: pt-table-usage [OPTIONS] [FILES] Usage: pt-table-usage [OPTIONS] [FILES]
pt-table-usage reads queries from slow query logs and analyzes how they use pt-table-usage reads queries from a log and analyzes how they use tables. If no
tables. If no FILE is specified, STDIN is read. Table usage for every query FILE is specified, it reads STDIN. It prints a report for each query.
is printed to STDOUT.
=head1 RISKS =head1 RISKS
pt-table-use is very low risk because it only reads and examines queries from pt-table-use is very low risk. By default, it simply reads queries from a log.
a log and executes C<EXPLAIN EXTENDED> if the L<"--explain-extended"> option It executes C<EXPLAIN EXTENDED> if you specify the L<"--explain-extended">
is specified. option.
At the time of this release, there are no known bugs that could cause serious At the time of this release, we know of no bugs that could harm users.
harm to users.
The authoritative source for updated information is always the online issue The authoritative source for updated information is always the online issue
tracking system. Issues that affect this tool will be marked as such. You can tracking system. Issues that affect this tool will be marked as such. You can
@@ -6708,40 +6706,32 @@ See also L<"BUGS"> for more information on filing bugs and getting help.
=head1 DESCRIPTION =head1 DESCRIPTION
pt-table-usage reads queries from slow query logs and analyzes how they use pt-table-usage reads queries from a log and analyzes how they use tables. The
tables. Table usage indicates more than just which tables are read from or log should be in MySQL's slow query log format.
written to by the query, it also indicates data flow: data in and data out.
Data flow is determined by the contexts in which tables are used by the query.
A single table can be used in several different contexts in the same query.
The reported table usage for each query lists every context for every table.
This CONTEXT-TABLE list tells how and where data flows, i.e. the query's table
usage. The L<"OUTPUT"> section lists the possible contexts and describes how
to read a table usage report.
Since this tool analyzes table usage, it's important that queries use Table usage is more than simply an indication of which tables the query reads or
table-qualified columns. If a query uses only one table, then all columns writes. It also indicates data flow: data in and data out. The tool determines
must be from that table and there's no problem. But if a query uses the data flow by the contexts in which tables appear. A single query can use a
multiple tables and the columns are not table-qualified, then that creates a table in several different contexts simultaneously. The tool's output lists
problem that can only be solved by knowing the query's database and specifying every context for every table. This CONTEXT-TABLE list indicates how data flows
L<"--explain-extended">. If the slow log does not specify the database between tables. The L<"OUTPUT"> section lists the possible contexts and
used by the query, then you can specify a default database with L<"--database">. describes how to read a table usage report.
There is no other way to know or guess the database, so the query will be
skipped. Secondly, if the database is known, then specifying
L<"--explain-extended"> causes pt-table-usage to do C<EXPLAIN EXTENDED ...>
C<SHOW WARNINGS> to get the fully qualified query as reported by MySQL
(i.e. all identifiers are fully database- and/or table-qualified). For
best results, you should specify L<"--explain-extended"> and
L<"--database"> if you know that all queries use the same database.
Each query is identified in the output by either an MD5 hex checksum The tool analyzes data flow down to the level of individual columns, so it is
of the query's fingerprint or the query's value for the specified helpful if columns are identified unambiguously in the query. If a query uses
L<"--id-attribute">. The query ID is for parsing and storing the table only one table, then all columns must be from that table, and there's no
usage reports in a table that is keyed on the query ID. See L<"OUTPUT"> difficulty. But if a query uses multiple tables and the column names are not
for more information. table-qualified, then it is necessary to use C<EXPLAIN EXTENDED>, followed by
C<SHOW WARNINGS>, to determine to which tables the columns belong.
If the tool does not know the query's default database, which can occur when the
database is not printed in the log, then C<EXPLAIN EXTENDED> can fail. In this
case, you can specify a default database with L<"--database">. You can also use
the L<"--create-table-definitions"> option to help resolve ambiguities.
=head1 OUTPUT =head1 OUTPUT
The table usage report that is printed for each query looks similar to the The tool prints a usage report for each table in every query, similar to the
following: following:
Query_id: 0x1CD27577D202A339.1 Query_id: 0x1CD27577D202A339.1
@@ -6758,45 +6748,43 @@ following:
JOIN t2 JOIN t2
WHERE t1 WHERE t1
Usage reports are separated by blank lines. The first line is always the The first line contains the query ID, which by default is the same as those
query ID: a unique ID that can be used to parse the output and store the shown in pt-query-digest reports. It is an MD5 checksum of the query's
usage reports in a table keyed on this ID. The query ID has two parts "fingerprint," which is what remains after removing literals, collapsing white
separated by a period: the query ID and the target table number. space, and a variety of other transformations. The query ID has two parts
separated by a period: the query ID and the table number. If you wish to use a
different value to identify the query, you can specify the L<"--id-attribute">
option.
If L<"--id-attribute"> is not specified, then query IDs are automatically The previous example shows two paragraphs for a single query, not two queries.
created by making an MD5 hex checksum of the query's fingerprint Note that the query ID is identical for the two, but the table number differs.
(as shown above, e.g. C<0x1CD27577D202A339>); otherwise, the query ID is the The table number increments by 1 for each table that the query updates. Only
query's value for the given attribute. multi-table UPDATE queries can update multiple tables with a single query, so
the table number is 1 for all other types of queries. (The tool does not
The target table number starts at 1 and increments by 1 for each table that support multi-table DELETE queries.) The example output above is from this
the query affects. Only multi-table UPDATE queries can affect query:
multiple tables with a single query, so this number is 1 for all other types
of queries. (Multi-table DELETE queries are not supported.)
The example output above is from this query:
UPDATE t1 AS a JOIN t2 AS b USING (id) UPDATE t1 AS a JOIN t2 AS b USING (id)
SET a.foo="bar", b.foo="bat" SET a.foo="bar", b.foo="bat"
WHERE a.id=1; WHERE a.id=1;
The C<SET> clause indicates that two tables are updated: C<a> aliased as C<t1>, The C<SET> clause indicates that the query updates two tables: C<a> aliased as
and C<b> aliased as C<t2>. So two usage reports are printed, one for each C<t1>, and C<b> aliased as C<t2>.
table, and this is indicated in the output by their common query ID but
incrementing target table number.
After the first line is a variable number of CONTEXT-TABLE lines. Possible After the first line, the tool prints a variable number of CONTEXT-TABLE lines.
contexts are: Possible contexts are as follows:
=over =over
=item * SELECT =item * SELECT
SELECT means that data is taken out of the table for one of two reasons: SELECT means that the query retrieves data from the table for one of two
to be returned to the user as part of a result set, or to be put into another reasons. The first is to be returned to the user as part of a result set. Only
table as part of an INSERT or UPDATE. In the first case, since only SELECT SELECT queries return result sets, so the report always shows a SELECT context
queries return result sets, a SELECT context is always listed for SELECT for SELECT queries.
queries. In the second case, data from one table is used to insert or
update rows in another table. For example, the UPDATE query in the example The second case is when data flows to another table as part of an INSERT or
above has the usage: UPDATE. For example, the UPDATE query in the example above has the usage:
SELECT DUAL SELECT DUAL
@@ -6804,9 +6792,9 @@ This refers to:
SET a.foo="bar", b.foo="bat" SET a.foo="bar", b.foo="bat"
DUAL is used for any values that does not originate in a table, in this case the The tool uses DUAL for any values that do not originate in a table, in this case
literal values "bar" and "bat". If that C<SET> clause were C<SET a.foo=b.foo> the literal values "bar" and "bat". If that C<SET> clause were C<SET
instead, then the complete usage would be: a.foo=b.foo> instead, then the complete usage would be:
Query_id: 0x1CD27577D202A339.1 Query_id: 0x1CD27577D202A339.1
UPDATE t1 UPDATE t1
@@ -6820,20 +6808,15 @@ INSERT, indicates where the UPDATE or INSERT retrieves its data. The example
immediately above reflects an UPDATE query that updates rows in table C<t1> immediately above reflects an UPDATE query that updates rows in table C<t1>
with data from table C<t2>. with data from table C<t2>.
=item * Any other query type =item * Any other verb
Any other query type, such as INSERT, UPDATE, DELETE, etc. may be a context. Any other verb, such as INSERT, UPDATE, DELETE, etc. may be a context. These
All these types indicate that the table is written or altered in some way. verbs indicate that the query modifies data in some way. If a SELECT context
If a SELECT context follows one of these types, then data is read from the follows one of these verbs, then the query reads data from the SELECT table and
SELECT table and written to this table. This happens, for example, with writes it to this table. This happens, for example, with INSERT..SELECT or
INSERT..SELECT or UPDATE queries that set column values using values from UPDATE queries that use values from tables instead of constant values.
tables instead of constant values.
These query types are not supported: These query types are not supported: SET, LOAD, and multi-table DELETE.
SET
LOAD
multi-table DELETE
=item * JOIN =item * JOIN
@@ -6853,14 +6836,14 @@ Results in:
WHERE t1 WHERE t1
WHERE t2 WHERE t2
Only unique tables are listed; that is why table C<t1> is listed only once. The tool lists only distinct tables; that is why table C<t1> is listed only
once.
=item * TLIST =item * TLIST
The TLIST context lists tables that are accessed by the query but do not The TLIST context lists tables that the query accesses, but which do not appear
appear in any other context. These tables are usually an implicit in any other context. These tables are usually an implicit cartesian join. For
full cartesian join, so they should be avoided. For example, the query example, the query C<SELECT * FROM t1, t2> results in:
C<SELECT * FROM t1, t2> results in:
Query_id: 0xBDDEB6EDA41897A8.1 Query_id: 0xBDDEB6EDA41897A8.1
SELECT t1 SELECT t1
@@ -6871,7 +6854,7 @@ C<SELECT * FROM t1, t2> results in:
First of all, there are two SELECT contexts, because C<SELECT *> selects First of all, there are two SELECT contexts, because C<SELECT *> selects
rows from all tables; C<t1> and C<t2> in this case. Secondly, the tables rows from all tables; C<t1> and C<t2> in this case. Secondly, the tables
are implicitly joined, but without any kind of join condition, which results are implicitly joined, but without any kind of join condition, which results
in a full cartesian join as indicated by the TLIST context for each. in a cartesian join as indicated by the TLIST context for each.
=back =back
@@ -6911,24 +6894,23 @@ first option on the command line.
type: string; default: DUAL type: string; default: DUAL
Value to print for constant data. Constant data means all data not Table to print as the source for constant data (literals). This is any data not
from tables (or subqueries since subqueries are not supported). For example, retrieved from tables (or subqueries, because subqueries are not supported).
real constant values like strings ("foo") and numbers (42), and data from This includes literal values such as strings ("foo") and numbers (42), or
functions like C<NOW()>. For example, in the query functions such as C<NOW()>. For example, in the query C<INSERT INTO t (c)
C<INSERT INTO t (c) VALUES ('a')>, the string 'a' is constant data, so the VALUES ('a')>, the string 'a' is constant data, so the table usage report is:
table usage report is:
INSERT t INSERT t
SELECT DUAL SELECT DUAL
The first line indicates that data is inserted into table C<t> and the second The first line indicates that the query inserts data into table C<t>, and the
line indicates that that data comes from some constant value. second line indicates that the inserted data comes from some constant value.
=item --[no]continue-on-error =item --[no]continue-on-error
default: yes default: yes
Continue parsing even if there is an error. Continue to work even if there is an error.
=item --create-table-definitions =item --create-table-definitions
@@ -6939,9 +6921,9 @@ If you cannot use L<"--explain-extended"> to fully qualify table and column
names, you can save the output of C<mysqldump --no-data> to one or more files names, you can save the output of C<mysqldump --no-data> to one or more files
and specify those files with this option. The tool will parse all and specify those files with this option. The tool will parse all
C<CREATE TABLE> definitions from the files and use this information to C<CREATE TABLE> definitions from the files and use this information to
qualify table and column names. If a column name is used in multiple tables, qualify table and column names. If a column name appears in multiple tables,
or table name is used in multiple databases, these duplicates cannot be or a table name appears in multiple databases, the ambiguities cannot be
qualified. resolved.
=item --daemonize =item --daemonize
@@ -6964,7 +6946,8 @@ Only read mysql options from the given file. You must give an absolute pathname
type: DSN type: DSN
EXPLAIN EXTENDED queries on this host to fully qualify table and column names. A server to execute EXPLAIN EXTENDED queries. This may be necessary to resolve
ambiguous (unqualified) column and table names.
=item --filter =item --filter
@@ -6972,89 +6955,13 @@ type: string
Discard events for which this Perl code doesn't return true. Discard events for which this Perl code doesn't return true.
This option is a string of Perl code or a file containing Perl code that gets This option is a string of Perl code or a file containing Perl code that is
compiled into a subroutine with one argument: $event. This is a hashref. compiled into a subroutine with one argument: $event. If the given value is a
If the given value is a readable file, then pt-table-usage reads the entire readable file, then pt-table-usage reads the entire file and uses its contents
file and uses its contents as the code. The file should not contain as the code.
a shebang (#!/usr/bin/perl) line.
If the code returns true, the chain of callbacks continues; otherwise it ends.
The code is the last statement in the subroutine other than C<return $event>.
The subroutine template is:
sub { $event = shift; filter && return $event; }
Filters given on the command line are wrapped inside parentheses like like
C<( filter )>. For complex, multi-line filters, you must put the code inside
a file so it will not be wrapped inside parentheses. Either way, the filter
must produce syntactically valid code given the template. For example, an
if-else branch given on the command line would not be valid:
--filter 'if () { } else { }' # WRONG
Since it's given on the command line, the if-else branch would be wrapped inside
parentheses which is not syntactically valid. So to accomplish something more
complex like this would require putting the code in a file, for example
filter.txt:
my $event_ok; if (...) { $event_ok=1; } else { $event_ok=0; } $event_ok
Then specify C<--filter filter.txt> to read the code from filter.txt.
If the filter code won't compile, pt-table-usage will die with an error.
If the filter code does compile, an error may still occur at runtime if the
code tries to do something wrong (like pattern match an undefined value).
pt-table-usage does not provide any safeguards so code carefully!
An example filter that discards everything but SELECT statements:
--filter '$event->{arg} =~ m/^select/i'
This is compiled into a subroutine like the following:
sub { $event = shift; ( $event->{arg} =~ m/^select/i ) && return $event; }
It is permissible for the code to have side effects (to alter C<$event>).
You can find an explanation of the structure of $event at
L<http://code.google.com/p/maatkit/wiki/EventAttributes>.
Here are more examples of filter code:
=over
=item Host/IP matches domain.com
--filter '($event->{host} || $event->{ip} || "") =~ m/domain.com/'
Sometimes MySQL logs the host where the IP is expected. Therefore, we
check both.
=item User matches john
--filter '($event->{user} || "") =~ m/john/'
=item More than 1 warning
--filter '($event->{Warning_count} || 0) > 1'
=item Query does full table scan or full join
--filter '(($event->{Full_scan} || "") eq "Yes") || (($event->{Full_join} || "") eq "Yes")'
=item Query was not served from query cache
--filter '($event->{QC_Hit} || "") eq "No"'
=item Query is 1 MB or larger
--filter '$event->{bytes} >= 1_048_576'
=back
Since L<"--filter"> allows you to alter C<$event>, you can use it to do other
things, like create new attributes.
Filters are implemented in the same fashion as in the pt-query-digest tool, so
please refer to its documentation for more information.
=item --help =item --help
@@ -7070,9 +6977,8 @@ Connect to host.
type: string type: string
Identify each event using this attribute. If not ID attribute is given, then Identify each event using this attribute. The default is to use a query ID,
events are identified with the query's checksum: an MD5 hex checksum of the which is an MD5 checksum of the query's fingerprint.
query's fingerprint.
=item --log =item --log
@@ -7115,10 +7021,7 @@ number of iterations.
type: string type: string
Analyze only this given query. If you want to analyze the table usage of Analyze the specified query instead of reading a log file.
one simple query by providing on the command line instead of reading it
from a slow log file, then specify that query with this option. The default
L<"--id-attribute"> will be used which is the query's checksum.
=item --read-timeout =item --read-timeout
@@ -7127,7 +7030,7 @@ type: time; default: 0
Wait this long for an event from the input; 0 to wait forever. Wait this long for an event from the input; 0 to wait forever.
This option sets the maximum time to wait for an event from the input. If an This option sets the maximum time to wait for an event from the input. If an
event is not received after the specified time, the script stops reading the event is not received after the specified time, the tool stops reading the
input and prints its reports. input and prints its reports.
This option requires the Perl POSIX module. This option requires the Perl POSIX module.