percona/percona-toolkit
1
0
Fork 0
mirror of https://github.com/percona/percona-toolkit.git synced 2025-09-01 18:25:59 +00:00
Code Issues Packages Projects Releases Wiki Activity

Make documentation for Go tools to use symlinks to README.rst

Browse Source 
- Currently, documentation for Go tools is duplicated in the user manual
  and README files. This PR makes README files single source of truth
and creates a symlincs inside docs directory.
- Fixed link to the forum in
  config/sphinx-build/_static/_templates/theme/relbar2.html (previously
mentioned PSMDB forum instead of Percona Toolkit's)
- Updated 3.6.0 release date in docs/percona-toolkit.pod
- Updated documentation for option --version in docs/percona-toolkit.pod
This commit is contained in:
Sveta Smirnova
2024-09-04 00:33:24 +03:00
parent e78da8d151
commit bf3d2ffab4
19 changed files with 562 additions and 1695 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
config/sphinx-build/_static/_templates/theme/relbar2.html
View File

@@ -2,7 +2,7 @@
<div class="md-content">
<article class="md-content__inner md-typeset" role="main">
<h4>Get Expert Help </h4>
<p>If you need assistance, visit the <a class="reference external" href="https://forums.percona.com/c/mongodb/percona-server-for-mongodb/17?utm_campaign=Doc%20pages" target="_blank">community forum</a> for comprehensive and free database knowledge.</p>
<p>If you need assistance, visit the <a class="reference external" href="https://forums.percona.com/c/polyglot-projects/percona-toolkit/13?utm_campaign=Doc%20pages" target="_blank">community forum</a> for comprehensive and free database knowledge.</p>
<p>Contact our <a class="reference external" href="https://www.percona.com/about/contact">Percona Database Experts</a> for professional support and services.</p>
<hr>
{# Add the last updated timestamp from git commits using gitstamp extension #}

8
docs/percona-toolkit.pod
View File

@@ -406,7 +406,7 @@ databases while running.
=item F
Defaults file for the MySQL client library (the C client library used by
DBD::mysql, I<not Percona Toolkit itself>). All tools all read the
DBD::mysql, I<not Percona Toolkit itself>). All tools read the
C<[client]> section within the defaults file. If you omit this, the standard
defaults files will be read in the usual order. "Standard" varies from system
to system, because the filenames to read are compiled into the client library.
@@ -441,7 +441,7 @@ lead to weird situations, like the server allowing LOCAL INFILE, but
the client throwing exceptions if it's used.
However, as long as the server allows LOAD DATA, clients can easily
re-enable it; see L<https://dev.mysql.com/doc/refman/5.0/en/load-data-local.html>
re-enable it; see L<https://dev.mysql.com/doc/refman/en/load-data-local-security.html>
and L<http://search.cpan.org/~capttofu/DBD-mysql/lib/DBD/mysql.pm>.
This option does exactly that.
@@ -539,7 +539,7 @@ Include the following information in your bug report:
=item * Tool C<--version>
=item * MySQL version of all servers involved
=item * MySQL, MongoDB, or PostgreSQL version of all servers involved
=item * Output from the tool including STDERR
@@ -603,6 +603,6 @@ Place, Suite 330, Boston, MA 02111-1307 USA.
=head1 VERSION
Percona Toolkit v3.6.0 released 2024-05-25
Percona Toolkit v3.6.0 released 2024-06-12
=cut

246
docs/pt-galera-log-explainer.rst
View File

@@ -1,246 +0,0 @@
.. _pt-galera-log-explainer:
==================================
:program:`pt-galera-log-explainer`
==================================
Filter, aggregate and summarize multiple galera logs together.
This is a toolbox to help navigating Galera logs.
Usage
=====
.. code-block:: bash
pt-galera-log-explainer [--since=] [--until=] [-vv] [--merge-by-directory] [--pxc-operator] <command> <paths ...>
Commands available
==================
list
~~~~
.. code-block:: bash
pt-galera-log-explainer [flags] list { --all | [--states] [--views] [--events] [--sst] [--applicative] } <paths ...>
List key events in chronological order from any number of nodes (sst, view changes, general errors, maintenance operations)
It will aggregates logs together by identifying them using node names, IPs and internal Galera identifiers.
It can be from a single node:
.. code-block:: bash
pt-galera-log-explainer list --all --since 2023-01-05T03:24:26.000000Z /var/log/mysql/*.log
or from multiple nodes.
.. code-block:: bash
pt-galera-log-explainer list --all *.log
You can filter by type of events
.. code-block:: bash
pt-galera-log-explainer list --sst --views *.log
whois
~~~~~
Find out information about nodes, using any type of information
.. code-block:: bash
pt-galera-log-explainer [flags] whois [--json] [--type { nodename | ip | uuid | auto }] <information to search> <paths ...>
.. code-block:: bash
pt-galera-log-explainer whois '218469b2' mysql.log
pt-galera-log-explainer whois '172.17.0.3' mysql.log
pt-galera-log-explainer whois 'galera-node2' mysql.log
conflicts
~~~~~~~~~
List every replication failure votes (Galera 4)
.. code-block:: bash
pt-galera-log-explainer conflicts [--json|--yaml] *.log
ctx
~~~
Get the tool crafted context for a single log.
It will contain everything the tool extracted from the log file: version, sst information, known uuid-ip-nodename mappings, ...
.. code-block:: bash
pt-galera-log-explainer ctx mysql.log
regex-list
~~~~~~~~~~
Will print every implemented regexes:
* regex: the regex that will be used against the log files
* internalRegex: the golang regex that will be used to extract piece of information
* type: the regex group it belong to
* verbosity: the required level of verbosity to which it will be printed
.. code-block:: bash
pt-galera-log-explainer regex-list
Available flags
~~~~~~~~~~~~~~~
``-h``, ``--help``
Show help and exit.
``--no-color``
Remove every color special characters
``--since``
Only list events after this date. It will affect the regex applied to the logs.
Format: 2023-01-23T03:53:40Z (RFC3339)
``--until``
Only list events before this date. This is only implemented in the tool loop, it does not alter regexes.
Format: 2023-01-23T03:53:40Z (RFC3339)
``--merge-by-directory``
Instead of relying on extracted information, logs will be merged by their base directory
It is useful when logs are very sparse and already organized by nodes.
``--skip-merge``
Disable the ability to merge log files together. Can be used when every nodes have the same ``wsrep_node_name``
``-v``, ``--verbosity``
``-v``: display in the timeline every mysql info the tool used
``-vv``: internal tool debug
``--pxc-operator``
Analyze logs from Percona PXC operator. Operator logs should be automatically detected (see ``--skip-operator-detection``).
It will prevent logs from being merged together, add operator specific regexes, and fine-tune regexes for logs taken from ``pt-k8s-debug-collector``.
Off by default because it negatively impacts performance for non-k8s setups.
``--skip-operator-detection``
Disable automatic detection of PXC operator logs. When detected, a message will be shown.
Detection is done using a prefix regex.
``--exclude-regexes``
Remove regexes from analysis. Use ``pt-galera-log-explainer regex-list | jq .`` to have the list
``--grep-cmd``
grep v3 binary command path. For Darwin systems, it could need to be set to ``ggrep``
Default: ``grep``
``--version``
Show version and exit.
``--custom-regexes``
Add custom regexes, printed in magenta. Format: (golang regex string)=[optional static message to display].
If the static message is left empty, the captured string will be printed instead. Custom regexes are separated using semi-colon.
Example: ``--custom-regexes="Page cleaner took [0-9]*ms to flush [0-9]* pages=;doesn't recommend.*pxc_strict_mode=unsafe query used"``
Example outputs
===============
.. code-block:: bash
$ pt-galera-log-explainer list --all --no-color --since=2023-03-12T19:41:28.493046Z --until=2023-03-12T19:44:59.855491Z tests/logs/upgrade/*
identifier 172.17.0.2 node2 tests/logs/upgrade/node3.log
current path tests/logs/upgrade/node1.log tests/logs/upgrade/node2.log tests/logs/upgrade/node3.log
last known ip 172.17.0.2
last known name node2
mysql version 8.0.28
2023-03-12T19:41:28.493046Z starting(8.0.28) | |
2023-03-12T19:41:28.500789Z started(cluster) | |
2023-03-12T19:43:17.630191Z | node3 joined |
2023-03-12T19:43:17.630208Z node3 joined | |
2023-03-12T19:43:17.630221Z node2 joined | |
2023-03-12T19:43:17.630243Z | node1 joined |
2023-03-12T19:43:17.634138Z | | node2 joined
2023-03-12T19:43:17.634229Z | | node1 joined
2023-03-12T19:43:17.643210Z | PRIMARY(n=3) |
2023-03-12T19:43:17.648163Z | | PRIMARY(n=3)
2023-03-12T19:43:18.130088Z CLOSED -> OPEN | |
2023-03-12T19:43:18.130230Z PRIMARY(n=3) | |
2023-03-12T19:43:18.130916Z OPEN -> PRIMARY | |
2023-03-12T19:43:18.904410Z will receive IST(seqno:178226792) | |
2023-03-12T19:43:18.913328Z | | node1 cannot find donor
2023-03-12T19:43:18.913429Z node1 cannot find donor | |
2023-03-12T19:43:18.913565Z | node1 cannot find donor |
2023-03-12T19:43:19.914122Z | | node1 cannot find donor
2023-03-12T19:43:19.914259Z node1 cannot find donor | |
2023-03-12T19:43:19.914362Z | node1 cannot find donor |
2023-03-12T19:43:20.914957Z | | (repeated x97)node1 cannot find donor
2023-03-12T19:43:20.915143Z (repeated x97)node1 cannot find donor | |
2023-03-12T19:43:20.915262Z | (repeated x97)node1 cannot find donor |
2023-03-12T19:44:58.999603Z | | node1 cannot find donor
2023-03-12T19:44:58.999791Z node1 cannot find donor | |
2023-03-12T19:44:58.999891Z | node1 cannot find donor |
2023-03-12T19:44:59.817822Z timeout from donor in gtid/keyring stage | |
2023-03-12T19:44:59.839692Z SST error | |
2023-03-12T19:44:59.840669Z | | node2 joined
2023-03-12T19:44:59.840745Z | | node1 left
2023-03-12T19:44:59.840933Z | node3 joined |
2023-03-12T19:44:59.841034Z | node1 left |
2023-03-12T19:44:59.841189Z NON-PRIMARY(n=1) | |
2023-03-12T19:44:59.841292Z PRIMARY -> OPEN | |
2023-03-12T19:44:59.841352Z OPEN -> CLOSED | |
2023-03-12T19:44:59.841515Z terminated | |
2023-03-12T19:44:59.841529Z former SST cancelled | |
2023-03-12T19:44:59.848349Z | | node1 left
2023-03-12T19:44:59.848409Z | | PRIMARY(n=2)
2023-03-12T19:44:59.855443Z | node1 left |
2023-03-12T19:44:59.855491Z | PRIMARY(n=2) |
$ pt-galera-log-explainer whois 172.17.0.2 --no-color tests/logs/upgrade/*
ip:
└── 172.17.0.2
├── nodename:
│ └── node1 (2023-03-12 19:35:07.644683 +0000 UTC)
│
└── uuid:
├── 1d3ea8f5 (2023-03-12 07:24:13.789261 +0000 UTC)
├── 54ab931e (2023-03-12 07:43:08.563339 +0000 UTC)
├── fecde235 (2023-03-12 08:46:48.963504 +0000 UTC)
├── a07872e1 (2023-03-12 08:49:41.206124 +0000 UTC)
├── 60da0bf9-aa9c (2023-03-12 12:29:48.873397 +0000 UTC)
├── 35b62086-902c (2023-03-12 13:04:23.979636 +0000 UTC)
├── ca2c2a5f-a82a (2023-03-12 19:35:05.878879 +0000 UTC)
└── eefb9c8a-b69a (2023-03-12 19:43:17.133756 +0000 UTC)
Requirements
============
grep, version 3
On Darwin based OS, grep is only version 2 due to license limitations. --grep-cmd can be used to point the correct grep binary, usually ggrep
Compatibility
=============
* Percona XtraDB Cluster: 5.5 to 8.0
* MariaDB Galera Cluster: 10.0 to 10.6
* logs from PXC operator pods (error.log, recovery.log, post.processing.log)
Known issues
============
* Nodes sharing the same ip, or nodes with identical names are not supported
* Sparse files identification can be missed, resulting in many columns displayed. ``--merge-by-directory`` can be used, but files need to be organized already in separate directories
This is mainly when the log file does not contain enough information.
* Some information will seems missed. Depending on the case, it may be simply unimplemented yet, or it was disabled later because it was found to be unreliable (node index numbers are not reliable for example)
* Columns width are sometimes too large to be easily readable. This usually happens when printing SST events with long node names
* When some display corner-cases seems broken (events not deduplicated, ...), it is because of extra hidden internal events.

1
docs/pt-galera-log-explainer.rst Symbolic link
View File

@@ -0,0 +1 @@
../src/go/pt-galera-log-explainer/README.rst

191
docs/pt-k8s-debug-collector.rst
View File

@@ -1,191 +0,0 @@
.. _pt-k8s-debug-collector:
==================================
:program:`pt-k8s-debug-collector`
==================================
Collects debug data (logs, resource statuses etc.) from a k8s/OpenShift cluster. Data is packed into the ``cluster-dump.tar.gz`` archive in the current working directory.
Data that will be collected
===========================
.. code-block:: bash
"pods",
"replicasets",
"deployments",
"statefulsets",
"replicationcontrollers",
"events",
"configmaps",
"cronjobs",
"jobs",
"podsecuritypolicies",
"poddisruptionbudgets",
"perconaxtradbbackups",
"perconaxtradbclusterbackups",
"perconaxtradbclusterrestores",
"perconaxtradbclusters",
"clusterrolebindings",
"clusterroles",
"rolebindings",
"roles",
"storageclasses",
"persistentvolumeclaims",
"persistentvolumes",
"modes",
"your-custom-resource" (depends on 'resource' flag)
Data, collected for PXC
~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
"perconaxtradbbackups",
"perconaxtradbclusterbackups",
"perconaxtradbclusterrestores",
"perconaxtradbclusters"
Summary, collected for PXC (available in file summary.txt)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
"pt-mysql-summary"
Individual files, collected for PXC
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
"var/lib/mysql/mysqld-error.log",
"var/lib/mysql/innobackup.backup.log",
"var/lib/mysql/innobackup.move.log",
"var/lib/mysql/innobackup.prepare.log",
"var/lib/mysql/grastate.dat",
"var/lib/mysql/gvwstate.dat",
"var/lib/mysql/mysqld.post.processing.log",
"var/lib/mysql/auto.cnf"
Data, collected for MySQL
~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
"perconaservermysqlbackups",
"perconaservermysqlrestores",
"perconaservermysqls"
Summary, collected for MySQL (available in file summary.txt)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
"pt-mysql-summary"
Data, collected for MongoDB
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
"perconaservermongodbbackups",
"perconaservermongodbrestores",
"perconaservermongodbs"
Summary, collected for MongoDB (available in file summary.txt)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
"pt-mongodb-summary"
Data, collected for PostgreSQL
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
"perconapgclusters",
"pgclusters",
"pgpolicies",
"pgreplicas",
"pgtasks"
Summary, collected for PostgreSQL (available in file summary.txt)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
"pg_gather"
Usage
=====
``pt-k8s-debug-collector <flags>``
Supported Flags
================
``--resource``
Targeted custom resource name. Supported values:
* ``pxc`` - PXC
* ``psmdb`` - MongoDB
* ``pg`` - PostgreSQL Operator v1 (deprecated)
* ``pgv2`` - PostgreSQL Operator v2
* ``ps`` - MySQL
* ``none`` - Collect only general Kubernetes data, do not collect anything specific to the particular operator).
* ``auto`` - Auto-detect custom resource
Default: ``auto``
``--namespace``
Targeted namespace. By default data will be collected from all namespaces
``--cluster``
Targeted cluster. By default data from all available clusters to be collected
``--kubeconfig``
Path to kubeconfig. Default configuration be used if none specified
``--forwardport``
Port to use when collecting database-specific summaries. By default, 3306 will be used for PXC and MySQL, 27017 for MongoDB, and 5432 for PostgreSQL
``--version``
Print version info
Requirements
============
- Installed, configured, and available in PATH ``kubectl``
- Installed, configured, and available in PATH ``pt-mysql-summary`` for PXC and MySQL
- Installed, configured, and available in PATH ``mysql`` for PXC and MySQL
- Installed, configured, and available in PATH ``pt-mongodb-summary`` for MongoDB
- Installed, configured, and available in PATH ``psql`` for PostgreSQL
Known Issues
============
On Kubernetes 1.21 - 1.24 warning is printed:
.. code-block:: bash
2022/12/15 17:43:16 Error: get resource podsecuritypolicies in namespace default: error: <nil>, stderr: Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
, stdout: apiVersion: v1
items: []
kind: List
metadata:
resourceVersion: ""
This warning is harmless and does not affect data collection. We will remove podsecuritypolicies once everyone upgrade to Kubernetes 1.25 or newer. Before that we advise to ignore this warning.

1
docs/pt-k8s-debug-collector.rst Symbolic link
View File

@@ -0,0 +1 @@
../src/go/pt-k8s-debug-collector/README.rst

72
docs/pt-mongodb-index-check.rst
View File

@@ -1,72 +0,0 @@
.. _pt-mongodb-index-check:
=================================
:program:`pt-mongodb-index-check`
=================================
Performs checks on MongoDB indexes.
Checks available
================
Duplicated indexes
~~~~~~~~~~~~~~~~~~
Check for indexes that are the prefix of other indexes. For example if we have these 2 indexes
.. code-block:: javascript
db.getSiblingDB("testdb").test_col.createIndex({"f1": 1, "f2": -1, "f3": 1, "f4": 1}, {"name": "idx_01"});
db.getSiblingDB("testdb").test_col.createIndex({"f1": 1, "f2": -1, "f3": 1}, {"name": "idx_02"});
The index ``idx_02`` is the prefix of ``idx_01`` because it has the same
keys in the same order so, ``idx_02`` can be dropped.
Unused indexes.
~~~~~~~~~~~~~~~
This check gets the ``$indexstats`` for all indexes and reports those
having ``accesses.ops`` = 0.
Usage
=====
Run the program as ``pt-mongodb-index-check <command> [flags]``
Available commands
~~~~~~~~~~~~~~~~~~
================ ==================================
Command Description
================ ==================================
check-duplicated Run checks for duplicated indexes.
check-unused Run check for unused indexes.
check-all Run all checks
================ ==================================
Available flags
~~~~~~~~~~~~~~~
+----------------------------+----------------------------------------+
| Flag | Description |
+============================+========================================+
| –all-databases | Check in all databases excluding |
| | system dbs. |
+----------------------------+----------------------------------------+
| –databases=DATABASES,… | Comma separated list of databases to |
| | check. |
+----------------------------+----------------------------------------+
| –all-collections | Check in all collections in the |
| | selected databases. |
+----------------------------+----------------------------------------+
| –collections=COLLECTIONS,… | Comma separated list of collections to |
| | check. |
+----------------------------+----------------------------------------+
| –mongodb.uri= | Connection URI |
+----------------------------+----------------------------------------+
| –json | Show output as JSON |
+----------------------------+----------------------------------------+
| –version | Show version information |
+----------------------------+----------------------------------------+

1
docs/pt-mongodb-index-check.rst Symbolic link
View File

@@ -0,0 +1 @@
../src/go/pt-mongodb-index-check/README.rst

101
docs/pt-mongodb-query-digest.rst
View File

@@ -1,101 +0,0 @@
.. _pt-mongodb-query-digest:
==================================
:program:`pt-mongodb-query-digest`
==================================
``pt-mongodb-query-digest`` reports query usage statistics
by aggregating queries from MongoDB query profiler.
Usage
=====
.. code-block:: bash
pt-mongodb-query-digest [OPTIONS]
It runs the following command::
db.getSiblingDB("samples").system.profile.find({"op":{"$nin":["getmore", "delete"]}});
Then the results are grouped by fingerprint and namespace
(database.collection).
The fingerprint is calculated as a sorted list of keys in the document
with a maximum depth level of 10.
By default, the results are sorted by ascending query count.
.. note:: ``pt-mongodb-query-digest`` cannot collect statistics
from MongoDB instances that require connection via SSL.
Support for SSL will be added in the future.
Options
-------
``-?``, ``--help``
Show help and exit
``-a``, ``--authenticationDatabase``
Specifies the database used to establish credentials and privileges
with a MongoDB server.
By default, the ``admin`` database is used.
``-c``, ``--no-version-check``
Don't check for updates
``-d``, ``--database``
Specifies which database to profile
``-f``, ``--output-format``
Specifies the report output format. Valid options are: ``text``, ``json``.
The default value is ``text``.
``-l``, ``--log-level``
Specifies the log level:
``panic``, ``fatal``, ``error``, ``warn``, ``info``, ``debug error``
``-n``, ``--limit``
Limits the number of queries to show
``-o``, ``--order-by``
Specifies the sorting order using fields:
``count``, ``ratio``, ``query-time``, ``docs-scanned``, ``docs-returned``.
Adding a hyphen (``-``) in front of a field denotes reverse order.
For example: ``--order-by="count,-ratio"``.
``-p``, ``--password``
Specifies the password to use when connecting to a server
with authentication enabled.
Do not add a space between the option and its value: ``-p<password>``.
If you specify the option without any value,
you will be prompted for the password.
``-u``, ``--user``
Specifies the user name for connecting to a server
with authentication enabled.
``-v``, ``--version``
Show version and exit
Output Example
==============
.. code-block:: none
# Query 2: 0.00 QPS, ID 1a6443c2db9661f3aad8edb6b877e45d
# Ratio 1.00 (docs scanned/returned)
# Time range: 2017-01-11 12:58:26.519 -0300 ART to 2017-01-11 12:58:26.686 -0300 ART
# Attribute pct total min max avg 95% stddev median
# ================== === ======== ======== ======== ======== ======== ======= ========
# Count (docs) 36
# Exec Time ms 0 0 0 0 0 0 0 0
# Docs Scanned 0 148.00 0.00 74.00 4.11 74.00 16.95 0.00
# Docs Returned 2 148.00 0.00 74.00 4.11 74.00 16.95 0.00
# Bytes recv 0 2.11M 215.00 1.05M 58.48K 1.05M 240.22K 215.00
# String:
# Namespaces samples.col1
# Fingerprint $gte,$lt,$meta,$sortKey,filter,find,projection,shardVersion,sort,user_id,user_id

1
docs/pt-mongodb-query-digest.rst Symbolic link
View File

@@ -0,0 +1 @@
../src/go/pt-mongodb-query-digest/README.rst

168
docs/pt-mongodb-summary.rst
View File

@@ -1,168 +0,0 @@
.. pt-mongodb-summary:
=============================
:program:`pt-mongodb-summary`
=============================
``pt-mongodb-summary`` collects information about a MongoDB cluster.
It collects information from several sources
to provide an overview of the cluster.
Usage
=====
.. code-block:: bash
pt-mongodb-summary [OPTIONS] [HOST[:PORT]]
By default, if you run ``pt-mongodb-summary`` without any parameters,
it will try to connect to ``localhost`` on port ``27017``.
The program collects information about MongoDB instances
by running administration commands and formatting the output.
.. note:: ``pt-mongodb-summary`` requires to be run by user
with the ``clusterAdmin`` or ``root`` built-in roles.
.. note:: ``pt-mongodb-summary`` cannot collect statistics
from MongoDB instances that require connection via SSL.
Support for SSL will be added in the future.
Options
-------
``-a``, ``--auth-db``
Specifies the database used to establish credentials and privileges
with a MongoDB server.
By default, the ``admin`` database is used.
``-f``, ``--output-format``
Specifies the report output format. Valid options are: ``text``, ``json``.
The default value is ``text``.
``-p``, ``--password``
Specifies the password to use when connecting to a server
with authentication enabled.
Do not add a space between the option and its value: ``-p<password>``.
If you specify the option without any value,
``pt-mongodb-summary`` will ask for password interactively.
``-u``, ``--user``
Specifies the user name for connecting to a server
with authentication enabled.
Output Example
==============
.. code-block:: none
# Instances ####################################################################################
ID Host Type ReplSet
0 localhost:17001 PRIMARY r1
1 localhost:17002 SECONDARY r1
2 localhost:17003 SECONDARY r1
0 localhost:18001 PRIMARY r2
1 localhost:18002 SECONDARY r2
2 localhost:18003 SECONDARY r2
# This host
# Mongo Executable #############################################################################
Path to executable | /home/karl/tmp/MongoDB32Labs/3.0/bin/mongos
# Report On 0 ########################################
User | karl
PID Owner | mongos
Time | 2016-10-30 00:18:49 -0300 ART
Hostname | karl-HP-ENVY
Version | 3.0.11
Built On | Linux x86_64
Started | 2016-10-30 00:18:49 -0300 ART
Datadir | /data/db
Process Type | mongos
# Running Ops ##################################################################################
Type Min Max Avg
Insert 0 0 0/5s
Query 0 0 0/5s
Update 0 0 0/5s
Delete 0 0 0/5s
GetMore 0 0 0/5s
Command 0 22 16/5s
# Security #####################################################################################
Users 0
Roles 0
Auth disabled
SSL disabled
# Oplog ########################################################################################
Oplog Size 18660 Mb
Oplog Used 55 Mb
Oplog Length 0.91 hours
Last Election 2016-10-30 00:18:44 -0300 ART
# Cluster wide #################################################################################
Databases: 3
Collections: 17
Sharded Collections: 1
Unsharded Collections: 16
Sharded Data Size: 68 GB
Unsharded Data Size: 0 KB
# Balancer (per day)
Success: 6
Failed: 0
Splits: 0
Drops: 0
Sections
--------
Output is separated into the following sections:
* **Instances**
This section lists all hosts connected to the current MongoDB instance.
For this, ``pt-mongodb-summary`` runs the ``listShards`` command
and then the ``replSetGetStatus`` on every instance
to collect its ID, type, and replica set.
* **This host**
This section provides an overview of the current MongoDB instance
and the underlying OS.
For this, ``pt-mongodb-summary`` groups information
collected from ``hostInfo``, ``getCmdLineOpts``, ``serverStatus``,
and the OS process (by process ID).
* **Running Ops**
This section provides minimum, maximum, and average operation counters
for ``insert``, ``query``, ``update``, ``delete``, ``getMore``,
and ``command`` operations.
For this, ``pt-mongodb-summary`` runs the ``serverStatus`` command
5 times at regular intervals (every second).
* **Security**
This section provides information about the security settings.
For this, ``pt-mongodb-summary``, parses ``getCmdLineOpts`` output
and queries the ``admin.system.users``
and ``admin.system.roles`` collections.
* **Oplog**
This section contains details about the MongoDB operations log (oplog).
For this, ``pt-mongodb-summary`` collects statistics
from the oplog on every host in the cluster,
and returns those with the smallest ``TimeDiffHours`` value.
* **Cluster wide**
This section provides information about the number of sharded and
unsharded databases, collections, and their size.
For this, ``pt-mongodb-summary`` runs the ``listDatabases`` command
and then runs ``collStats`` for every collection in every database.

1
docs/pt-mongodb-summary.rst Symbolic link
View File

@@ -0,0 +1 @@
../src/go/pt-mongodb-summary/README.rst

475
docs/pt-pg-summary.rst
View File

@@ -1,475 +0,0 @@
========================
:program:`pt-pg-summary`
========================
**pt-pg-summary** collects information about a PostgreSQL cluster.
Usage
-----
``pt-pg-summary [options] [host:[port]]``
Binaries
--------
Please check the `releases <https://github.com/percona/toolkit-go/releases>`_ tab to download the binaries.
Parameters
^^^^^^^^^^
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| Short | Long | Default | Description |
+========+=================================+=========+============================================================================+
| | ``--help`` | | Show context-sensitive help (also try ``--help-long`` and ``--help-man``). |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| | ``--version`` | | Show application version. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| | ``--databases=DATABASES`` | | Summarize this comma-separated list of databases. All if not specified. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| ``-h`` | ``--host=HOST`` | | Host to connect to or local Unix socket for connection |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| ``-W`` | ``--password=PASSWORD`` | | Password to use when connecting. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| ``-p`` | ``--port=PORT`` | | Port number to use for connection. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| | ``--sleep=SLEEP`` | 10 | Seconds to sleep when gathering status counters. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| ``-U`` | ``--username=USERNAME`` | | User for login if not current user. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| | ``--disable-ssl`` | true | Disable SSL for the connection. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| | ``--verbose`` | false | Show verbose log. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| | ``--debug`` | false | Show debug information in the logs. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
.. Currently hidden
.. --list-encrypted-tables Include a list of the encrypted tables in all databases
.. --ask-pass Prompt for a password when connecting to PostgreSQL
.. --config Config file
.. --defaults-file Only read PostgreSQL options from the given file
.. --read-samples Create a report from the files found in this directory
.. --save-samples Save the data files used to generate the summary in this directory
Output
^^^^^^
The output is grouped into these categories:
AllDatabases
Selects ``datname`` from ``pg_database`` where ``datistemplate`` is false.
ClusterInfo
Selects cluster information from ``pg_stat_activity``.
ConnectedClients
Counts the connected clients by selecting from ``pg_stat_activity``.
Connections
Selects ``state`` from ``pg_stat_activity`` and counts them.
Counters
Selects various counter values from ``pg_stat_database``.
DatabaseWaitEvents
Shows database wait events from ``pg_locks``, ``pg_stat_database``, ``pg_class``, and ``pg_stat_activity``.
Databases
Shows the name and size of databases from ``pg_stat_database``.
GlobalWaitEvents
Shows global wait evens from ``pg_stat_activity``.
IndexCacheHitRatio
Shows index hit ratios from ``pg_statio_user_indexes``.
PortAndDatadir
Shows port and data directory name from ``pg_settings``.
ServerVersion
Shows the value of ``server_version_num``.
Setting
Selects ``name`` and ``setting`` from ``pg_settings``.
SlaveHosts10
Selects information for PostgreSQL version 10.
SlaveHosts96
Selects information for PostgreSQL version 9.6.
TableAccess
Shows table access information by selecting from ``pg_locks``, ``pg_stat_database`` and ``pg_class``.
TableCacheHitRatio
Shows table cache hit ratio information from ``pg_statio_user_tables``.
Tablespaces
Show owner and location from ``pg_catalog.pg_tablespace``.
Output example
""""""""""""""
.. code-block:: html
##### --- Database Port and Data_Directory --- ####
+----------------------+----------------------------------------------------+
| Name | Setting |
+----------------------+----------------------------------------------------+
| data_directory | /var/lib/postgresql/data |
+----------------------+----------------------------------------------------+
##### --- List of Tablespaces ---- ######
+----------------------+----------------------+----------------------------------------------------+
| Name | Owner | Location |
+----------------------+----------------------+----------------------------------------------------+
| pg_default | postgres | |
| pg_global | postgres | |
+----------------------+----------------------+----------------------------------------------------+
##### --- Cluster Information --- ####
+------------------------------------------------------------------------------------------------------+
Usename : postgres
Time : 2020-04-21 13:38:22.770077 +0000 UTC
Client Address : 172.19.0.1
Client Hostname:
Version : PostgreSQL 9.6.17 on x86_64-pc-linux-gnu (Debian 9.6.17-2.pgdg90+1), compiled by
Started : 2020-04-21 13:36:59.909175 +0000 UTC
Is Slave : false
+------------------------------------------------------------------------------------------------------+
##### --- Databases --- ####
+----------------------+------------+
| Dat Name | Size |
+----------------------+------------+
| postgres | 7071 kB |
| template1 | 6961 kB |
| template0 | 6961 kB |
+----------------------+------------+
##### --- Index Cache Hit Ratios --- ####
Database: postgres
+----------------------+------------+
| Index Name | Ratio |
+----------------------+------------+
| index hit rate | 0.00 |
+----------------------+------------+
##### --- Table Cache Hit Ratios --- ####
Database: postgres
+----------------------+------------+
| Index Name | Ratio |
+----------------------+------------+
| cache hit rate | 0.00 |
+----------------------+------------+
##### --- List of Wait_events for the entire Cluster - all-databases --- ####
No stats available
##### --- List of users and client_addr or client_hostname connected to --all-databases --- ####
+----------------------+------------+---------+----------------------+---------+
| Wait Event Type | Client | State | Count |
+----------------------+------------+---------+----------------------+---------+
| postgres | 172.19.0.1/32 | active | 1 |
+----------------------+------------+---------+----------------------+---------+
##### --- Counters diff after 10 seconds --- ####
+----------------------+-------------+------------+--------------+-------------+------------+-------------+------------+-------------+------------+------------+-----------+-----------+-----------+------------+
| Database | Numbackends | XactCommit | XactRollback | BlksRead | BlksHit | TupReturned | TupFetched | TupInserted | TupUpdated | TupDeleted | Conflicts | TempFiles | TempBytes | Deadlocks |
+----------------------+-------------+------------+--------------+-------------+------------+-------------+------------+-------------+------------+------------+-----------+-----------+-----------+------------+
| postgres | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| template0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| template1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+----------------------+-------------+------------+--------------+-------------+------------+-------------+------------+-------------+------------+------------+-----------+-----------+-----------+------------+
##### --- Table access per database --- ####
Database: postgres
+----------------------------------------------------+------+--------------------------------+---------+
| Relname | Kind | Datname | Count |
+----------------------------------------------------+------+--------------------------------+---------+
| pg_class | r | postgres | 1 |
| pg_stat_database | v | postgres | 1 |
| pg_locks | v | postgres | 1 |
| pg_class_tblspc_relfilenode_index | i | postgres | 1 |
| pg_class_relname_nsp_index | i | postgres | 1 |
| pg_class_oid_index | i | postgres | 1 |
+----------------------------------------------------+------+--------------------------------+---------+
##### --- Instance settings --- ####
Setting Value
allow_system_table_mods : off
application_name :
archive_command : (disabled)
archive_mode : off
archive_timeout : 0
array_nulls : on
authentication_timeout : 60
autovacuum : on
autovacuum_analyze_scale_factor : 0.1
autovacuum_analyze_threshold : 50
autovacuum_freeze_max_age : 200000000
autovacuum_max_workers : 3
autovacuum_multixact_freeze_max_age : 400000000
autovacuum_naptime : 60
autovacuum_vacuum_cost_delay : 20
autovacuum_vacuum_cost_limit : -1
autovacuum_vacuum_scale_factor : 0.2
autovacuum_vacuum_threshold : 50
autovacuum_work_mem : -1
backend_flush_after : 0
backslash_quote : safe_encoding
bgwriter_delay : 200
bgwriter_flush_after : 64
bgwriter_lru_maxpages : 100
bgwriter_lru_multiplier : 2
block_size : 8192
bonjour : off
bonjour_name :
bytea_output : hex
check_function_bodies : on
checkpoint_completion_target : 0.5
checkpoint_flush_after : 32
checkpoint_timeout : 300
checkpoint_warning : 30
client_encoding : UTF8
client_min_messages : notice
cluster_name :
commit_delay : 0
commit_siblings : 5
config_file : /var/lib/postgresql/data/postgresql.conf
constraint_exclusion : partition
cpu_index_tuple_cost : 0.005
cpu_operator_cost : 0.0025
cpu_tuple_cost : 0.01
cursor_tuple_fraction : 0.1
data_checksums : off
data_directory : /var/lib/postgresql/data
data_sync_retry : off
DateStyle : ISO, MDY
db_user_namespace : off
deadlock_timeout : 1000
debug_assertions : off
debug_pretty_print : on
debug_print_parse : off
debug_print_plan : off
debug_print_rewritten : off
default_statistics_target : 100
default_tablespace :
default_text_search_config : pg_catalog.english
default_transaction_deferrable : off
default_transaction_isolation : read committed
default_transaction_read_only : off
default_with_oids : off
dynamic_library_path : $libdir
dynamic_shared_memory_type : posix
effective_cache_size : 524288
effective_io_concurrency : 1
enable_bitmapscan : on
enable_hashagg : on
enable_hashjoin : on
enable_indexonlyscan : on
enable_indexscan : on
enable_material : on
enable_mergejoin : on
enable_nestloop : on
enable_seqscan : on
enable_sort : on
enable_tidscan : on
escape_string_warning : on
event_source : PostgreSQL
exit_on_error : off
external_pid_file :
extra_float_digits : 2
force_parallel_mode : off
from_collapse_limit : 8
fsync : on
full_page_writes : on
geqo : on
geqo_effort : 5
geqo_generations : 0
geqo_pool_size : 0
geqo_seed : 0
geqo_selection_bias : 2
geqo_threshold : 12
gin_fuzzy_search_limit : 0
gin_pending_list_limit : 4096
hba_file : /var/lib/postgresql/data/pg_hba.conf
hot_standby : off
hot_standby_feedback : off
huge_pages : try
ident_file : /var/lib/postgresql/data/pg_ident.conf
idle_in_transaction_session_timeout : 0
ignore_checksum_failure : off
ignore_system_indexes : off
integer_datetimes : on
IntervalStyle : postgres
join_collapse_limit : 8
krb_caseins_users : off
krb_server_keyfile : FILE:/etc/postgresql-common/krb5.keytab
lc_collate : en_US.utf8
lc_ctype : en_US.utf8
lc_messages : en_US.utf8
lc_monetary : en_US.utf8
lc_numeric : en_US.utf8
lc_time : en_US.utf8
listen_addresses : *
lo_compat_privileges : off
local_preload_libraries :
lock_timeout : 0
log_autovacuum_min_duration : -1
log_checkpoints : off
log_connections : off
log_destination : stderr
log_directory : pg_log
log_disconnections : off
log_duration : off
log_error_verbosity : default
log_executor_stats : off
log_file_mode : 0600
log_filename : postgresql-%Y-%m-%d_%H%M%S.log
log_hostname : off
log_line_prefix :
log_lock_waits : off
log_min_duration_statement : -1
log_min_error_statement : error
log_min_messages : warning
log_parser_stats : off
log_planner_stats : off
log_replication_commands : off
log_rotation_age : 1440
log_rotation_size : 10240
log_statement : none
log_statement_stats : off
log_temp_files : -1
log_timezone : Etc/UTC
log_truncate_on_rotation : off
logging_collector : off
maintenance_work_mem : 65536
max_connections : 100
max_files_per_process : 1000
max_function_args : 100
max_identifier_length : 63
max_index_keys : 32
max_locks_per_transaction : 64
max_parallel_workers_per_gather : 0
max_pred_locks_per_transaction : 64
max_prepared_transactions : 0
max_replication_slots : 0
max_stack_depth : 2048
max_standby_archive_delay : 30000
max_standby_streaming_delay : 30000
max_wal_senders : 0
max_wal_size : 64
max_worker_processes : 8
min_parallel_relation_size : 1024
min_wal_size : 5
old_snapshot_threshold : -1
operator_precedence_warning : off
parallel_setup_cost : 1000
parallel_tuple_cost : 0.1
password_encryption : on
port : 5432
post_auth_delay : 0
pre_auth_delay : 0
quote_all_identifiers : off
random_page_cost : 4
replacement_sort_tuples : 150000
restart_after_crash : on
row_security : on
search_path : "$user", public
segment_size : 131072
seq_page_cost : 1
server_encoding : UTF8
server_version : 9.6.17
server_version_num : 90617
session_preload_libraries :
session_replication_role : origin
shared_buffers : 16384
shared_preload_libraries :
sql_inheritance : on
ssl : off
ssl_ca_file :
ssl_cert_file : server.crt
ssl_ciphers : HIGH:MEDIUM:+3DES:!aNULL
ssl_crl_file :
ssl_ecdh_curve : prime256v1
ssl_key_file : server.key
ssl_prefer_server_ciphers : on
standard_conforming_strings : on
statement_timeout : 0
stats_temp_directory : pg_stat_tmp
superuser_reserved_connections : 3
synchronize_seqscans : on
synchronous_commit : on
synchronous_standby_names :
syslog_facility : local0
syslog_ident : postgres
syslog_sequence_numbers : on
syslog_split_messages : on
tcp_keepalives_count : 9
tcp_keepalives_idle : 7200
tcp_keepalives_interval : 75
temp_buffers : 1024
temp_file_limit : -1
temp_tablespaces :
TimeZone : Etc/UTC
timezone_abbreviations : Default
trace_notify : off
trace_recovery_messages : log
trace_sort : off
track_activities : on
track_activity_query_size : 1024
track_commit_timestamp : off
track_counts : on
track_functions : none
track_io_timing : off
transaction_deferrable : off
transaction_isolation : read committed
transaction_read_only : off
transform_null_equals : off
unix_socket_directories : /var/run/postgresql
unix_socket_group :
unix_socket_permissions : 0777
update_process_title : on
vacuum_cost_delay : 0
vacuum_cost_limit : 200
vacuum_cost_page_dirty : 20
vacuum_cost_page_hit : 1
vacuum_cost_page_miss : 10
vacuum_defer_cleanup_age : 0
vacuum_freeze_min_age : 50000000
vacuum_freeze_table_age : 150000000
vacuum_multixact_freeze_min_age : 5000000
vacuum_multixact_freeze_table_age : 150000000
wal_block_size : 8192
wal_buffers : 512
wal_compression : off
wal_keep_segments : 0
wal_level : minimal
wal_log_hints : off
wal_receiver_status_interval : 10
wal_receiver_timeout : 60000
wal_retrieve_retry_interval : 5000
wal_segment_size : 2048
wal_sender_timeout : 60000
wal_sync_method : fdatasync
wal_writer_delay : 200
wal_writer_flush_after : 128
work_mem : 4096
xmlbinary : base64
xmloption : content
zero_damaged_pages : off
##### --- Processes start up command --- ####
No postgres process found
Minimum auth role
^^^^^^^^^^^^^^^^^
This program needs to run some commands like ``getShardMap`` and to be able to run those commands
it needs to run under a user with the ``clusterAdmin`` or ``root`` built-in roles.

1
docs/pt-pg-summary.rst Symbolic link
View File

@@ -0,0 +1 @@
../src/go/pt-pg-summary/README.rst

173
docs/pt-secure-collect.rst
View File

@@ -1,173 +0,0 @@
============================
:program:`pt-secure-collect`
============================
NAME
====
:program:`pt-secure-collect` - collect, sanitize, pack and encrypt data.
SYNOPSIS
========
Usage
-----
::
pt-secure-data [<flags>] <command> [<args> ...]
By default, :program:`pt-secure-collect` will collect the output of:
- ``pt-stalk``
- ``pt-summary``
- ``pt-mysql-summary``
Global flags
------------
.. option:: --help
Show context-sensitive help (also try --help-long and --help-man).
.. option:: --debug
Enable debug log level.
COMMANDS
========
* **Help command**
Show help
* **Collect command**
Collect, sanitize, pack and encrypt data from pt-tools. Usage:
::
pt-secure-collect collect <flags>
.. option:: --bin-dir
Directory having the Percona Toolkit binaries (if they are not in PATH).
.. option:: --temp-dir
Temporary directory used for the data collection.
Default: ``${HOME}/data_collection_{timestamp}``
.. option:: --include-dir
Include this dir into the sanitized tar file.
.. option:: --config-file
Path to the config file. Default: ``~/.my.cnf``
.. option:: --mysql-host
MySQL host. Default: ``127.0.0.1``
.. option:: --mysql-port
MySQL port. Default: ``3306``
.. option:: --mysql-user
MySQL user name.
.. option:: --mysql-password
MySQL password.
.. option:: --ask-mysql-pass
password.
.. option:: --extra-cmd
Also run this command as part of the data collection. This parameter can
be used more than once.
.. option:: --encrypt-password
Encrypt the output file using this password. If omitted, it will be asked
in the command line.
.. option:: --no-collect
Do not collect data
.. option:: --no-sanitize
Do not sanitize data
.. option:: --no-encrypt
Do not encrypt the output file.
.. option:: --no-sanitize-hostnames
Do not sanitize hostnames.
.. option:: --no-sanitize-queries
Do not replace queries by their fingerprints.
.. option:: --no-remove-temp-files
Do not remove temporary files.
* **Decrypt command**
Decrypt an encrypted file. The password will be requested from the
terminal. Usage:
::
pt-secure-collect decrypt [flags] <input file>
.. option:: --outfile
Write the output to this file. If omitted, the output file
name will be the same as the input file, adding the ``.aes`` extension.
* **Encrypt command**
Encrypt a file. The password will be requested from the terminal. Usage:
::
pt-secure-collect encrypt [flags] <input file>
.. option:: --outfile
Write the output to this file. If omitted, the output file
name will be the same as the input file, without the ``.aes`` extension.
* **Sanitize command**
Replace queries in a file by their fingerprints and obfuscate hostnames.
Usage:
::
pt-secure-collect sanitize [flags]
.. option:: --input-file
Input file. If not specified, the input will be Stdin.
.. option:: --output-file
Output file. If not specified, the input will be Stdout.
.. option:: --no-sanitize-hostnames
Do not sanitize host names.
.. option:: --no-sanitize-queries
Do not replace queries by their fingerprints.

1
docs/pt-secure-collect.rst Symbolic link
View File

@@ -0,0 +1 @@
../src/go/pt-secure-collect/README.rst

2
docs/release_notes.rst
View File

@@ -1,4 +1,4 @@
Percona Toolkit
Release Notes
***************
v3.6.0 released 2024-06-12

5
src/go/pt-galera-log-explainer/README.rst
View File

@@ -244,3 +244,8 @@ Known issues
* Some information will seems missed. Depending on the case, it may be simply unimplemented yet, or it was disabled later because it was found to be unreliable (node index numbers are not reliable for example)
* Columns width are sometimes too large to be easily readable. This usually happens when printing SST events with long node names
* When some display corner-cases seems broken (events not deduplicated, ...), it is because of extra hidden internal events.
Authors
=======
Yoann La Cancellera

5
src/go/pt-k8s-debug-collector/README.rst
View File

@@ -189,3 +189,8 @@ On Kubernetes 1.21 - 1.24 warning is printed:
resourceVersion: ""
This warning is harmless and does not affect data collection. We will remove podsecuritypolicies once everyone upgrade to Kubernetes 1.25 or newer. Before that we advise to ignore this warning.
Authors
=======
Max Dudin, Andrii Dema, Carlos Salguero, Sveta Smirnova

4
src/go/pt-mongodb-index-check/README.rst
View File

@@ -70,3 +70,7 @@ Available flags
| –version | Show version information |
+----------------------------+----------------------------------------+
Authors
=======
Carlos Salguero

49
src/go/pt-mongodb-query-digest/README.md
View File

@@ -1,49 +0,0 @@
#pt-mongodb-query-digest
This program reports query usage statistics by aggregating queries from MongoDB query profiler.
The queries are the result of running:
```javascript
db.getSiblingDB("samples").system.profile.find({"op":{"$nin":["getmore", "delete"]}});
```
and then, the results are grouped by fingerprint and namespace (database.collection).
The fingerprint is calculated as the **sorted list** of the keys in the document. The max depth level is 10.
The last step is sorting the results. The default sort order is by ascending query count.
##Sample output
```
# Query 3: 0.06 QPS, ID 0b906bd86148def663d11b402f3e41fa
# Ratio 1.00 (docs scanned/returned)
# Time range: 2017-02-03 16:01:37.484 -0300 ART to 2017-02-03 16:02:08.43 -0300 ART
# Attribute pct total min max avg 95% stddev median
# ================== === ======== ======== ======== ======== ======== ======= ========
# Count (docs) 100
# Exec Time ms 2 3 0 1 0 0 0 0
# Docs Scanned 5 7.50K 75.00 75.00 75.00 75.00 0.00 75.00
# Docs Returned 92 7.50K 75.00 75.00 75.00 75.00 0.00 75.00
# Bytes recv 1 106.12M 1.06M 1.06M 1.06M 1.06M 0.00 1.06M
# String:
# Namespaces samples.col1
# Operation query
# Fingerprint find,shardVersion
# Query {"find":"col1","shardVersion":[0,"000000000000000000000000"]}
```
##Command line parameters
|Short|Long|Help|
|-----|----|----|
|-?|--help|Show help|
|-a|--authenticationDatabase|database used to establish credentials and privileges with a MongoDB server admin|
|-c|--no-version-check|Don't check for updates|
|-d|--database|database to profile|
|-f|--output-format|report output format. Valid values are text, json. Default: text|
|-l|--log-level|Log level:, panic, fatal, error, warn, info, debug error|
|-n|--limit|show the first n queries|
|-o|--order-by|comma separated list of order by fields (max values): `count`, `ratio`, `query-time`, `docs-scanned`, `docs-returned`.<br> A `-` in front of the field name denotes reverse order.<br> Example:`--order-by="count,-ratio"`).|
|-p|--password[=password]|Password (optional). If it is not specified it will be asked|
|-s|--skip-collections|Comma separated list of collections to skip. Default: `system.profile`. It is possible to use an empty list by setting `--skip-collections=""`|
|-u|--user|Username|
|-v|--version|Show version & exit|

115
src/go/pt-mongodb-query-digest/README.rst Normal file
View File

@@ -0,0 +1,115 @@
.. _pt-mongodb-query-digest:
==================================
:program:`pt-mongodb-query-digest`
==================================
``pt-mongodb-query-digest`` reports query usage statistics
by aggregating queries from MongoDB query profiler.
Usage
=====
.. code-block:: bash
pt-mongodb-query-digest [OPTIONS]
It runs the following command::
db.getSiblingDB("samples").system.profile.find({"op":{"$nin":["getmore", "delete"]}});
Then the results are grouped by fingerprint and namespace
(database.collection).
The fingerprint is calculated as a sorted list of keys in the document
with a maximum depth level of 10.
By default, the results are sorted by ascending query count.
Options
-------
``-?``, ``--help``
Show help and exit
``-a``, ``--authenticationDatabase``
Specifies the database used to establish credentials and privileges
with a MongoDB server.
By default, the ``admin`` database is used.
``-c``, ``--no-version-check``
Don't check for updates
``-d``, ``--database``
Specifies which database to profile
``-f``, ``--output-format``
Specifies the report output format. Valid options are: ``text``, ``json``.
The default value is ``text``.
``-l``, ``--log-level``
Specifies the log level:
``panic``, ``fatal``, ``error``, ``warn``, ``info``, ``debug error``
``-n``, ``--limit``
Limits the number of queries to show
``-o``, ``--order-by``
Specifies the sorting order using fields:
``count``, ``ratio``, ``query-time``, ``docs-scanned``, ``docs-returned``.
Adding a hyphen (``-``) in front of a field denotes reverse order.
For example: ``--order-by="count,-ratio"``.
``-p``, ``--password``
Specifies the password to use when connecting to a server
with authentication enabled.
Do not add a space between the option and its value: ``-p<password>``.
If you specify the option without any value,
you will be prompted for the password.
``-s``, ``--skip-collections``
Comma separated list of collections to skip.
Collection ``system.profile`` is skipped by default.
It is possible to use an empty list by setting ``--skip-collections=""``.
``--sslCAFile``
Specifies SSL CA cert file used for authentication.
``--sslPEMKeyFile``
Specifies SSL client PEM file used for authentication.
``-u``, ``--user``
Specifies the user name for connecting to a server
with authentication enabled.
``-v``, ``--version``
Show version and exit
Output Example
==============
.. code-block:: none
# Query 3: 0.06 QPS, ID 0b906bd86148def663d11b402f3e41fa
# Ratio 1.00 (docs scanned/returned)
# Time range: 2017-02-03 16:01:37.484 -0300 ART to 2017-02-03 16:02:08.43 -0300 ART
# Attribute pct total min max avg 95% stddev median
# ================== === ======== ======== ======== ======== ======== ======= ========
# Count (docs) 100
# Exec Time ms 2 3 0 1 0 0 0 0
# Docs Scanned 5 7.50K 75.00 75.00 75.00 75.00 0.00 75.00
# Docs Returned 92 7.50K 75.00 75.00 75.00 75.00 0.00 75.00
# Bytes recv 1 106.12M 1.06M 1.06M 1.06M 1.06M 0.00 1.06M
# String:
# Namespaces samples.col1
# Operation query
# Fingerprint find,shardVersion
# Query {"find":"col1","shardVersion":[0,"000000000000000000000000"]}
Authors
=======
Carlos Salguero

126
src/go/pt-mongodb-summary/README.rst
View File

@@ -1,36 +1,63 @@
pt-mongodb-summary
==================
**pt-mongodb-summary** collects information about a MongoDB cluster.
.. pt-mongodb-summary:
=============================
:program:`pt-mongodb-summary`
=============================
``pt-mongodb-summary`` collects information about a MongoDB cluster.
It collects information from several sources
to provide an overview of the cluster.
Usage
-----
pt-mongodb-summary [options] [host:[port]]
=====
.. code-block:: bash
pt-mongodb-summary [OPTIONS] [HOST:[PORT]]
By default, if you run ``pt-mongodb-summary`` without any parameters,
it will try to connect to ``localhost`` on port ``27017``.
The program collects information about MongoDB instances
by running administration commands and formatting the output.
.. note:: ``pt-mongodb-summary`` requires to be run by user
with the ``clusterAdmin`` or ``root`` built-in roles.
.. note:: ``pt-mongodb-summary`` cannot collect statistics
from MongoDB instances that require connection via SSL.
Support for SSL will be added in the future.
Default host:port is `localhost:27017`.
For better results, host must be a **mongos** server.
Binaries
--------
Please check the `releases <https://github.com/percona/toolkit-go/releases>`_ tab to download the binaries.
Options
-------
Parameters
^^^^^^^^^
|Short|Long|Default|Description|
|-----|----|-------|-----------|
|-a|--auth-db|admin|database used to establish credentials and privileges with a MongoDB server|
|-f|--output-format|report output format|Valid values are text, json. Default: text|
|-f|--output-format|text|output format: text, json. Default: text|
|-p|--password|empty|password to use when connecting if DB auth is enabled|
|-u|--user|empty|user name to use when connecting if DB auth is enabled|
``-a``, ``--auth-db``
Specifies the database used to establish credentials and privileges
with a MongoDB server.
By default, the ``admin`` database is used.
``-f``, ``--output-format``
Specifies the report output format. Valid options are: ``text``, ``json``.
The default value is ``text``.
``-p`` is an optional parameter. If it is used it shouldn't have a blank between the parameter and its value: `-p<password>`
It can be also used as `-p` without specifying a password; in that case, the program will ask the password to avoid using a password in the command line.
``-p``, ``--password``
Specifies the password to use when connecting to a server
with authentication enabled.
Do not add a space between the option and its value: ``-p<password>``.
If you specify the option without any value,
``pt-mongodb-summary`` will ask for password interactively.
``-u``, ``--user``
Specifies the user name for connecting to a server
with authentication enabled.
Output example
""""""""""""""
.. code-block:: html
==============
.. code-block:: none
# Instances ####################################################################################
ID Host Type ReplSet
@@ -90,9 +117,56 @@ Output example
Splits: 0
Drops: 0
Minimum auth role
^^^^^^^^^^^^^^^^^
Sections
--------
This program needs to run some commands like ``getShardMap`` and to be able to run those commands
it needs to run under a user with the ``clusterAdmin`` or ``root`` built-in roles.
Output is separated into the following sections:
* **Instances**
This section lists all hosts connected to the current MongoDB instance.
For this, ``pt-mongodb-summary`` runs the ``listShards`` command
and then the ``replSetGetStatus`` on every instance
to collect its ID, type, and replica set.
* **This host**
This section provides an overview of the current MongoDB instance
and the underlying OS.
For this, ``pt-mongodb-summary`` groups information
collected from ``hostInfo``, ``getCmdLineOpts``, ``serverStatus``,
and the OS process (by process ID).
* **Running Ops**
This section provides minimum, maximum, and average operation counters
for ``insert``, ``query``, ``update``, ``delete``, ``getMore``,
and ``command`` operations.
For this, ``pt-mongodb-summary`` runs the ``serverStatus`` command
5 times at regular intervals (every second).
* **Security**
This section provides information about the security settings.
For this, ``pt-mongodb-summary``, parses ``getCmdLineOpts`` output
and queries the ``admin.system.users``
and ``admin.system.roles`` collections.
* **Oplog**
This section contains details about the MongoDB operations log (oplog).
For this, ``pt-mongodb-summary`` collects statistics
from the oplog on every host in the cluster,
and returns those with the smallest ``TimeDiffHours`` value.
* **Cluster wide**
This section provides information about the number of sharded and
unsharded databases, collections, and their size.
For this, ``pt-mongodb-summary`` runs the ``listDatabases`` command
and then runs ``collStats`` for every collection in every database.
Authors
=======
Carlos Salguero

242
src/go/pt-pg-summary/README.rst
View File

@@ -1,116 +1,86 @@
pt-pg-summary
=============
**pt-pg-summary** collects information about a PostgreSQL cluster.
.. pt-pg-summary:
========================
:program:`pt-pg-summary`
========================
``pt-pg-summary`` collects information about a PostgreSQL cluster.
Usage
-----
=====
``pt-pg-summary [options] [host:[port]]``
.. code-block:: bash
Binaries
--------
Please check the `releases <https://github.com/percona/toolkit-go/releases>`_ tab to download the binaries.
pt-pg-summary [OPTIONS] [HOST:[PORT]]
Parameters
^^^^^^^^^^
Options
-------
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| Short | Long | Default | Description |
+========+=================================+=========+============================================================================+
| | ``--help`` | | Show context-sensitive help (also try ``--help-long`` and ``--help-man``). |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| | ``--version`` | | Show application version. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| | ``--databases=DATABASES`` | | Summarize this comma-separated list of databases. All if not specified. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| ``-h`` | ``--host=HOST`` | | Host to connect to. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| ``-W`` | ``--password=PASSWORD`` | | Password to use when connecting. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| ``-p`` | ``--port=PORT`` | | Port number to use for connection. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| | ``--sleep=SLEEP`` | 10 | Seconds to sleep when gathering status counters. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| ``-U`` | ``--username=USERNAME`` | | User for login if not current user. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| | ``--disable-ssl`` | true | Disable SSL for the connection. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| | ``--verbose`` | false | Show verbose log. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
| | ``--debug`` | false | Show debug information in the logs. |
+--------+---------------------------------+---------+----------------------------------------------------------------------------+
``--help``, ``--help-long``, ``--help-man``
Shows context-sensitive help. ``--help-long`` and ``--help-man`` provide more verbose output.
``--version``
Show application version and exit. |
``--databases``
Summarizes this comma-separated list of databases.
All if not specified.
``-h``, ``--host``
Host or local Unix socket for connection.
``-W``, ``--password``
Password to use when connecting. |
``-p``, ``--port``
Port number to use for connection. |
``--sleep``
Seconds to sleep when gathering status counters.
Sleeps 10 seconds if not provided.
``-U``, ``--username``
User for login if not current user.
``--disable-ssl``
Disable SSL for the connection.
Enabled by default.
``--verbose``
Show verbose log.
``--debug``
Show debug information in the logs.
.. Currently hidden
.. --list-encrypted-tables Include a list of the encrypted tables in all databases
.. --ask-pass Prompt for a password when connecting to PostgreSQL
.. --config Config file
.. --defaults-file Only read PostgreSQL options from the given file
.. --read-samples Create a report from the files found in this directory
.. --save-samples Save the data files used to generate the summary in this directory
Experimental Options
--------------------
``--list-encrypted-tables``
Include a list of the encrypted tables in all databases.
Output
^^^^^^
``--ask-pass``
Prompt for a password when connecting to PostgreSQL.
The output is grouped into these categories:
``--config``
Configuration file.
AllDatabases
Selects ``datname`` from ``pg_database`` where ``datistemplate`` is false.
``--defaults-file``
Only read PostgreSQL options from the given file.
ClusterInfo
Selects cluster information from ``pg_stat_activity``.
ConnectedClients
Counts the connected clients by selecting from ``pg_stat_activity``.
Connections
Selects ``state`` from ``pg_stat_activity`` and counts them.
Counters
Selects various counter values from ``pg_stat_database``.
DatabaseWaitEvents
Shows database wait events from ``pg_locks``, ``pg_stat_database``, ``pg_class``, and ``pg_stat_activity``.
Databases
Shows the name and size of databases from ``pg_stat_database``.
GlobalWaitEvents
Shows global wait evens from ``pg_stat_activity``.
IndexCacheHitRatio
Shows index hit ratios from ``pg_statio_user_indexes``.
PortAndDatadir
Shows port and data directory name from ``pg_settings``.
ServerVersion
Shows the value of ``server_version_num``.
Setting
Selects ``name`` and ``setting`` from ``pg_settings``.
SlaveHosts10
Selects information for PostgreSQL version 10.
SlaveHosts96
Selects information for PostgreSQL version 9.6.
TableAccess
Shows table access information by selecting from ``pg_locks``, ``pg_stat_database`` and ``pg_class``.
TableCacheHitRatio
Shows table cache hit ratio information from ``pg_statio_user_tables``.
Tablespaces
Show owner and location from ``pg_catalog.pg_tablespace``.
``--read-samples``
Create a report from the files found in this directory.
``--save-samples``
Save the data files used to generate the summary in this directory.
Output example
""""""""""""""
==============
.. code-block:: html
.. code-block:: none
##### --- Database Port and Data_Directory --- ####
+----------------------+----------------------------------------------------+
@@ -466,8 +436,80 @@ Output example
##### --- Processes start up command --- ####
No postgres process found
Minimum auth role
^^^^^^^^^^^^^^^^^
Sections
--------
This program needs to run some commands like ``getShardMap`` and to be able to run those commands
it needs to run under a user with the ``clusterAdmin`` or ``root`` built-in roles.
Output is separated into the following sections:
* **AllDatabases**
Selects ``datname`` from ``pg_database`` where ``datistemplate`` is false.
* **ClusterInfo**
Selects cluster information from ``pg_stat_activity``.
* **ConnectedClients**
Counts the connected clients by selecting from ``pg_stat_activity``.
* **Connections**
Selects ``state`` from ``pg_stat_activity`` and counts them.
* **Counters**
Selects various counter values from ``pg_stat_database``.
* **DatabaseWaitEvents**
Shows database wait events from ``pg_locks``, ``pg_stat_database``, ``pg_class``, and ``pg_stat_activity``.
* **Databases**
Shows the name and size of databases from ``pg_stat_database``.
* **GlobalWaitEvents**
Shows global wait evens from ``pg_stat_activity``.
* **IndexCacheHitRatio**
Shows index hit ratios from ``pg_statio_user_indexes``.
* **PortAndDatadir**
Shows port and data directory name from ``pg_settings``.
* **ServerVersion**
Shows the value of ``server_version_num``.
* **Setting**
Selects ``name`` and ``setting`` from ``pg_settings``.
* **SlaveHosts10**
Selects information for PostgreSQL version 10.
* **SlaveHosts96**
Selects information for PostgreSQL version 9.6.
* **TableAccess**
Shows table access information by selecting from ``pg_locks``, ``pg_stat_database`` and ``pg_class``.
* **TableCacheHitRatio**
Shows table cache hit ratio information from ``pg_statio_user_tables``.
* **Tablespaces**
Show owner and location from ``pg_catalog.pg_tablespace``.
Authors
=======
Carlos Salguero

88
src/go/pt-secure-collect/README.md
View File

@@ -1,88 +0,0 @@
# pt-secure-collect
Collect, sanitize, pack and encrypt data. By default, this program will collect the output of:
- `pt-stalk --no-stalk --iterations=2 --sleep=30 --host=$mysql-host --dest=$temp-dir --port=$mysql-port --user=$mysql-user --password=$mysql-pass`
- `pt-summary`
- `pt-mysql-summary --host=$mysql-host --port=$mysql-port --user=$mysql-user --password=$mysql-pass`
Internal variables placeholders will be replaced with the corresponding flag values. For example, `$mysql-host` will be replaced with the values specified in the `--mysql-host` flag.
Usage:
```
pt-secure-collect [<flags>] <command> [<args> ...]
```
### Global flags
|Flag|Description|
|-----|-----|
|--help|Show context-sensitive help (also try --help-long and --help-man).|
|--debug|Enable debug log level.|
### **Commands**
#### **Help command**
Show help
#### **Collect command**
Collect, sanitize, pack and encrypt data from pt-tools.
Usage:
```
pt-secure-collect collect <flags>
```
|Flag|Description|
|-----|-----|
|--bin-dir|Directory having the Percona Toolkit binaries (if they are not in PATH).|
|--temp-dir|Temporary directory used for the data collection. Default: ${HOME}/data_collection\_{timestamp}|
|--include-dir|Include this dir into the sanitized tar file|
|--config-file|Path to the config file. Default: `~/.my.cnf`|
|--mysql-host|MySQL host. Default: `127.0.0.1`|
|--mysql-port|MySQL port. Default: `3306`|
|--mysql-user|MySQL user name.|
|--mysql-password|MySQL password.|
|--ask-mysql-pass|Ask MySQL password.|
|--extra-cmd|Also run this command as part of the data collection. This parameter can be used more than once.|
|--encrypt-password|Encrypt the output file using this password.<br>If omitted, it will be asked in the command line.|
|--no-collect|Do not collect data|
|--no-sanitize|Do not sanitize data|
|--no-encrypt|Do not encrypt the output file.|
|--no-sanitize-hostnames|Do not sanitize host names.|
|--no-sanitize-queries|Do not replace queries by their fingerprints.|
|--no-remove-temp-files|Do not remove temporary files.|
#### **Decrypt command**
Decrypt an encrypted file. The password will be requested from the terminal.
Usage:
```
pt-secure-collect decrypt [flags] <input file>
```
|Flag|Description|
|-----|------|
|--outfile|Write the output to this file.<br>If omitted, the output file name will be the same as the input file, adding the `.aes` extension|
#### **Encrypt command**
Encrypt a file. The password will be requested from the terminal.
Usage:
```
pt-secure-collect encrypt [flags] <input file>
```
|Flag|Description|
|-----|------|
|--outfile|Write the output to this file.<br>If omitted, the output file name will be the same as the input file, without the `.aes` extension|
#### **Sanitize command**
Replace queries in a file by their fingerprints and obfuscate hostnames.
Usage:
```
pt-secure-collect sanitize [flags]
```
|Flag|Description|
|-----|-----|
|--input-file| Input file. If not specified, the input will be Stdin.|
|--output-file|Output file. If not specified, the input will be Stdout.|
|--no-sanitize-hostnames|Do not sanitize host names.|
|--no-sanitize-queries|Do not replace queries by their fingerprints.|

178
src/go/pt-secure-collect/README.rst Normal file
View File

@@ -0,0 +1,178 @@
============================
:program:`pt-secure-collect`
============================
NAME
====
:program:`pt-secure-collect` - collect, sanitize, pack and encrypt data.
SYNOPSIS
========
Usage
-----
::
pt-secure-data [<flags>] <command> [<args> ...]
By default, :program:`pt-secure-collect` will collect the output of:
- ``pt-stalk --no-stalk --iterations=2 --sleep=30``
- ``pt-summary``
- ``pt-mysql-summary``
Global flags
------------
.. option:: --help
Show context-sensitive help (also try --help-long and --help-man).
.. option:: --debug
Enable debug log level.
COMMANDS
========
* **Help command**
Show help
* **Collect command**
Collect, sanitize, pack and encrypt data from pt-tools. Usage:
::
pt-secure-collect collect <flags>
.. option:: --bin-dir
Directory having the Percona Toolkit binaries (if they are not in PATH).
.. option:: --temp-dir
Temporary directory used for the data collection.
Default: ``${HOME}/data_collection_{timestamp}``
.. option:: --include-dir
Include this dir into the sanitized tar file.
.. option:: --config-file
Path to the config file. Default: ``~/.my.cnf``
.. option:: --mysql-host
MySQL host. Default: ``127.0.0.1``
.. option:: --mysql-port
MySQL port. Default: ``3306``
.. option:: --mysql-user
MySQL user name.
.. option:: --mysql-password
MySQL password.
.. option:: --ask-mysql-pass
Ask MySQL password.
.. option:: --extra-cmd
Also run this command as part of the data collection. This parameter can
be used more than once.
.. option:: --encrypt-password
Encrypt the output file using this password. If omitted, it will be asked
in the command line.
.. option:: --no-collect
Do not collect data
.. option:: --no-sanitize
Do not sanitize data
.. option:: --no-encrypt
Do not encrypt the output file.
.. option:: --no-sanitize-hostnames
Do not sanitize hostnames.
.. option:: --no-sanitize-queries
Do not replace queries by their fingerprints.
.. option:: --no-remove-temp-files
Do not remove temporary files.
* **Decrypt command**
Decrypt an encrypted file. The password will be requested from the
terminal. Usage:
::
pt-secure-collect decrypt [flags] <input file>
.. option:: --outfile
Write the output to this file. If omitted, the output file
name will be the same as the input file, adding the ``.aes`` extension.
* **Encrypt command**
Encrypt a file. The password will be requested from the terminal. Usage:
::
pt-secure-collect encrypt [flags] <input file>
.. option:: --outfile
Write the output to this file. If omitted, the output file
name will be the same as the input file, without the ``.aes`` extension.
* **Sanitize command**
Replace queries in a file by their fingerprints and obfuscate hostnames.
Usage:
::
pt-secure-collect sanitize [flags]
.. option:: --input-file
Input file. If not specified, the input will be Stdin.
.. option:: --output-file
Output file. If not specified, the input will be Stdout.
.. option:: --no-sanitize-hostnames
Do not sanitize host names.
.. option:: --no-sanitize-queries
Do not replace queries by their fingerprints.
Authors
=======
Carlos Salguero