Compare commits

...

41 Commits

Author SHA1 Message Date
Max Dudin
6596920085 Merge pull request #452 from percona/CLOUD-535
PT-1865, PT-1866 Added pt-k8s-debug-collector
2020-08-05 18:41:48 +03:00
Max Dudin
08c9621c14 PT-1865 fix typo 2020-08-05 18:00:30 +03:00
Max Dudin
8ed3b1cdc9 CLOUD-535 Updated Gopkg.lock 2020-08-05 14:01:34 +03:00
Max Dudin
61ba3c729e CLOUD-535 gopath added, changed suumary func name 2020-08-05 13:59:46 +03:00
Max Dudin
212d2032ed PT-1865 small changes 2020-08-04 00:00:54 +03:00
Max Dudin
9b90408451 PT-1865 Changed tool name 2020-08-03 20:39:12 +03:00
Max Dudin
0d2a2f8fdc PT-1865 Updated README 2020-07-17 14:29:17 +03:00
Max Dudin
309bb24f8a PT-1865 Added pt-mysql-summury and pt-mobgodb-summary 2020-07-17 14:28:13 +03:00
Max Dudin
57c7769c5b PT-1865 Update readme, rename tool 2020-07-07 12:06:01 +03:00
Max Dudin
dbb1982321 PT-1865 Clean-up 2020-07-06 12:19:32 +03:00
Max Dudin
36e9a2f07f PT-1865 add README file 2020-07-03 12:12:45 +03:00
Max Dudin
118110a671 PT-1865 small changes 2020-07-03 12:02:30 +03:00
Max Dudin
faffa70867 PT-1865 Remove archive package 2020-07-01 10:53:28 +03:00
Max Dudin
a65dbefbd9 CLOUD-535 Add contexts to errors, small fixes 2020-06-19 16:22:24 +03:00
Max Dudin
1d6da2fdca CLOUD-535 Rework archive and dumper 2020-06-19 15:53:55 +03:00
Max Dudin
136e506549 CLOUD-535 Add cluster debug collector 2020-06-17 16:14:55 +03:00
Anton Kucherov
0cd918a835 Merge pull request #396 from percona/PMM-3880-p99-for-qan
PMM-3880 qan p99.
2019-04-25 10:50:30 +03:00
Anton Kucherov
e4f3b1b7d1 PMM-3880 fix tests. 2019-04-25 01:30:44 +03:00
Anton Kucherov
0252710c45 PMM-3880 qan p99. 2019-04-24 16:13:07 +03:00
Carlos Salguero
106b1edebb Merge pull request #394 from percona/PT-1715
PT-1715 Updated pt-upgrade docucumentation
2019-04-11 15:02:25 -03:00
Carlos Salguero
9a57f5a479 PT-1715 Updated pt-upgrade docucumentation
Added tcpdump into the --type parameter
2019-04-11 15:00:54 -03:00
Carlos Salguero
a9eeeb6fb1 Merge pull request #392 from percona/PT-1706
PT-1706 Made mongo tools compile with Go 1.12
2019-04-03 11:48:40 -03:00
Carlos Salguero
3b88222428 PT-1706 Added go 1.12.x to travis.yml 2019-04-03 10:40:38 -03:00
Carlos Salguero
12c866c9a4 PT-1706 Updated travis.yml 2019-04-03 10:12:18 -03:00
Carlos Salguero
cdea59e688 PT-1706 Made mongo tools compile with Go 1.12
- Removed t.parallel from tests
- Added lock to stats  pkg
- Fixed timeout channels usage in stats pkg
- Removed non-compatible dependency (badfix/slice) and its dependencies
- Made code improvements for linter
2019-03-31 04:20:31 -03:00
Carlos Salguero
3f29746483 Merge pull request #391 from percona/PT-1575
PT-1575 Fix pt-mysql-summary to recognize PXC
2019-03-20 12:33:54 -03:00
Carlos Salguero
06582008fd PT-1575 Fix pt-mysql-summary to recognize PXC 2019-03-20 12:32:25 -03:00
Eduardo Casarero
39aa1f88f0 Update README.md 2019-03-15 18:37:49 -03:00
Viacheslav Sarzhan
033fd5df9d Merge pull request #389 from hors/PT-1701
PT-1701 build Percona Toolkit 3.0 for RHEL8
2019-03-11 22:12:40 +02:00
Viacheslav Sarzhan
8d64a845d4 PT-1701 build Percona Toolkit 3.0 for RHEL8 2019-03-11 21:53:28 +02:00
Carlos Salguero
e9a9e787ea Merge pull request #387 from percona/PT-1114-new-fix
pt-table-checksum fails when table is empty
2019-03-04 11:10:04 -03:00
Carlos Salguero
b7905fcb74 Updated changelog to include PT-1633 2019-01-29 09:02:54 -03:00
Carlos Salguero
4d87947852 Merge pull request #386 from ruleant/patch-1
PT-1633 fix incorrect parsing of a variable with number + K,M,G,T
2019-01-29 08:58:33 -03:00
Carlos Salguero
424a48518c PT-1114 test fix 2019-01-11 15:43:38 -03:00
Carlos Salguero
a9381ba2ea Merge pull request #383 from percona/release-3.0.13
RM-422 Updated changelog & library version
2019-01-10 09:26:48 -03:00
Dieter Adriaenssens
5487319fd1 PT-1633 fix incorrect parsing of a variable with number + K,M,G,T
This fixes bug https://jira.percona.com/browse/PT-1633

A regexp to convert numbers matches any variable ending in a number followed by character K, M, G or T, even if the variable contains other characters, fe. 'ibtmp1:12M:autoextend:max:5G'.
This fix changes the regexp so it matches only those variables that contain only a number followed by K,M,G or T
2019-01-09 16:37:40 +01:00
Roma Novikov
a59993182f Merge pull request #384 from fiowro/RN-3.0.13
PT 3.0.13 release notes
2019-01-09 16:02:41 +02:00
Dima
8f43937f18 Remove PT-1114 2019-01-09 16:24:12 +03:00
Carlos Salguero
7748cc765d Revert "Merge pull request #380 from percona/PT-1114"
This reverts commit bcbc175d0c, reversing
changes made to cf5c661d46.
2019-01-08 14:37:05 -03:00
Roma Novikov
4c28e71092 Merge pull request #376 from percona/PT-157
PT-157 Specifying the index to use for pt-archiver ignores --primary-key-only
2019-01-03 11:50:15 +02:00
Dima
6445f76166 PT 3.0.13 release notes 2019-01-02 16:18:52 +03:00
24 changed files with 913 additions and 164 deletions

View File

@@ -3,6 +3,7 @@ language: go
go:
- 1.9.x
- 1.10.x
- 1.12.x
services:
- docker
@@ -31,7 +32,7 @@ before_script:
- dep ensure
script:
- go test -timeout 1m ./src/...
- go test -timeout 20m ./src/...
allow_failures:
- tip

View File

@@ -1,5 +1,7 @@
Changelog for Percona Toolkit
* Fixed bug PT-1633: Fix incorrect parsing of a variable with number + K,M,G,T (Thanks Dieter Adriaenssens)
v3.0.13 released 2018-12-28
* Fixed bug PT-1673: Fix pt-show-grants for MariaDB 10+ (thanks Tim Birkett)

270
Gopkg.lock generated
View File

@@ -2,138 +2,206 @@
[[projects]]
digest = "1:b856d8248663c39265a764561c1a1a149783f6cc815feb54a1f3a591b91f6eca"
name = "github.com/Masterminds/semver"
packages = ["."]
pruneopts = ""
revision = "c7af12943936e8c39859482e61f0574c2fd7fc75"
version = "v1.4.2"
[[projects]]
digest = "1:f82b8ac36058904227087141017bb82f4b0fc58272990a4cdae3e2d6d222644e"
name = "github.com/StackExchange/wmi"
packages = ["."]
pruneopts = ""
revision = "5d049714c4a64225c3c79a7cf7d02f7fb5b96338"
version = "1.0.0"
[[projects]]
digest = "1:15d017551627c8bb091bde628215b2861bed128855343fdd570c62d08871f6e1"
name = "github.com/alecthomas/kingpin"
packages = ["."]
pruneopts = ""
revision = "947dcec5ba9c011838740e680966fd7087a71d0d"
version = "v2.2.6"
[[projects]]
branch = "master"
digest = "1:a74730e052a45a3fab1d310fdef2ec17ae3d6af16228421e238320846f2aaec8"
name = "github.com/alecthomas/template"
packages = [
".",
"parse"
"parse",
]
pruneopts = ""
revision = "a0175ee3bccc567396460bf5acd36800cb10c49c"
[[projects]]
branch = "master"
digest = "1:8483994d21404c8a1d489f6be756e25bfccd3b45d65821f25695577791a08e68"
name = "github.com/alecthomas/units"
packages = ["."]
pruneopts = ""
revision = "2efee857e7cfd4f3d0138cc3cbb1b4966962b93a"
[[projects]]
branch = "master"
name = "github.com/bradfitz/slice"
packages = ["."]
revision = "d9036e2120b5ddfa53f3ebccd618c4af275f47da"
[[projects]]
digest = "1:03edf882162b807cdf1bc558c66226167fa2f8eb44359eac2eeb3794a91cb168"
name = "github.com/go-ini/ini"
packages = ["."]
revision = "ace140f73450505f33e8b8418216792275ae82a7"
version = "v1.35.0"
pruneopts = ""
revision = "c85607071cf08ca1adaf48319cd1aa322e81d8c1"
version = "v1.42.0"
[[projects]]
digest = "1:fd2ee29b7807f198e72dbd6371267b34d05aa83151c8c81b2ade14854e50f4ee"
name = "github.com/go-logr/logr"
packages = ["."]
pruneopts = ""
revision = "d18fcbf02861580d05a1f23601145b272c4e7b4b"
version = "v0.2.0"
[[projects]]
digest = "1:b6581f9180e0f2d5549280d71819ab951db9d511478c87daca95669589d505c0"
name = "github.com/go-ole/go-ole"
packages = [
".",
"oleutil"
"oleutil",
]
revision = "a41e3c4b706f6ae8dfbff342b06e40fa4d2d0506"
version = "v1.2.1"
pruneopts = ""
revision = "97b6244175ae18ea6eef668034fd6565847501c9"
version = "v1.2.4"
[[projects]]
digest = "1:d69d2ba23955582a64e367ff2b0808cdbd048458c178cea48f11ab8c40bd7aea"
name = "github.com/gogo/protobuf"
packages = [
"proto",
"sortkeys",
]
pruneopts = ""
revision = "5628607bb4c51c3157aacc3a50f0ab707582b805"
version = "v1.3.1"
[[projects]]
digest = "1:530233672f656641b365f8efb38ed9fba80e420baff2ce87633813ab3755ed6d"
name = "github.com/golang/mock"
packages = ["gomock"]
revision = "c34cdb4725f4c3844d095133c6e40e448b86589b"
version = "v1.1.1"
pruneopts = ""
revision = "51421b967af1f557f93a59e0057aaf15ca02e29c"
version = "v1.2.0"
[[projects]]
digest = "1:16ecf9e89b8b1310d9566a53484c31c5241bb47c32162eba780b46c0dfb58fef"
name = "github.com/google/gofuzz"
packages = ["."]
pruneopts = ""
revision = "db92cf7ae75e4a7a28abc005addab2b394362888"
version = "v1.1.0"
[[projects]]
branch = "master"
digest = "1:b759103c9b4135568253c17d2866064cde398e93764b611caabf5aa8e3059685"
name = "github.com/hashicorp/go-version"
packages = ["."]
revision = "23480c0665776210b5fbbac6eaaee40e3e6a96b7"
pruneopts = ""
revision = "d40cf49b3a77bba84a7afdbd7f1dc295d114efb1"
[[projects]]
branch = "master"
digest = "1:f81c8d7354cc0c6340f2f7a48724ee6c2b3db3e918ecd441c985b4d2d97dd3e7"
name = "github.com/howeyc/gopass"
packages = ["."]
pruneopts = ""
revision = "bf9dde6d0d2c004a008c27aaee91170c786f6db8"
[[projects]]
digest = "1:0f51cee70b0d254dbc93c22666ea2abf211af81c1701a96d04e2284b408621db"
name = "github.com/konsorten/go-windows-terminal-sequences"
packages = ["."]
pruneopts = ""
revision = "f55edac94c9bbba5d6182a4be46d86a2c9b5b50e"
version = "v1.0.2"
[[projects]]
digest = "1:3108ec0946181c60040ff51b811908f89d03e521e2b4ade5ef5c65b3c0e911ae"
name = "github.com/kr/pretty"
packages = ["."]
pruneopts = ""
revision = "73f6ac0b30a98e433b289500d779f50c1a6f0712"
version = "v0.1.0"
[[projects]]
digest = "1:11b056b4421396ab14e384ab8ab8c2079b03f1e51aa5eb4d9b81f9e0d1aa8fbf"
name = "github.com/kr/text"
packages = ["."]
pruneopts = ""
revision = "e2ffdb16a802fe2bb95e2e35ff34f0e53aeef34f"
version = "v0.1.0"
[[projects]]
digest = "1:0093a7c66d5b9e0cdaf4be5c20e0a9b889d1d839148eeed1d587e99b4cfd90ff"
name = "github.com/mattn/go-shellwords"
packages = ["."]
revision = "02e3cf038dcea8290e44424da473dd12be796a8a"
version = "v1.0.3"
pruneopts = ""
revision = "a72fbe27a1b0ed0df2f02754945044ce1456608b"
version = "v1.0.5"
[[projects]]
digest = "1:a067513044dc491395a58f56f39cedddb5ad35789b832b570c283a64d712f81b"
name = "github.com/montanaflynn/stats"
packages = ["."]
pruneopts = ""
revision = "eeaced052adbcfeea372c749c281099ed7fdaa38"
version = "0.2.0"
[[projects]]
branch = "master"
digest = "1:020f67c818cb9c3fdc77d92c5744fb2d5b90930280cc43311ba43c6459fd0b98"
name = "github.com/pborman/getopt"
packages = [
".",
"v2"
"v2",
]
revision = "7148bc3a4c3008adfcab60cbebfd0576018f330b"
pruneopts = ""
revision = "fd6d657c3083960b8d604310c34a621ec24bdc6a"
[[projects]]
branch = "master"
digest = "1:7a840dbacabd648e5b511010dea5da9eed99030dd185b3c7c7195fdadb3051a8"
name = "github.com/percona/go-mysql"
packages = ["query"]
revision = "82ed67a1d0f1779cd60a025c54e0827da0c0838b"
pruneopts = ""
revision = "f5cfaf6a5e55b754b7b106f4488e1bc24cb8c2d6"
[[projects]]
digest = "1:16b4510ba61ab0bb7a4e694ea6396a7b2879f5fabb21e93066e182691f790173"
name = "github.com/percona/pmgo"
packages = [
".",
"pmgomock"
"pmgomock",
]
pruneopts = ""
revision = "497d06e28f910fbe26d5d60f59d36284a6901c6f"
version = "0.5.2"
[[projects]]
digest = "1:1d7e1867c49a6dd9856598ef7c3123604ea3daabf5b83f303ff457bcbc410b1d"
name = "github.com/pkg/errors"
packages = ["."]
revision = "645ef00459ed84a119197bfb8d8205042c6df63d"
version = "v0.8.0"
pruneopts = ""
revision = "ba968bfe8b2f7e042a574c888954fccecfa385b4"
version = "v0.8.1"
[[projects]]
digest = "1:7f569d906bdd20d906b606415b7d794f798f91a62fcfb6a4daa6d50690fb7a3f"
name = "github.com/satori/go.uuid"
packages = ["."]
pruneopts = ""
revision = "f58768cc1a7a7e77a3bd49e98cdd21419399b6a3"
version = "v1.2.0"
[[projects]]
digest = "1:d77a85cf43b70ae61fa2543d402d782b40dca0f5f41413839b5f916782b0fab9"
name = "github.com/shirou/gopsutil"
packages = [
"cpu",
@@ -141,52 +209,97 @@
"internal/common",
"mem",
"net",
"process"
"process",
]
revision = "fc04d2dd9a512906a2604242b35275179e250eda"
version = "v2.18.03"
pruneopts = ""
revision = "6c6abd6d1666d6b27f1c261e0f850441ba22aa3a"
version = "v2.19.02"
[[projects]]
branch = "master"
digest = "1:99c6a6dab47067c9b898e8c8b13d130c6ab4ffbcc4b7cc6236c2cd0b1e344f5b"
name = "github.com/shirou/w32"
packages = ["."]
pruneopts = ""
revision = "bb4de0191aa41b5507caa14b0650cdbddcd9280b"
[[projects]]
digest = "1:b73fe282e350b3ef2c71d8ff08e929e0b9670b1bb5b7fde1d3c1b4cd6e6dc8b1"
name = "github.com/sirupsen/logrus"
packages = ["."]
revision = "d682213848ed68c0a260ca37d6dd5ace8423f5ba"
version = "v1.0.4"
[[projects]]
branch = "master"
name = "go4.org"
packages = ["reflectutil"]
revision = "9599cf28b011184741f249bd9f9330756b506cbc"
pruneopts = ""
revision = "dae0fa8d5b0c810a8ab733fbd5510c7cae84eca4"
version = "v1.4.0"
[[projects]]
branch = "master"
digest = "1:36ef1d8645934b1744cc7d8726e00d3dd9d8d84c18617bf7367a3a6d532f3370"
name = "golang.org/x/crypto"
packages = ["ssh/terminal"]
revision = "d6449816ce06963d9d136eee5a56fca5b0616e7e"
pruneopts = ""
revision = "a5d413f7728c81fb97d96a2b722368945f651e78"
[[projects]]
branch = "master"
digest = "1:adcb9e84ce154ef1d45851b57c40f8a211db3e36373a65b7c4f10c79b7428718"
name = "golang.org/x/net"
packages = ["context"]
revision = "d41e8174641f662c5a2d1c7a5f9e828788eb8706"
packages = [
"context",
"http/httpguts",
"http2",
"http2/hpack",
"idna",
]
pruneopts = ""
revision = "74de082e2cca95839e88aa0aeee5aadf6ce7710f"
[[projects]]
branch = "master"
digest = "1:1b0de777d8ddd63356d5a4d76799ea8f47e811aa9dda85ddc72b2a061c799cc9"
name = "golang.org/x/sys"
packages = [
"unix",
"windows"
"windows",
]
revision = "3ccc7e5779793fd54564baf60c51bf017955e0ba"
pruneopts = ""
revision = "9eb1bfa1ce65ae8a6ff3114b0aaf9a41a6cf3560"
[[projects]]
digest = "1:fccda34e4c58111b1908d8d69bf8d57c41c8e2542bc18ec8cd38c4fa21057f71"
name = "golang.org/x/text"
packages = [
"collate",
"collate/build",
"internal/colltab",
"internal/gen",
"internal/language",
"internal/language/compact",
"internal/tag",
"internal/triegen",
"internal/ucd",
"language",
"secure/bidirule",
"transform",
"unicode/bidi",
"unicode/cldr",
"unicode/norm",
"unicode/rangetable",
]
pruneopts = ""
revision = "23ae387dee1f90d29a23c0e87ee0b46038fbed0e"
version = "v0.3.3"
[[projects]]
digest = "1:75fb3fcfc73a8c723efde7777b40e8e8ff9babf30d8c56160d01beffea8a95a6"
name = "gopkg.in/inf.v0"
packages = ["."]
pruneopts = ""
revision = "d2d2541c53f18d2a059457998ce2876cc8e67cbf"
version = "v0.9.1"
[[projects]]
branch = "v2"
digest = "1:f54ba71a035aac92ced3e902d2bff3734a15d1891daff73ec0f90ef236750139"
name = "gopkg.in/mgo.v2"
packages = [
".",
@@ -194,19 +307,92 @@
"dbtest",
"internal/json",
"internal/sasl",
"internal/scram"
"internal/scram",
]
revision = "3f83fa5005286a7fe593b055f0d7771a7dce4655"
pruneopts = ""
revision = "9856a29383ce1c59f308dd1cf0363a79b5bef6b5"
[[projects]]
branch = "v2"
digest = "1:61a650a53e5e865a91ae9581f02990a4b6e3afcb8d280f19b1e67a3c284944e6"
name = "gopkg.in/tomb.v2"
packages = ["."]
pruneopts = ""
revision = "d5d1b5820637886def9eef33e03a27a9f166942c"
[[projects]]
digest = "1:a249e341b9bf261a982ab262c69f08223e839302d0a21cfe6e00f2ef2e8695a2"
name = "k8s.io/api"
packages = ["core/v1"]
pruneopts = ""
revision = "f822fed505d4c9dd4eb2c5f4ca2f4c49c19ea394"
version = "v0.18.6"
[[projects]]
digest = "1:74eeecf1188777314a92348555adcb977912d530269130143daf7fc0e80bb512"
name = "k8s.io/apimachinery"
packages = [
"pkg/api/resource",
"pkg/apis/meta/v1",
"pkg/conversion",
"pkg/conversion/queryparams",
"pkg/fields",
"pkg/labels",
"pkg/runtime",
"pkg/runtime/schema",
"pkg/selection",
"pkg/types",
"pkg/util/errors",
"pkg/util/intstr",
"pkg/util/json",
"pkg/util/naming",
"pkg/util/net",
"pkg/util/runtime",
"pkg/util/sets",
"pkg/util/validation",
"pkg/util/validation/field",
"pkg/watch",
"third_party/forked/golang/reflect",
]
pruneopts = ""
revision = "fbe88689c3c2735e949f67884a4f58cb99379159"
version = "v0.17.9"
[[projects]]
digest = "1:5ad0a3bf1b13f9b8bd99f4079c635cb813d87b70db65b98fe5503762e1d39735"
name = "k8s.io/klog"
packages = ["."]
pruneopts = ""
revision = "b5c3182dac44f851522e32c97c86ac32755c296d"
version = "v2.3.0"
[solve-meta]
analyzer-name = "dep"
analyzer-version = 1
inputs-digest = "35a93d0003438bc707c085101ccfca910f349d201cd01edc986d0ed64ef2a12f"
input-imports = [
"github.com/Masterminds/semver",
"github.com/alecthomas/kingpin",
"github.com/go-ini/ini",
"github.com/golang/mock/gomock",
"github.com/hashicorp/go-version",
"github.com/howeyc/gopass",
"github.com/kr/pretty",
"github.com/mattn/go-shellwords",
"github.com/montanaflynn/stats",
"github.com/pborman/getopt",
"github.com/pborman/getopt/v2",
"github.com/percona/go-mysql/query",
"github.com/percona/pmgo",
"github.com/percona/pmgo/pmgomock",
"github.com/pkg/errors",
"github.com/satori/go.uuid",
"github.com/shirou/gopsutil/process",
"github.com/sirupsen/logrus",
"golang.org/x/crypto/ssh/terminal",
"gopkg.in/mgo.v2",
"gopkg.in/mgo.v2/bson",
"gopkg.in/mgo.v2/dbtest",
"k8s.io/api/core/v1",
]
solver-name = "gps-cdcl"
solver-version = 1

View File

@@ -24,10 +24,6 @@
name = "github.com/Masterminds/semver"
version = "1.4.0"
[[constraint]]
branch = "master"
name = "github.com/bradfitz/slice"
[[constraint]]
name = "github.com/golang/mock"
version = "1.0.0"

View File

@@ -1,4 +1,5 @@
# Percona Toolkit
[![CLA assistant](https://cla-assistant.percona.com/readme/badge/percona/percona-toolkit)](https://cla-assistant.percona.com/percona/percona-toolkit)
*Percona Toolkit* is a collection of advanced command-line tools used by
[Percona](http://www.percona.com/) support staff to perform a variety of

View File

@@ -3244,7 +3244,7 @@ sub _process_val {
$val =~ s/\s*#.*//;
}
if ( my ($num, $factor) = $val =~ m/(\d+)([KMGT])b?$/i ) {
if ( my ($num, $factor) = $val =~ m/^(\d+)([KMGT])b?$/i ) {
my %factor_for = (
k => 1_024,
m => 1_048_576,

View File

@@ -2358,10 +2358,9 @@ report_mysql_summary () {
section_percona_server_features "$dir/mysql-variables"
section "Percona XtraDB Cluster"
local has_wsrep="$(get_var "wsrep_on" "$dir/mysql-variables")"
local has_wsrep=$($CMD_MYSQL $EXT_ARGV -ss -e 'show session variables like "%wsrep_on%";' | cut -f2 | grep -i "on")
if [ -n "${has_wsrep:-""}" ]; then
local wsrep_on="$(feat_on "$dir/mysql-variables" "wsrep_on")"
if [ "${wsrep_on:-""}" = "Enabled" ]; then
if [ "${has_wsrep:-""}" = "ON" ]; then
section_percona_xtradb_cluster "$dir/mysql-variables" "$dir/mysql-status"
else
name_val "wsrep_on" "OFF"

View File

@@ -6292,6 +6292,7 @@ use strict;
use warnings FATAL => 'all';
use English qw(-no_match_vars);
use constant PTDEBUG => $ENV{PTDEBUG} || 0;
use IndexLength;
use Data::Dumper;
$Data::Dumper::Indent = 1;
@@ -6690,11 +6691,11 @@ sub row_estimate {
sub can_nibble {
my (%args) = @_;
my @required_args = qw(Cxn tbl chunk_size OptionParser TableParser);
my @required_args = qw(Cxn tbl chunk_size OptionParser TableParser Quoter);
foreach my $arg ( @required_args ) {
die "I need a $arg argument" unless $args{$arg};
}
my ($cxn, $tbl, $chunk_size, $o) = @args{@required_args};
my ($cxn, $tbl, $chunk_size, $o, $q) = @args{@required_args};
my $where = $o->has('where') ? $o->get('where') : '';
@@ -6705,13 +6706,30 @@ sub can_nibble {
);
if ( !$where ) {
$mysql_index = undef;
}
$mysql_index = undef;
}
my $chunk_size_limit = $o->get('chunk-size-limit') || 1;
my $one_nibble = !defined $args{one_nibble} || $args{one_nibble}
? $row_est <= $chunk_size * $chunk_size_limit
: 0;
if ($mysql_index) {
my $idx_len = IndexLength->new(Quoter => $q);
my ($key_len, $key) = $idx_len->index_length(
Cxn => $args{Cxn},
tbl => $tbl,
index => $mysql_index,
n_index_cols => $o->get('chunk-index-columns'),
);
if ( !$key || !$key_len || lc($key) ne lc($mysql_index)) {
$one_nibble = 1;
}
} else {
$one_nibble = 1;
}
PTDEBUG && _d('One nibble:', $one_nibble ? 'yes' : 'no');
if ( $args{resume}
@@ -9660,14 +9678,14 @@ sub _get_first_values {
. "WHERE " . join(' AND ', @where)
. " ORDER BY $index_columns "
. "LIMIT 1 /*key_len*/"; # only need 1 row
PTDEBUG && _d($sql);
PTDEBUG && _d("_get_first_values: $sql");
my $vals = $cxn->dbh()->selectrow_arrayref($sql);
return $vals;
}
sub _make_range_query {
my ($self, %args) = @_;
my @required_args = qw(tbl index n_index_cols vals);
my @required_args = qw(tbl index n_index_cols);
foreach my $arg ( @required_args ) {
die "I need a $arg argument" unless $args{$arg};
}
@@ -9682,13 +9700,11 @@ sub _make_range_query {
if ( $n_index_cols > 1 ) {
foreach my $n ( 0..($n_index_cols - 2) ) {
my $col = $index_cols->[$n];
my $val = $vals->[$n];
push @where, $q->quote($col) . " = ?";
}
}
my $col = $index_cols->[$n_index_cols - 1];
my $val = $vals->[-1]; # should only be as many vals as cols
push @where, $q->quote($col) . " >= ?";
my $sql = "EXPLAIN SELECT /*!40001 SQL_NO_CACHE */ * "

View File

@@ -11172,6 +11172,7 @@ Type of log files. Valid types are:
slowlog MySQL slow log
genlog MySQL general log
binlog MySQL binary log (converted by mysqlbinlog)
tcpdump TCP dump file generated by tcpdump command
rawlog Custom log with one SQL statement per line
=item --upgrade-table

View File

@@ -1,3 +1,6 @@
%undefine _missing_build_ids_terminate_build
%define debug_package %{nil}
Name: percona-toolkit
Summary: Advanced MySQL and system command-line tools
Version: %{version}
@@ -9,6 +12,8 @@ URL: http://www.percona.com/software/percona-toolkit/
Source: percona-toolkit-%{version}.tar.gz
BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root
BuildArch: x86_64
BuildRequires: perl(ExtUtils::MakeMaker) make
Requires: perl(DBI) >= 1.13, perl(DBD::mysql) >= 1.0, perl(Time::HiRes), perl(IO::Socket::SSL), perl(Digest::MD5), perl(Term::ReadKey)
AutoReq: no

26
docs/rn.3-0-13.txt Normal file
View File

@@ -0,0 +1,26 @@
v3.0.13 released 2019-01-09
===========================
Improvements
* :jirabug:`PT-1340`: ``pt-stalk`` now doesn't call ``mysqladmin debug`` command
by default to avoid flooding in the error log.
``CMD_MYSQLADMIN="mysqladmin debug"`` environment variable reverts
``pt-stalk`` to the previous way of operation.
* :jirabug:`PT-1637`: A new ``--fail-on-stopped-replication`` option allows
``pt-table-checksum`` to detect failing slave nodes.
Fixed bugs
* :jirabug:`PT-1673`: ``pt-show-grants`` was incompatible with MariaDB 10+
(thanks `Tim Birkett <https://github.com/pysysops>`_)
* :jirabug:`PT-1638`: ``pt-online-schema-change`` was erroneously taking MariaDB
10.x for MySQL 8.0 and rejecting to work with it to avoid the upstream bug
`#89441 <https://bugs.mysql.com/bug.php?id=89441>`_ scope.
* :jirabug:`PT-1616`: ``pt-table-checksum`` failed to resume on large tables
with binary strings containing invalid UTF-8 characters.
* :jirabug:`PT-1573`: ``pt-query-digest`` didn't work in case of
``log_timestamps = SYSTEM`` my.cnf option.
* :jirabug:`PT-157`: Specifying a non-primary key index with the ``i`` part of
the ``--source`` argument made ``pt-archiver`` to ignore the
``--primary-key-only`` option presence.

View File

@@ -26,6 +26,7 @@ use strict;
use warnings FATAL => 'all';
use English qw(-no_match_vars);
use constant PTDEBUG => $ENV{PTDEBUG} || 0;
use IndexLength;
use Data::Dumper;
$Data::Dumper::Indent = 1;
@@ -487,11 +488,11 @@ sub row_estimate {
sub can_nibble {
my (%args) = @_;
my @required_args = qw(Cxn tbl chunk_size OptionParser TableParser);
my @required_args = qw(Cxn tbl chunk_size OptionParser TableParser Quoter);
foreach my $arg ( @required_args ) {
die "I need a $arg argument" unless $args{$arg};
}
my ($cxn, $tbl, $chunk_size, $o) = @args{@required_args};
my ($cxn, $tbl, $chunk_size, $o, $q) = @args{@required_args};
my $where = $o->has('where') ? $o->get('where') : '';
@@ -502,6 +503,23 @@ sub can_nibble {
where => $where,
);
my $can_get_keys;
if ($mysql_index) {
my $idx_len = IndexLength->new(Quoter => $q);
my ($key_len, $key) = $idx_len->index_length(
Cxn => $args{Cxn},
tbl => $tbl,
index => $mysql_index,
n_index_cols => $o->get('chunk-index-columns'),
);
if ( !$key || !$key_len || lc($key) ne lc($mysql_index)) {
$can_get_keys = 0;
} else {
$can_get_keys = 1;
}
}
# MySQL's chosen index is only something we should prefer
# if --where is used. Else, we can chose our own index
# and disregard the MySQL index from the row estimate.
@@ -521,6 +539,10 @@ sub can_nibble {
my $one_nibble = !defined $args{one_nibble} || $args{one_nibble}
? $row_est <= $chunk_size * $chunk_size_limit
: 0;
if (!$can_get_keys) {
$one_nibble = 1;
}
PTDEBUG && _d('One nibble:', $one_nibble ? 'yes' : 'no');
# Special case: we're resuming and there's no boundaries, so the table

View File

@@ -52,6 +52,7 @@ func LoadBson(filename string, destination interface{}) error {
if err != nil {
return err
}
defer file.Close()
buf, err := ioutil.ReadAll(file)
if err != nil {

View File

@@ -57,7 +57,7 @@ func NewProfiler(iterator pmgo.IterManager, filters []filter.Filter, ticker <-ch
// internal
docsChan: make(chan proto.SystemProfile, DocsBufferSize),
timeoutsChan: nil,
timeoutsChan: make(chan time.Time),
keyFilters: []string{"^shardVersion$", "^\\$"},
}
}
@@ -97,9 +97,6 @@ func (p *Profile) Stop() {
func (p *Profile) TimeoutsChan() <-chan time.Time {
p.lock.Lock()
defer p.lock.Unlock()
if p.timeoutsChan == nil {
p.timeoutsChan = make(chan time.Time)
}
return p.timeoutsChan
}
@@ -129,14 +126,15 @@ func (p *Profile) getDocs() {
for p.iterator.Next(&doc) || p.iterator.Timeout() {
if p.iterator.Timeout() {
if p.timeoutsChan != nil {
p.timeoutsChan <- time.Now().UTC()
select {
case p.timeoutsChan <- time.Now().UTC():
default:
}
continue
}
valid := true
for _, filter := range p.filters {
if filter(doc) == false {
if !filter(doc) {
valid = false
break
}

View File

@@ -8,8 +8,9 @@ import (
"time"
"github.com/montanaflynn/stats"
"github.com/percona/percona-toolkit/src/go/mongolib/proto"
"gopkg.in/mgo.v2/bson"
"github.com/percona/percona-toolkit/src/go/mongolib/proto"
)
type StatsError struct {
@@ -91,6 +92,7 @@ func (s *Stats) Add(doc proto.SystemProfile) error {
qiac.Count++
// docsExamined is renamed from nscannedObjects in 3.2.0.
// https://docs.mongodb.com/manual/reference/database-profiler/#system.profile.docsExamined
s.Lock()
if doc.NscannedObjects > 0 {
qiac.NScanned = append(qiac.NScanned, float64(doc.NscannedObjects))
} else {
@@ -105,14 +107,15 @@ func (s *Stats) Add(doc proto.SystemProfile) error {
if qiac.LastSeen.IsZero() || qiac.LastSeen.Before(doc.Ts) {
qiac.LastSeen = doc.Ts
}
s.Unlock()
return nil
}
// Queries returns all collected statistics
func (s *Stats) Queries() Queries {
s.RLock()
defer s.RUnlock()
s.Lock()
defer s.Unlock()
keys := GroupKeys{}
for key := range s.queryInfoAndCounters {
@@ -244,6 +247,7 @@ type Statistics struct {
Max float64
Avg float64
Pct95 float64
Pct99 float64
StdDev float64
Median float64
}
@@ -323,6 +327,7 @@ func calcStats(samples []float64) Statistics {
s.Max, _ = stats.Max(samples)
s.Avg, _ = stats.Mean(samples)
s.Pct95, _ = stats.PercentileNearestRank(samples, 95)
s.Pct99, _ = stats.PercentileNearestRank(samples, 99)
s.StdDev, _ = stats.StandardDeviation(samples)
s.Median, _ = stats.Median(samples)
return s

View File

@@ -188,8 +188,6 @@ func TestStatsSingle(t *testing.T) {
for _, file := range files {
f := file.Name()
t.Run(f, func(t *testing.T) {
t.Parallel()
doc := proto.SystemProfile{}
err = tutil.LoadBson(dir+f, &doc)
if err != nil {

View File

@@ -1,10 +1,10 @@
package util
import (
"sort"
"strings"
"time"
"github.com/bradfitz/slice"
"github.com/percona/percona-toolkit/src/go/mongolib/proto"
"github.com/percona/pmgo"
"github.com/pkg/errors"
@@ -58,9 +58,7 @@ func GetReplicasetMembers(dialer pmgo.Dialer, di *pmgo.DialInfo) ([]proto.Member
if serverStatus, err := GetServerStatus(dialer, di, m.Name); err == nil {
m.ID = serverStatus.Pid
m.StorageEngine = serverStatus.StorageEngine
if cmdOpts.Parsed.Sharding.ClusterRole == "" {
m.StateStr = m.StateStr
} else {
if cmdOpts.Parsed.Sharding.ClusterRole != "" {
m.StateStr = cmdOpts.Parsed.Sharding.ClusterRole + "/" + m.StateStr
}
m.StateStr = strings.ToUpper(m.StateStr)
@@ -75,7 +73,7 @@ func GetReplicasetMembers(dialer pmgo.Dialer, di *pmgo.DialInfo) ([]proto.Member
members = append(members, member)
}
slice.Sort(members, func(i, j int) bool { return members[i].Name < members[j].Name })
sort.Slice(members, func(i, j int) bool { return members[i].Name < members[j].Name })
return members, nil
}
@@ -145,13 +143,13 @@ func buildHostsListFromShardMap(shardsMap proto.ShardsMap) []string {
/* Example
mongos> db.getSiblingDB('admin').runCommand('getShardMap')
{
"map" : {
"config" : "localhost:19001,localhost:19002,localhost:19003",
"localhost:17001" : "r1/localhost:17001,localhost:17002,localhost:17003",
"r1" : "r1/localhost:17001,localhost:17002,localhost:17003",
"r1/localhost:17001,localhost:17002,localhost:17003" : "r1/localhost:17001,localhost:17002,localhost:17003",
},
"ok" : 1
"map" : {
"config" : "localhost:19001,localhost:19002,localhost:19003",
"localhost:17001" : "r1/localhost:17001,localhost:17002,localhost:17003",
"r1" : "r1/localhost:17001,localhost:17002,localhost:17003",
"r1/localhost:17001,localhost:17002,localhost:17003" : "r1/localhost:17001,localhost:17002,localhost:17003",
},
"ok" : 1
}
*/
@@ -200,12 +198,10 @@ func GetShardedHosts(dialer pmgo.Dialer, di *pmgo.DialInfo) ([]string, error) {
return hostnames, errors.Wrap(err, "cannot list shards")
}
if shardsInfo != nil {
for _, shardInfo := range shardsInfo.Shards {
m := strings.Split(shardInfo.Host, "/")
h := strings.Split(m[1], ",")
hostnames = append(hostnames, h[0])
}
for _, shardInfo := range shardsInfo.Shards {
m := strings.Split(shardInfo.Host, "/")
h := strings.Split(m[1], ",")
hostnames = append(hostnames, h[0])
}
return hostnames, nil
}
@@ -230,7 +226,11 @@ func GetServerStatus(dialer pmgo.Dialer, di *pmgo.DialInfo, hostname string) (pr
defer session.Close()
session.SetMode(mgo.Monotonic, true)
if err := session.DB("admin").Run(bson.D{{"serverStatus", 1}, {"recordStats", 1}}, &ss); err != nil {
query := bson.D{
{Name: "serverStatus", Value: 1},
{Name: "recordStats", Value: 1},
}
if err := session.DB("admin").Run(query, &ss); err != nil {
return ss, errors.Wrap(err, "GetHostInfo.serverStatus")
}

View File

@@ -0,0 +1,44 @@
# Debug collector tool
Collects debug data (logs, resource statuses etc.) from a k8s/opeshift cluster. Data packed into "cluster-dump.tar.gz" archive in the current working directory.
### Data that will be collected
"pods",
"replicasets",
"deployments",
"statefulsets",
"replicationcontrollers",
"events",
"configmaps",
"secrets",
"cronjobs",
"jobs",
"podsecuritypolicies",
"poddisruptionbudgets",
"perconaxtradbbackups",
"perconaxtradbclusterbackups",
"perconaxtradbclusterrestores",
"perconaxtradbclusters",
"clusterrolebindings",
"clusterroles",
"rolebindings",
"roles",
"storageclasses",
"persistentvolumeclaims",
"persistentvolumes",
"modes",
"your-custom-resource" (depend on 'resource' flag)
### Usage
`pt-k8s-debug-collector <flags>`
Flags:
`--resource` targeted custom resource name (default "pxc")
`--namespace` targeted namespace. By default data will be collected from all namespaces
`--cluster` targeted pxc/psmdb cluster. By default data from all available clusters to be collected
### Requirements
Installed and configured 'kubectl'
Installed and configured 'pt-mysql-summary'
Installed and configured 'pt-mongodb-summary'

View File

@@ -0,0 +1,355 @@
package dumper
import (
"archive/tar"
"bytes"
"compress/gzip"
"encoding/base64"
"encoding/json"
"fmt"
"log"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
"github.com/pkg/errors"
corev1 "k8s.io/api/core/v1"
)
// Dumper struct is for dumping cluster
type Dumper struct {
cmd string
resources []string
namespace string
location string
errors string
mode int64
crType string
}
// New return new Dumper object
func New(location, namespace, resource string) Dumper {
resources := []string{
"pods",
"replicasets",
"deployments",
"statefulsets",
"replicationcontrollers",
"events",
"configmaps",
"secrets",
"cronjobs",
"jobs",
"podsecuritypolicies",
"poddisruptionbudgets",
"perconaxtradbbackups",
"perconaxtradbclusterbackups",
"perconaxtradbclusterrestores",
"perconaxtradbclusters",
"clusterrolebindings",
"clusterroles",
"rolebindings",
"roles",
"storageclasses",
"persistentvolumeclaims",
"persistentvolumes",
}
if len(resource) > 0 {
resources = append(resources, resource)
}
return Dumper{
cmd: "kubectl",
resources: resources,
location: "cluster-dump",
mode: int64(0777),
namespace: namespace,
crType: resource,
}
}
type k8sPods struct {
Items []corev1.Pod `json:"items"`
}
type namespaces struct {
Items []corev1.Namespace `json:"items"`
}
// DumpCluster create dump of a cluster in Dumper.location
func (d *Dumper) DumpCluster() error {
file, err := os.Create(d.location + ".tar.gz")
if err != nil {
return errors.Wrap(err, "create tar file")
}
zr := gzip.NewWriter(file)
tw := tar.NewWriter(zr)
defer func() {
err = addToArchive(d.location+"/errors.txt", d.mode, []byte(d.errors), tw)
if err != nil {
log.Println("Error: add errors.txt to archive:", err)
}
err = tw.Close()
if err != nil {
log.Println("close tar writer", err)
return
}
err = zr.Close()
if err != nil {
log.Println("close gzip writer", err)
return
}
err = file.Close()
if err != nil {
log.Println("close file", err)
return
}
}()
var nss namespaces
if len(d.namespace) > 0 {
ns := corev1.Namespace{}
ns.Name = d.namespace
nss.Items = append(nss.Items, ns)
} else {
args := []string{"get", "namespaces", "-o", "json"}
output, err := d.runCmd(args...)
if err != nil {
d.logError(err.Error(), args...)
return errors.Wrap(err, "get namespaces")
}
err = json.Unmarshal(output, &nss)
if err != nil {
d.logError(err.Error(), "unmarshal namespaces")
return errors.Wrap(err, "unmarshal namespaces")
}
}
for _, ns := range nss.Items {
args := []string{"get", "pods", "-o", "json", "--namespace", ns.Name}
output, err := d.runCmd(args...)
if err != nil {
d.logError(err.Error(), args...)
continue
}
var pods k8sPods
err = json.Unmarshal(output, &pods)
if err != nil {
d.logError(err.Error(), "unmarshal pods from namespace", ns.Name)
log.Printf("Error: unmarshal pods in namespace %s: %v", ns.Name, err)
}
for _, pod := range pods.Items {
location := filepath.Join(d.location, ns.Name, pod.Name, "logs.txt")
args := []string{"logs", pod.Name, "--namespace", ns.Name, "--all-containers"}
output, err = d.runCmd(args...)
if err != nil {
d.logError(err.Error(), args...)
err = addToArchive(location, d.mode, []byte(err.Error()), tw)
if err != nil {
log.Printf("Error: create archive with logs for pod %s in namespace %s: %v", pod.Name, ns.Name, err)
}
continue
}
err = addToArchive(location, d.mode, output, tw)
if err != nil {
d.logError(err.Error(), "create archive for pod "+pod.Name)
log.Printf("Error: create archive for pod %s: %v", pod.Name, err)
}
if len(pod.Labels) == 0 {
continue
}
location = filepath.Join(d.location, ns.Name, pod.Name, "/pt-summary.txt")
component := d.crType
if d.crType == "psmdb" {
component = "mongod"
}
if pod.Labels["app.kubernetes.io/component"] == component {
output, err = d.getPodSummary(d.crType, pod.Name, pod.Labels["app.kubernetes.io/instance"], tw)
if err != nil {
d.logError(err.Error(), d.crType, pod.Name)
err = addToArchive(location, d.mode, []byte(err.Error()), tw)
if err != nil {
log.Printf("Error: create pt-summary errors archive for pod %s in namespace %s: %v", pod.Name, ns.Name, err)
}
continue
}
err = addToArchive(location, d.mode, output, tw)
if err != nil {
d.logError(err.Error(), "create pt-summary archive for pod "+pod.Name)
log.Printf("Error: create pt-summary archive for pod %s: %v", pod.Name, err)
}
}
}
for _, resource := range d.resources {
err = d.getResource(resource, ns.Name, tw)
if err != nil {
log.Printf("Error: get %s resource: %v", resource, err)
}
}
}
err = d.getResource("nodes", "", tw)
if err != nil {
return errors.Wrapf(err, "get nodes")
}
return nil
}
// runCmd run command (Dumper.cmd) with given args, return it output
func (d *Dumper) runCmd(args ...string) ([]byte, error) {
var outb, errb bytes.Buffer
cmd := exec.Command(d.cmd, args...)
cmd.Stdout = &outb
cmd.Stderr = &errb
err := cmd.Run()
if err != nil || errb.Len() > 0 {
return nil, errors.Errorf("error: %v, stderr: %s, stdout: %s", err, errb.String(), outb.String())
}
return outb.Bytes(), nil
}
func (d *Dumper) getResource(name, namespace string, tw *tar.Writer) error {
location := d.location
args := []string{"get", name, "-o", "yaml"}
if len(namespace) > 0 {
args = append(args, "--namespace", namespace)
location = filepath.Join(d.location, namespace)
}
location = filepath.Join(location, name+".yaml")
output, err := d.runCmd(args...)
if err != nil {
d.logError(err.Error(), args...)
log.Printf("Error: get resource %s in namespace %s: %v", name, namespace, err)
return addToArchive(location, d.mode, []byte(err.Error()), tw)
}
return addToArchive(location, d.mode, output, tw)
}
func (d *Dumper) logError(err string, args ...string) {
d.errors += d.cmd + " " + strings.Join(args, " ") + ": " + err + "\n"
}
func addToArchive(location string, mode int64, content []byte, tw *tar.Writer) error {
hdr := &tar.Header{
Name: location,
Mode: mode,
Size: int64(len(content)),
}
if err := tw.WriteHeader(hdr); err != nil {
return errors.Wrapf(err, "write header to %s", location)
}
if _, err := tw.Write(content); err != nil {
return errors.Wrapf(err, "write content to %s", location)
}
return nil
}
type crSecrets struct {
Spec struct {
SecretName string `json:"secretsName,omitempty"`
Secrets struct {
Users string `json:"users,omitempty"`
} `json:"secrets,omitempty"`
} `json:"spec"`
}
func (d *Dumper) getPodSummary(resource, podName, crName string, tw *tar.Writer) ([]byte, error) {
var (
summCmdName string
ports string
summCmdArgs []string
)
switch resource {
case "pxc":
cr, err := d.getCR("pxc/" + crName)
if err != nil {
return nil, errors.Wrap(err, "get cr")
}
pass, err := d.getDataFromSecret(cr.Spec.SecretName, "root")
if err != nil {
return nil, errors.Wrap(err, "get password from pxc users secret")
}
ports = "3306:3306"
summCmdName = "pt-mysql-summary"
summCmdArgs = []string{"--host=127.0.0.1", "--port=3306", "--user=root", "--password=" + string(pass)}
case "psmdb":
cr, err := d.getCR("psmdb/" + crName)
if err != nil {
return nil, errors.Wrap(err, "get cr")
}
pass, err := d.getDataFromSecret(cr.Spec.Secrets.Users, "MONGODB_CLUSTER_ADMIN_PASSWORD")
if err != nil {
return nil, errors.Wrap(err, "get password from psmdb users secret")
}
ports = "27017:27017"
summCmdName = "pt-mongodb-summary"
summCmdArgs = []string{"--username=clusterAdmin", "--password=" + pass, "--authenticationDatabase=admin", "127.0.0.1:27017"}
}
cmdPortFwd := exec.Command(d.cmd, "port-forward", "pod/"+podName, ports)
go func() {
err := cmdPortFwd.Run()
if err != nil {
d.logError(err.Error(), "port-forward")
}
}()
defer func() {
err := cmdPortFwd.Process.Kill()
if err != nil {
d.logError(err.Error(), "kill port-forward")
}
}()
time.Sleep(3 * time.Second) // wait for port-forward command
var outb, errb bytes.Buffer
cmd := exec.Command(summCmdName, summCmdArgs...)
cmd.Stdout = &outb
cmd.Stderr = &errb
err := cmd.Run()
if err != nil {
return nil, errors.Errorf("error: %v, stderr: %s, stdout: %s", err, errb.String(), outb.String())
}
return []byte(fmt.Sprintf("stderr: %s, stdout: %s", errb.String(), outb.String())), nil
}
func (d *Dumper) getCR(crName string) (crSecrets, error) {
var cr crSecrets
output, err := d.runCmd("get", crName, "-o", "json")
if err != nil {
return cr, errors.Wrap(err, "get "+crName)
}
err = json.Unmarshal(output, &cr)
if err != nil {
return cr, errors.Wrap(err, "unmarshal psmdb cr")
}
return cr, nil
}
func (d *Dumper) getDataFromSecret(secretName, dataName string) (string, error) {
passEncoded, err := d.runCmd("get", "secrets/"+secretName, "--template={{.data."+dataName+"}}")
if err != nil {
return "", errors.Wrap(err, "run get secret cmd")
}
pass, err := base64.StdEncoding.DecodeString(string(passEncoded))
if err != nil {
return "", errors.Wrap(err, "decode data")
}
return string(pass), nil
}

View File

@@ -0,0 +1,35 @@
package main
import (
"flag"
"log"
"os"
"github.com/percona/percona-toolkit/src/go/pt-k8s-debug-collector/dumper"
)
func main() {
namespace := ""
resource := ""
clusterName := ""
flag.StringVar(&namespace, "namespace", "", "Namespace for collecting data. If empty data will be collected from all namespaces")
flag.StringVar(&resource, "resource", "pxc", "Resource name. Default value - 'pxc'")
flag.StringVar(&clusterName, "cluster", "", "Cluster name")
flag.Parse()
if len(clusterName) > 0 {
resource += "/" + clusterName
}
d := dumper.New("", namespace, resource)
log.Println("Start collecting cluster data")
err := d.DumpCluster()
if err != nil {
log.Println("Error:", err)
os.Exit(1)
}
log.Println("Done")
}

View File

@@ -30,12 +30,13 @@ import (
const (
TOOLNAME = "pt-mongodb-summary"
DEFAULT_AUTHDB = "admin"
DEFAULT_HOST = "localhost:27017"
DEFAULT_LOGLEVEL = "warn"
DEFAULT_RUNNINGOPSINTERVAL = 1000 // milliseconds
DEFAULT_RUNNINGOPSSAMPLES = 5
DEFAULT_OUTPUT_FORMAT = "text"
DefaultAuthDB = "admin"
DefaultHost = "localhost:27017"
DefaultLogLevel = "warn"
DefaultRunningOpsInterval = 1000 // milliseconds
DefaultRunningOpsSamples = 5
DefaultOutputFormat = "text"
typeMongos = "mongos"
)
var (
@@ -99,10 +100,10 @@ type security struct {
type databases struct {
Databases []struct {
Name string `bson:"name"`
SizeOnDisk int64 `bson:"sizeOnDisk"`
Empty bool `bson:"empty"`
Shards map[string]int64 `bson:"shards"`
Name string `bson:"name"`
// SizeOnDisk int64 `bson:"sizeOnDisk"`
// Empty bool `bson:"empty"`
// Shards map[string]int64 `bson:"shards"`
} `bson:"databases"`
TotalSize int64 `bson:"totalSize"`
TotalSizeMb int64 `bson:"totalSizeMb"`
@@ -227,7 +228,7 @@ func main() {
ci := &collectedInfo{}
ci.HostInfo, err = GetHostinfo(session)
ci.HostInfo, err = getHostinfo(session)
if err != nil {
message := fmt.Sprintf("Cannot get host info for %q: %s", di.Addrs[0], err.Error())
log.Errorf(message)
@@ -241,13 +242,18 @@ func main() {
log.Debugf("replicaMembers:\n%+v\n", ci.ReplicaMembers)
if opts.RunningOpsSamples > 0 && opts.RunningOpsInterval > 0 {
if ci.RunningOps, err = GetOpCountersStats(session, opts.RunningOpsSamples, time.Duration(opts.RunningOpsInterval)*time.Millisecond); err != nil {
ci.RunningOps, err = getOpCountersStats(
session,
opts.RunningOpsSamples,
time.Duration(opts.RunningOpsInterval)*time.Millisecond,
)
if err != nil {
log.Printf("[Error] cannot get Opcounters stats: %v\n", err)
}
}
if ci.HostInfo != nil {
if ci.SecuritySettings, err = GetSecuritySettings(session, ci.HostInfo.Version); err != nil {
if ci.SecuritySettings, err = getSecuritySettings(session, ci.HostInfo.Version); err != nil {
log.Errorf("[Error] cannot get security settings: %v\n", err)
}
} else {
@@ -255,7 +261,7 @@ func main() {
}
if ci.OplogInfo, err = oplog.GetOplogInfo(hostnames, di); err != nil {
log.Info("Cannot get Oplog info: %v\n", err)
log.Infof("Cannot get Oplog info: %s\n", err)
} else {
if len(ci.OplogInfo) == 0 {
log.Info("oplog info is empty. Skipping")
@@ -265,13 +271,13 @@ func main() {
}
// individual servers won't know about this info
if ci.HostInfo.NodeType == "mongos" {
if ci.ClusterWideInfo, err = GetClusterwideInfo(session); err != nil {
if ci.HostInfo.NodeType == typeMongos {
if ci.ClusterWideInfo, err = getClusterwideInfo(session); err != nil {
log.Printf("[Error] cannot get cluster wide info: %v\n", err)
}
}
if ci.HostInfo.NodeType == "mongos" {
if ci.HostInfo.NodeType == typeMongos {
if ci.BalancerStats, err = GetBalancerStats(session); err != nil {
log.Printf("[Error] cannot get balancer stats: %v\n", err)
}
@@ -326,7 +332,7 @@ func formatResults(ci *collectedInfo, format string) ([]byte, error) {
return buf.Bytes(), nil
}
func GetHostinfo(session pmgo.SessionManager) (*hostInfo, error) {
func getHostinfo(session pmgo.SessionManager) (*hostInfo, error) {
hi := proto.HostInfo{}
if err := session.Run(bson.M{"hostInfo": 1}, &hi); err != nil {
@@ -357,7 +363,7 @@ func GetHostinfo(session pmgo.SessionManager) (*hostInfo, error) {
Hostname: hi.System.Hostname,
HostOsType: hi.Os.Type,
HostSystemCPUArch: hi.System.CpuArch,
DBPath: "", // Sets default. It will be overriden later if necessary
DBPath: "", // Sets default. It will be overridden later if necessary
ProcessName: ss.Process,
ProcProcessCount: procCount,
@@ -390,19 +396,19 @@ func countMongodProcesses() (int, error) {
if err != nil {
continue
}
if name, _ := p.Name(); name == "mongod" || name == "mongos" {
if name, _ := p.Name(); name == "mongod" || name == typeMongos {
count++
}
}
return count, nil
}
func GetClusterwideInfo(session pmgo.SessionManager) (*clusterwideInfo, error) {
func getClusterwideInfo(session pmgo.SessionManager) (*clusterwideInfo, error) {
var databases databases
err := session.Run(bson.M{"listDatabases": 1}, &databases)
if err != nil {
return nil, errors.Wrap(err, "GetClusterwideInfo.listDatabases ")
return nil, errors.Wrap(err, "getClusterwideInfo.listDatabases ")
}
cwi := &clusterwideInfo{
@@ -456,7 +462,7 @@ func sizeAndUnit(size int64) (float64, string) {
return newSize, unit[idx]
}
func GetSecuritySettings(session pmgo.SessionManager, ver string) (*security, error) {
func getSecuritySettings(session pmgo.SessionManager, ver string) (*security, error) {
s := security{
Auth: "disabled",
SSL: "disabled",
@@ -498,13 +504,22 @@ func GetSecuritySettings(session pmgo.SessionManager, ver string) (*security, er
isPrivate, err := isPrivateNetwork(strings.TrimSpace(ip))
if !isPrivate && err == nil {
if s.Auth == "enabled" {
s.WarningMsgs = append(s.WarningMsgs, fmt.Sprintf("Warning: You might be insecure (bind ip %s is public)", ip))
s.WarningMsgs = append(
s.WarningMsgs,
fmt.Sprintf("Warning: You might be insecure (bind ip %s is public)", ip),
)
} else {
s.WarningMsgs = append(s.WarningMsgs, fmt.Sprintf("Error. You are insecure: bind ip %s is public and auth is disabled", ip))
s.WarningMsgs = append(
s.WarningMsgs,
fmt.Sprintf("Error. You are insecure: bind ip %s is public and auth is disabled", ip),
)
}
} else {
if ip != "127.0.0.1" && ip != extIP {
s.WarningMsgs = append(s.WarningMsgs, fmt.Sprintf("WARNING: You might be insecure. IP binding %s is not localhost", ip))
s.WarningMsgs = append(
s.WarningMsgs,
fmt.Sprintf("WARNING: You might be insecure. IP binding %s is not localhost", ip),
)
}
}
}
@@ -553,12 +568,12 @@ func getNodeType(session pmgo.SessionManager) (string, error) {
} else if md.Msg == "isdbgrid" {
// isdbgrid is always the msg value when calling isMaster on a mongos
// see http://docs.mongodb.org/manual/core/sharded-cluster-query-router/
return "mongos", nil
return typeMongos, nil
}
return "mongod", nil
}
func GetOpCountersStats(session pmgo.SessionManager, count int, sleep time.Duration) (*opCounters, error) {
func getOpCountersStats(session pmgo.SessionManager, count int, sleep time.Duration) (*opCounters, error) {
oc := &opCounters{}
prevOpCount := &opCounters{}
ss := proto.ServerStatus{}
@@ -694,7 +709,7 @@ func getProcInfo(pid int32, templateData *procInfo) error {
//proc, err := process.NewProcess(templateData.ServerStatus.Pid)
proc, err := process.NewProcess(pid)
if err != nil {
return fmt.Errorf("cannot get process %d\n", pid)
return fmt.Errorf("cannot get process %d", pid)
}
ct, err := proc.CreateTime()
if err != nil {
@@ -850,12 +865,12 @@ func externalIP() (string, error) {
func parseFlags() (*options, error) {
opts := &options{
Host: DEFAULT_HOST,
LogLevel: DEFAULT_LOGLEVEL,
RunningOpsSamples: DEFAULT_RUNNINGOPSSAMPLES,
RunningOpsInterval: DEFAULT_RUNNINGOPSINTERVAL, // milliseconds
AuthDB: DEFAULT_AUTHDB,
OutputFormat: DEFAULT_OUTPUT_FORMAT,
Host: DefaultHost,
LogLevel: DefaultLogLevel,
RunningOpsSamples: DefaultRunningOpsSamples,
RunningOpsInterval: DefaultRunningOpsInterval, // milliseconds
AuthDB: DefaultAuthDB,
OutputFormat: DefaultOutputFormat,
}
gop := getopt.New()
@@ -864,17 +879,22 @@ func parseFlags() (*options, error) {
gop.BoolVarLong(&opts.NoVersionCheck, "no-version-check", 'c', "", "Default: Don't check for updates")
gop.StringVarLong(&opts.User, "username", 'u', "", "Username to use for optional MongoDB authentication")
gop.StringVarLong(&opts.Password, "password", 'p', "", "Password to use for optional MongoDB authentication").SetOptional()
gop.StringVarLong(&opts.Password, "password", 'p', "", "Password to use for optional MongoDB authentication").
SetOptional()
gop.StringVarLong(&opts.AuthDB, "authenticationDatabase", 'a', "admin",
"Database to use for optional MongoDB authentication. Default: admin")
gop.StringVarLong(&opts.LogLevel, "log-level", 'l', "error", "Log level: panic, fatal, error, warn, info, debug. Default: error")
gop.StringVarLong(&opts.LogLevel, "log-level", 'l', "error",
"Log level: panic, fatal, error, warn, info, debug. Default: error")
gop.StringVarLong(&opts.OutputFormat, "output-format", 'f', "text", "Output format: text, json. Default: text")
gop.IntVarLong(&opts.RunningOpsSamples, "running-ops-samples", 's',
fmt.Sprintf("Number of samples to collect for running ops. Default: %d", opts.RunningOpsSamples))
fmt.Sprintf("Number of samples to collect for running ops. Default: %d", opts.RunningOpsSamples),
)
gop.IntVarLong(&opts.RunningOpsInterval, "running-ops-interval", 'i',
fmt.Sprintf("Interval to wait betwwen running ops samples in milliseconds. Default %d milliseconds", opts.RunningOpsInterval))
fmt.Sprintf("Interval to wait betwwen running ops samples in milliseconds. Default %d milliseconds",
opts.RunningOpsInterval),
)
gop.StringVarLong(&opts.SSLCAFile, "sslCAFile", 0, "SSL CA cert file used for authentication")
gop.StringVarLong(&opts.SSLPEMKeyFile, "sslPEMKeyFile", 0, "SSL client PEM file used for authentication")

View File

@@ -29,32 +29,51 @@ func TestGetOpCounterStats(t *testing.T) {
database := pmgomock.NewMockDatabaseManager(ctrl)
ss := proto.ServerStatus{}
tutil.LoadJson("test/sample/serverstatus.json", &ss)
if err := tutil.LoadJson("test/sample/serverstatus.json", &ss); err != nil {
t.Fatalf("Cannot load sample file: %s", err)
}
session.EXPECT().DB("admin").Return(database)
database.EXPECT().Run(bson.D{{"serverStatus", 1}, {"recordStats", 1}}, gomock.Any()).SetArg(1, ss)
database.EXPECT().Run(bson.D{
{Name: "serverStatus", Value: 1},
{Name: "recordStats", Value: 1},
}, gomock.Any()).SetArg(1, ss)
session.EXPECT().DB("admin").Return(database)
database.EXPECT().Run(bson.D{{"serverStatus", 1}, {"recordStats", 1}}, gomock.Any()).SetArg(1, ss)
database.EXPECT().Run(bson.D{
{Name: "serverStatus", Value: 1},
{Name: "recordStats", Value: 1},
}, gomock.Any()).SetArg(1, ss)
session.EXPECT().DB("admin").Return(database)
database.EXPECT().Run(bson.D{{"serverStatus", 1}, {"recordStats", 1}}, gomock.Any()).SetArg(1, ss)
database.EXPECT().Run(bson.D{
{Name: "serverStatus", Value: 1},
{Name: "recordStats", Value: 1},
}, gomock.Any()).SetArg(1, ss)
session.EXPECT().DB("admin").Return(database)
database.EXPECT().Run(bson.D{{"serverStatus", 1}, {"recordStats", 1}}, gomock.Any()).SetArg(1, ss)
database.EXPECT().Run(bson.D{
{Name: "serverStatus", Value: 1},
{Name: "recordStats", Value: 1},
}, gomock.Any()).SetArg(1, ss)
session.EXPECT().DB("admin").Return(database)
database.EXPECT().Run(bson.D{{"serverStatus", 1}, {"recordStats", 1}}, gomock.Any()).SetArg(1, ss)
database.EXPECT().Run(bson.D{
{Name: "serverStatus", Value: 1},
{Name: "recordStats", Value: 1},
}, gomock.Any()).SetArg(1, ss)
ss = addToCounters(ss, 1)
session.EXPECT().DB("admin").Return(database)
database.EXPECT().Run(bson.D{{"serverStatus", 1}, {"recordStats", 1}}, gomock.Any()).SetArg(1, ss)
database.EXPECT().Run(bson.D{
{Name: "serverStatus", Value: 1}, {Name: "recordStats", Value: 1},
}, gomock.Any()).SetArg(1, ss)
var sampleCount int = 5
var sampleRate time.Duration = 10 * time.Millisecond // in seconds
sampleCount := 5
sampleRate := 10 * time.Millisecond // in seconds
expect := TimedStats{Min: 0, Max: 0, Total: 0, Avg: 0}
os, err := GetOpCountersStats(session, sampleCount, sampleRate)
os, err := getOpCountersStats(session, sampleCount, sampleRate)
if err != nil {
t.Error(err)
}
@@ -67,7 +86,7 @@ func TestGetOpCounterStats(t *testing.T) {
func TestSecurityOpts(t *testing.T) {
cmdopts := []proto.CommandLineOptions{
// 1
proto.CommandLineOptions{
{
Parsed: proto.Parsed{
Net: proto.Net{
SSL: proto.SSL{
@@ -81,7 +100,7 @@ func TestSecurityOpts(t *testing.T) {
},
},
// 2
proto.CommandLineOptions{
{
Parsed: proto.Parsed{
Net: proto.Net{
SSL: proto.SSL{
@@ -95,7 +114,7 @@ func TestSecurityOpts(t *testing.T) {
},
},
// 3
proto.CommandLineOptions{
{
Parsed: proto.Parsed{
Net: proto.Net{
SSL: proto.SSL{
@@ -109,7 +128,7 @@ func TestSecurityOpts(t *testing.T) {
},
},
// 4
proto.CommandLineOptions{
{
Parsed: proto.Parsed{
Net: proto.Net{
SSL: proto.SSL{
@@ -123,7 +142,7 @@ func TestSecurityOpts(t *testing.T) {
},
},
// 5
proto.CommandLineOptions{
{
Parsed: proto.Parsed{
Net: proto.Net{
SSL: proto.SSL{
@@ -143,7 +162,7 @@ func TestSecurityOpts(t *testing.T) {
expect := []*security{
// 1
&security{
{
Users: 1,
Roles: 2,
Auth: "disabled",
@@ -153,7 +172,7 @@ func TestSecurityOpts(t *testing.T) {
WarningMsgs: nil,
},
// 2
&security{
{
Users: 1,
Roles: 2,
Auth: "enabled",
@@ -162,7 +181,7 @@ func TestSecurityOpts(t *testing.T) {
WarningMsgs: nil,
},
// 3
&security{
{
Users: 1,
Roles: 2,
Auth: "enabled",
@@ -172,7 +191,7 @@ func TestSecurityOpts(t *testing.T) {
WarningMsgs: nil,
},
// 4
&security{
{
Users: 1,
Roles: 2,
Auth: "disabled",
@@ -182,7 +201,7 @@ func TestSecurityOpts(t *testing.T) {
WarningMsgs: nil,
},
// 5
&security{
{
Users: 1,
Roles: 2,
Auth: "enabled",
@@ -204,7 +223,9 @@ func TestSecurityOpts(t *testing.T) {
for i, cmd := range cmdopts {
session.EXPECT().DB("admin").Return(database)
database.EXPECT().Run(bson.D{{"getCmdLineOpts", 1}, {"recordStats", 1}}, gomock.Any()).SetArg(1, cmd)
database.EXPECT().Run(bson.D{
{Name: "getCmdLineOpts", Value: 1}, {Name: "recordStats", Value: 1},
}, gomock.Any()).SetArg(1, cmd)
session.EXPECT().Clone().Return(session)
session.EXPECT().SetMode(mgo.Strong, true)
@@ -218,7 +239,7 @@ func TestSecurityOpts(t *testing.T) {
rolesCol.EXPECT().Count().Return(2, nil)
session.EXPECT().Close().Return()
got, err := GetSecuritySettings(session, "3.2")
got, err := getSecuritySettings(session, "3.2")
if err != nil {
t.Errorf("cannot get sec settings: %v", err)
@@ -326,7 +347,9 @@ func TestGetChunks(t *testing.T) {
col := pmgomock.NewMockCollectionManager(ctrl)
var res []proto.ChunksByCollection
tutil.LoadJson("test/sample/chunks.json", &res)
if err := tutil.LoadJson("test/sample/chunks.json", &res); err != nil {
t.Errorf("Cannot load samples file: %s", err)
}
pipe.EXPECT().All(gomock.Any()).SetArg(0, res)
@@ -356,10 +379,12 @@ func TestIntegrationGetChunks(t *testing.T) {
server.SetPath(tempDir)
session := pmgo.NewSessionManager(server.Session())
session.DB("config").C("chunks").Insert(bson.M{"ns": "samples.col1", "count": 2})
if err := session.DB("config").C("chunks").Insert(bson.M{"ns": "samples.col1", "count": 2}); err != nil {
t.Errorf("Cannot insert sample data: %s", err)
}
want := []proto.ChunksByCollection{
proto.ChunksByCollection{
{
ID: "samples.col1",
Count: 1,
},
@@ -373,7 +398,9 @@ func TestIntegrationGetChunks(t *testing.T) {
t.Errorf("Invalid integration chunks count.\ngot: %+v\nwant: %+v", got, want)
}
server.Session().DB("config").DropDatabase()
if err := server.Session().DB("config").DropDatabase(); err != nil {
t.Logf("Cannot drop config database (cleanup): %s", err)
}
server.Session().Close()
server.Stop()
@@ -397,11 +424,11 @@ func TestParseArgs(t *testing.T) {
{
args: []string{TOOLNAME}, // arg[0] is the command itself
want: &options{
Host: DEFAULT_HOST,
LogLevel: DEFAULT_LOGLEVEL,
AuthDB: DEFAULT_AUTHDB,
RunningOpsSamples: DEFAULT_RUNNINGOPSSAMPLES,
RunningOpsInterval: DEFAULT_RUNNINGOPSINTERVAL,
Host: DefaultHost,
LogLevel: DefaultLogLevel,
AuthDB: DefaultAuthDB,
RunningOpsSamples: DefaultRunningOpsSamples,
RunningOpsInterval: DefaultRunningOpsInterval,
OutputFormat: "text",
},
},

View File

@@ -28,6 +28,7 @@
"Max": 1061230,
"Avg": 1061230,
"Pct95": 1061230,
"Pct99": 1061230,
"StdDev": 0,
"Median": 1061230
},
@@ -38,6 +39,7 @@
"Max": 75,
"Avg": 75,
"Pct95": 75,
"Pct99": 75,
"StdDev": 0,
"Median": 75
},
@@ -48,6 +50,7 @@
"Max": 75,
"Avg": 75,
"Pct95": 75,
"Pct99": 75,
"StdDev": 0,
"Median": 75
}
@@ -71,6 +74,7 @@
"Max": 7,
"Avg": 7,
"Pct95": 7,
"Pct99": 7,
"StdDev": 0,
"Median": 7
},
@@ -81,6 +85,7 @@
"Max": 215,
"Avg": 215,
"Pct95": 215,
"Pct99": 215,
"StdDev": 0,
"Median": 215
},
@@ -101,6 +106,7 @@
"Max": 10000,
"Avg": 10000,
"Pct95": 10000,
"Pct99": 10000,
"StdDev": 0,
"Median": 10000
}
@@ -134,6 +140,7 @@
"Max": 215,
"Avg": 215,
"Pct95": 215,
"Pct99": 215,
"StdDev": 0,
"Median": 215
},
@@ -158,4 +165,4 @@
"Median": 0
}
}
]
]

View File

@@ -17,6 +17,7 @@
"Max": 7,
"Avg": 2.3333333333333335,
"Pct95": 7,
"Pct99": 7,
"StdDev": 3.2998316455372216,
"Median": 0
},
@@ -27,6 +28,7 @@
"Max": 1061230,
"Avg": 353886.6666666667,
"Pct95": 1061230,
"Pct99": 1061230,
"StdDev": 500167.26762709644,
"Median": 215
},
@@ -37,6 +39,7 @@
"Max": 75,
"Avg": 25,
"Pct95": 75,
"Pct99": 75,
"StdDev": 35.35533905932738,
"Median": 0
},
@@ -47,7 +50,8 @@
"Max": 10000,
"Avg": 3358.3333333333335,
"Pct95": 10000,
"Pct99": 10000,
"StdDev": 4696.46734850308,
"Median": 75
}
}
}