* PT-2114 Incorrect casting of BIT columns by pt-archiver
Added special handling of BIT columns, because by default Perl does not properly recognizes
this type and creates invalid query for MySQL.
Removed debugging comment from t/pt-table-checksum/pt-226.t
Added test case.
* PT-2114 Incorrect casting of BIT columns by pt-archiver
Removed debugging comments from t/pt-deadlock-logger/standard_options.t and t/pt-table-checksum/fnv_64.t
* PT-2114 Incorrect casting of BIT columns by pt-archiver
Added test for archiving BIT columns.
* PT-2114 Incorrect casting of BIT columns by pt-archiver
Added test for bulk operations
* PT-2114 Incorrect casting of BIT columns by pt-archiver
Added more tests for BIT columns
* PT-2114 Incorrect casting of BIT columns by pt-archiver
- Improved fix for PT-2123, so it works with 5.7
- Fixed tests for PT-2114, so they work with 5.7
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias
Added check if source DSN has character set UTF specified while option --charset is not provided
In this case it is safe to open bulk insert data file in utf8 mode.
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias
Removed unrelated row in t/pt-archiver/samples/pt-2123.sql
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias
util/update-modules for pt-archiver
* PT-2120 - pt-stalk with --system-only displaying MYSQL_ONLY: message on the screen
Now we are printing MYSQL_ONLY or SYSTEM_ONLY if option is specified
Also added a warning in case if both options provided together
Added test cases
* Update pt-stalk
Identation as was requested by @denisok
Bug itself is fixed by commits 7eaca8a98e and ee6a9da438
But the documentation still listed Debian 8 (jessie) which is, again, outdated.
So I adopted example for Percona 3.5.1 on Debian 11 (bullseye)
* Create toolkit.yml
Added github action that will build go binaries on each commit. After that we will scan all binaries on CVEs. And if there are no CVEs binaries will be available for downloads
* Update toolkit.yml
Update GA
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias
Added check if source DSN has character set UTF specified while option --charset is not provided
In this case it is safe to open bulk insert data file in utf8 mode.
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias
Removed unrelated row in t/pt-archiver/samples/pt-2123.sql
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias
util/update-modules for pt-archiver
* PT-2150 Fix tests for pt-mysql-summary
- Added POSIX environment to tests, so they do not depend on the local user environment
- Enabled keyring plugin on 8.0
- Fixed encryption information collection in 8.0, see also PT-1588
* PT-2150 Fix tests for pt-mysql-summary
- Added support of the environment variable DISABLE_KEYRING, so we can diable keyring plugin in sandbox/servers/start
* PT-2150 Fix tests for pt-mysql-summary
- Reverted changes in the expected output for tests 006 and 007 made by commit 9190b5e6ac
Because the tool now works properly again and can parse Percona XtraDB Cluster data
* PT-2150 Fix tests for pt-mysql-summary
- Added comment '--read-samples' back to the test cases
* PT-2141 Fix tests for pt-archiver
Fixed tests t/pt-archiver/bulk_insert.t, t/pt-archiver/issue_1225.t, t/pt-archiver/issue_1229.t
Their failures were due to change of utf8 alias from utf8mb3 in MySQL 5.7 to utf8mb4 in MySQL 8.0.
And also due to character set match check between connection ad source table.
* PT-2141 Fix tests for pt-archiver
Removed test for PT-1898, because it was broken from the beginning:
- it called pt-online-schema-change instead of pt-archiver
- it announced it will run 6 tests while was running only 2
- the test case itself did not test the issue, described in PT-1898
* PT-1059 - Tools can't parse index names containing newlines
Fixed regular expressions in TableParser.
Added test case, including test for new lines in the column name
* PT-1059 - Tools can't parse index names containing newlines
Disabled pt-1637.t until PT-2174 is fixed.
Updated number of tables in b/t/pt-table-checksum/issue_1485195.t
* Patch newlines in table columns (#369)
Will accept this change as part of the fix for PT-1059 - Tools cannot parse index names containing new lines. We will later fix the issue with the patch ourselves.
mysql 5.6.40 allows newlines in column names however the following code:
my @defs = $ddl =~ m/^(\s+`.*?),?$/gm;
breaks due to it detecting newlines as line ends. The 'm' argument at the end does this by auto-detecting lines by newline characters.
To correct this issue I've made use of zero-length assertions known as " positive lookback"
https://www.regular-expressions.info/lookaround.html
what does it do?
m/(?:(?<=,\n)|(?<=\(\n))(\s+`(?:.|\n)+?`.+?),?\n/g;
TLDR:
Treat the string as one long string and don't treat \n as the end of a line.
look for (\s+`(?:.|\n)+?`.+?),?\n
if one of those matches look at what precedes the string
if it's ',\n' or ')\n' the string matches. Only save what's in (\s+`(?:.|\n)+?`.+?),?\n
m/ is declaring this a matching regex.
(?:(?<=,\n)|(?<=(\n)) This is an OR statement including two look-behind clauses. The ?: tells the enclosing parentheses to not store the result as a variable. I've put the two look-behinds in this OR statement below this line:
(?<=,\n) Look behind the matched string for a comma followed by a newline, the comma must be there for this look behind to match.
(?<=(\n) Look behind the matched string for a open parentheses followed by a newline, the open parentheses must be there.
(\s+`(?:.|\n)+?`.+?),?\n This is the actual match. Match newline character followed by one or more spaces followed by back-tick followed by a character which can be any character or a newline one or more times, but don't be greedy and take the rest of the match into consideration. Followed by a back tick and any character one or more times. This match stops where there is a comma or failing that a newline following a back tick and some characters.
,?\n match a comma that may not be there followed by a newline.
/g don't stop if this pattern matches keep looking for more patterns to the end of the string.
* PT-1059 - Tools can't parse index names containing newlines
Placed fix from PR-369 into proper place and created test case for this fix.
---------
Co-authored-by: geneguido <31323560+geneguido@users.noreply.github.com>
* PT-2084 Readding SHOW SLAVE STATUS outputs to pt-stalk
Before, they were only gathered if version was 5.6 or prior.
Now, they are always gathered, no matter what version.
Since ${mysql_version} is only major.minor, we can't yet add
SHOW REPLICA STATUS, because it starts on 8.0.22 and we have
no way to check that. We will need to wait for 8.1 release.
* PT-2084 Ran update-modules to generate binaries with changes.
* PT-2084 Added checks for preemptive 8.1 SHOW SLAVE STATUS removal.
* PT-2084 Adding test case for SHOW SLAVE STATUS
* Update t/pt-stalk/pt-stalk-replication.t
---------
Co-authored-by: Sveta Smirnova <svetasmirnova@users.noreply.github.com>
* PT-2164 - pt-k8s-debug-collector does not have version flag
Added flag
Updated README
* PT-2164 - pt-k8s-debug-collector does not have version flag
Added test case
* PT-2164 version flag for pt-k8s-debug-collector
go fmt for src/go/pt-k8s-debug-collector/main.go
* PT-2164 version flag for pt-k8s-debug-collector
Updated docs/pt-k8s-debug-collector.rst
Old documentation was incomplete: old rows would not use the new user defined default value for the altered column.
Only new rows inserted after pt-osc would use the user defined default value.
Existing rows would only be converted to a default value derived from the datatype, as it used to be the case with mysql 5.6
* RM-1153 - Percona Toolkit 3.5.1
Updated version for Perl files to 3.5.1
Updated Perl module Percona::Toolkit to version 3.5.1 and updated all scripts using this module
Manually updated Go programs
Moved ONLY_UPDATE_VERSION after definition of DATE and SERIES variables, so the script does not fail because of undefined variables
* RM-1153 - Percona Toolkit 3.5.1
Updated Changelog
Updated release date in docs/percona-toolkit.pod
* PT-2162 Release Notes 3.5.1 (#571)
modified: Makefile.PL
modified: config/sphinx-build/conf.py
modified: docs/release_notes.rst
new file: docs/rn.3-5-1.txt
Co-authored-by: Alina Derkach <“alina.derkach@percona.com”>
* PT-7 Fix syntax (#573)
Co-authored-by: Alina Derkach <81975178+alina-derkach-oaza@users.noreply.github.com>
Co-authored-by: Alina Derkach <“alina.derkach@percona.com”>
Co-authored-by: EvgeniyPatlan <evgeniy.patlan@percona.com>
* PT-2139 - CVEs in golang binaries
Updated version for golang.org/x/text to 0.3.8
golang.org/x/net is already in the newest version v0.0.0-20221114191408-850992195362
* PT-2139 - CVEs in golang binaries
- Updated golang.org/x/net as requested Tomislav Plavcic
- golang.org/x/sys, golang.org/x/term, and golang.org/x/text also updated
* Ran tidy
Co-authored-by: Carlos Salguero <carlos.salguero@percona.com>
* PT-2138 - Fix tests for pt-table-checksum
- Updated t/pt-table-checksum/samples/default-results-8.0.txt and
t/pt-table-checksum/samples/static-chunk-size-results-8.0.txt
to support latest MySQL 8.0 version. Tests are now incompatible with elder 8.0 releases.
- Put fix for PT-136 into package RowChecksum
- Added execution bit for pt-online-schema-change
* PT-2138 - Fix tests for pt-table-checksum
- Fixed percona_test.load_data, so it really tests if LOAD DATA LOCAL INFILE is enabled
- Fixed option --[no]create-replicate-table, broken by commit c9836d5962
* PT-2138 - Fix tests for pt-table-checksum
- Enabled t/pt-table-checksum/error_handling.t for MySQL 8.0
- Fixed test t/pt-table-checksum/fnv_64.t and it's samples file
t/pt-table-checksum/samples/fnv64-sakila-city.txt
to reflect new function name convention and
changes after 62d84e5dba
* PT-2138 - Fix tests for pt-table-checksum
- Fixed t/pt-table-checksum/issue_1485195.t, so it checks only one table and
isn't get broken when we add tables into percona_test database
- Fixed typo in error output of bin/pt-table-checksum
- Skipped issue_47.t in 8.0 until https://jira.percona.com/browse/PT-1805 is fixed
* PT-2138 - Fix tests for pt-table-checksum
- Disabled pt-131.t for 8.0, because it does not have the QUERY_RESPONSE_TIME plugins
- Added SLOW_TESTS check to pt-1616.t
- Updated pt-226.t to include the fix for PT-1766
- For replication_filters.t: excluded false positive expression for tests 10 and 11 and added sys schema to the list of checked databases for 8.0
- Changed get_slaves in lib/MasterSlave.pm, so it returns slave's parent, required for wait_for_slaves in pt-table-checksum to work properly with chained slaves
* PT-2138 - Fix tests for pt-table-checksum
- Modified pt-204.t to support 8.0 and diffs in system tables due to timestamps
Moved the fix for PT-1616 into the proper place: lib/NibbleIterator.pm
* PT-2138 - Fix tests for pt-table-checksum
- pxc.t -added mysql.proxies_priv into ignore list, because its timestamp is different on node
- pxc.t - removed FORK=pxc from statup options for slave (non-cluster) nodes
- pxc.t - disabled wsrep replication with help of the variable wsrep_on: sql_log_bin doesn't disable wsrep replication anymore. See https://jira.percona.com/browse/PXC-3464 for details
- Removed data.tar.gz from 5.7 sandbox configuration, because it has an outdated definition for Performance Schema
- Disabled pxc.t for version 8.0 until PT-1699 is fixed
- start-sandbox - removed the first line (ALTER USER) from the init file, because it was rewritten by the next echo command, and then repeated later.
* PT-2138 - Fix tests for pt-table-checksum
- Adopted issue_1485195.t and basics.t for MyRocks-enabled setup
- replication_filters.t - added sys schema to the list of expected schemas for 5.7 and 8.0
- issue_1485195.t - added checks for the existence of mysql.plugin, func, and proxies_priv tables
- added samples/pt-131-wipe.sql that uninstalls QRT plugin if it was earlier installed by this test
-adjusted return code in pt-204.t, because expected differences in mysql.proxies_priv
* Update lib/PerconaTest.pm
removed diagnostic code
Co-authored-by: Carlos Salguero <carlos.salguero@percona.com>
Co-authored-by: Carlos Salguero <carlos.salguero@percona.com>
* PT-2134 - Support for PostgreSQL and PS MySQL operators
For PS MySQL operator: added support of pt-mysql-summary
For PostgreSQL operator: added support for pg_gather
Options added:
- kubeconfig - path to kubeconfig
- forwardport - port to use when calling pt-*-summary tools ad pt_gather
Options changed:
- resource - now default value is (was pxc). New values added:
psql - for PostgreSQL operator
ps - for PS MySQL operator
if default value () is used: only K8 information is collected
Otherwise resource-specific summary is collected
* PT-2134_support_for_PostgreSQL_and_MySQL_operators
Fixed summary collection when connecting to PostgreSQL opertor and no resource specified
* PT-2134 - support for PostgreSQL and MySQL operators
Added test case for full supported set of values for option --resource,
added --kubeconfig and --forwardport options for test cases,
added support for environment variables KUBECONFIG_PXC, KUBECONFIG_PS, KUBECONFIG_PSMDB, KUBECONFIG_PSQL, FORWARDPORT for test cases
* PT-2134 - support for PostgreSQL and MySQL operators
updated docs/pt-k8s-debug-collector.rst,
replaced README.md in src/go/pt-k8s-debug-collector wih symbolic link to docs/pt-k8s-debug-collector.rst
to avoid having inconsistent documentation
* PT-2134 - support for PostgreSQL and MySQL operators
Removed curl STDERR from the command output
* PT-2134 - support for PostgreSQL and MySQL operators
typo in pt-k8s-debug-collector.rst
* PT-2134 - support for PostgreSQL and MySQL operators
Renamed --resource=psql to --resource=pg after review suggestion by Ege
and collecting feedback from potential users
Co-authored-by: EvgeniyPatlan <evgeniy.patlan@percona.com>
Co-authored-by: Carlos Salguero <carlos.salguero@percona.com>
* Changelog and version for Release 3.5.0
* PT-2076 add rhel9 dependency
* PT-2128 Updated version to 3.5.0
Co-authored-by: svetasmirnova <sveta.smirnova@percona.com>
Co-authored-by: EvgeniyPatlan <evgeniy.patlan@percona.com>
* PT-2105 - Collect individual log files for PXC
Combined log files are hard to read by humans. Since pt-k8-debug-collector is the tool that
accesses data from the running pods, it can copy raw log files, necessary for troubleshooting PXC issues.
So, in addition to collecting logs.txt, this adds method getIndividualFiles for the Dumper that
reads individual files from PXC pods and stores them in the resulting archive.
Additionally, this commit fixes invalid timestamps in the resulting archive.
* PT-2105 - added support for non-default namespaces
* PT-2105 Let pt-k8-debug-collector to collect individual logs in PXC pods
Added test case for this new collection.
* Update go.mod
Co-authored-by: Viacheslav Sarzhan <slava.sarzhan@percona.com>
Co-authored-by: Viacheslav Sarzhan <slava.sarzhan@percona.com>
* Correcting specified package type in error message (#58)
* Docs table regex (#53)
* Add more specific info about regex
* noop change to trigger checks
* Fix typo (#298)
* Updated readme
* Fix typo in comment
Co-authored-by: Carlos Salguero <carlos.salguero@percona.com>
* Fix for PT-1799 (#466)
Co-authored-by: Daniel Hoherd <daniel.hoherd@gmail.com>
Co-authored-by: Daniël van Eeden <git@myname.nl>
Co-authored-by: Moritz Lenz <moritz.lenz@noris.de>
Co-authored-by: Carlos Salguero <carlos.salguero@percona.com>
The previous package name "percona-toolkit" is not a valid package name for
ExtUtils::MakeMaker:
$ perl Makefile.PL
Checking if your kit is complete...
Looks good
Warning: NAME must be a package name
[...]
Let us use the same package name (Percona::Toolkit) various scripts in bin/
are already using.
* Add JSON output support to pt-kill
Introduces new flags `--log-json` and `--log-json-fields` to control JSON output:
- `--log-json`: when combined with `--print`, outputs kill information in JSON format instead of plain text, with the same fields as the `kill_log` table when using `--log-dsn`
- `--log-json-fields`: when combined with `--log-json`, adds user-defined key-value pairs to the JSON document that is output (such as cluster name or hostname); parameter value is in the format of `key1:value1,key2:value2`
* Convert --log-json to write to file, remove dependency on --print, update docs
The `--log-json` parameter now takes a file path value, and writes JSON output to this file.
The `--print` parameter is now independent and does not need to be specified to use `--log-json`.
Docs updated for clarity.
* Add chksm to JSON output, make JSON key names consistent
- Adds chksm outout to JSON output to make it easier to group similar queries
- Moves chksm code so that it is always executed, is accessible within the `log-json` scope, and isn't duplicated
- Change JSON key names to consistently use uppercase letters at start of words, change `ts` to `Timestamp`
* Update docs to match changed field names
* Rename JSON parameter, output to STDOUT, fix tests, add test for --json, bug fix for --query-id
- `--json` parameter must now be used in conjunction with `--print`; JSON output replaces normal `--print` output when `--json` is also specified
- `--json` output is now sent to STDOUT instead of a specified file, as its new behaviour is to change the output format of `--print` instead of logging to a file
- Renamed `--log-json` and `--log-json-fields` parameters to `--json` and `--json-fields` to better represent their new behaviour
- Refactored checksum code to prevent test failures when `$query->{'Info'}` is empty
- Added test for `--json` and `--json-fields` parameters
- Fixed bug where specifying `--query-id` would cause errors when `$query->{'Info'}` is empty
* Fix typo: Unkown -> Unknown
