* PT-2101 - pt-mongodb-query-digest doesn't work on standalone server
Restoring test case, disabled for new sandbox that was never created.
Added debugging output to find out why the tool behaves not as expected.
Most of changes into main.go will be removed after the fix is done.
* PT-2101 - pt-mongodb-query-digest doesn't work on standalone server
- Changed code so it works with the standalone server
- Updated main_test.go so it works for MongoDB 5.0
- Removed eval.js and group.js, because these command are not supported since MongoDB 4.2
* PT-2101 - pt-mongodb-query-digest doesn't work on standalone server
Updated go.mod and go.sum from the 3.x branch
* Test adding meta tags
modified: config/sphinx-build/_static/_templates/theme/layout.html
new file: config/sphinx-build/_static/toolkit.png
* Test adding meta tags
* PT-2048 - pt-osc spans excessive connections to the replica when executing in the source
The bug itself is fixed by PT-2160: 'fix tests for pt online schema change'
* PT-2048 - pt-osc spans excessive connections to the replica when executing in the source
Suggestions by Saikumar
* PT-2048 - pt-osc spans excessive connections to the replica when executing in the source
Typos in the test file
* PT-2169 - pt-k8s-debug-collector integration of pg_gather requires croping first line of the output file
Modified pt-k8s-debug-collector so it redirects only STDOUT to summary.txt
STDERR is stored in the logs and recorded in summary.txt only if summary fails with the error
Modified Makefile, so it does not include closing bracket into the version string
Created test case for the fix
* PT-2169 - pt-k8s-debug-collector integration of pg_gather requires croping first line of the output
Removed else as requested by Ege
Some chunk-related documentation was copy-pasted from pt-table-checksum.
I updated it, so it reflects pt-osc behavior: the tool dies instead of skipping chunk
* PT-2156 - Fix tests for lib
- Fixed tests broken for lib/TableParser.pm after fix for PT-1059
- Updated tests for lib/TableParser.pm that are broken due to SHOW CREATE TABLE output format change in 8.0
- Updated modules for all tools that use lib/TableParser.pm
* Revert "Fixed pt-archiver tests"
This reverts commit a3ab87b12e.
This commit wa needed, because removed code in sandbox/slave_channels.sql broked the test. Proper fix would be to do not remove channel names rather than chaging the test. So revertig it.
* PT-2156 - Fix tests for lib
- Updated test t/lib/TableChecksum.t so it reflects changes, introduced in the fix for PT-2016
- Updated test t/lib/RowChecksum.t so it reflects changes added to in the fix for PT-2138: UTF8 support
- Uncommented SQL in sandbox/slave_channels.sql that made t/lib/MasterSlave.t to fail
- Added check for undef into t/pt-archiver/archive_using_channels.t
- Updated lib/Cxn.pm so it uses $dbh->{Active} after issue with the ping() call, reported at https://github.com/perl5-dbi/DBD-mysql/issues/306
* PT-2156 - Fix tests for lib
- Impoved the fix for PT-2016, so it does not files with keys with USING keyword
- Added brackets to expression in lib/TableNibbler.pm, so it does not crap query wit many indexes with OR keyword
- Adjusted test t/lib/TableNibbler.t, so it reflects above chages
- Modified lib/Cxn.pm, so it has workaround for https://github.com/perl5-dbi/DBD-mysql/issues/306 , introduced in DBD::mysql 4.0.50
- Updated tests: added debugging code and cleanups
- Updated modules for tools
* PT-2165 - Actual Version, GoVersion, and Build for Go tools
Removed outdated numbers from main.go, because we update these numbers in the Makefile.
* PT-2165 - Actual Version, GoVersion, and Build for Go tools
Added template version into Makefile for the latest tag.
* PT-2160 - Fix tests for pt-online-schema-change
Re-enabled tests after MySQL bug #89441 fix in 8.0.14 and later
Updated t/pt-online-schema-change/samples/basic_no_fks_innodb.sql, so its unique index is NOT NULL
Added synchonizations when tests are waiting for too long due to slave catch up
Added innodb_directories option for t/pt-online-schema-change/pt-244.t
* PT-2160 - Fix tests for pt-online-schema-change
PT-2048 - pt-osc spans excessive connections to the replica when executing in the source
t/pt-online-schema-change/slave_lag.t was failing due to PT-2048. I added code that reuses already created connections when checks for slave in the wait loop. It fixes slave_lag.t too.
I also added that prints more detailed error when fails to connect to the replica.
* PT-2160 - Fix tests for pt-online-schema-change
Modified t/pt-online-schema-change/slave_lag.t, so it runs and clean ups faster.
* PT-2160 - Fix tests for pt-online-schema-change
Modified t/pt-online-schema-change/pt-1455.t, so it runs and clean ups faster.
* PT-2160 - Fix tests for pt-online-schema-change
Fixed search for HASH and BTREE keys broken by fix for PT-2123
Updated modules for pt-archiver
Fixed die message for sandbox/start-sandbox
Added global_grants table to list of exceptions for the ok function in lib/Sandbox.pm
Added wait_for_slaves to bugs.t, so changes on the source are copied to replica before pt-osc starts working
Updated PXC tests
Added wait_for_slaves to pt-1455.t
Fixed regular expression in pt-229.t, so it works with both 5.x and 8.0 versions
Added innodb_directories option for pt-244.t, so it does not fail on 8.0
Modified slave_lag.t, so it is more stable and slow enough, so pt-osc can print message about delayed replicas
* PT-2102 pt-mysql-summary should support include* in config file
includedir directories are now printed in the order of which they appear in root my.cnf
* PT-2102 pt-mysql-summary should support include* in config file
!include was not supported. Changed 'for' to 'while' to properly
test for '!includedir' and '!include'. Supports extra spaces between
files and !include.
* PT-2102 pt-mysql-summary should support include* in config file
Updates modules, added test case for !include and !includedir
Changed find_my_cnf function to make it try every regexes and only
returned when it find one
* Update pt-mysql-summary
Identation fix as suggested by review
* Update collect_mysql_info.sh
Identation as suggested by review
* PT-2102 pt-mysql-summary should support include* in config file
Fix formatting after review
---------
Co-authored-by: Sveta Smirnova <svetasmirnova@users.noreply.github.com>
Added REPLICATION_THREADS=0 to tests which suppose that replication is single-threaded,
because new default in 8.0 is 4 replication threads.
Fixed clean up code in t/pt-slave-restart/pt-slave-restart.t
* PT-2114 Incorrect casting of BIT columns by pt-archiver
Added special handling of BIT columns, because by default Perl does not properly recognizes
this type and creates invalid query for MySQL.
Removed debugging comment from t/pt-table-checksum/pt-226.t
Added test case.
* PT-2114 Incorrect casting of BIT columns by pt-archiver
Removed debugging comments from t/pt-deadlock-logger/standard_options.t and t/pt-table-checksum/fnv_64.t
* PT-2114 Incorrect casting of BIT columns by pt-archiver
Added test for archiving BIT columns.
* PT-2114 Incorrect casting of BIT columns by pt-archiver
Added test for bulk operations
* PT-2114 Incorrect casting of BIT columns by pt-archiver
Added more tests for BIT columns
* PT-2114 Incorrect casting of BIT columns by pt-archiver
- Improved fix for PT-2123, so it works with 5.7
- Fixed tests for PT-2114, so they work with 5.7
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias
Added check if source DSN has character set UTF specified while option --charset is not provided
In this case it is safe to open bulk insert data file in utf8 mode.
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias
Removed unrelated row in t/pt-archiver/samples/pt-2123.sql
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias
util/update-modules for pt-archiver
* PT-2120 - pt-stalk with --system-only displaying MYSQL_ONLY: message on the screen
Now we are printing MYSQL_ONLY or SYSTEM_ONLY if option is specified
Also added a warning in case if both options provided together
Added test cases
* Update pt-stalk
Identation as was requested by @denisok
Bug itself is fixed by commits 7eaca8a98e and ee6a9da438
But the documentation still listed Debian 8 (jessie) which is, again, outdated.
So I adopted example for Percona 3.5.1 on Debian 11 (bullseye)
* Create toolkit.yml
Added github action that will build go binaries on each commit. After that we will scan all binaries on CVEs. And if there are no CVEs binaries will be available for downloads
* Update toolkit.yml
Update GA
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias
Added check if source DSN has character set UTF specified while option --charset is not provided
In this case it is safe to open bulk insert data file in utf8 mode.
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias
Removed unrelated row in t/pt-archiver/samples/pt-2123.sql
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias
util/update-modules for pt-archiver
* PT-2150 Fix tests for pt-mysql-summary
- Added POSIX environment to tests, so they do not depend on the local user environment
- Enabled keyring plugin on 8.0
- Fixed encryption information collection in 8.0, see also PT-1588
* PT-2150 Fix tests for pt-mysql-summary
- Added support of the environment variable DISABLE_KEYRING, so we can diable keyring plugin in sandbox/servers/start
* PT-2150 Fix tests for pt-mysql-summary
- Reverted changes in the expected output for tests 006 and 007 made by commit 9190b5e6ac
Because the tool now works properly again and can parse Percona XtraDB Cluster data
* PT-2150 Fix tests for pt-mysql-summary
- Added comment '--read-samples' back to the test cases
* PT-2141 Fix tests for pt-archiver
Fixed tests t/pt-archiver/bulk_insert.t, t/pt-archiver/issue_1225.t, t/pt-archiver/issue_1229.t
Their failures were due to change of utf8 alias from utf8mb3 in MySQL 5.7 to utf8mb4 in MySQL 8.0.
And also due to character set match check between connection ad source table.
* PT-2141 Fix tests for pt-archiver
Removed test for PT-1898, because it was broken from the beginning:
- it called pt-online-schema-change instead of pt-archiver
- it announced it will run 6 tests while was running only 2
- the test case itself did not test the issue, described in PT-1898
* PT-1059 - Tools can't parse index names containing newlines
Fixed regular expressions in TableParser.
Added test case, including test for new lines in the column name
* PT-1059 - Tools can't parse index names containing newlines
Disabled pt-1637.t until PT-2174 is fixed.
Updated number of tables in b/t/pt-table-checksum/issue_1485195.t
* Patch newlines in table columns (#369)
Will accept this change as part of the fix for PT-1059 - Tools cannot parse index names containing new lines. We will later fix the issue with the patch ourselves.
mysql 5.6.40 allows newlines in column names however the following code:
my @defs = $ddl =~ m/^(\s+`.*?),?$/gm;
breaks due to it detecting newlines as line ends. The 'm' argument at the end does this by auto-detecting lines by newline characters.
To correct this issue I've made use of zero-length assertions known as " positive lookback"
https://www.regular-expressions.info/lookaround.html
what does it do?
m/(?:(?<=,\n)|(?<=\(\n))(\s+`(?:.|\n)+?`.+?),?\n/g;
TLDR:
Treat the string as one long string and don't treat \n as the end of a line.
look for (\s+`(?:.|\n)+?`.+?),?\n
if one of those matches look at what precedes the string
if it's ',\n' or ')\n' the string matches. Only save what's in (\s+`(?:.|\n)+?`.+?),?\n
m/ is declaring this a matching regex.
(?:(?<=,\n)|(?<=(\n)) This is an OR statement including two look-behind clauses. The ?: tells the enclosing parentheses to not store the result as a variable. I've put the two look-behinds in this OR statement below this line:
(?<=,\n) Look behind the matched string for a comma followed by a newline, the comma must be there for this look behind to match.
(?<=(\n) Look behind the matched string for a open parentheses followed by a newline, the open parentheses must be there.
(\s+`(?:.|\n)+?`.+?),?\n This is the actual match. Match newline character followed by one or more spaces followed by back-tick followed by a character which can be any character or a newline one or more times, but don't be greedy and take the rest of the match into consideration. Followed by a back tick and any character one or more times. This match stops where there is a comma or failing that a newline following a back tick and some characters.
,?\n match a comma that may not be there followed by a newline.
/g don't stop if this pattern matches keep looking for more patterns to the end of the string.
* PT-1059 - Tools can't parse index names containing newlines
Placed fix from PR-369 into proper place and created test case for this fix.
---------
Co-authored-by: geneguido <31323560+geneguido@users.noreply.github.com>
* PT-2084 Readding SHOW SLAVE STATUS outputs to pt-stalk
Before, they were only gathered if version was 5.6 or prior.
Now, they are always gathered, no matter what version.
Since ${mysql_version} is only major.minor, we can't yet add
SHOW REPLICA STATUS, because it starts on 8.0.22 and we have
no way to check that. We will need to wait for 8.1 release.
* PT-2084 Ran update-modules to generate binaries with changes.
* PT-2084 Added checks for preemptive 8.1 SHOW SLAVE STATUS removal.
* PT-2084 Adding test case for SHOW SLAVE STATUS
* Update t/pt-stalk/pt-stalk-replication.t
---------
Co-authored-by: Sveta Smirnova <svetasmirnova@users.noreply.github.com>
* PT-2164 - pt-k8s-debug-collector does not have version flag
Added flag
Updated README
* PT-2164 - pt-k8s-debug-collector does not have version flag
Added test case
* PT-2164 version flag for pt-k8s-debug-collector
go fmt for src/go/pt-k8s-debug-collector/main.go
* PT-2164 version flag for pt-k8s-debug-collector
Updated docs/pt-k8s-debug-collector.rst
Old documentation was incomplete: old rows would not use the new user defined default value for the altered column.
Only new rows inserted after pt-osc would use the user defined default value.
Existing rows would only be converted to a default value derived from the datatype, as it used to be the case with mysql 5.6
* RM-1153 - Percona Toolkit 3.5.1
Updated version for Perl files to 3.5.1
Updated Perl module Percona::Toolkit to version 3.5.1 and updated all scripts using this module
Manually updated Go programs
Moved ONLY_UPDATE_VERSION after definition of DATE and SERIES variables, so the script does not fail because of undefined variables
* RM-1153 - Percona Toolkit 3.5.1
Updated Changelog
Updated release date in docs/percona-toolkit.pod
* PT-2162 Release Notes 3.5.1 (#571)
modified: Makefile.PL
modified: config/sphinx-build/conf.py
modified: docs/release_notes.rst
new file: docs/rn.3-5-1.txt
Co-authored-by: Alina Derkach <“alina.derkach@percona.com”>
* PT-7 Fix syntax (#573)
Co-authored-by: Alina Derkach <81975178+alina-derkach-oaza@users.noreply.github.com>
Co-authored-by: Alina Derkach <“alina.derkach@percona.com”>
Co-authored-by: EvgeniyPatlan <evgeniy.patlan@percona.com>
