- Merge branch '3.x' into HEAD
- Improve the fix, so it does not fail when $slave_to_skip->{P} is not
defined while $slave->{dsn}->{P} is defined
- Improve the fix for pt-table-checksum (PR-230)
- Added test cases for this fix for pt-table-checksum and pt-osc
- Adjusted replica info repository options in test for pt-1760.t, so it
does not break regression tests for pt-osc on 5.7
* added new hook before_die
before die, the script calls this hook in oder to execute extra user's
operations
* PR-509 - added new hook before_die
- Added test cases
---------
Co-authored-by: Sveta Smirnova <sveta.smirnova@percona.com>
* PT-2109 - pt-mysql-summary throws an error when using sql_mode="ANSI_QUOTES"
- Fixed issues still visible with SQL mode ANSI_QUOTES
- Added test cases for this mode
* PT-2109 - pt-mysql-summary throws an error when using sql_mode="ANSI_QUOTES"
- Removed keyring plugin check in the beginning of the test, because
reported error would show up even on servers that do not have keyring
plugin installed. Same applies to wsrep_on check and Percona Server
features checks.
- Removed temporary dir setup, because not needed for this test.
* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node
- Proof of concept
- Fixed regular expression in lib/TableParser.pm mistakenly chaged in the tool's code
* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node
- Added basic test case for PT-2168
- Added more details for replica lag information
- Disconnecting replica if lag is not checked. This prevents "Too many
connections" error
* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node
- Implemented option --wait-lost-replicas for pt-osc, added test case
* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node
- Added more tests for situations where connection to the replica can
fail
* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node
- Removed extra checks for wait_no_die variable
- Added test cases for SQL queries that pt-osc sends to replicas
* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node
- Allow to reload dsns table while waiting for missed replica if
--recursion-method is dsn
- Fixed logic in replica rediscovery, so it works with replicas on the
same host but with different ports
- Renamed option wait-lost-replicas to fail-on-stopped-replication, so
it is in line with pt-table-checksum
- Adjusted tests
- Removed debug code for PT-1760
- Added test case for PT-1760
- Added exception for variable Open_tables_with_triggers in
lib/bash/collect.sh due to failed test in Percona Server 8.0.34+
- Updated pt-stalk
* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node
- Updated modules
- Fixed typo in t/pt-table-sync/bidirectional.t
- Removed trailing whitespaces in lib/MasterSlave.pm
* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node
- Help for option --fail-on-stopped-replication
* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node
- Added check for availability of the simple_rewrite_plugin in t/pt-online-schema-change/pt-2168.t
* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node
- Added link to the simple_rewrite_plugin source code
- Removed tests for code that runs only in the beginning of pt-osc
action, so should not be affected by the option fail-on-stopped-replication
* Fix typos in lib/ directory
* Update generated bin/ files
* PR 655 - War on typos Act 1 #655
- Updated modules in tools that were not updated
- Fixed tests to reflect proposed changes
---------
Co-authored-by: Sveta Smirnova <sveta.smirnova@percona.com>
* PT-2156 - Fix tests for lib
Fixed tests, broken by putting fix for PT-1554 into the library code, mostly by updating checksums.
Added AutoCommit option for test lib/QueryReview.t
Reset SQL mode to empty to allow TableChunker to work with invalid and zero dates
* PT-2156 - Fix tests for lib
Adjusted t/lib/SchemaIterator.t to work with version 8.0
* PT-2156 - Fix tests for lib
Adjusted b/t/lib/Processlist.t, so it reflects fix for PT-981
* PT-2156 - Fix tests for lib
Adjusted t/lib/HTTP/Micro.t, so it works with different order of parameters, returned at v.percona.com/
Adjusted test for empty files in t/lib/bash/collect.t
Disabled mysqladmin debug test in t/lib/bash/collect.t, because of PT-2242
* PT-2156 - Fix tests for lib
Added LC_NUMERRIC=POSIX into t/lib/bash/report_system_info.sh, so reports in tests are not environment-dependent
Updated expected results in t/lib/bash/report_system_info.sh, so they reflect new information, collected by pt-summary
* PT-2156 - Fix tests for lib
- Improved fix for PT-76, so it handles inline comments
- Added test case for PT-76
- Improved fix for PT-1720, so it ignores unrecognizable option only if it comes from the toolkit-wide files and still errors out if wrong option was passed via command line or the tool-specific option file.
* PT-2156 - Fix tests for lib
- Improved fix for PT-2102, so it finds running instance configuration file using PID and also does not break t/lib/bash/report_mysql_info.t test
- Removed unnecessary copy-paste from t/pt-mysql-summary/pt-2102.t test
- Adjusted number of collected files in t/lib/bash/collect_mysql_info.sh
* PT-2156 - Fix tests for lib
- Fix for PT-1543 and MyRocks collection were originally put only into lib/bash/report_mysql_info.shthat broke the logic of collecting data first, then formatting report from this data. This, in its turn, broke test t/lib/bash/report_mysql_info.sh/t/lib/bash/report_mysql_info.t, because CMD_MYSQL is not defined in this library. I rewrote these fixes, so they follow original logic of the tool (pt-mysql-summary)
- Added tests for keyring plugin, encrypted tables, and MyRocks for t/lib/bash/report_mysql_info.sh and t/pt-mysql-summary/pt-mysql-summary_encryption.t
* PT-2156 - Fix tests for lib
- Added FLUSH TABLES to t/lib/bash/collect.sh, so it does not fail opentables tests if run when more than 1000 tables open in the sandbox environment
- Changed number of expected sample files to reflect keyring colletion file
* PT-2156 - Fix tests for lib
- Added skip to some tests in lib that file when run with PXC, because not supported to work with PXC
- Adjusted configuration files for PXC, so they allow LOAD DATA/SELECT INTO OUTFILE commands
- Adjusted data samples, so they do not depend on auto increment values
- Fixed lib tests, failing with PXC
* PT-2156 - Fix tests for lib
Updated tests for pt-online-schema-change, so they work with PXC and skipped if designed for semi-synchronous replication setup
* PT-2156 - Fix tests for lib
- Added cluster-specific samples for t/lib/SchemaIterator.t
- Removed extra debugging print from t/pt-table-checksum/pt-1728.t
* PT-2156 - Fix tests for lib
Evgeniy's review on July 20
* Update t/lib/TableSyncer.t
Removed comment left after debugging
* Update t/pt-mysql-summary/pt-mysql-summary_encryption.t
Removed extra debug output
* PT-2156 - Fix tests for lib
Tabs to spaces
* fix(pt-online-schema-change): Fix the issue where using --skip-check-slave-lag multiple times does not work as expected.
* PT-2241: add test cases for pt-online-schema-change
* Update t/pt-online-schema-change/pt-2241.t
fix():reset the delay on replicas
Co-authored-by: Sveta Smirnova <svetasmirnova@users.noreply.github.com>
---------
Co-authored-by: Sveta Smirnova <svetasmirnova@users.noreply.github.com>
* Version updated
* PT-2232 Release Notes 3.5.4 (#633)
modified: Changelog
modified: Makefile.PL
modified: config/sphinx-build/conf.py
modified: docs/release_notes.rst
* Updated version in the second place
---------
Co-authored-by: Alina Derkach <alina.derkach@percona.com>
* PT-2151 - Fix tests for pt-query-digest
Put fix for PT-1908 into the proper place
* PT-2151 - Fix tests for pt-query-digest
Put fix for PT-1554 into the proper place
* PT-2151 - Fix tests for pt-query-digest
Adjusted expected results for the default test t/pt-table-checksum/basics.t, so they do not depend on number of rows in the help tables
* PT-2151 - Fix tests for pt-query-digest
Added additional check for both replicas into t/pt-online-schema-change/preserve_triggers.t to avoid deadlock when ->ok() is doing CHECKSUM
* PT-2151 - Fix tests for pt-query-digest
Added delay to t/pt-table-sync/wait.t, so it waits for the child process to start lagging
* PT-2151 - Fix tests for pt-query-digest
Updated t/pt-query-digest/samples/issue_1196-output-8.0.txt, so it reflects changes in the latest 8.0
* PT-2151 - Fix tests for pt-query-digest
Updated queries against query history table, so they don't fail after e2cf183762
* PT-2151 - Fix tests for pt-query-digest
Fixed PT-813 by comparing query text. Since order itself does not matter, it is not essential to compare by the fingerprint or use any other function that changes the query.
* PT-2151 - Fix tests for pt-query-digest
Adjusted samples files which now should have consistent order after fix for PT-813
* PT-2151 - Fix tests for pt-query-digest
Fixed typo in the SELECT query in the QueryReview package
* PT-2151 - Fix tests for pt-query-digest
Fix for PT-981
* PT-2151 - Fix tests for pt-query-digest
Updated modules for pt-index-usage and fixed tests, because checksum is calculated differently after fix for PT-1554
* PT-2151 - Fix tests for pt-query-digest
Updated modules for pt-diskstats, pt-fk-error-logger, pt-heartbeat, pt-online-schema-change, pt-slave-delay, pt-slave-find, pt-table-checksum, pt-table-sync, pt-upgrade
* PT-2151 - Fix tests for pt-query-digest
Updated lib/IndexUsage.pm, so fix for pt-index-usage is in the correct place
Some chunk-related documentation was copy-pasted from pt-table-checksum.
I updated it, so it reflects pt-osc behavior: the tool dies instead of skipping chunk
* PT-2156 - Fix tests for lib
- Fixed tests broken for lib/TableParser.pm after fix for PT-1059
- Updated tests for lib/TableParser.pm that are broken due to SHOW CREATE TABLE output format change in 8.0
- Updated modules for all tools that use lib/TableParser.pm
* Revert "Fixed pt-archiver tests"
This reverts commit a3ab87b12e.
This commit wa needed, because removed code in sandbox/slave_channels.sql broked the test. Proper fix would be to do not remove channel names rather than chaging the test. So revertig it.
* PT-2156 - Fix tests for lib
- Updated test t/lib/TableChecksum.t so it reflects changes, introduced in the fix for PT-2016
- Updated test t/lib/RowChecksum.t so it reflects changes added to in the fix for PT-2138: UTF8 support
- Uncommented SQL in sandbox/slave_channels.sql that made t/lib/MasterSlave.t to fail
- Added check for undef into t/pt-archiver/archive_using_channels.t
- Updated lib/Cxn.pm so it uses $dbh->{Active} after issue with the ping() call, reported at https://github.com/perl5-dbi/DBD-mysql/issues/306
* PT-2156 - Fix tests for lib
- Impoved the fix for PT-2016, so it does not files with keys with USING keyword
- Added brackets to expression in lib/TableNibbler.pm, so it does not crap query wit many indexes with OR keyword
- Adjusted test t/lib/TableNibbler.t, so it reflects above chages
- Modified lib/Cxn.pm, so it has workaround for https://github.com/perl5-dbi/DBD-mysql/issues/306 , introduced in DBD::mysql 4.0.50
- Updated tests: added debugging code and cleanups
- Updated modules for tools
This change addresses https://jira.percona.com/browse/PT-2064 which causes PT-archiver to crash when hitting an error. We use pt-archiver to delete rows that shouldn't be moved to another destination, so `--dest` is not provided.
In most places, we seem to check if `$dst` is actually defined before using it, so we might want to do the same in the `do_with_retries` function?
* PT-2160 - Fix tests for pt-online-schema-change
Re-enabled tests after MySQL bug #89441 fix in 8.0.14 and later
Updated t/pt-online-schema-change/samples/basic_no_fks_innodb.sql, so its unique index is NOT NULL
Added synchonizations when tests are waiting for too long due to slave catch up
Added innodb_directories option for t/pt-online-schema-change/pt-244.t
* PT-2160 - Fix tests for pt-online-schema-change
PT-2048 - pt-osc spans excessive connections to the replica when executing in the source
t/pt-online-schema-change/slave_lag.t was failing due to PT-2048. I added code that reuses already created connections when checks for slave in the wait loop. It fixes slave_lag.t too.
I also added that prints more detailed error when fails to connect to the replica.
* PT-2160 - Fix tests for pt-online-schema-change
Modified t/pt-online-schema-change/slave_lag.t, so it runs and clean ups faster.
* PT-2160 - Fix tests for pt-online-schema-change
Modified t/pt-online-schema-change/pt-1455.t, so it runs and clean ups faster.
* PT-2160 - Fix tests for pt-online-schema-change
Fixed search for HASH and BTREE keys broken by fix for PT-2123
Updated modules for pt-archiver
Fixed die message for sandbox/start-sandbox
Added global_grants table to list of exceptions for the ok function in lib/Sandbox.pm
Added wait_for_slaves to bugs.t, so changes on the source are copied to replica before pt-osc starts working
Updated PXC tests
Added wait_for_slaves to pt-1455.t
Fixed regular expression in pt-229.t, so it works with both 5.x and 8.0 versions
Added innodb_directories option for pt-244.t, so it does not fail on 8.0
Modified slave_lag.t, so it is more stable and slow enough, so pt-osc can print message about delayed replicas
* PT-2102 pt-mysql-summary should support include* in config file
includedir directories are now printed in the order of which they appear in root my.cnf
* PT-2102 pt-mysql-summary should support include* in config file
!include was not supported. Changed 'for' to 'while' to properly
test for '!includedir' and '!include'. Supports extra spaces between
files and !include.
* PT-2102 pt-mysql-summary should support include* in config file
Updates modules, added test case for !include and !includedir
Changed find_my_cnf function to make it try every regexes and only
returned when it find one
* Update pt-mysql-summary
Identation fix as suggested by review
* Update collect_mysql_info.sh
Identation as suggested by review
* PT-2102 pt-mysql-summary should support include* in config file
Fix formatting after review
---------
Co-authored-by: Sveta Smirnova <svetasmirnova@users.noreply.github.com>
* PT-2114 Incorrect casting of BIT columns by pt-archiver
Added special handling of BIT columns, because by default Perl does not properly recognizes
this type and creates invalid query for MySQL.
Removed debugging comment from t/pt-table-checksum/pt-226.t
Added test case.
* PT-2114 Incorrect casting of BIT columns by pt-archiver
Removed debugging comments from t/pt-deadlock-logger/standard_options.t and t/pt-table-checksum/fnv_64.t
* PT-2114 Incorrect casting of BIT columns by pt-archiver
Added test for archiving BIT columns.
* PT-2114 Incorrect casting of BIT columns by pt-archiver
Added test for bulk operations
* PT-2114 Incorrect casting of BIT columns by pt-archiver
Added more tests for BIT columns
* PT-2114 Incorrect casting of BIT columns by pt-archiver
- Improved fix for PT-2123, so it works with 5.7
- Fixed tests for PT-2114, so they work with 5.7
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias
Added check if source DSN has character set UTF specified while option --charset is not provided
In this case it is safe to open bulk insert data file in utf8 mode.
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias
Removed unrelated row in t/pt-archiver/samples/pt-2123.sql
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias
util/update-modules for pt-archiver
* PT-2120 - pt-stalk with --system-only displaying MYSQL_ONLY: message on the screen
Now we are printing MYSQL_ONLY or SYSTEM_ONLY if option is specified
Also added a warning in case if both options provided together
Added test cases
* Update pt-stalk
Identation as was requested by @denisok
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias
Added check if source DSN has character set UTF specified while option --charset is not provided
In this case it is safe to open bulk insert data file in utf8 mode.
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias
Removed unrelated row in t/pt-archiver/samples/pt-2123.sql
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias
util/update-modules for pt-archiver
* PT-2150 Fix tests for pt-mysql-summary
- Added POSIX environment to tests, so they do not depend on the local user environment
- Enabled keyring plugin on 8.0
- Fixed encryption information collection in 8.0, see also PT-1588
* PT-2150 Fix tests for pt-mysql-summary
- Added support of the environment variable DISABLE_KEYRING, so we can diable keyring plugin in sandbox/servers/start
* PT-2150 Fix tests for pt-mysql-summary
- Reverted changes in the expected output for tests 006 and 007 made by commit 9190b5e6ac
Because the tool now works properly again and can parse Percona XtraDB Cluster data
* PT-2150 Fix tests for pt-mysql-summary
- Added comment '--read-samples' back to the test cases
* PT-1059 - Tools can't parse index names containing newlines
Fixed regular expressions in TableParser.
Added test case, including test for new lines in the column name
* PT-1059 - Tools can't parse index names containing newlines
Disabled pt-1637.t until PT-2174 is fixed.
Updated number of tables in b/t/pt-table-checksum/issue_1485195.t
* Patch newlines in table columns (#369)
Will accept this change as part of the fix for PT-1059 - Tools cannot parse index names containing new lines. We will later fix the issue with the patch ourselves.
mysql 5.6.40 allows newlines in column names however the following code:
my @defs = $ddl =~ m/^(\s+`.*?),?$/gm;
breaks due to it detecting newlines as line ends. The 'm' argument at the end does this by auto-detecting lines by newline characters.
To correct this issue I've made use of zero-length assertions known as " positive lookback"
https://www.regular-expressions.info/lookaround.html
what does it do?
m/(?:(?<=,\n)|(?<=\(\n))(\s+`(?:.|\n)+?`.+?),?\n/g;
TLDR:
Treat the string as one long string and don't treat \n as the end of a line.
look for (\s+`(?:.|\n)+?`.+?),?\n
if one of those matches look at what precedes the string
if it's ',\n' or ')\n' the string matches. Only save what's in (\s+`(?:.|\n)+?`.+?),?\n
m/ is declaring this a matching regex.
(?:(?<=,\n)|(?<=(\n)) This is an OR statement including two look-behind clauses. The ?: tells the enclosing parentheses to not store the result as a variable. I've put the two look-behinds in this OR statement below this line:
(?<=,\n) Look behind the matched string for a comma followed by a newline, the comma must be there for this look behind to match.
(?<=(\n) Look behind the matched string for a open parentheses followed by a newline, the open parentheses must be there.
(\s+`(?:.|\n)+?`.+?),?\n This is the actual match. Match newline character followed by one or more spaces followed by back-tick followed by a character which can be any character or a newline one or more times, but don't be greedy and take the rest of the match into consideration. Followed by a back tick and any character one or more times. This match stops where there is a comma or failing that a newline following a back tick and some characters.
,?\n match a comma that may not be there followed by a newline.
/g don't stop if this pattern matches keep looking for more patterns to the end of the string.
* PT-1059 - Tools can't parse index names containing newlines
Placed fix from PR-369 into proper place and created test case for this fix.
---------
Co-authored-by: geneguido <31323560+geneguido@users.noreply.github.com>
* PT-2084 Readding SHOW SLAVE STATUS outputs to pt-stalk
Before, they were only gathered if version was 5.6 or prior.
Now, they are always gathered, no matter what version.
Since ${mysql_version} is only major.minor, we can't yet add
SHOW REPLICA STATUS, because it starts on 8.0.22 and we have
no way to check that. We will need to wait for 8.1 release.
* PT-2084 Ran update-modules to generate binaries with changes.
* PT-2084 Added checks for preemptive 8.1 SHOW SLAVE STATUS removal.
* PT-2084 Adding test case for SHOW SLAVE STATUS
* Update t/pt-stalk/pt-stalk-replication.t
---------
Co-authored-by: Sveta Smirnova <svetasmirnova@users.noreply.github.com>
Old documentation was incomplete: old rows would not use the new user defined default value for the altered column.
Only new rows inserted after pt-osc would use the user defined default value.
Existing rows would only be converted to a default value derived from the datatype, as it used to be the case with mysql 5.6
* RM-1153 - Percona Toolkit 3.5.1
Updated version for Perl files to 3.5.1
Updated Perl module Percona::Toolkit to version 3.5.1 and updated all scripts using this module
Manually updated Go programs
Moved ONLY_UPDATE_VERSION after definition of DATE and SERIES variables, so the script does not fail because of undefined variables
* RM-1153 - Percona Toolkit 3.5.1
Updated Changelog
Updated release date in docs/percona-toolkit.pod
* PT-2162 Release Notes 3.5.1 (#571)
modified: Makefile.PL
modified: config/sphinx-build/conf.py
modified: docs/release_notes.rst
new file: docs/rn.3-5-1.txt
Co-authored-by: Alina Derkach <“alina.derkach@percona.com”>
* PT-7 Fix syntax (#573)
Co-authored-by: Alina Derkach <81975178+alina-derkach-oaza@users.noreply.github.com>
Co-authored-by: Alina Derkach <“alina.derkach@percona.com”>
Co-authored-by: EvgeniyPatlan <evgeniy.patlan@percona.com>
