Commit Graph

2584 Commits

Author SHA1 Message Date
Sveta Smirnova
46fd033994 PT-2258 - Pass tbl as argument for on_copy_rows_after_nibble (#682)
- Implemented the feature
- Created test case
2023-09-22 13:35:29 +03:00
Marco Tusa
3be972e518 Fixed one issue (I introduced filtering the PASSWORD keyword)
correct an IF condition
removed a delete action on mysql.user. Always use DROP not delete
2023-09-21 17:03:49 +02:00
Marco Tusa
275ceb0051 Fixing other small things and MariaDB specific,
- Use of VIA
- Use of USING
- Incompatible syntax with MySQL
- ADD IF NOT EXISTS
- convert the script to us AS to maintain the hash as they are

Add a parameter --convert-MariaDB as parameter (default false)
2023-09-21 16:35:57 +02:00
Sveta Smirnova
f8c43118bd PT-2168 pt-osc shouldnt fail while unable to monitor a replica node (#676)
* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node

- Proof of concept
- Fixed regular expression in lib/TableParser.pm mistakenly chaged in the tool's code

* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node

- Added basic test case for PT-2168
- Added more details for replica lag information
- Disconnecting replica if lag is not checked. This prevents "Too many
  connections" error

* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node

- Implemented option --wait-lost-replicas for pt-osc, added test case

* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node

- Added more tests for situations where connection to the replica can
  fail

* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node

- Removed extra checks for wait_no_die variable
- Added test cases for SQL queries that pt-osc sends to replicas

* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node

- Allow to reload dsns table while waiting for missed replica if
  --recursion-method is dsn
- Fixed logic in replica rediscovery, so it works with replicas on the
  same host but with different ports
- Renamed option wait-lost-replicas to fail-on-stopped-replication, so
  it is in line with pt-table-checksum
- Adjusted tests
- Removed debug code for PT-1760
- Added test case for PT-1760
- Added exception for variable Open_tables_with_triggers in
  lib/bash/collect.sh due to failed test in Percona Server 8.0.34+
- Updated pt-stalk

* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node

- Updated modules
- Fixed typo in t/pt-table-sync/bidirectional.t
- Removed trailing whitespaces in lib/MasterSlave.pm

* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node

- Help for option --fail-on-stopped-replication

* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node

- Added check for availability of the simple_rewrite_plugin in t/pt-online-schema-change/pt-2168.t

* PT-2168 - PT-OSC shouldn't fail while unable to monitor a replica node

- Added link to the simple_rewrite_plugin source code
- Removed tests for code that runs only in the beginning of pt-osc
  action, so should not be affected by the option fail-on-stopped-replication
2023-09-19 23:31:33 +03:00
Marco Tusa
8ad0bc0827 I had to correct few things in the script to have it working correctly:
1) OR instead and in the if line 2042
2) evaluate array create_user if it has some value
3) filter out the user creation in Mariadb inside the Grants
2023-09-18 15:16:56 +02:00
Viktor Szépe
f2f7a7993d Add final newlines and remove trailing spaces (#672)
* Add final newlines and remove trailing spaces

* Replace TAB characters with spaces

* Fix indent
2023-09-15 17:31:40 +03:00
Viktor Szépe
2bd40d8c39 Remove trailing spaces (#665)
* Remove trailing spaces

* PR-665 -  Remove trailing spaces

- Updated not stable test t/pt-online-schema-change/preserve_triggers.t
- Updated utilities in bin directory

* PR-665 -  Remove trailing spaces

- Fixed typos

* PR-665 -  Remove trailing spaces

- Fixed typos

---------

Co-authored-by: Sveta Smirnova <sveta.smirnova@percona.com>
2023-09-06 01:15:12 +03:00
Viktor Szépe
c57441be8c War on typos Act 1 (#655)
* Fix typos in lib/ directory

* Update generated bin/ files

* PR 655 -  War on typos Act 1 #655

- Updated modules in tools that were not updated
- Fixed tests to reflect proposed changes

---------

Co-authored-by: Sveta Smirnova <sveta.smirnova@percona.com>
2023-08-22 16:18:38 +03:00
Sveta Smirnova
9755074cd8 PT-2156 Fix tests for lib, part2 (#650)
* PT-2156 - Fix tests for lib

Fixed tests, broken by putting fix for PT-1554 into the library code, mostly by updating checksums.
Added AutoCommit option for test lib/QueryReview.t
Reset SQL mode to empty to allow TableChunker to work with invalid and zero dates

* PT-2156 - Fix tests for lib

Adjusted t/lib/SchemaIterator.t to work with version 8.0

* PT-2156 - Fix tests for lib

Adjusted b/t/lib/Processlist.t, so it reflects fix for PT-981

* PT-2156 - Fix tests for lib

Adjusted t/lib/HTTP/Micro.t, so it works with different order of parameters, returned at v.percona.com/
Adjusted test for empty files in t/lib/bash/collect.t
Disabled mysqladmin debug test in t/lib/bash/collect.t, because of PT-2242

* PT-2156 - Fix tests for lib

Added LC_NUMERRIC=POSIX into t/lib/bash/report_system_info.sh, so reports in tests are not environment-dependent
Updated expected results in t/lib/bash/report_system_info.sh, so they reflect new information, collected by pt-summary

* PT-2156 - Fix tests for lib

- Improved fix for PT-76, so it handles inline comments
- Added test case for PT-76
- Improved fix for PT-1720, so it ignores unrecognizable option only if it comes from the toolkit-wide files and still errors out if wrong option was passed via command line or the tool-specific option file.

* PT-2156 - Fix tests for lib

- Improved fix for PT-2102, so it finds running instance configuration file using PID and also does not break t/lib/bash/report_mysql_info.t test
- Removed unnecessary copy-paste from t/pt-mysql-summary/pt-2102.t test
- Adjusted number of collected files in t/lib/bash/collect_mysql_info.sh

* PT-2156 - Fix tests for lib

- Fix for PT-1543 and MyRocks collection were originally put only into lib/bash/report_mysql_info.shthat broke the logic of collecting data first, then formatting report from this data. This, in its turn, broke test t/lib/bash/report_mysql_info.sh/t/lib/bash/report_mysql_info.t, because CMD_MYSQL is not defined in this library. I rewrote these fixes, so they follow original logic of the tool (pt-mysql-summary)
- Added tests for keyring plugin, encrypted tables, and MyRocks for t/lib/bash/report_mysql_info.sh and t/pt-mysql-summary/pt-mysql-summary_encryption.t

* PT-2156 - Fix tests for lib

- Added FLUSH TABLES to t/lib/bash/collect.sh, so it does not fail opentables tests if run when more than 1000 tables open in the sandbox environment
- Changed number of expected sample files to reflect keyring colletion file

* PT-2156 - Fix tests for lib

- Added skip to some tests in lib that file when run with PXC, because not supported to work with PXC
- Adjusted configuration files for PXC, so they allow LOAD DATA/SELECT INTO OUTFILE commands
- Adjusted data samples, so they do not depend on auto increment values
- Fixed lib tests, failing with PXC

* PT-2156 - Fix tests for lib

Updated tests for pt-online-schema-change, so they work with PXC and skipped if designed for semi-synchronous replication setup

* PT-2156 - Fix tests for lib

- Added cluster-specific samples for t/lib/SchemaIterator.t
- Removed extra debugging print from t/pt-table-checksum/pt-1728.t

* PT-2156 - Fix tests for lib

Evgeniy's review on July 20

* Update t/lib/TableSyncer.t

Removed comment left after debugging

* Update t/pt-mysql-summary/pt-mysql-summary_encryption.t

Removed extra debug output

* PT-2156 - Fix tests for lib

Tabs to spaces
2023-08-02 14:50:21 +03:00
PinoCao
91f9e27255 PT-2241 - fix(pt-online-schema-change): using --skip-check-slave-lag multiple times does not work as expected (#622)
* fix(pt-online-schema-change): Fix the issue where using --skip-check-slave-lag multiple times does not work as expected.

* PT-2241: add test cases for pt-online-schema-change

* Update t/pt-online-schema-change/pt-2241.t

fix():reset the delay on replicas

Co-authored-by: Sveta Smirnova <svetasmirnova@users.noreply.github.com>

---------

Co-authored-by: Sveta Smirnova <svetasmirnova@users.noreply.github.com>
2023-07-31 16:59:09 +03:00
Marcelo HP Ferreira
8ece30ff2f Add after nibble hook to pt-online-schema-change (#645)
* Include hook after nibble on copy rows

* Update tests with new hook for pt-online-schema-change
2023-07-31 16:30:24 +03:00
Sveta Smirnova
205d7baf5b Release v3.5.4 (#647)
* Version updated

* PT-2232 Release Notes 3.5.4 (#633)

modified:   Changelog
	modified:   Makefile.PL
	modified:   config/sphinx-build/conf.py
	modified:   docs/release_notes.rst

* Updated version in the second place

---------

Co-authored-by: Alina Derkach <alina.derkach@percona.com>
2023-06-30 18:44:59 +03:00
Sveta Smirnova
25c969542b PT-2151 fix tests for pt-query-digest (#631)
* PT-2151 - Fix tests for pt-query-digest

Put fix for PT-1908 into the proper place

* PT-2151 - Fix tests for pt-query-digest

Put fix for PT-1554 into the proper place

* PT-2151 - Fix tests for pt-query-digest

Adjusted expected results for the default test t/pt-table-checksum/basics.t, so they do not depend on number of rows in the help tables

* PT-2151 - Fix tests for pt-query-digest

Added additional check for both replicas into t/pt-online-schema-change/preserve_triggers.t to avoid deadlock when ->ok() is doing CHECKSUM

* PT-2151 - Fix tests for pt-query-digest

Added delay to t/pt-table-sync/wait.t, so it waits for the child process to start lagging

* PT-2151 - Fix tests for pt-query-digest

Updated t/pt-query-digest/samples/issue_1196-output-8.0.txt, so it reflects changes in the latest 8.0

* PT-2151 - Fix tests for pt-query-digest

Updated queries against query history table, so they don't fail after e2cf183762

* PT-2151 - Fix tests for pt-query-digest

Fixed PT-813 by comparing query text. Since order itself does not matter, it is not essential to compare by the fingerprint or use any other function that changes the query.

* PT-2151 - Fix tests for pt-query-digest

Adjusted samples files which now should have consistent order after fix for PT-813

* PT-2151 - Fix tests for pt-query-digest

Fixed typo in the SELECT query in the QueryReview package

* PT-2151 - Fix tests for pt-query-digest

Fix for PT-981

* PT-2151 - Fix tests for pt-query-digest

Updated modules for pt-index-usage and fixed tests, because checksum is calculated differently after fix for PT-1554

* PT-2151 - Fix tests for pt-query-digest

Updated modules for pt-diskstats, pt-fk-error-logger, pt-heartbeat, pt-online-schema-change, pt-slave-delay, pt-slave-find, pt-table-checksum, pt-table-sync, pt-upgrade

* PT-2151 - Fix tests for pt-query-digest

Updated lib/IndexUsage.pm, so fix for pt-index-usage is in the correct place
2023-06-23 15:48:21 +03:00
Ricardo
01e7b1819e Remove $has_child_tables var (#617)
Remove $has_child_tables check as this seems a duplicate of the check within the alter_fk_method block.
2023-06-21 16:54:22 +03:00
Sveta Smirnova
b566350b64 Release v3.5.3 (#629)
* PT-2206 Release Notes 3.5.3 (#621)

* PT-2206 Release Notes 3.5.3
modified:   Changelog
	modified:   Makefile.PL
	modified:   config/sphinx-build/conf.py
	modified:   docs/release_notes.rst

* Update Changelog

* Update docs/release_notes.rst

---------

Co-authored-by: Sveta Smirnova <svetasmirnova@users.noreply.github.com>

* RM-1222 - Percona Toolkit 3.5.3

Updated versions

* Add the release date for 3.5.3 release notes (#624)

modified:   Changelog
	modified:   docs/release_notes.rst

* ENG-7 fix toolkit build (#626)

* Release v3.5.3 (#627)

* ENG-7 fix toolkit build

* ENG-7 fix build

* Update the RNs date (#625)

modified:   Changelog
	modified:   docs/release_notes.rst

---------

Co-authored-by: Alina Derkach <alina.derkach@percona.com>
Co-authored-by: EvgeniyPatlan <evgeniy.patlan@percona.com>
2023-06-05 22:48:13 +03:00
baotiao
b7d1337476 PT-2211 This pstack return error code line number 2023-06-02 20:11:08 +08:00
Sveta Smirnova
85eb1411d1 PT-1583 - Skipping chunks when using pt-online-schema-change (#616)
Some chunk-related documentation was copy-pasted from pt-table-checksum.
I updated it, so it reflects pt-osc behavior: the tool dies instead of skipping chunk
2023-04-19 17:33:58 +03:00
Carlos Salguero
a85b7aa32c Release v3.5.2 (#609)
* Release v3.5.1 Go 1.20 and upgrade deps

* Release v3.5.2

* PT-2193 Release Notes 3.5.2 (#610)

modified:   Changelog
	modified:   Makefile.PL
	modified:   config/sphinx-build/conf.py
	modified:   docs/release_notes.rst
	new file:   docs/rn.3-5-2.txt

* Updated version properly (#611)

* Release v3.5.1 Go 1.20 and upgrade deps

* Release v3.5.2

* PT-2193 Release Notes 3.5.2 (#610)

modified:   Changelog
	modified:   Makefile.PL
	modified:   config/sphinx-build/conf.py
	modified:   docs/release_notes.rst
	new file:   docs/rn.3-5-2.txt

* Updated version properly

---------

Co-authored-by: Carlos Salguero <carlos.salguero@percona.com>
Co-authored-by: Alina Derkach <alina.derkach@percona.com>

---------

Co-authored-by: Alina Derkach <alina.derkach@percona.com>
Co-authored-by: Sveta Smirnova <svetasmirnova@users.noreply.github.com>
2023-03-28 17:33:37 +03:00
Sveta Smirnova
642253ce52 PT-2156 Fix tests for lib (#606)
* PT-2156 - Fix tests for lib

- Fixed tests broken for lib/TableParser.pm after fix for PT-1059
- Updated tests for lib/TableParser.pm that are broken due to SHOW CREATE TABLE output format change in 8.0
- Updated modules for all tools that use lib/TableParser.pm

* Revert "Fixed pt-archiver tests"

This reverts commit a3ab87b12e.

This commit wa needed, because removed code in sandbox/slave_channels.sql broked the test. Proper fix would be to do not remove channel names rather than chaging the test. So revertig it.

* PT-2156 - Fix tests for lib

- Updated test t/lib/TableChecksum.t so it reflects changes, introduced in the fix for PT-2016
- Updated test t/lib/RowChecksum.t so it reflects changes added to in the fix for PT-2138: UTF8 support
- Uncommented SQL in sandbox/slave_channels.sql that made t/lib/MasterSlave.t to fail
- Added check for undef into t/pt-archiver/archive_using_channels.t
- Updated lib/Cxn.pm so it uses $dbh->{Active} after issue with the ping() call, reported at https://github.com/perl5-dbi/DBD-mysql/issues/306

* PT-2156 - Fix tests for lib

- Impoved the fix for PT-2016, so it does not files with keys with USING keyword
- Added brackets to expression in lib/TableNibbler.pm, so it does not crap query wit many indexes with OR keyword
- Adjusted test t/lib/TableNibbler.t, so it reflects above chages
- Modified lib/Cxn.pm, so it has workaround for https://github.com/perl5-dbi/DBD-mysql/issues/306 , introduced in DBD::mysql 4.0.50
- Updated tests: added debugging code and cleanups
- Updated modules for tools
2023-03-27 14:18:26 +03:00
Jakob
79757865e3 Only rollback destination if it's defined
This change addresses https://jira.percona.com/browse/PT-2064 which causes PT-archiver to crash when hitting an error. We use pt-archiver to delete rows that shouldn't be moved to another destination, so `--dest` is not provided.

In most places, we seem to check  if `$dst` is actually defined before using it, so we might want to do the same in the `do_with_retries` function?
2023-03-09 15:24:46 +01:00
Sveta Smirnova
bfc00b93a8 PT-2160 fix tests for pt online schema change (#602)
* PT-2160 - Fix tests for pt-online-schema-change

Re-enabled tests after MySQL bug #89441 fix in 8.0.14 and later
Updated t/pt-online-schema-change/samples/basic_no_fks_innodb.sql, so its unique index is NOT NULL
Added synchonizations when tests are waiting for too long due to slave catch up
Added innodb_directories option for t/pt-online-schema-change/pt-244.t

* PT-2160 - Fix tests for pt-online-schema-change
PT-2048 - pt-osc spans excessive connections to the replica when executing in the source

t/pt-online-schema-change/slave_lag.t was failing due to PT-2048. I added code that reuses already created connections when checks for slave in the wait loop. It fixes slave_lag.t too.
I also added that prints more detailed error when fails to connect to the replica.

* PT-2160 - Fix tests for pt-online-schema-change

Modified t/pt-online-schema-change/slave_lag.t, so it runs and clean ups faster.

* PT-2160 - Fix tests for pt-online-schema-change

Modified t/pt-online-schema-change/pt-1455.t, so it runs and clean ups faster.

* PT-2160 - Fix tests for pt-online-schema-change

Fixed search for HASH and BTREE keys broken by fix for PT-2123
Updated modules for pt-archiver
Fixed die message for sandbox/start-sandbox
Added global_grants table to list of exceptions for the ok function in lib/Sandbox.pm
Added wait_for_slaves to bugs.t, so changes on the source are copied to replica before pt-osc starts working
Updated PXC tests
Added wait_for_slaves to pt-1455.t
Fixed regular expression in pt-229.t, so it works with both 5.x and 8.0 versions
Added innodb_directories option for pt-244.t, so it does not fail on 8.0
Modified slave_lag.t, so it is more stable and slow enough, so pt-osc can print message about delayed replicas
2023-02-27 01:19:11 +03:00
Yoann La Cancellera
7ecdae6ef6 PT-2102_pt-mysql-summary should support include in config files (#601)
* PT-2102 pt-mysql-summary should support include* in config file

includedir directories are now printed in the order of which they appear in root my.cnf

* PT-2102 pt-mysql-summary should support include* in config file

!include was not supported. Changed 'for' to 'while' to properly
test for '!includedir' and '!include'. Supports extra spaces between
files and !include.

* PT-2102 pt-mysql-summary should support include* in config file

Updates modules, added test case for !include and !includedir
Changed find_my_cnf function to make it try every regexes and only
returned when it find one

* Update pt-mysql-summary

Identation fix as suggested by review

* Update collect_mysql_info.sh

Identation as suggested by review

* PT-2102 pt-mysql-summary should support include* in config file

Fix formatting after review

---------

Co-authored-by: Sveta Smirnova <svetasmirnova@users.noreply.github.com>
2023-02-23 16:00:44 +03:00
Sveta Smirnova
91ef89f904 PT-2114 Incorrect casting of bit columns by pt archiver (#587)
* PT-2114 Incorrect casting of BIT columns by pt-archiver

Added special handling of BIT columns, because by default Perl does not properly recognizes
this type and creates invalid query for MySQL.
Removed debugging comment from t/pt-table-checksum/pt-226.t
Added test case.

* PT-2114 Incorrect casting of BIT columns by pt-archiver

Removed debugging comments from t/pt-deadlock-logger/standard_options.t and t/pt-table-checksum/fnv_64.t

* PT-2114 Incorrect casting of BIT columns by pt-archiver

Added test for archiving BIT columns.

* PT-2114 Incorrect casting of BIT columns by pt-archiver

Added test for bulk operations

* PT-2114 Incorrect casting of BIT columns by pt-archiver

Added more tests for BIT columns

* PT-2114 Incorrect casting of BIT columns by pt-archiver

- Improved fix for PT-2123, so it works with 5.7
- Fixed tests for PT-2114, so they work with 5.7
2023-02-22 18:59:05 +03:00
Sveta Smirnova
61a53edc54 PT-188-Use percona.checksums in examples (#594)
Replaced test.checksum with percona.checksum in examples
2023-02-13 12:47:35 +03:00
Sveta Smirnova
2d6b22a1a0 PT-1389 - pt-variable-advisor Docs Typos (#593)
Fixed typo
2023-02-13 12:46:52 +03:00
Sveta Smirnova
a38fd1d998 PT-1594 - Need correction in the Documentation - pt-mysql-summary (#592)
Implemented suggested correction
2023-02-13 12:46:07 +03:00
Sveta Smirnova
0accdb3bd3 PT-1595-percona toolkit docs still reference Maatkit (#591)
- Removed Maatkit left overs
- Keeped links to Maatkit bugs and wiki
- Updated brokn links
2023-02-13 12:44:57 +03:00
Sveta Smirnova
bbe3170f2f PT-2120 pt-stalk with --system-only displaying mysql_only message on the screen (#584)
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias

Added check if source DSN has character set UTF specified while option --charset is not provided
In this case it is safe to open bulk insert data file in utf8 mode.

* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias

Removed unrelated row in t/pt-archiver/samples/pt-2123.sql

* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias

util/update-modules for pt-archiver

* PT-2120 - pt-stalk with --system-only displaying MYSQL_ONLY: message on the screen

Now we are printing MYSQL_ONLY or SYSTEM_ONLY if option is specified
Also added a warning in case if both options provided together
Added test cases

* Update pt-stalk

Identation as was requested by @denisok
2023-02-13 12:44:12 +03:00
Sveta Smirnova
64ca4de973 PT-1642-Update PT docs to reference bug reporting location as Percona JIRA (#590)
Updated bug tracker URL and also changed "open issues" list to https://jira.percona.com/projects/PT/issues
2023-02-11 00:34:01 +03:00
Sveta Smirnova
c0b208ff94 PT-1700 pt-table-checksum REPLICATION_STOPPED exit status not documented (#588)
Fixed documntaiton in bin/pt-table-checksum
2023-02-10 21:09:20 +03:00
Sveta Smirnova
798eaaa208 PT-2123 wide character in print for pt archiver (#583)
* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias

Added check if source DSN has character set UTF specified while option --charset is not provided
In this case it is safe to open bulk insert data file in utf8 mode.

* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias

Removed unrelated row in t/pt-archiver/samples/pt-2123.sql

* PT-2123 pt-archiver gives error "Wide character in print at /usr/bin/pt-archiver line 6815" when using --bulk-insert while using character set alias

util/update-modules for pt-archiver
2023-02-07 18:00:32 +03:00
Sveta Smirnova
a12d4aecea PT-2150 Fix tests for pt-mysql-summary (#582)
* PT-2150 Fix tests for pt-mysql-summary

- Added POSIX environment to tests, so they do not depend on the local user environment
- Enabled keyring plugin on 8.0
- Fixed encryption information collection in 8.0, see also PT-1588

* PT-2150 Fix tests for pt-mysql-summary

- Added support of the environment variable DISABLE_KEYRING, so we can diable keyring plugin in sandbox/servers/start

* PT-2150 Fix tests for pt-mysql-summary

- Reverted changes in the expected output for tests 006 and 007 made by commit 9190b5e6ac
Because the tool now works properly again and can parse Percona XtraDB Cluster data

* PT-2150 Fix tests for pt-mysql-summary

- Added comment '--read-samples' back to the test cases
2023-02-06 17:11:42 +03:00
Sveta Smirnova
f9726e75cc PT-1059 tools cannot parse index names containing newlines (#578)
* PT-1059 - Tools can't parse index names containing newlines

Fixed regular expressions in TableParser.
Added test case, including test for new lines in the column name

* PT-1059 - Tools can't parse index names containing newlines

Disabled pt-1637.t until PT-2174 is fixed.
Updated number of tables in b/t/pt-table-checksum/issue_1485195.t

* Patch newlines in table columns (#369)

Will accept this change as part of the fix for PT-1059 - Tools cannot parse index names containing new lines. We will later fix the issue with the patch ourselves.

mysql 5.6.40 allows newlines in column names however the following code:

my @defs = $ddl =~ m/^(\s+`.*?),?$/gm;

breaks due to it detecting newlines as line ends. The 'm' argument at the end does this by auto-detecting lines by newline characters.

To correct this issue I've made use of zero-length assertions known as " positive lookback"

https://www.regular-expressions.info/lookaround.html

what does it do?

m/(?:(?<=,\n)|(?<=\(\n))(\s+`(?:.|\n)+?`.+?),?\n/g;

TLDR:

Treat the string as one long string and don't treat \n as the end of a line.

look for (\s+`(?:.|\n)+?`.+?),?\n

if one of those matches look at what precedes the string

if it's ',\n' or ')\n' the string matches. Only save what's in (\s+`(?:.|\n)+?`.+?),?\n

m/ is declaring this a matching regex.

(?:(?<=,\n)|(?<=(\n)) This is an OR statement including two look-behind clauses. The ?: tells the enclosing parentheses to not store the result as a variable. I've put the two look-behinds in this OR statement below this line:

(?<=,\n) Look behind the matched string for a comma followed by a newline, the comma must be there for this look behind to match.

(?<=(\n) Look behind the matched string for a open parentheses followed by a newline, the open parentheses must be there.

(\s+`(?:.|\n)+?`.+?),?\n This is the actual match. Match newline character followed by one or more spaces followed by back-tick followed by a character which can be any character or a newline one or more times, but don't be greedy and take the rest of the match into consideration. Followed by a back tick and any character one or more times. This match stops where there is a comma or failing that a newline following a back tick and some characters.

,?\n match a comma that may not be there followed by a newline.
/g don't stop if this pattern matches keep looking for more patterns to the end of the string.

* PT-1059 - Tools can't parse index names containing newlines

Placed fix from PR-369 into proper place and created test case for this fix.

---------

Co-authored-by: geneguido <31323560+geneguido@users.noreply.github.com>
2023-02-02 17:09:13 +03:00
グリアンドロ
163376af69 PT-2084 show slave status (#545)
* PT-2084 Readding SHOW SLAVE STATUS outputs to pt-stalk

Before, they were only gathered if version was 5.6 or prior.
Now, they are always gathered, no matter what version.
Since ${mysql_version} is only major.minor, we can't yet add
SHOW REPLICA STATUS, because it starts on 8.0.22 and we have
no way to check that. We will need to wait for 8.1 release.

* PT-2084 Ran update-modules to generate binaries with changes.

* PT-2084 Added checks for preemptive 8.1 SHOW SLAVE STATUS removal.

* PT-2084 Adding test case for SHOW SLAVE STATUS

* Update t/pt-stalk/pt-stalk-replication.t

---------

Co-authored-by: Sveta Smirnova <svetasmirnova@users.noreply.github.com>
2023-02-02 17:08:54 +03:00
Yoann La Cancellera
a7531c2cb9 PT-2116 pt-osc --null-to-not-null description is wrong (#580)
Old documentation was incomplete: old rows would not use the new user defined default value for the altered column.
Only new rows inserted after pt-osc would use the user defined default value.
Existing rows would only be converted to a default value derived from the datatype, as it used to be the case with mysql 5.6
2023-01-31 15:49:20 +03:00
Sveta Smirnova
f585c9922d PT-215-Documentation fix (#579)
Replaced pt-table-checksum with pt-archiver in the pt-archiver docs
2023-01-30 15:34:40 +03:00
Sveta Smirnova
c1696fd9f8 Release 3.5.1 (#574)
* RM-1153 - Percona Toolkit 3.5.1

Updated version for Perl files to 3.5.1
Updated Perl module Percona::Toolkit to version 3.5.1 and updated all scripts using this module
Manually updated Go programs
Moved ONLY_UPDATE_VERSION after definition of DATE and SERIES variables, so the script does not fail because of undefined variables

* RM-1153 - Percona Toolkit 3.5.1

Updated Changelog
Updated release date in docs/percona-toolkit.pod

* PT-2162 Release Notes 3.5.1 (#571)

modified:   Makefile.PL
	modified:   config/sphinx-build/conf.py
	modified:   docs/release_notes.rst
	new file:   docs/rn.3-5-1.txt

Co-authored-by: Alina Derkach <“alina.derkach@percona.com”>

* PT-7 Fix syntax (#573)

Co-authored-by: Alina Derkach <81975178+alina-derkach-oaza@users.noreply.github.com>
Co-authored-by: Alina Derkach <“alina.derkach@percona.com”>
Co-authored-by: EvgeniyPatlan <evgeniy.patlan@percona.com>
2023-01-23 17:51:17 +03:00
Sveta Smirnova
2f584c85db PT-2140 - Update modules in the main branch (#570)
* PT-2140 - Update modules in the main branch

Updated modules for:
- pt-align
- pt-archiver
- pt-config-diff
- pt-deadlock-logger
- pt-diskstats
- pt-duplicate-key-checker
- pt-fifo-split
- pt-find
- pt-fingerprint
- pt-fk-error-logger
- pt-heartbeat
- pt-index-usage
- pt-ioprofile
- pt-kill
- pt-mext
- pt-mysql-summary
- pt-online-schema-change
- pt-pmp
- pt-query-digest
- pt-show-grants
- pt-sift
- pt-slave-delay
- pt-slave-find
- pt-slave-restart
- pt-summary
- pt-table-sync
- pt-upgrade
- pt-variable-advisor

Added execute bit for pt-query-digest

* PT-2140 Update modules in the main branch

Fixed Daemon.pm plugin usage
Updated all tests, related to the Daemon plugin.
I intentionally did not fix failing tests, not related to Daemon plugin,
to avoid making this PR too big.

- bin/pt-archiver
	- PT-2141 - Fixed usage of Daemon.pm
	- PT-2141 - Updated t/pt-archiver/standard_options.t
- bin/pt-deadlock-logger
	- PT-2143 - Fixed usage of Daemon.pm
	- PT-2143 - Updated t/pt-deadlock-logger/standard_options.t
- bin/pt-fifo-split
	- PT-2144 - Fixed usage of Daemon.pm
	- PT-2144 - Updated t/pt-fifo-split/pt-fifo-split.t
- bin/pt-find
	- PT-2145 - Fixed usage of Daemon.pm
	- PT-2145 - Updated t/pt-find/pt-find.t
- bin/pt-fk-error-logger
	- PT-2146 - Fixed usage of Daemon.pm
	- PT-2146 - Updated t/pt-fk-error-logger/basics.t
- bin/pt-heartbeat
	- PT-2147 - Fixed usage of Daemon.pm
	- PT-2147 - Updated t/pt-heartbeat/standard_options.t, t/pt-heartbeat/basics.t
- bin/pt-kill
	- PT-2148 - Fixed usage of Daemon.pm
	- PT-2148 - Updated t/pt-kill/standard_options.t
- bin/pt-show-grants
	- PT-2152 - Fixed usage of Daemon.pm
	- PT-2152 - Updated t/pt-show-grants/standard_options.t
- bin/pt-slave-delay
	- Fixed usage of Daemon.pm
	- Updated t/pt-slave-delay/standard_options.t
- bin/pt-slave-find
	- PT-2153 - Fixed usage of Daemon.pm
	- PT-2153 -Updated t/pt-slave-find/pt-slave-find.t
- bin/pt-slave-restart
	- Fixed usage of Daemon.pm
	- Updated t/pt-slave-restart/pt-slave-restart.t
- bin/pt-table-sync
	- PT-2154 - Fixed usage of Daemon.pm
	- PT-2154 -Updated t/pt-table-sync/standard_options.t
2023-01-06 15:04:10 +03:00
Sveta Smirnova
a7efd8c94e PT-2138 - fix tests for pt table checksum (#568)
* PT-2138 - Fix tests for pt-table-checksum

- Updated t/pt-table-checksum/samples/default-results-8.0.txt and
t/pt-table-checksum/samples/static-chunk-size-results-8.0.txt
to support latest MySQL 8.0 version. Tests are now incompatible with elder 8.0 releases.
- Put fix for PT-136 into package RowChecksum
- Added execution bit for pt-online-schema-change

* PT-2138 - Fix tests for pt-table-checksum

- Fixed percona_test.load_data, so it really tests if LOAD DATA LOCAL INFILE is enabled
- Fixed option --[no]create-replicate-table, broken by commit c9836d5962

* PT-2138 - Fix tests for pt-table-checksum

- Enabled t/pt-table-checksum/error_handling.t for MySQL 8.0
- Fixed test t/pt-table-checksum/fnv_64.t and it's samples file
t/pt-table-checksum/samples/fnv64-sakila-city.txt
to reflect new function name convention and
changes after 62d84e5dba

* PT-2138 - Fix tests for pt-table-checksum

- Fixed t/pt-table-checksum/issue_1485195.t, so it checks only one table and
isn't get broken when we add tables into percona_test database
- Fixed typo in error output of bin/pt-table-checksum
- Skipped issue_47.t in 8.0 until https://jira.percona.com/browse/PT-1805 is fixed

* PT-2138 - Fix tests for pt-table-checksum

- Disabled pt-131.t for 8.0, because it does not have the QUERY_RESPONSE_TIME plugins
- Added SLOW_TESTS check to pt-1616.t
- Updated pt-226.t to include the fix for PT-1766
- For replication_filters.t: excluded false positive expression for tests 10 and 11 and added sys schema to the list of checked databases for 8.0
- Changed get_slaves in lib/MasterSlave.pm, so it returns slave's parent, required for wait_for_slaves in pt-table-checksum to work properly with chained slaves

* PT-2138 - Fix tests for pt-table-checksum

- Modified pt-204.t  to support 8.0 and diffs in system tables due to timestamps
Moved the fix for PT-1616 into the proper place: lib/NibbleIterator.pm

* PT-2138 - Fix tests for pt-table-checksum

- pxc.t -added mysql.proxies_priv into ignore list, because its timestamp is different on node
- pxc.t - removed FORK=pxc from statup options for slave (non-cluster) nodes
- pxc.t - disabled wsrep replication with help of the variable wsrep_on: sql_log_bin doesn't disable wsrep replication anymore. See https://jira.percona.com/browse/PXC-3464 for details
- Removed data.tar.gz from 5.7 sandbox configuration, because it has an outdated definition for Performance Schema
- Disabled pxc.t for version 8.0 until PT-1699 is fixed
- start-sandbox - removed the first line (ALTER USER) from the init file, because it was rewritten by the next echo command, and then repeated later.

* PT-2138 - Fix tests for pt-table-checksum

- Adopted issue_1485195.t and basics.t for MyRocks-enabled setup
- replication_filters.t - added sys schema to the list of expected schemas for 5.7 and 8.0
- issue_1485195.t - added checks for the existence of mysql.plugin, func, and proxies_priv tables
- added samples/pt-131-wipe.sql that uninstalls QRT plugin if it was earlier installed by this test
-adjusted return code in pt-204.t, because expected differences in mysql.proxies_priv

* Update lib/PerconaTest.pm

removed diagnostic code

Co-authored-by: Carlos Salguero <carlos.salguero@percona.com>

Co-authored-by: Carlos Salguero <carlos.salguero@percona.com>
2022-12-28 23:09:13 +03:00
Dario Minnucci
7e5c51d0fb Fix some spelling errors (#462)
Co-authored-by: Sveta Smirnova <svetasmirnova@users.noreply.github.com>
2022-12-01 15:54:24 +03:00
Carlos Salguero
882d8787c7 Release 3.5.0 (#563)
* Changelog and version for Release 3.5.0

* PT-2076 add rhel9 dependency

* PT-2128 Updated version to 3.5.0

Co-authored-by: svetasmirnova <sveta.smirnova@percona.com>
Co-authored-by: EvgeniyPatlan <evgeniy.patlan@percona.com>
2022-12-01 14:31:46 +03:00
Kazuya Yokogawa
bc3056f29d PT-2110 - Fixed PTDEBUG output with --ignore-engines option (#555) 2022-11-30 14:51:49 +03:00
Sveta Smirnova
cf8f824d5e PT-2107 merge community prs 466 298 58 53 (#554)
* Correcting specified package type in error message (#58)

* Docs table regex (#53)

* Add more specific info about regex

* noop change to trigger checks

* Fix typo (#298)

* Updated readme

* Fix typo in comment

Co-authored-by: Carlos Salguero <carlos.salguero@percona.com>

* Fix for PT-1799 (#466)

Co-authored-by: Daniel Hoherd <daniel.hoherd@gmail.com>
Co-authored-by: Daniël van Eeden <git@myname.nl>
Co-authored-by: Moritz Lenz <moritz.lenz@noris.de>
Co-authored-by: Carlos Salguero <carlos.salguero@percona.com>
2022-11-16 20:48:17 +03:00
E. Souhrada
9bd0fef2c2 PT-1739 - fix unexpected QRT disabling on slaves (#416) 2022-09-28 20:55:20 +03:00
kokopellis
60b3750d96 PT-1631 fix ioprofile ubuntu18.04, ubuntu16.04 support (#431) 2022-09-28 02:58:05 +03:00
dm-2
26522d20fd PT-2080 Add JSON logging to pt-kill (#543)
* Add JSON output support to pt-kill

Introduces new flags `--log-json` and `--log-json-fields` to control JSON output:
- `--log-json`: when combined with `--print`, outputs kill information in JSON format instead of plain text, with the same fields as the `kill_log` table when using `--log-dsn`
- `--log-json-fields`: when combined with `--log-json`, adds user-defined key-value pairs to the JSON document that is output (such as cluster name or hostname); parameter value is in the format of `key1:value1,key2:value2`

* Convert --log-json to write to file, remove dependency on --print, update docs

The `--log-json` parameter now takes a file path value, and writes JSON output to this file.

The `--print` parameter is now independent and does not need to be specified to use `--log-json`.

Docs updated for clarity.

* Add chksm to JSON output, make JSON key names consistent

- Adds chksm outout to JSON output to make it easier to group similar queries
- Moves chksm code so that it is always executed, is accessible within the `log-json` scope, and isn't duplicated
- Change JSON key names to consistently use uppercase letters at start of words, change `ts` to `Timestamp`

* Update docs to match changed field names

* Rename JSON parameter, output to STDOUT, fix tests, add test for --json, bug fix for --query-id

- `--json` parameter must now be used in conjunction with `--print`; JSON output replaces normal `--print` output when `--json` is also specified
- `--json` output is now sent to STDOUT instead of a specified file, as its new behaviour is to change the output format of `--print` instead of logging to a file
- Renamed `--log-json` and `--log-json-fields` parameters to `--json` and `--json-fields` to better represent their new behaviour
- Refactored checksum code to prevent test failures when `$query->{'Info'}` is empty
- Added test for `--json` and `--json-fields` parameters
- Fixed bug where specifying `--query-id` would cause errors when `$query->{'Info'}` is empty

* Fix typo: Unkown -> Unknown
2022-09-27 23:41:17 +03:00
Christian Bartolomäus
db32ba584b Fix some typos in documentation of pt-query-digest (#553) 2022-09-12 17:46:35 +03:00
Sveta Smirnova
19aa46d1a5 PT-1052 include numa information in pt tool output (#547)
* PT-1052: fix for pt-summary

* Re-added fix from commit 71fae6d117 into proper place: library directory
Added tests for PT-1052 (pt-summary only)

* Fixed tests for pt-summary,
set locale for pt-summary tests, so they are not affected by user environment.
Finished tests for PT-1052

* Added fix for PT-1983 into the proper place: library file

* PT-1052: fix for pt-stalk which now includes numastat data
2022-08-15 15:45:24 +03:00
I
a950b223ff Typo: Intsance -> Instance (#546) 2022-08-01 09:35:44 -03:00
Sveta Smirnova
ac3843bcb7 Pt 1897 lock information in8.0 in pt stalk (#544)
* PT-1897 pt-stalk on MySQL 8 not collecting lock information

For version prior 8.0 pt-stalk continue using Information Schema INNODB_LOCKS and INNODB_LOCK_WAITS tables for collecting lock information.
For version 8.0 and higher pt-stalk uses tables data_locks and data_lock_waits in Performance Schema

* PT-1897 - Better MariaDB support

We cannot simply compare version number with 8.0 to identify if lock tables are in Performance Schema,
because MariaDB 10.x still have them in the Information Schema.
Therefore added additional flag, indicating which syntax we should use

* Tests for PT-1897
2022-07-25 10:01:14 -03:00