PT-2138 - fix tests for pt table checksum (#568)

* PT-2138 - Fix tests for pt-table-checksum

- Updated t/pt-table-checksum/samples/default-results-8.0.txt and
t/pt-table-checksum/samples/static-chunk-size-results-8.0.txt
to support latest MySQL 8.0 version. Tests are now incompatible with elder 8.0 releases.
- Put fix for PT-136 into package RowChecksum
- Added execution bit for pt-online-schema-change

* PT-2138 - Fix tests for pt-table-checksum

- Fixed percona_test.load_data, so it really tests if LOAD DATA LOCAL INFILE is enabled
- Fixed option --[no]create-replicate-table, broken by commit c9836d5962

* PT-2138 - Fix tests for pt-table-checksum

- Enabled t/pt-table-checksum/error_handling.t for MySQL 8.0
- Fixed test t/pt-table-checksum/fnv_64.t and it's samples file
t/pt-table-checksum/samples/fnv64-sakila-city.txt
to reflect new function name convention and
changes after 62d84e5dba

* PT-2138 - Fix tests for pt-table-checksum

- Fixed t/pt-table-checksum/issue_1485195.t, so it checks only one table and
isn't get broken when we add tables into percona_test database
- Fixed typo in error output of bin/pt-table-checksum
- Skipped issue_47.t in 8.0 until https://jira.percona.com/browse/PT-1805 is fixed

* PT-2138 - Fix tests for pt-table-checksum

- Disabled pt-131.t for 8.0, because it does not have the QUERY_RESPONSE_TIME plugins
- Added SLOW_TESTS check to pt-1616.t
- Updated pt-226.t to include the fix for PT-1766
- For replication_filters.t: excluded false positive expression for tests 10 and 11 and added sys schema to the list of checked databases for 8.0
- Changed get_slaves in lib/MasterSlave.pm, so it returns slave's parent, required for wait_for_slaves in pt-table-checksum to work properly with chained slaves

* PT-2138 - Fix tests for pt-table-checksum

- Modified pt-204.t  to support 8.0 and diffs in system tables due to timestamps
Moved the fix for PT-1616 into the proper place: lib/NibbleIterator.pm

* PT-2138 - Fix tests for pt-table-checksum

- pxc.t -added mysql.proxies_priv into ignore list, because its timestamp is different on node
- pxc.t - removed FORK=pxc from statup options for slave (non-cluster) nodes
- pxc.t - disabled wsrep replication with help of the variable wsrep_on: sql_log_bin doesn't disable wsrep replication anymore. See https://jira.percona.com/browse/PXC-3464 for details
- Removed data.tar.gz from 5.7 sandbox configuration, because it has an outdated definition for Performance Schema
- Disabled pxc.t for version 8.0 until PT-1699 is fixed
- start-sandbox - removed the first line (ALTER USER) from the init file, because it was rewritten by the next echo command, and then repeated later.

* PT-2138 - Fix tests for pt-table-checksum

- Adopted issue_1485195.t and basics.t for MyRocks-enabled setup
- replication_filters.t - added sys schema to the list of expected schemas for 5.7 and 8.0
- issue_1485195.t - added checks for the existence of mysql.plugin, func, and proxies_priv tables
- added samples/pt-131-wipe.sql that uninstalls QRT plugin if it was earlier installed by this test
-adjusted return code in pt-204.t, because expected differences in mysql.proxies_priv

* Update lib/PerconaTest.pm

removed diagnostic code

Co-authored-by: Carlos Salguero <carlos.salguero@percona.com>

Co-authored-by: Carlos Salguero <carlos.salguero@percona.com>

bin/pt-online-schema-change

@@ -112,10 +112,10 @@ sub _d {
 # ###########################################################################
 # VersionCompare package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/VersionCompare.pm
 #   t/lib/VersionCompare.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package VersionCompare;
@@ -159,10 +159,10 @@ sub cmp {
 # ###########################################################################
 # OptionParser package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/OptionParser.pm
 #   t/lib/OptionParser.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package OptionParser;
@@ -1271,10 +1271,10 @@ if ( PTDEBUG ) {
 # ###########################################################################
 # Lmo::Utils package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Lmo/Utils.pm
 #   t/lib/Lmo/Utils.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Lmo::Utils;
@@ -1331,10 +1331,10 @@ sub _unimport_coderefs {
 # ###########################################################################
 # Lmo::Meta package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Lmo/Meta.pm
 #   t/lib/Lmo/Meta.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Lmo::Meta;
@@ -1388,10 +1388,10 @@ sub attributes_for_new {
 # ###########################################################################
 # Lmo::Object package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Lmo/Object.pm
 #   t/lib/Lmo/Object.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Lmo::Object;
@@ -1484,10 +1484,10 @@ sub meta {
 # ###########################################################################
 # Lmo::Types package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Lmo/Types.pm
 #   t/lib/Lmo/Types.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Lmo::Types;
@@ -1585,10 +1585,10 @@ sub _nested_constraints {
 # ###########################################################################
 # Lmo package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Lmo.pm
 #   t/lib/Lmo.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 BEGIN {
@@ -1920,10 +1920,10 @@ sub override {
 # ###########################################################################
 # VersionParser package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/VersionParser.pm
 #   t/lib/VersionParser.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package VersionParser;
@@ -2112,10 +2112,10 @@ no Lmo;
 # ###########################################################################
 # DSNParser package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/DSNParser.pm
 #   t/lib/DSNParser.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package DSNParser;
@@ -2557,10 +2557,10 @@ sub _d {
 # ###########################################################################
 # Daemon package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Daemon.pm
 #   t/lib/Daemon.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Daemon;
@@ -2811,10 +2811,10 @@ sub _d {
 # ###########################################################################
 # Quoter package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Quoter.pm
 #   t/lib/Quoter.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Quoter;
@@ -2964,10 +2964,10 @@ sub _d {
 # ###########################################################################
 # TableNibbler package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/TableNibbler.pm
 #   t/lib/TableNibbler.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package TableNibbler;
@@ -3230,10 +3230,10 @@ sub _d {
 # ###########################################################################
 # TableParser package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/TableParser.pm
 #   t/lib/TableParser.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package TableParser;
@@ -3663,10 +3663,10 @@ sub _d {
 # ###########################################################################
 # Progress package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Progress.pm
 #   t/lib/Progress.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Progress;
@@ -3810,10 +3810,10 @@ sub _d {
 # ###########################################################################
 # Retry package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Retry.pm
 #   t/lib/Retry.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Retry;
@@ -3890,10 +3890,10 @@ sub _d {
 # ###########################################################################
 # Cxn package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Cxn.pm
 #   t/lib/Cxn.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Cxn;
@@ -4154,10 +4154,10 @@ sub _d {
 # ###########################################################################
 # MasterSlave package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/MasterSlave.pm
 #   t/lib/MasterSlave.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package MasterSlave;
@@ -4294,7 +4294,7 @@ sub recurse_to_slaves {
    my $slave_dsn = $dsn;
    if ($slave_user) {
       $slave_dsn->{u} = $slave_user;
-      PTDEBUG && _d("Using slave user $slave_user on ".$slave_dsn->{h}.":".($slave_dsn->{P}?$slave_dsn->{P}:""));
+      PTDEBUG && _d("Using slave user $slave_user on ".$slave_dsn->{h}.":".$slave_dsn->{P});
    }
    if ($slave_password) {
       $slave_dsn->{p} = $slave_password;
@@ -4969,10 +4969,10 @@ sub _d {
 # ###########################################################################
 # ReplicaLagWaiter package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/ReplicaLagWaiter.pm
 #   t/lib/ReplicaLagWaiter.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package ReplicaLagWaiter;
@@ -5017,8 +5017,6 @@ sub wait {
    my $pr_callback;
    my $pr_first_report;
 
-   ### refresh list of slaves. In: self passed to wait()
-   ### Returns: new slave list
    my $pr_refresh_slave_list = sub {
       my ($self) = @_;
       my ($slaves, $refresher) = ($self->{slaves}, $self->{get_slaves_cb});
@@ -5037,8 +5035,6 @@ sub wait {
    $slaves = $pr_refresh_slave_list->($self);
 
    if ( $pr ) {
-      # If you use the default Progress report callback, you'll need to
-      # to add Transformers.pm to this tool.
       $pr_callback = sub {
          my ($fraction, $elapsed, $remaining, $eta, $completed) = @_;
          my $dsn_name = $worst->{cxn}->name();
@@ -5050,33 +5046,28 @@ sub wait {
             if ($self->{fail_on_stopped_replication}) {
                 die 'replication is stopped';
             }
-            print STDERR "(1) Replica '$dsn_name' is stopped.  Waiting.\n";
+            print STDERR "Replica $dsn_name is stopped.  Waiting.\n";
          }
          return;
       };
       $pr->set_callback($pr_callback);
 
-      # If a replic is stopped, don't wait 30s (or whatever interval)
-      # to report this.  Instead, report it once, immediately, then
-      # keep reporting it every interval.
       $pr_first_report = sub {
          my $dsn_name = $worst->{cxn}->name();
          if ( !defined $worst->{lag} ) {
             if ($self->{fail_on_stopped_replication}) {
                 die 'replication is stopped';
             }
-            print STDERR "(2) Replica '$dsn_name' is stopped.  Waiting.\n";
+            print STDERR "Replica $dsn_name is stopped.  Waiting.\n";
          }
          return;
       };
    }
 
-   # First check all slaves.
    my @lagged_slaves = map { {cxn=>$_, lag=>undef} } @$slaves;
    while ( $oktorun->() && @lagged_slaves ) {
       PTDEBUG && _d('Checking slave lag');
 
-      ### while we were waiting our list of slaves may have changed
       $slaves = $pr_refresh_slave_list->($self);
       my $watched = 0;
       @lagged_slaves = grep {
@@ -5102,10 +5093,8 @@ sub wait {
          }
       }
 
-      # Remove slaves that aren't lagging.
       @lagged_slaves = grep { defined $_ } @lagged_slaves;
       if ( @lagged_slaves ) {
-         # Sort lag, undef is highest because it means the slave is stopped.
          @lagged_slaves = reverse sort {
               defined $a->{lag} && defined $b->{lag} ? $a->{lag} <=> $b->{lag}
             : defined $a->{lag}                      ? -1
@@ -5116,10 +5105,6 @@ sub wait {
             $worst->{lag}, 'on', Dumper($worst->{cxn}->dsn()));
 
          if ( $pr ) {
-            # There's no real progress because we can't estimate how long
-            # it will take all slaves to catch up.  The progress reports
-            # are just to inform the user every 30s which slave is still
-            # lagging this most.
             $pr->update(
                sub { return 0; },
                first_report => $pr_first_report,
@@ -5153,10 +5138,10 @@ sub _d {
 # ###########################################################################
 # FlowControlWaiter package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/FlowControlWaiter.pm
 #   t/lib/FlowControlWaiter.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package FlowControlWaiter;
@@ -5258,10 +5243,10 @@ sub _d {
 # ###########################################################################
 # MySQLStatusWaiter package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/MySQLStatusWaiter.pm
 #   t/lib/MySQLStatusWaiter.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package MySQLStatusWaiter;
@@ -5389,7 +5374,7 @@ sub wait {
             die "$var=$val exceeds its critical threshold "
                . "$self->{critical_val_for}->{$var}\n";
          }
-         if ( $val >= $self->{max_val_for}->{$var} ) {
+         if ( $val && $val >= $self->{max_val_for}->{$var} ) {
             $vals_too_high{$var} = $val;
          }
          else {
@@ -5458,10 +5443,10 @@ sub _d {
 # ###########################################################################
 # WeightedAvgRate package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/WeightedAvgRate.pm
 #   t/lib/WeightedAvgRate.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package WeightedAvgRate;
@@ -5527,10 +5512,10 @@ sub _d {
 # ###########################################################################
 # NibbleIterator package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/NibbleIterator.pm
 #   t/lib/NibbleIterator.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package NibbleIterator;
@@ -6323,10 +6308,10 @@ sub _d {
 # ###########################################################################
 # Transformers package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Transformers.pm
 #   t/lib/Transformers.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Transformers;
@@ -6676,10 +6661,10 @@ sub _d {
 # ###########################################################################
 # CleanupTask package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/CleanupTask.pm
 #   t/lib/CleanupTask.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package CleanupTask;
@@ -6738,10 +6723,10 @@ sub _d {
 # ###########################################################################
 # IndexLength package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/IndexLength.pm
 #   t/lib/IndexLength.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 
@@ -6900,10 +6885,10 @@ sub _d {
 # ###########################################################################
 # HTTP::Micro package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/HTTP/Micro.pm
 #   t/lib/HTTP/Micro.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package HTTP::Micro;
@@ -7553,10 +7538,10 @@ if ( $INC{"IO/Socket/SSL.pm"} ) {
 # ###########################################################################
 # VersionCheck package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/VersionCheck.pm
 #   t/lib/VersionCheck.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package VersionCheck;
@@ -7657,13 +7642,15 @@ sub version_check {
          return;
       }
       PTDEBUG && _d('Using', $protocol);
+      my $url = $args{url}                       # testing
+                || $ENV{PERCONA_VERSION_CHECK_URL}  # testing
+                || "$protocol://v.percona.com";
+      PTDEBUG && _d('API URL:', $url);
 
       my $advice = pingback(
          instances => $instances_to_check,
          protocol  => $protocol,
-         url       => $args{url}                       # testing
+         url       => $url,
-                   || $ENV{PERCONA_VERSION_CHECK_URL}  # testing
-                   || "$protocol://v.percona.com",
       );
       if ( $advice ) {
          PTDEBUG && _d('Advice:', Dumper($advice));
@@ -7821,9 +7808,14 @@ sub get_uuid {
     my $filename = $ENV{"HOME"} . $uuid_file;
     my $uuid = _generate_uuid();
 
-    open(my $fh, '>', $filename) or die "Could not open file '$filename' $!";
+    my $fh;
+    eval {
+        open($fh, '>', $filename);
+    };
+    if (!$EVAL_ERROR) {
         print $fh $uuid;
         close $fh;
+    }
 
     return $uuid;
 }   
@@ -7889,8 +7881,9 @@ sub pingback {
       general_id => get_uuid(),
    );
 
+   my $tool_name = $ENV{XTRABACKUP_VERSION} ? "Percona XtraBackup" : File::Basename::basename($0);
    my $client_response = {
-      headers => { "X-Percona-Toolkit-Tool" => File::Basename::basename($0) },
+      headers => { "X-Percona-Toolkit-Tool" => $tool_name },
       content => $client_content,
    };
    PTDEBUG && _d('Client response:', Dumper($client_response));
@@ -7973,6 +7966,7 @@ my %sub_for_type = (
    perl_version        => \&get_perl_version,
    perl_module_version => \&get_perl_module_version,
    mysql_variable      => \&get_mysql_variable,
+   xtrabackup          => \&get_xtrabackup_version,
 );
 
 sub valid_item {
@@ -8100,6 +8094,10 @@ sub get_perl_version {
    return $version;
 }
 
+sub get_xtrabackup_version {
+    return $ENV{XTRABACKUP_VERSION};
+}
+
 sub get_perl_module_version {
    my (%args) = @_;
    my $item = $args{item};
@@ -8177,10 +8175,10 @@ sub _d {
 # ###########################################################################
 # Percona::XtraDB::Cluster package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Percona/XtraDB/Cluster.pm
 #   t/lib/Percona/XtraDB/Cluster.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Percona::XtraDB::Cluster;

bin/pt-table-checksum

@@ -113,10 +113,10 @@ sub _d {
 # ###########################################################################
 # HTTP::Micro package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/HTTP/Micro.pm
 #   t/lib/HTTP/Micro.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package HTTP::Micro;
@@ -766,10 +766,10 @@ if ( $INC{"IO/Socket/SSL.pm"} ) {
 # ###########################################################################
 # VersionCheck package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/VersionCheck.pm
 #   t/lib/VersionCheck.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package VersionCheck;
@@ -870,13 +870,15 @@ sub version_check {
          return;
       }
       PTDEBUG && _d('Using', $protocol);
+      my $url = $args{url}                       # testing
+                || $ENV{PERCONA_VERSION_CHECK_URL}  # testing
+                || "$protocol://v.percona.com";
+      PTDEBUG && _d('API URL:', $url);
 
       my $advice = pingback(
          instances => $instances_to_check,
          protocol  => $protocol,
-         url       => $args{url}                       # testing
+         url       => $url,
-                   || $ENV{PERCONA_VERSION_CHECK_URL}  # testing
-                   || "$protocol://v.percona.com",
       );
       if ( $advice ) {
          PTDEBUG && _d('Advice:', Dumper($advice));
@@ -1034,9 +1036,14 @@ sub get_uuid {
     my $filename = $ENV{"HOME"} . $uuid_file;
     my $uuid = _generate_uuid();
 
-    open(my $fh, '>', $filename) or die "Could not open file '$filename' $!";
+    my $fh;
+    eval {
+        open($fh, '>', $filename);
+    };
+    if (!$EVAL_ERROR) {
         print $fh $uuid;
         close $fh;
+    }
 
     return $uuid;
 }   
@@ -1102,8 +1109,9 @@ sub pingback {
       general_id => get_uuid(),
    );
 
+   my $tool_name = $ENV{XTRABACKUP_VERSION} ? "Percona XtraBackup" : File::Basename::basename($0);
    my $client_response = {
-      headers => { "X-Percona-Toolkit-Tool" => File::Basename::basename($0) },
+      headers => { "X-Percona-Toolkit-Tool" => $tool_name },
       content => $client_content,
    };
    PTDEBUG && _d('Client response:', Dumper($client_response));
@@ -1186,6 +1194,7 @@ my %sub_for_type = (
    perl_version        => \&get_perl_version,
    perl_module_version => \&get_perl_module_version,
    mysql_variable      => \&get_mysql_variable,
+   xtrabackup          => \&get_xtrabackup_version,
 );
 
 sub valid_item {
@@ -1313,6 +1322,10 @@ sub get_perl_version {
    return $version;
 }
 
+sub get_xtrabackup_version {
+    return $ENV{XTRABACKUP_VERSION};
+}
+
 sub get_perl_module_version {
    my (%args) = @_;
    my $item = $args{item};
@@ -1390,10 +1403,10 @@ sub _d {
 # ###########################################################################
 # DSNParser package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/DSNParser.pm
 #   t/lib/DSNParser.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package DSNParser;
@@ -1835,10 +1848,10 @@ sub _d {
 # ###########################################################################
 # OptionParser package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/OptionParser.pm
 #   t/lib/OptionParser.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package OptionParser;
@@ -2946,10 +2959,10 @@ if ( PTDEBUG ) {
 # ###########################################################################
 # Lmo::Utils package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Lmo/Utils.pm
 #   t/lib/Lmo/Utils.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Lmo::Utils;
@@ -3006,10 +3019,10 @@ sub _unimport_coderefs {
 # ###########################################################################
 # Lmo::Meta package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Lmo/Meta.pm
 #   t/lib/Lmo/Meta.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Lmo::Meta;
@@ -3063,10 +3076,10 @@ sub attributes_for_new {
 # ###########################################################################
 # Lmo::Object package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Lmo/Object.pm
 #   t/lib/Lmo/Object.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Lmo::Object;
@@ -3159,10 +3172,10 @@ sub meta {
 # ###########################################################################
 # Lmo::Types package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Lmo/Types.pm
 #   t/lib/Lmo/Types.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Lmo::Types;
@@ -3260,10 +3273,10 @@ sub _nested_constraints {
 # ###########################################################################
 # Lmo package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Lmo.pm
 #   t/lib/Lmo.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 BEGIN {
@@ -3595,10 +3608,10 @@ sub override {
 # ###########################################################################
 # Cxn package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Cxn.pm
 #   t/lib/Cxn.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Cxn;
@@ -3746,7 +3759,7 @@ sub name {
 
 sub description {
    my ($self) = @_;
-   return sprintf("%s -> %s:%s", $self->name(), $self->{dsn}->{h}, $self->{dsn}->{P} || 'socket');
+   return sprintf("%s -> %s:%s", $self->name(), $self->{dsn}->{h} || 'localhost' , $self->{dsn}->{P} || 'socket');
 }
 
 sub get_id {
@@ -3859,10 +3872,10 @@ sub _d {
 # ###########################################################################
 # Percona::XtraDB::Cluster package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Percona/XtraDB/Cluster.pm
 #   t/lib/Percona/XtraDB/Cluster.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Percona::XtraDB::Cluster;
@@ -4060,10 +4073,10 @@ sub _d {
 # ###########################################################################
 # Quoter package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Quoter.pm
 #   t/lib/Quoter.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Quoter;
@@ -4099,6 +4112,8 @@ sub quote_val {
    return $val if $val =~ m/^0x[0-9a-fA-F]+$/  # quote hex data
                   && !$args{is_char};          # unless is_char is true
 
+   return $val if $args{is_float};
+
    $val =~ s/(['\\])/\\$1/g;
    return "'$val'";
 }
@@ -4211,10 +4226,10 @@ sub _d {
 # ###########################################################################
 # VersionParser package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/VersionParser.pm
 #   t/lib/VersionParser.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package VersionParser;
@@ -4233,8 +4248,6 @@ use overload (
 
 use Carp ();
 
-our $VERSION = 0.01;
-
 has major => (
     is       => 'ro',
     isa      => 'Int',
@@ -4405,10 +4418,10 @@ no Lmo;
 # ###########################################################################
 # TableParser package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/TableParser.pm
 #   t/lib/TableParser.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package TableParser;
@@ -4838,10 +4851,10 @@ sub _d {
 # ###########################################################################
 # TableNibbler package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/TableNibbler.pm
 #   t/lib/TableNibbler.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package TableNibbler;
@@ -5104,10 +5117,10 @@ sub _d {
 # ###########################################################################
 # MasterSlave package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/MasterSlave.pm
 #   t/lib/MasterSlave.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package MasterSlave;
@@ -5188,7 +5201,7 @@ sub get_slaves {
                   $slave_dsn->{p} = $o->get('slave-password');
                   PTDEBUG && _d("Slave password set");
                }
-               push @$slaves, $make_cxn->(dsn => $slave_dsn, dbh => $dbh);
+               push @$slaves, $make_cxn->(dsn => $slave_dsn, dbh => $dbh, parent => $parent);
                return;
             },
          }
@@ -5919,10 +5932,10 @@ sub _d {
 # ###########################################################################
 # RowChecksum package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/RowChecksum.pm
 #   t/lib/RowChecksum.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package RowChecksum;
@@ -6287,9 +6300,9 @@ sub find_replication_differences {
    }
    my ($dbh, $repl_table) = @args{@required_args};
 
+    
    my $tries = $self->{'OptionParser'}->get('replicate-check-retries') || 1; 
    my $diffs;
-
    while ($tries--) {
       my $sql
          = "SELECT CONCAT(db, '.', tbl) AS `table`, "
@@ -6329,10 +6342,10 @@ sub _d {
 # ###########################################################################
 # NibbleIterator package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/NibbleIterator.pm
 #   t/lib/NibbleIterator.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package NibbleIterator;
@@ -6493,6 +6506,9 @@ sub _nibble_params {
       );
       PTDEBUG && _d('Ascend params:', Dumper($asc));
 
+      my $force_concat_enums;
+
+
       my $from     = "$tbl->{name} FORCE INDEX(`$index`)";
       my $order_by = join(', ', map {$q->quote($_)} @{$index_cols});
       my $order_by_dec = join(' DESC,', map {$q->quote($_)} @{$index_cols});
@@ -6974,7 +6990,7 @@ sub _get_bounds {
       if (    defined $nibble->{lower_boundary}
            && defined $nibble->{upper_boundary} ) {
          my $sth = $dbh->prepare($self->{resume_lb_sql});
-         my @ub  = split ',', $nibble->{upper_boundary};
+         my @ub = $self->{Quoter}->deserialize_list($nibble->{upper_boundary});
          PTDEBUG && _d($sth->{Statement}, 'params:', @ub);
          $sth->execute(@ub);
          $self->{next_lower} = $sth->fetchrow_arrayref();
@@ -7122,10 +7138,10 @@ sub _d {
 # ###########################################################################
 # OobNibbleIterator package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/OobNibbleIterator.pm
 #   t/lib/OobNibbleIterator.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package OobNibbleIterator;
@@ -7312,10 +7328,10 @@ sub _d {
 # ###########################################################################
 # Daemon package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Daemon.pm
 #   t/lib/Daemon.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Daemon;
@@ -8044,10 +8060,10 @@ sub _d {
 # ###########################################################################
 # Retry package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Retry.pm
 #   t/lib/Retry.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Retry;
@@ -8124,10 +8140,10 @@ sub _d {
 # ###########################################################################
 # Transformers package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Transformers.pm
 #   t/lib/Transformers.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Transformers;
@@ -8477,10 +8493,10 @@ sub _d {
 # ###########################################################################
 # Progress package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Progress.pm
 #   t/lib/Progress.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Progress;
@@ -8624,10 +8640,10 @@ sub _d {
 # ###########################################################################
 # ReplicaLagWaiter package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/ReplicaLagWaiter.pm
 #   t/lib/ReplicaLagWaiter.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package ReplicaLagWaiter;
@@ -8671,6 +8687,24 @@ sub wait {
    my $worst;  # most lagging slave
    my $pr_callback;
    my $pr_first_report;
+
+   my $pr_refresh_slave_list = sub {
+      my ($self) = @_;
+      my ($slaves, $refresher) = ($self->{slaves}, $self->{get_slaves_cb});
+      return $slaves if ( not defined $refresher );
+      my $before = join ' ', sort map {$_->name()} @$slaves;
+      $slaves = $refresher->();
+      my $after = join ' ', sort map {$_->name()} @$slaves;
+      if ($before ne $after) {
+         $self->{slaves} = $slaves;
+         printf STDERR "Slave set to watch has changed\n  Was: %s\n  Now: %s\n",
+            $before, $after;
+      }
+      return($self->{slaves});
+   };
+
+   $slaves = $pr_refresh_slave_list->($self);
+
    if ( $pr ) {
       $pr_callback = sub {
          my ($fraction, $elapsed, $remaining, $eta, $completed) = @_;
@@ -8704,8 +8738,22 @@ sub wait {
    my @lagged_slaves = map { {cxn=>$_, lag=>undef} } @$slaves;
    while ( $oktorun->() && @lagged_slaves ) {
       PTDEBUG && _d('Checking slave lag');
+
+      $slaves = $pr_refresh_slave_list->($self);
+      my $watched = 0;
+      @lagged_slaves = grep {
+         my $slave_name = $_->{cxn}->name();
+         grep {$slave_name eq $_->name()} @{$slaves // []}
+                            } @lagged_slaves;
+
       for my $i ( 0..$#lagged_slaves ) {
-         my $lag = $get_lag->($lagged_slaves[$i]->{cxn});
+         my $lag;
+         eval {
+             $lag = $get_lag->($lagged_slaves[$i]->{cxn});
+         };
+         if ($EVAL_ERROR) {
+             die $EVAL_ERROR;
+         }
          PTDEBUG && _d($lagged_slaves[$i]->{cxn}->name(),
             'slave lag:', $lag);
          if ( !defined $lag || $lag > $max_lag ) {
@@ -8775,23 +8823,13 @@ sub _d {
 # Place, Suite 330, Boston, MA  02111-1307  USA.
 # ###########################################################################
 # MySQLConfig package
+# This package is a copy without comments from the original.  The original
+# with comments and its test file can be found in the GitHub repository at,
+#   lib/MySQLConfig.pm
+#   t/lib/MySQLConfig.t
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
-# Package: MySQLConfig
-# MySQLConfig parses and encapsulates system variables and values from
-# SHOW VARIABLES, option files, mysqld --help --verbose or my_print_defaults.
-# A MySQLConfig object represents how MySQL is or would be configured given
-# one of those inputs.  If the input is SHOW VARIABLES, then the config is
-# acive, i.e. MySQL's running config.  All other inputs are inactive, i.e.
-# how MySQL should or would be running if started with the config.
-#
-# Inactive configs are made to mimic SHOW VARIABLES so that MySQLConfig
-# objects can be reliably compared with MySQLConfigComparer.  This is
-# necessary because the inputs are different in how they list values,
-# how they treat variables with optional values, etc.
-#
-# Only variables present in the input are saved in the MySQLConfig object.
-# So if <has()> returns false, then the variable did not appear in the input.
 package MySQLConfig;
 
 use strict;
@@ -8809,22 +8847,6 @@ my %can_be_duplicate = (
    replicate_do_db             => 1,
 );
 
-# Sub: new
-#
-# Parameters:
-#   %args - Arguments
-#
-# Arguments:
-#   file       - Filename of an option file, or containing output of
-#                mysqld --help --verbose, my_print_defaults or SHOW VARIABLES
-#   output     - Text output of one of ^ if you want to slurp the file manually
-#   result_set - Arrayref of SHOW VARIABLES
-#   dbh        - dbh to get SHOW VARIABLES from
-#   TextResultSetParser - <TextResultSetParser> object if file or output
-#                         arg is given
-#
-# Returns:
-#   MySQLConfig object
 sub new {
    my ( $class, %args ) = @_;
    my @requires_one_of = qw(file output result_set dbh);
@@ -8901,7 +8923,6 @@ sub handle_special_vars {
             $vars->{$var} = $val;
          }
       }
-      # Delete from vars, but not from dupes, since we still want that
       delete $vars->{wsrep_provider_options};
    }
 
@@ -9010,10 +9031,6 @@ sub parse_show_variables {
    return \%config;
 }
 
-# Parse "mysqld --help --verbose" and return a hashref of variable=>values
-# and an arrayref of default defaults files if possible.  The "default
-# defaults files" are the defaults file that mysqld reads by default if no
-# defaults file is explicitly given by --default-file.
 sub parse_mysqld {
    my ( %args ) = @_;
    my @required_args = qw(output);
@@ -9022,9 +9039,6 @@ sub parse_mysqld {
    }
    my ($output) = @args{@required_args};
 
-   # First look for the list of option files like
-   #   Default options are read from the following files in the given order:
-   #   /etc/my.cnf /usr/local/mysql/etc/my.cnf ~/.my.cnf
    my @opt_files;
    if ( $output =~ m/^Default options are read.+\n/mg ) {
       my ($opt_files) = $output =~ m/\G^(.+)\n/m;
@@ -9036,32 +9050,13 @@ sub parse_mysqld {
       PTDEBUG && _d("mysqld help output doesn't list option files");
    }
 
-   # The list of sys vars and their default vals begins like:
-   #   Variables (--variable-name=value)
-   #   and boolean options {FALSE|TRUE}  Value (after reading options)
-   #   --------------------------------- -----------------------------
-   #   help                              TRUE
-   #   abort-slave-event-count           0
-   # So we search for that line of hypens.
-   #
-   # It also ends with something like
-   #
-   #   wait_timeout                      28800
-   #
-   #   To see what values a running MySQL server is using, type
-   #   'mysqladmin variables' instead of 'mysqld --verbose --help'.
-   #
-   # So try to find it by locating a double newline, and strip it away
    if ( $output !~ m/^-+ -+$(.+?)(?:\n\n.+)?\z/sm ) {
       PTDEBUG && _d("mysqld help output doesn't list vars and vals");
       return;
    }
 
-   # Grab the varval list.
    my $varvals = $1;
 
-   # Parse the "var  val" lines.  2nd retval is duplicates but there
-   # shouldn't be any with mysqld.
    my ($config, undef) = _parse_varvals(
       qr/^(\S+)(.*)$/,
       $varvals,
@@ -9070,8 +9065,6 @@ sub parse_mysqld {
    return $config, \@opt_files;
 }
 
-# Parse "my_print_defaults" output and return a hashref of variable=>values
-# and a hashref of any duplicated variables.
 sub parse_my_print_defaults {
    my ( %args ) = @_;
    my @required_args = qw(output);
@@ -9080,7 +9073,6 @@ sub parse_my_print_defaults {
    }
    my ($output) = @args{@required_args};
 
-   # Parse the "--var=val" lines.
    my ($config, $dupes) = _parse_varvals(
       qr/^--([^=]+)(?:=(.*))?$/,
       $output,
@@ -9089,8 +9081,6 @@ sub parse_my_print_defaults {
    return $config, $dupes;
 }
 
-# Parse the [mysqld] section of an option file and return a hashref of
-# variable=>values and a hashref of any duplicated variables.
 sub parse_option_file {
    my ( %args ) = @_;
    my @required_args = qw(output);
@@ -9102,7 +9092,6 @@ sub parse_option_file {
    my ($mysqld_section) = $output =~ m/\[mysqld\](.+?)(?:^\s*\[\w+\]|\Z)/xms;
    die "Failed to parse the [mysqld] section" unless $mysqld_section;
 
-   # Parse the "var=val" lines.
    my ($config, $dupes) = _parse_varvals(
       qr/^([^=]+)(?:=(.*))?$/,
       $mysqld_section,
@@ -9111,13 +9100,6 @@ sub parse_option_file {
    return $config, $dupes;
 }
 
-# Called by _parse_varvals(), takes two arguments: a regex, and
-# a string to match against. The string will be split in lines,
-# and each line will be matched against the regex.
-# The regex must return to captures, although the second doesn't
-# have to match anything.
-# Returns a hashref of arrayrefs ala
-# { port => [ 12345, 12346 ], key_buffer_size => [ "16M" ] }
 sub _preprocess_varvals {
    my ($re, $to_parse) = @_;
 
@@ -9134,18 +9116,14 @@ sub _preprocess_varvals {
 
       my ($var, $val) = ($1, $2);
       
-      # Variable names are usually specified like "log-bin"
-      # but in SHOW VARIABLES they're all like "log_bin".
       $var =~ tr/-/_/;
 
-      # Remove trailing comments
       $var =~ s/\s*#.*$//;
 
       if ( !defined $val ) {
          $val = '';
       }
       
-      # Strip leading and trailing whitespace.
       for my $item ($var, $val) {
          $item =~ s/^\s+//;
          $item =~ s/\s+$//;
@@ -9157,26 +9135,16 @@ sub _preprocess_varvals {
    return \%vars;
 }
 
-# Parses a string of variables and their values ("varvals"), returns two
-# hashrefs: one with normalized variable=>value, the other with duplicate
-# vars.
 sub _parse_varvals {
    my ( $vars ) = _preprocess_varvals(@_);
 
-   # Config built from parsing the given varvals.
    my %config;
 
-   # Discover duplicate vars.
    my %duplicates;
 
    while ( my ($var, $vals) = each %$vars ) {
       my $val = _process_val( pop @$vals );
-      # If the variable has duplicates, then @$vals will contain
-      # the rest of the values
       if ( @$vals && !$can_be_duplicate{$var} ) {
-         # The var is a duplicate (in the bad sense, i.e. where user is
-         # probably unaware that there's two different values for this var
-         # but only the last is used).
          PTDEBUG && _d("Duplicate var:", $var);
          foreach my $current_val ( map { _process_val($_) } @$vals ) {
             push @{$duplicates{$var} ||= []}, $current_val;
@@ -9185,7 +9153,6 @@ sub _parse_varvals {
 
       PTDEBUG && _d("Var:", $var, "val:", $val);
 
-      # Save this var-val.
       $config{$var} = $val;
    }
 
@@ -9204,16 +9171,13 @@ sub _process_val {
    my ($val) = @_;
 
    if ( $val =~ $quote_re ) {
-      # If it matches the quote re, then $2 holds the value
       $val = $2;
    }
    else {
-      # Otherwise, remove possible trailing comments
       $val =~ s/\s*#.*//;
    }
 
    if ( my ($num, $factor) = $val =~ m/(\d+)([KMGT])b?$/i ) {
-      # value is a size like 1k, 16M, etc.
       my %factor_for = (
          k => 1_024,
          m => 1_048_576,
@@ -9228,17 +9192,6 @@ sub _process_val {
    return $val;
 }
 
-# Sub: _mimic_show_variables
-#   Make the variables' values mimic SHOW VARIABLES.  Different output formats
-#   list values differently.  To make comparisons easier, outputs are made to
-#   mimic SHOW VARIABLES.
-#
-# Parameters:
-#   %args - Arguments
-#
-# Required Arguments:
-#   vars   - Hashref of variables-values
-#   format - Config output format (mysqld, option_file, etc.)
 sub _mimic_show_variables {
    my ( %args ) = @_;
    my @required_args = qw(vars format);
@@ -9250,21 +9203,11 @@ sub _mimic_show_variables {
    foreach my $var ( keys %$vars ) {
       if ( $vars->{$var} eq '' ) {
          if ( $format eq 'mysqld' ) {
-            # mysqld lists "(No default value)" for certain variables
-            # that are not set/configured.  _parse_varvals() turns this
-            # into a blank string.  For most vars this means there's no
-            # value and SHOW VARIABLES will similarly show no value.
-            # But for log*, skip* and ignore* vars, SHOW VARIABLES will
-            # show OFF.  But, log_error is an exception--it's practically
-            # always on.
             if ( $var ne 'log_error' && $var =~ m/^(?:log|skip|ignore)/ ) {
                $vars->{$var} = 'OFF';
             }
          }
          else {
-            # Output formats other than mysqld (e.g. option file), if
-            # a variable is listed then it's enabled, like --skip-federated.
-            # SHOW VARIBLES will show ON for these.
             $vars->{$var} = 'ON';
          }
       }
@@ -9292,55 +9235,43 @@ sub _get_version {
    return $version;
 }
 
-# #############################################################################
-# Accessor methods.
-# #############################################################################
 
-# Returns true if this MySQLConfig obj has the given variable.
 sub has {
    my ( $self, $var ) = @_;
    return exists $self->{vars}->{$var};
 }
 
-# Return the value of the given variable.
 sub value_of {
    my ( $self, $var ) = @_;
    return unless $var;
    return $self->{vars}->{$var};
 }
 
-# Return hashref of all variables.
 sub variables {
    my ( $self, %args ) = @_;
    return $self->{vars};
 }
 
-# Return hashref of duplicate variables.
 sub duplicate_variables {
    my ( $self ) = @_;
    return $self->{duplicate_vars};
 }
 
-# Return arrayref of option files.
 sub option_files {
    my ( $self ) = @_;
    return $self->{option_files};
 }
 
-# Return MySQL version.
 sub mysql_version {
    my ( $self ) = @_;
    return $self->{mysql_version};
 }
 
-# Return the config file format (mysqld, option file, etc.)
 sub format {
    my ( $self ) = @_;
    return $self->{format};
 }
 
-# Return true if the config is active (i.e. the effective config
-# that MySQL is using; only true if config is from SHOW VARIABLES).
 sub is_active {
    my ( $self ) = @_;
    return $self->{dbh} ? 1 : 0;
@@ -9379,10 +9310,10 @@ sub _d {
 # ###########################################################################
 # MySQLStatusWaiter package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/MySQLStatusWaiter.pm
 #   t/lib/MySQLStatusWaiter.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package MySQLStatusWaiter;
@@ -9510,7 +9441,7 @@ sub wait {
             die "$var=$val exceeds its critical threshold "
                . "$self->{critical_val_for}->{$var}\n";
          }
-         if ( $val >= $self->{max_val_for}->{$var} ) {
+         if ( $val && $val >= $self->{max_val_for}->{$var} ) {
             $vals_too_high{$var} = $val;
          }
          else {
@@ -9579,10 +9510,10 @@ sub _d {
 # ###########################################################################
 # WeightedAvgRate package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/WeightedAvgRate.pm
 #   t/lib/WeightedAvgRate.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package WeightedAvgRate;
@@ -9648,10 +9579,10 @@ sub _d {
 # ###########################################################################
 # IndexLength package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/IndexLength.pm
 #   t/lib/IndexLength.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 
@@ -9810,10 +9741,10 @@ sub _d {
 # ###########################################################################
 # Runtime package
 # This package is a copy without comments from the original.  The original
-# with comments and its test file can be found in the Bazaar repository at,
+# with comments and its test file can be found in the GitHub repository at,
 #   lib/Runtime.pm
 #   t/lib/Runtime.t
-# See https://launchpad.net/percona-toolkit for more information.
+# See https://github.com/percona/percona-toolkit for more information.
 # ###########################################################################
 {
 package Runtime;
@@ -11698,7 +11629,7 @@ sub nibble_is_safe {
          warn ts("Skipping chunk " . $nibble_iter->nibble_number()
             . " of $tbl->{db}.$tbl->{tbl} because MySQL chose "
             . ($expl->{key} ? "the $expl->{key}" : "no") . " index "
-            . " instead of the " . $nibble_iter->nibble_index() . "index.\n");
+            . " instead of the " . $nibble_iter->nibble_index() . " index.\n");
       }
       $exit_status |= $PTC_EXIT_STATUS{SKIP_CHUNK};
       return 0; # not safe
@@ -12489,9 +12420,12 @@ sub wait_for_slaves {
    }
    my ($master_dbh, $ms, $slaves) = @args{@required_args};
 
-   my $master_status = $ms->get_master_status($master_dbh);
    foreach my $slave ( @$slaves ) {
+       my $dp = %$slave{DSNParser};
+       my $mdbh = $dp->get_dbh($dp->get_cxn_params($slave->{parent}));
+       my $master_status = $ms->get_master_status($mdbh);
        $ms->wait_for_master(master_status => $master_status, slave_dbh => $slave->dbh());
+       $dp->disconnect($mdbh);
    }
 }
 
@@ -13332,8 +13266,7 @@ first option on the command line.
 
 See the L<"--help"> output for a list of default config files.
 
-=item --create-replicate-table
+=item --[no]create-replicate-table
-=item --no-create-replicate-table
 
 default: yes
 

lib/MasterSlave.pm

@@ -100,7 +100,7 @@ sub get_slaves {
                   $slave_dsn->{p} = $o->get('slave-password');
                   PTDEBUG && _d("Slave password set");
                }
-               push @$slaves, $make_cxn->(dsn => $slave_dsn, dbh => $dbh);
+               push @$slaves, $make_cxn->(dsn => $slave_dsn, dbh => $dbh, parent => $parent);
                return;
             },
          }

lib/NibbleIterator.pm

@@ -763,7 +763,8 @@ sub _get_bounds {
       if (    defined $nibble->{lower_boundary}
            && defined $nibble->{upper_boundary} ) {
          my $sth = $dbh->prepare($self->{resume_lb_sql});
-         my @ub  = split ',', $nibble->{upper_boundary};
+         #my @ub  = split ',', $nibble->{upper_boundary};
+         my @ub = $self->{Quoter}->deserialize_list($nibble->{upper_boundary});
          PTDEBUG && _d($sth->{Statement}, 'params:', @ub);
          $sth->execute(@ub);
          $self->{next_lower} = $sth->fetchrow_arrayref();

lib/RowChecksum.pm

@@ -102,18 +102,30 @@ sub make_row_checksum {
       $sep    =~ s/'//g;
       $sep  ||= '#';
 
+      my @converted_cols;
+      for my $col(@{$cols->{select}}) {
+          my $colname = $col;
+          $colname =~ s/`//g;
+          my $type = $tbl_struct->{type_for}->{$colname} || '';
+          if ($type =~ m/^(CHAR|VARCHAR|BINARY|VARBINARY|BLOB|TEXT|ENUM|SET|JSON)$/i) {
+              push @converted_cols, "convert($col using utf8mb4)";
+          } else {
+              push @converted_cols, "$col";
+          }
+      }
+
       # Add a bitmap of which nullable columns are NULL.
       my @nulls = grep { $cols->{allowed}->{$_} } @{$tbl_struct->{null_cols}};
       if ( @nulls ) {
          my $bitmap = "CONCAT("
             . join(', ', map { 'ISNULL(' . $q->quote($_) . ')' } @nulls)
             . ")";
-         push @{$cols->{select}}, $bitmap;
+         push @converted_cols, $bitmap;
       }
 
-      $query .= @{$cols->{select}} > 1
+      $query .= scalar @converted_cols > 1
-              ? "$func(CONCAT_WS('$sep', " . join(', ', @{$cols->{select}}) . '))'
+              ? "$func(CONCAT_WS('$sep', " . join(', ', @converted_cols) . '))'
-              : "$func($cols->{select}->[0])";
+              : "$func($converted_cols[0])";
    }
    else {
       # As a special case, FNV1A_64/FNV_64 doesn't need its arguments

sandbox/servers/pxc/8.0/my.sandbox.cnf

@@ -0,0 +1,43 @@
+[client]
+user     = msandbox
+password = msandbox
+port     = PORT
+socket   = /tmp/PORT/mysql_sandboxPORT.sock
+
+[mysqld]
+port                       = PORT
+socket                     = /tmp/PORT/mysql_sandboxPORT.sock
+pid-file                   = /tmp/PORT/data/mysql_sandboxPORT.pid
+basedir                    = PERCONA_TOOLKIT_SANDBOX
+datadir                    = /tmp/PORT/data
+key_buffer_size            = 16M
+innodb_buffer_pool_size    = 16M
+innodb_data_home_dir       = /tmp/PORT/data
+innodb_log_group_home_dir  = /tmp/PORT/data
+innodb_data_file_path      = ibdata1:10M:autoextend
+innodb_log_file_size       = 5M
+log-bin                    = mysql-bin
+relay_log                  = mysql-relay-bin
+log_slave_updates
+server-id                  = PORT
+report-host                = 127.0.0.1
+report-port                = PORT
+log-error                  = /tmp/PORT/data/mysqld.log
+innodb_lock_wait_timeout   = 3
+general_log
+general_log_file           = genlog
+
+binlog_format              = ROW
+wsrep_provider             = LIBGALERA
+wsrep_cluster_address      = CLUSTER_AD
+wsrep_sst_receive_address  = ADDR:RECEIVE_PRT
+wsrep_node_incoming_address= ADDR:PORT
+wsrep_slave_threads        = 2
+wsrep_cluster_name         = CLUSTER_NAME
+wsrep_provider_options     = "gmcast.listen_addr=tcp://ADDR:LISTEN_PRT;"
+wsrep_sst_method           = xtrabackup-v2
+wsrep_node_name            = PORT
+innodb_autoinc_lock_mode   = 2
+wsrep-replicate-myisam
+pxc_strict_mode            = DISABLED
+pxc_encrypt_cluster_traffic = 0

sandbox/start-sandbox

@@ -164,7 +164,6 @@ make_sandbox() {
 
         $PERCONA_TOOLKIT_BRANCH/util/version_cmp $minor_version 5.7.5
         if [ $? -eq 2 ]; then
-            echo "ALTER USER 'root'@'localhost' IDENTIFIED BY 'msandbox';" > $init_file
             echo "CREATE USER IF NOT EXISTS 'msandbox'@'%';" > $init_file
             echo "ALTER USER 'msandbox'@'%' IDENTIFIED BY 'msandbox' REQUIRE NONE PASSWORD EXPIRE DEFAULT ACCOUNT UNLOCK;" >> $init_file
             echo "GRANT ALL PRIVILEGES ON *.* TO 'msandbox'@'%';" >> $init_file

t/pt-table-checksum/basics.t

@@ -69,11 +69,16 @@ sub reset_repl_db {
 # ############################################################################
 
 # 1
+# We need to remove mysql.plugin and percona_test.checksums tables from the 
+# result and the sample, because they have different number of rows than default
+# if run test with enabled MyRocks or TokuDB SE
 ok(
    no_diff(
       sub { pt_table_checksum::main(@args) },
       "$sample/default-results-$sandbox_version.txt",
-      post_pipe => 'awk \'{print $2 " " $3 " " $4 " " $7 " " $9}\'',
+      sed_out => '\'/mysql.plugin$/d; /percona_test.checksums$/d\'',
+      post_pipe => 'sed \'/mysql.plugin$/d; /percona_test.checksums$/d\' | ' .
+                   'awk \'{print $2 " " $3 " " $4 " " $7 " " $9}\'',
    ),
    "Default checksum"
 );
@@ -88,18 +93,23 @@ my $max_chunks = $sandbox_version < '5.7' ? 60 : 100;
 
 ok(
    $row->[0] > 25 && $row->[0] < $max_chunks,
-   'Between 25 and 60 chunks'
+   "Between 25 and $max_chunks chunks"
 ) or diag($row->[0]);
 
 # ############################################################################
 # Static chunk size (disable --chunk-time)
 # ############################################################################
 # 3
+# We need to remove mysql.plugin and percona_test.checksums tables from the 
+# result and the sample, because they have different number of rows than default
+# if run test with enabled MyRocks or TokuDB SE
 ok(
    no_diff(
       sub { pt_table_checksum::main(@args, qw(--chunk-time 0 --ignore-databases mysql)) },
       "$sample/static-chunk-size-results-$sandbox_version.txt",
-      post_pipe => 'awk \'{print $2 " " $3 " " $4 " " $6 " " $7 " " $9}\'',
+      sed_out => '\'/mysql.plugin$/d; /percona_test.checksums$/d\'',
+      post_pipe => 'sed \'/mysql.plugin$/d; /percona_test.checksums$/d\' | ' .
+                   'awk \'{print $2 " " $3 " " $4 " " $6 " " $7 " " $9}\'',
    ),
    "Static chunk size (--chunk-time 0)"
 );

t/pt-table-checksum/error_handling.t

@@ -30,10 +30,6 @@ elsif ( !$slave1_dbh ) {
    plan skip_all => 'Cannot connect to sandbox slave';
 }
 
-if ($sandbox_version ge '8.0') {
-    plan skip_all => "TODO master master sandbox is failing with MySQL 8.0+. FIX ME !!!!";
-}
-
 # The sandbox servers run with lock_wait_timeout=3 and it's not dynamic
 # so we need to specify --set-vars innodb_lock_wait_timeout=3 else the tool will die.
 # And --max-load "" prevents waiting for status variables.

t/pt-table-checksum/fnv_64.t

@@ -32,10 +32,13 @@ my $sample     = "t/pt-table-checksum/samples/";
 my $row;
 my $output;
 
+$sb->create_dbs($master_dbh, [qw(test)]);
 
-eval { $master_dbh->do('DROP FUNCTION test.fnv_64'); };
+eval { $master_dbh->do('DROP FUNCTION IF EXISTS fnv_64'); };
-eval { $master_dbh->do("CREATE FUNCTION fnv_64 RETURNS INTEGER SONAME 'fnv_udf.so';"); };
+eval { $master_dbh->do("CREATE FUNCTION fnv_64 RETURNS INTEGER SONAME 'libfnv_udf.so';"); };
 if ( $EVAL_ERROR ) {
+#REMOVEME
+print  $EVAL_ERROR;
    chomp $EVAL_ERROR;
    plan skip_all => "No FNV_64 UDF lib"
 }
@@ -43,8 +46,6 @@ else {
    plan tests => 7;
 }
 
-$sb->create_dbs($master_dbh, [qw(test)]);
-
 # ############################################################################
 # First test the the FNV function works in MySQL and gives the correct results.
 # ############################################################################

t/pt-table-checksum/issue_1485195.t

@@ -32,7 +32,7 @@ $sb->load_file('master', 't/pt-table-checksum/samples/issue_1485195.sql');
 # so we need to specify --set-vars innodb_lock_wait_timeout=3 else the tool will die.
 # And --max-load "" prevents waiting for status variables.
 my $master_dsn = 'h=127.1,P=12345,u=msandbox,p=msandbox,D=my_binary_database';
-my @args       = ($master_dsn, qw(--replicate my_binary_database.my_table -d percona_test)); 
+my @args       = ($master_dsn, qw(--replicate my_binary_database.my_table -t percona_test.checksums)); 
 my $output;
 
 $output = output(
@@ -40,9 +40,13 @@ $output = output(
    stderr => 1,
 );
 
+# We do not count these tables by default, because their presense depends from 
+# previously running tests
+my $extra_tables = $dbh->selectrow_arrayref("select count(*) from percona_test.checksums where db_tbl in ('mysql.plugin', 'mysql.func', 'mysql.proxies_priv');")->[0];
+
 is(
    PerconaTest::count_checksum_results($output, 'rows'),
-   $sandbox_version ge '8.0' ? 29 : $sandbox_version lt '5.7' ? 24 : 25,
+   $sandbox_version ge '8.0' ? 27 + $extra_tables : $sandbox_version lt '5.7' ? 24 : 23  + $extra_tables,
    "Large BLOB/TEXT/BINARY Checksum"
 );
 

t/pt-table-checksum/issue_47.t

@@ -21,8 +21,9 @@ my $master_dbh = $sb->get_dbh_for('master');
 
 if ( !$master_dbh ) {
    plan skip_all => 'Cannot connect to sandbox master';
-}
+} elsif ($sandbox_version >= '8.0') { 
-else {
+   plan skip_all => "8.0 requires fix for https://jira.percona.com/browse/PT-1805";
+} else {
    plan tests => 3;
 }
 

t/pt-table-checksum/pt-131.t

@@ -21,11 +21,13 @@ my $sb  = new Sandbox(basedir => '/tmp', DSNParser => $dp);
 my $dbh = $sb->get_dbh_for('master');
 my $sb_version = VersionParser->new($dbh);
 my $rows = $dbh->selectall_hashref("SHOW VARIABLES LIKE '%version%'", ['variable_name']);
+my $remove_plugin = 0;
 
 if ( !$dbh ) {
    plan skip_all => 'Cannot connect to sandbox master';
-} elsif ( $sb_version < '5.7.21' || !($rows->{version_comment}->{value} =~ m/percona server/i) ) {
+} elsif ( $sb_version < '5.7.21' || $sb_version >= '8.0' ||
-   plan skip_all => 'This test file needs Percona Server 5.7.21.21+';
+      !($rows->{version_comment}->{value} =~ m/percona server/i) ) {
+   plan skip_all => 'This test file needs 5.7 Percona Server, starting from 5.7.21';
 } else {
    plan tests => 3;
 }
@@ -35,6 +37,7 @@ eval {
 };
 if ($EVAL_ERROR) {
     $sb->load_file('master', 't/pt-table-checksum/samples/pt-131.sql');
+    $remove_plugin = 1;
 }
 # The sandbox servers run with lock_wait_timeout=3 and it is not dynamic
 # so we need to specify --set-vars innodb_lock_wait_timeout=3 else the tool will die.
@@ -63,6 +66,9 @@ delete $ENV{PTDEBUG};
 # #############################################################################
 # Done.
 # #############################################################################
+if ($remove_plugin) {
+   $sb->load_file('master', 't/pt-table-checksum/samples/pt-131-wipe.sql');
+}
 $sb->wipe_clean($dbh);
 ok($sb->ok(), "Sandbox servers") or BAIL_OUT(__FILE__ . " broke the sandbox");
 exit;

t/pt-table-checksum/pt-1616.t

@@ -11,6 +11,10 @@ use warnings FATAL => 'all';
 use English qw(-no_match_vars);
 use Test::More;
 
+if ( !$ENV{SLOW_TESTS} ) { 
+   plan skip_all => "pt-table-checksum/replication_filters.t is one of the top slowest files; set SLOW_TESTS=1 to enable it.";
+}
+
 use PerconaTest;
 use Sandbox;
 use SqlModes;

t/pt-table-checksum/pt-204.t

@@ -70,9 +70,11 @@ like(
    stderr => 1,
 );
 
+my $return_code = ($sandbox_version >= '5.7') ? 16 : 0;
+
 is(
    $exit_status,
-   0,
+   $return_code,
    "PT-204 Starting checksum since RocksDB table was skipped with --ignore-tables",
 );
 
@@ -96,7 +98,7 @@ unlike(
 
 is(
    $exit_status,
-   0,
+   $return_code,
    "PT-204 Starting checksum since RocksDB table was skipped with --ignore-engines",
 );
 

t/pt-table-checksum/pt-226.t

@@ -28,7 +28,6 @@ else {
 }
 
 diag("loading samples");
-#$sb->load_file('master', 't/pt-table-checksum/samples/pt-226.sql');
 $sb->load_file('master', 't/pt-table-checksum/samples/pt-226.sql');
 
 # The sandbox servers run with lock_wait_timeout=3 and it's not dynamic
@@ -55,6 +54,8 @@ $output = output(
    stderr => 1,
 );
 
+#REMOVEME
+diag($exit_status);
 isnt(
    $exit_status,
    0,
@@ -63,7 +64,7 @@ isnt(
 
 like(
     $output,
-    qr/1\s+100\s+0\s+1\s+0\s+.*test.joinit/,
+    qr/1\s+100\s+10\s+1\s+0\s+.*test.joinit/,
     "PT-226 table joinit has differences",
 );
 

t/pt-table-checksum/pxc.t

@@ -27,9 +27,14 @@ my $ip = qr/\Q127.1\E|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}/;
 my $dp = new DSNParser(opts=>$dsn_opts);
 my $sb = new Sandbox(basedir => '/tmp', DSNParser => $dp);
 my $node1 = $sb->get_dbh_for('node1');
+my $sb_version = VersionParser->new($node1);
 my $node2 = $sb->get_dbh_for('node2');
 my $node3 = $sb->get_dbh_for('node3');
 
+if ($sb_version >= '8.0') {
+   plan skip_all => 'Cannot run tests on PXC 8.0 until PT-1699 is fixed';
+}
+
 if ( !$node1 ) {
    plan skip_all => 'Cannot connect to cluster node1';
 }
@@ -93,8 +98,8 @@ ok (
 );
 
 for my $args (
-      ["using recusion-method=dsn", '--recursion-method', "dsn=$node1_dsn,D=dsns,t=dsns"],
+      ["using recusion-method=dsn", '--recursion-method', "dsn=$node1_dsn,D=dsns,t=dsns", '--ignore-tables=mysql.proxies_priv'],
-      ["using recursion-method=cluster", '--recursion-method', 'cluster']
+      ["using recursion-method=cluster", '--recursion-method', 'cluster', '--ignore-tables=mysql.proxies_priv']
    )
 {
    my $test = shift @$args;
@@ -125,14 +130,12 @@ for my $args (
 }
 
 # Now really test checksumming a cluster.  To create a diff we have to disable
-# the binlog.  Although PXC doesn't need or use the binlog to communicate
+# wsrep replication, so we can make a change on one node without
-# (it has its own broadcast-based protocol implemented via the Galera lib)
-# it still respects sql_log_bin, so we can make a change on one node without
 # affecting the others.
 $sb->load_file('node1', "$sample/a-z.sql");
-$node2->do("set sql_log_bin=0");
+$node2->do("set wsrep_on=0");
 $node2->do("update test.t set c='zebra' where c='z'");
-$node2->do("set sql_log_bin=1");
+$node2->do("set wsrep_on=1");
 
 my ($row) = $node2->selectrow_array("select c from test.t order by c desc limit 1");
 is(
@@ -188,8 +191,8 @@ sub test_recursion_methods {
    }
 
    for my $args (
-         ["using recusion-method=dsn", '--recursion-method', "dsn=$node1_dsn,D=dsns,t=dsns"],
+         ["using recusion-method=dsn", '--recursion-method', "dsn=$node1_dsn,D=dsns,t=dsns", '--ignore-tables=mysql.proxies_priv'],
-         ["using recursion-method=cluster", '--recursion-method', 'cluster']
+         ["using recursion-method=cluster", '--recursion-method', 'cluster', '--ignore-tables=mysql.proxies_priv']
       )
    {
       my $test = shift @$args;
@@ -227,6 +230,7 @@ sub test_recursion_methods {
             0\s+     # errors
             1\s+     # diffs
             26\s+    # rows
+            0\s+     # diff_rows
             \d+\s+   # chunks
             0\s+     # skipped
             \S+\s+   # time
@@ -264,16 +268,16 @@ my ($slave_dbh, $slave_dsn) = $sb->start_sandbox(
    server => 'cslave1',
    type   => 'slave',
    master => 'node1',
-   env    => q/FORK="pxc" BINLOG_FORMAT="ROW"/,
+   env    => q/BINLOG_FORMAT="ROW"/,
 );
 
 # Add the slave to the DSN table.
 $node1->do(qq/INSERT INTO dsns.dsns VALUES (4, 3, '$slave_dsn')/);
 
 # Fix what we changed earlier on node2 so the cluster is consistent.
-$node2->do("set sql_log_bin=0");
+$node2->do("set wsrep_on=0");
 $node2->do("update test.t set c='z' where c='zebra'");
-$node2->do("set sql_log_bin=1");
+$node2->do("set wsrep_on=1");
 
 # Wait for the slave to apply the binlogs from node1 (its master).
 # Then change it so it's not consistent.
@@ -287,8 +291,8 @@ $slave_dbh->do("update test.t set c='zebra' where c='z'");
 # Cluster nodes default to ROW format because that's what Galeara
 # works best with, even though it doesn't really use binlogs.
 for my $args (
-      ["using recusion-method=dsn", '--recursion-method', "dsn=$node1_dsn,D=dsns,t=dsns"],
+      ["using recusion-method=dsn", '--recursion-method', "dsn=$node1_dsn,D=dsns,t=dsns", '--ignore-tables=mysql.user'],
-      ["using recursion-method=cluster,hosts", '--recursion-method', 'cluster,hosts']
+      ["using recursion-method=cluster,hosts", '--recursion-method', 'cluster,hosts', '--ignore-tables=mysql.user']
    )
 {
    my $test = shift @$args;
@@ -347,7 +351,7 @@ $sb->stop_sandbox('cslave1');
    server => 'cslave1',
    type   => 'slave',
    master => 'node2',
-   env    => q/FORK="pxc" BINLOG_FORMAT="ROW"/,
+   env    => q/BINLOG_FORMAT="ROW"/,
 );
 
 # Wait for the slave to apply the binlogs from node2 (its master).
@@ -364,8 +368,8 @@ is(
 );
 
 for my $args (
-      ["using recusion-method=dsn", '--recursion-method', "dsn=$node1_dsn,D=dsns,t=dsns"],
+      ["using recusion-method=dsn", '--recursion-method', "dsn=$node1_dsn,D=dsns,t=dsns", '--ignore-tables=mysql.user'],
-      ["using recursion-method=cluster,hosts", '--recursion-method', 'cluster,hosts']
+      ["using recursion-method=cluster,hosts", '--recursion-method', 'cluster,hosts', '--ignore-tables=mysql.user']
    )
 {
    my $test = shift @$args;
@@ -404,7 +408,7 @@ $node1->do(qq/DELETE FROM dsns.dsns WHERE id=4/);
 my ($master_dbh, $master_dsn) = $sb->start_sandbox(
    server => 'cmaster',
    type   => 'master',
-   env    => q/FORK="pxc" BINLOG_FORMAT="ROW"/,
+   env    => q/BINLOG_FORMAT="ROW"/,
 );
 
 # Since master is new, node1 shouldn't have binlog to replay.
@@ -466,9 +470,9 @@ is(
 
 # Make a diff on node1.  If ptc is really auto-detecting node1, then it
 # should report this diff.
-$node1->do("set sql_log_bin=0");
+$node1->do("set wsrep_on=0");
 $node1->do("update test.t set c='zebra' where c='z'");
-$node1->do("set sql_log_bin=1");
+$node1->do("set wsrep_on=1");
 
 $output = output(
    sub { pt_table_checksum::main($master_dsn,
@@ -502,6 +506,7 @@ like(
       0\s+     # errors
       1\s+     # diffs
       26\s+    # rows
+      0\s+     # diff_rows
       \d+\s+   # chunks
       0\s+     # skipped
       \S+\s+   # time
@@ -548,6 +553,7 @@ like(
       0\s+     # errors
       1\s+     # diffs
       26\s+    # rows
+      0\s+     # diff_rows
       \d+\s+   # chunks
       0\s+     # skipped
       \S+\s+   # time
@@ -575,9 +581,9 @@ for my $args (
 
    # Make a diff on node1.  If ptc is really auto-detecting node1, then it
    # should report this diff.
-   $node1->do("set sql_log_bin=0");
+   $node1->do("set wsrep_on=0");
    $node1->do("update test.t set c='zebra' where c='z'");
-   $node1->do("set sql_log_bin=1");
+   $node1->do("set wsrep_on=1");
    
    $output = output(
       sub { pt_table_checksum::main($master_dsn,
@@ -600,6 +606,7 @@ for my $args (
          0\s+     # errors
          1\s+     # diffs
          26\s+    # rows
+         0\s+     # diff_rows
          \d+\s+   # chunks
          0\s+     # skipped
          \S+\s+   # time
@@ -619,13 +626,13 @@ for my $args (
    # to node1, node1 isn't different, so it broadcasts the result in ROW format
    # that all is ok, which node2 gets and thus false reports.  This is why
    # those ^ warnings exist.
-   $node1->do("set sql_log_bin=0");
+   $node1->do("set wsrep_on=0");
    $node1->do("update test.t set c='z' where c='zebra'");
-   $node1->do("set sql_log_bin=1");
+   $node1->do("set wsrep_on=1");
 
-   $node2->do("set sql_log_bin=0");
+   $node2->do("set wsrep_on=0");
    $node2->do("update test.t set c='zebra' where c='z'");
-   $node2->do("set sql_log_bin=1");
+   $node2->do("set wsrep_on=1");
 
    ($row) = $node2->selectrow_array("select c from test.t order by c desc limit 1");
    is(

t/pt-table-checksum/replication_filters.t

@@ -31,6 +31,7 @@ use Data::Dumper;
 my $dp = new DSNParser(opts=>$dsn_opts);
 my $sb = new Sandbox(basedir => '/tmp', DSNParser => $dp);
 my $master_dbh = $sb->get_dbh_for('master');
+my $sb_version = VersionParser->new($master_dbh);
 my $slave1_dbh = $sb->get_dbh_for('slave1');
 my $slave2_dbh = $sb->get_dbh_for('slave2');
 
@@ -282,7 +283,7 @@ elsif ( -x "$ENV{PERCONA_TOOLKIT_SANDBOX}/bin/mysqlbinlog" ) {
    $mysqlbinlog = "$ENV{PERCONA_TOOLKIT_SANDBOX}/bin/mysqlbinlog";
 }
 
-$output = `$mysqlbinlog /tmp/12345/data/$row->{file} --start-position=$row->{position} | grep 'use ' | grep -v '^# Warning' |  sort -u | sed -e 's/\`//g'`;
+$output = `$mysqlbinlog /tmp/12345/data/$row->{file} --start-position=$row->{position} | grep 'use ' | grep -v '^# Warning' | grep -v 'pseudo_replica_mode' | sort -u | sed -e 's/\`//g'`;
 
 my $use_dbs = "use mysql/*!*/;
 use percona/*!*/;
@@ -290,6 +291,10 @@ use percona_test/*!*/;
 use sakila/*!*/;
 ";
 
+if ($sb_version >= '5.7') {
+   $use_dbs .= "use sys/*!*/;\n";
+}
+
 is(
    $output,
    $use_dbs,
@@ -301,7 +306,7 @@ $row = $master_dbh->selectrow_hashref('show master status');
 
 pt_table_checksum::main(@args, qw(--quiet --replicate-database percona));
 
-$output = `$mysqlbinlog /tmp/12345/data/$row->{file} --start-position=$row->{position} | grep 'use ' | grep -v '^# Warning' | sort -u | sed -e 's/\`//g'`;
+$output = `$mysqlbinlog /tmp/12345/data/$row->{file} --start-position=$row->{position} | grep 'use ' | grep -v '^# Warning' | grep -v 'pseudo_replica_mode' | sort -u | sed -e 's/\`//g'`;
 
 is(
    $output,

t/pt-table-checksum/samples/default-results-5.7.txt

@@ -6,10 +6,10 @@ ERRORS DIFFS ROWS SKIPPED TABLE
 0 0 2 0 mysql.engine_cost
 0 0 0 0 mysql.event
 0 0 0 0 mysql.func
-0 0 40 0 mysql.help_category
+0 0 50 0 mysql.help_category
-0 0 682 0 mysql.help_keyword
+0 0 908 0 mysql.help_keyword
-0 0 1340 0 mysql.help_relation
+0 0 1803 0 mysql.help_relation
-0 0 637 0 mysql.help_topic
+0 0 659 0 mysql.help_topic
 0 0 0 0 mysql.ndb_binlog_index
 0 0 0 0 mysql.plugin
 0 0 0 0 mysql.proc
@@ -23,7 +23,7 @@ ERRORS DIFFS ROWS SKIPPED TABLE
 0 0 0 0 mysql.time_zone_name
 0 0 0 0 mysql.time_zone_transition
 0 0 0 0 mysql.time_zone_transition_type
-0 0 2 0 mysql.user
+0 0 4 0 mysql.user
 0 0 23 0 percona_test.checksums
 0 0 1 0 percona_test.sentinel
 0 0 200 0 sakila.actor

t/pt-table-checksum/samples/default-results-8.0.txt

@@ -5,19 +5,23 @@ ERRORS DIFFS ROWS SKIPPED TABLE
 0 0 0 0 mysql.component
 0 0 2 0 mysql.db
 0 0 0 0 mysql.default_roles
-0 0 2 0 mysql.engine_cost
+0 1 2 0 mysql.engine_cost
 0 0 0 0 mysql.func
-0 0 26 0 mysql.global_grants
+0 0 85 0 mysql.global_grants
-0 0 40 0 mysql.help_category
+0 0 53 0 mysql.help_category
-0 0 695 0 mysql.help_keyword
+0 0 985 0 mysql.help_keyword
-0 0 1457 0 mysql.help_relation
+0 0 2043 0 mysql.help_relation
-0 0 644 0 mysql.help_topic
+0 0 701 0 mysql.help_topic
 0 0 0 0 mysql.password_history
 0 0 0 0 mysql.plugin
 0 0 0 0 mysql.procs_priv
-0 0 1 0 mysql.proxies_priv
+0 1 1 0 mysql.proxies_priv
+0 0 0 0 mysql.replication_asynchronous_connection_failover
+0 0 0 0 mysql.replication_asynchronous_connection_failover_managed
+0 0 1 0 mysql.replication_group_configuration_version
+0 0 2 0 mysql.replication_group_member_actions
 0 0 0 0 mysql.role_edges
-0 0 6 0 mysql.server_cost
+0 1 6 0 mysql.server_cost
 0 0 0 0 mysql.servers
 0 0 2 0 mysql.tables_priv
 0 0 0 0 mysql.time_zone
@@ -25,8 +29,8 @@ ERRORS DIFFS ROWS SKIPPED TABLE
 0 0 0 0 mysql.time_zone_name
 0 0 0 0 mysql.time_zone_transition
 0 0 0 0 mysql.time_zone_transition_type
-0 0 5 0 mysql.user
+0 1 5 0 mysql.user
-0 0 27 0 percona_test.checksums
+0 0 29 0 percona_test.checksums
 0 0 1 0 percona_test.sentinel
 0 0 200 0 sakila.actor
 0 0 603 0 sakila.address

t/pt-table-checksum/samples/fnv64-sakila-city.txt

@@ -1,8 +1,10 @@
+Checking if all tables can be checksummed ...
+Starting checksum ...
 --
 -- sakila.city
 --
 
-REPLACE INTO `percona`.`checksums` (db, tbl, chunk, chunk_index, lower_boundary, upper_boundary, this_cnt, this_crc) SELECT ?, ?, ?, ?, ?, ?, COUNT(*) AS cnt, COALESCE(LOWER(CONV(BIT_XOR(CAST(FNV_64(`city_id`, `city`, `country_id`, `last_update` + 0) AS UNSIGNED)), 10, 16)), 0) AS crc FROM `sakila`.`city` FORCE INDEX(`PRIMARY`) WHERE ((`city_id` >= ?)) AND ((`city_id` <= ?)) /*checksum chunk*/
+REPLACE INTO `percona`.`checksums` (db, tbl, chunk, chunk_index, lower_boundary, upper_boundary, this_cnt, this_crc) SELECT ?, ?, ?, ?, ?, ?, COUNT(*) AS cnt, COALESCE(LOWER(CONV(BIT_XOR(CAST(FNV_64(`city_id`, `city`, `country_id`, UNIX_TIMESTAMP(`last_update`)) AS UNSIGNED)), 10, 16)), 0) AS crc FROM `sakila`.`city` FORCE INDEX(`PRIMARY`) WHERE ((`city_id` >= ?)) AND ((`city_id` <= ?)) /*checksum chunk*/
 
 REPLACE INTO `percona`.`checksums` (db, tbl, chunk, chunk_index, lower_boundary, upper_boundary, this_cnt, this_crc) SELECT ?, ?, ?, ?, ?, ?, COUNT(*), '0' FROM `sakila`.`city` FORCE INDEX(`PRIMARY`) WHERE ((`city_id` < ?)) ORDER BY `city_id` /*past lower chunk*/
 
@@ -14,7 +16,7 @@ SELECT /*!40001 SQL_NO_CACHE */ `city_id` FROM `sakila`.`city` FORCE INDEX(`PRIM
 -- sakila.film_actor
 --
 
-REPLACE INTO `percona`.`checksums` (db, tbl, chunk, chunk_index, lower_boundary, upper_boundary, this_cnt, this_crc) SELECT ?, ?, ?, ?, ?, ?, COUNT(*) AS cnt, COALESCE(LOWER(CONV(BIT_XOR(CAST(FNV_64(`actor_id`, `film_id`, `last_update` + 0) AS UNSIGNED)), 10, 16)), 0) AS crc FROM `sakila`.`film_actor` FORCE INDEX(`PRIMARY`) WHERE ((`actor_id` > ?) OR (`actor_id` = ? AND `film_id` >= ?)) AND ((`actor_id` < ?) OR (`actor_id` = ? AND `film_id` <= ?)) /*checksum chunk*/
+REPLACE INTO `percona`.`checksums` (db, tbl, chunk, chunk_index, lower_boundary, upper_boundary, this_cnt, this_crc) SELECT ?, ?, ?, ?, ?, ?, COUNT(*) AS cnt, COALESCE(LOWER(CONV(BIT_XOR(CAST(FNV_64(`actor_id`, `film_id`, UNIX_TIMESTAMP(`last_update`)) AS UNSIGNED)), 10, 16)), 0) AS crc FROM `sakila`.`film_actor` FORCE INDEX(`PRIMARY`) WHERE ((`actor_id` > ?) OR (`actor_id` = ? AND `film_id` >= ?)) AND ((`actor_id` < ?) OR (`actor_id` = ? AND `film_id` <= ?)) /*checksum chunk*/
 
 REPLACE INTO `percona`.`checksums` (db, tbl, chunk, chunk_index, lower_boundary, upper_boundary, this_cnt, this_crc) SELECT ?, ?, ?, ?, ?, ?, COUNT(*), '0' FROM `sakila`.`film_actor` FORCE INDEX(`PRIMARY`) WHERE ((`actor_id` < ?) OR (`actor_id` = ? AND `film_id` < ?)) ORDER BY `actor_id`, `film_id` /*past lower chunk*/
 

t/pt-table-checksum/samples/pt-131-wipe.sql

@@ -0,0 +1,13 @@
+-- See https://www.percona.com/doc/percona-server/LATEST/diagnostics/response_time_distribution.html
+
+-- This plugin is used for gathering statistics.
+UNINSTALL PLUGIN QUERY_RESPONSE_TIME_AUDIT;
+
+-- This plugin provides the interface (QUERY_RESPONSE_TIME) to output gathered statistics.
+UNINSTALL PLUGIN QUERY_RESPONSE_TIME;
+
+-- This plugin provides the interface (QUERY_RESPONSE_TIME_READ) to output gathered statistics.
+UNINSTALL PLUGIN QUERY_RESPONSE_TIME_READ;
+
+-- This plugin provides the interface (QUERY_RESPONSE_TIME_WRITE) to output gathered statistics.
+UNINSTALL PLUGIN QUERY_RESPONSE_TIME_WRITE;

t/pt-table-checksum/samples/static-chunk-size-results-8.0.txt

@@ -1,7 +1,7 @@
 if all tables be checksummed 
 checksum ...    
 ERRORS DIFFS ROWS CHUNKS SKIPPED TABLE
-0 0 27 1 0 percona_test.checksums
+0 0 29 1 0 percona_test.checksums
 0 0 1 1 0 percona_test.sentinel
 0 0 200 1 0 sakila.actor
 0 0 603 1 0 sakila.address

util/check-load-data

@@ -45,7 +45,7 @@ $dbh->do("CREATE TABLE IF NOT EXISTS percona_test.load_data (i int)");
 `echo 1 > /tmp/load_data_test.$$`;
 
 eval {
-    $dbh->do("LOAD DATA INFILE '/tmp/load_data_test.$$' INTO TABLE percona_test.load_data");
+    $dbh->do("LOAD DATA LOCAL INFILE '/tmp/load_data_test.$$' INTO TABLE percona_test.load_data");
 };
 
 if ( $EVAL_ERROR ) {