HI, i am back
An error occurred when I migrated data from MySQL to OceanBase:【 DBD::mysql::db do failed: You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use near ':= @OLD_SQL_MODE, @@SQL_QUOTE_SHOW_CREATE := @OLD_QUOTE /' at line 1 [for Statement "/!40101 SET @@SQL_MODE := @OLD_SQL_MODE, @@SQL_QUOTE_SHOW_CREATE := @OLD_QUOTE */"] at /usr/bin/pt-archiver line 1932.】
I'm not a Perl programmer. But by reading the source code and making some changes, the migration was done successfully, so I submit this report
- [x] The contributed code is licensed under GPL v2.0
- [x] Contributor Licence Agreement (CLA) is signed
- [x] util/update-modules has been ran
- [x] Not involved Documention updated
- [x] Not involved Test suite update
* PT-2156 - Fix tests for lib
- Fixed tests broken for lib/TableParser.pm after fix for PT-1059
- Updated tests for lib/TableParser.pm that are broken due to SHOW CREATE TABLE output format change in 8.0
- Updated modules for all tools that use lib/TableParser.pm
* Revert "Fixed pt-archiver tests"
This reverts commit a3ab87b12e.
This commit wa needed, because removed code in sandbox/slave_channels.sql broked the test. Proper fix would be to do not remove channel names rather than chaging the test. So revertig it.
* PT-2156 - Fix tests for lib
- Updated test t/lib/TableChecksum.t so it reflects changes, introduced in the fix for PT-2016
- Updated test t/lib/RowChecksum.t so it reflects changes added to in the fix for PT-2138: UTF8 support
- Uncommented SQL in sandbox/slave_channels.sql that made t/lib/MasterSlave.t to fail
- Added check for undef into t/pt-archiver/archive_using_channels.t
- Updated lib/Cxn.pm so it uses $dbh->{Active} after issue with the ping() call, reported at https://github.com/perl5-dbi/DBD-mysql/issues/306
* PT-2156 - Fix tests for lib
- Impoved the fix for PT-2016, so it does not files with keys with USING keyword
- Added brackets to expression in lib/TableNibbler.pm, so it does not crap query wit many indexes with OR keyword
- Adjusted test t/lib/TableNibbler.t, so it reflects above chages
- Modified lib/Cxn.pm, so it has workaround for https://github.com/perl5-dbi/DBD-mysql/issues/306 , introduced in DBD::mysql 4.0.50
- Updated tests: added debugging code and cleanups
- Updated modules for tools
* PT-2160 - Fix tests for pt-online-schema-change
Re-enabled tests after MySQL bug #89441 fix in 8.0.14 and later
Updated t/pt-online-schema-change/samples/basic_no_fks_innodb.sql, so its unique index is NOT NULL
Added synchonizations when tests are waiting for too long due to slave catch up
Added innodb_directories option for t/pt-online-schema-change/pt-244.t
* PT-2160 - Fix tests for pt-online-schema-change
PT-2048 - pt-osc spans excessive connections to the replica when executing in the source
t/pt-online-schema-change/slave_lag.t was failing due to PT-2048. I added code that reuses already created connections when checks for slave in the wait loop. It fixes slave_lag.t too.
I also added that prints more detailed error when fails to connect to the replica.
* PT-2160 - Fix tests for pt-online-schema-change
Modified t/pt-online-schema-change/slave_lag.t, so it runs and clean ups faster.
* PT-2160 - Fix tests for pt-online-schema-change
Modified t/pt-online-schema-change/pt-1455.t, so it runs and clean ups faster.
* PT-2160 - Fix tests for pt-online-schema-change
Fixed search for HASH and BTREE keys broken by fix for PT-2123
Updated modules for pt-archiver
Fixed die message for sandbox/start-sandbox
Added global_grants table to list of exceptions for the ok function in lib/Sandbox.pm
Added wait_for_slaves to bugs.t, so changes on the source are copied to replica before pt-osc starts working
Updated PXC tests
Added wait_for_slaves to pt-1455.t
Fixed regular expression in pt-229.t, so it works with both 5.x and 8.0 versions
Added innodb_directories option for pt-244.t, so it does not fail on 8.0
Modified slave_lag.t, so it is more stable and slow enough, so pt-osc can print message about delayed replicas
* PT-1059 - Tools can't parse index names containing newlines
Fixed regular expressions in TableParser.
Added test case, including test for new lines in the column name
* PT-1059 - Tools can't parse index names containing newlines
Disabled pt-1637.t until PT-2174 is fixed.
Updated number of tables in b/t/pt-table-checksum/issue_1485195.t
* Patch newlines in table columns (#369)
Will accept this change as part of the fix for PT-1059 - Tools cannot parse index names containing new lines. We will later fix the issue with the patch ourselves.
mysql 5.6.40 allows newlines in column names however the following code:
my @defs = $ddl =~ m/^(\s+`.*?),?$/gm;
breaks due to it detecting newlines as line ends. The 'm' argument at the end does this by auto-detecting lines by newline characters.
To correct this issue I've made use of zero-length assertions known as " positive lookback"
https://www.regular-expressions.info/lookaround.html
what does it do?
m/(?:(?<=,\n)|(?<=\(\n))(\s+`(?:.|\n)+?`.+?),?\n/g;
TLDR:
Treat the string as one long string and don't treat \n as the end of a line.
look for (\s+`(?:.|\n)+?`.+?),?\n
if one of those matches look at what precedes the string
if it's ',\n' or ')\n' the string matches. Only save what's in (\s+`(?:.|\n)+?`.+?),?\n
m/ is declaring this a matching regex.
(?:(?<=,\n)|(?<=(\n)) This is an OR statement including two look-behind clauses. The ?: tells the enclosing parentheses to not store the result as a variable. I've put the two look-behinds in this OR statement below this line:
(?<=,\n) Look behind the matched string for a comma followed by a newline, the comma must be there for this look behind to match.
(?<=(\n) Look behind the matched string for a open parentheses followed by a newline, the open parentheses must be there.
(\s+`(?:.|\n)+?`.+?),?\n This is the actual match. Match newline character followed by one or more spaces followed by back-tick followed by a character which can be any character or a newline one or more times, but don't be greedy and take the rest of the match into consideration. Followed by a back tick and any character one or more times. This match stops where there is a comma or failing that a newline following a back tick and some characters.
,?\n match a comma that may not be there followed by a newline.
/g don't stop if this pattern matches keep looking for more patterns to the end of the string.
* PT-1059 - Tools can't parse index names containing newlines
Placed fix from PR-369 into proper place and created test case for this fix.
---------
Co-authored-by: geneguido <31323560+geneguido@users.noreply.github.com>
* PMM-1914 Fixed column parsing having generated
Fixed table parser code that errouneously considered a column as
generated when the default was empty DEFAULT '' and the COMMENT had
the word 'Generated'.
* PMM-1914 Updated TableParser in all programs
* PT-1914 Updated changelog
* PT-1914 Added test
TableParser's parse function was failing while trying to lowercase
column names in the provided 'SHOW CREATE TABLE'.
The problem was it was trying to lowercase everything between backticks
but lines like these:
`field_name` int comment "here is a ` in the comment"
`second_field_name` int
made the original regex to fail, matching `in the coment"` as an
expression to be lowercased while second_file_name was considered as
outside backticks.
