Author: Daniel Black
Author Date: 2023-10-01 07:18:09 UTC

MDEV-32336 deb default config - use uca1400_ai_ci for utf8mb4

utf8mb4_general_ci has been outdated for a while and contained loosely
standardized collations.

UCA-14.0.0 has a more defined collation with multiple benefit that new
users may not immediately consider, or may assume to be default.

By defining default collation for utf8mb4 to be uc1400_ai_ci newly
created tables will have a modern standard collation.

Author: Sergei Golubchik
Author Date: 2023-10-19 15:02:37 UTC

MDEV-32500 Information schema leaks table names and structure to unauthorized users

standard table KEY_COLUMN_USAGE should only show keys where
a user has some privileges on every column of the key

standard table TABLE_CONSTRAINTS should show tables where
a user has any non-SELECT privilege on the table or on any column
of the table

standard table REFERENTIAL_CONSTRAINTS is defined in terms of
TABLE_CONSTRAINTS, so the same rule applies. If the user
has no rights to see the REFERENCED_TABLE_NAME value, it should be NULL

SHOW INDEX (and STATISTICS table) is non-standard, but it seems
reasonable to use the same logic as for KEY_COLUMN_USAGE.

Author: midenok
Author Date: 2023-10-18 17:39:19 UTC

Mydumper compilation as MariaDB submodule

Requires commit "Compilation as MariaDB submodule fixes" from midenok
branch in git@github.com:mariadb-corporation/mydumper.git

Author: Igor Babaev
Author Date: 2023-10-19 19:09:41 UTC

MDEV-32351 Significant slowdown for query with many outer joins

This patch fixes a performance regression introduced in the patch for the
bug MDEV-21104. The performance regression could affect queries for which
join buffer was used for an outer join such that its on expression from
which a conjunctive condition depended only on outer tables can be
extracted. If the number of records in the join buffer for which this
condition was false greatly exceeded the number of other records the
slowdown could be significant.

If there is a conjunctive condition extracted from the ON expression
depending only on outer tables this condition is evaluated when interesting
fields of each survived record of outer tables are put into the join buffer.
Each such set of fields for any join operation is supplied with a match
flag field used to generate null complemented rows. If the result of the
evaluation of the condition is false the flag is set to MATCH_IMPOSSIBLE.
When looking in the join buffer for records matching a record of the
right operand of the outer join operation the records with such flags
are not needed to be unpacked into record buffers for evaluation of on
expressions.

The patch for MDEV-21104 fixing some problem of wrong results when
'not exists' optimization by mistake broke the code that allowed to
ignore records with the match flag set to MATCH_IMPOSSIBLE when looking
for matching records. As a result such records were unpacked for each
record of the right operand of the outer join operation. This caused
significant execution penalty in some cases.

One of the test cases added in the patch can be used only for demonstration
of the restored performance for the reported query. The second test case is
needed to demonstrate the validity of the fix.

Author: Nikita Malyavin
Author Date: 2023-10-19 17:58:26 UTC

add period to buildbot_suites.bat

Author: Elena Stepanova
Author Date: 2023-10-19 16:01:13 UTC

Test commit: run feature-related suites on Windows

Author: Sergey Petrunia
Author Date: 2023-09-19 15:22:49 UTC

MDEV-32113: utf8mb3_key_col=utf8mb4_value cannot be used for ref

(Variant#3: Allow cross-charset comparisons, use a special
CHARSET_INFO to create lookup keys. Review input addressed.)

Equalities that compare utf8mb{3,4}_general_ci strings, like:

  WHERE ... utf8mb3_key_col=utf8mb4_value (MB3-4-CMP)

can now be used to construct ref[const] access and also participate
in multiple-equalities.
This means that utf8mb3_key_col can be used for key-lookups when
compared with an utf8mb4 constant, field or expression using '=' or
'<=>' comparison operators.

This is controlled by optimizer_switch='cset_narrowing=on', which is
OFF by default.

IMPLEMENTATION
Item value comparison in (MB3-4-CMP) is done using utf8mb4_general_ci.
This is valid as any utf8mb3 value is also an utf8mb4 value.

When making index lookup value for utf8mb3_key_col, we do "Charset
Narrowing": characters that are in the Basic Multilingual Plane (=BMP) are
copied as-is, as they can be represented in utf8mb3. Characters that are
outside the BMP cannot be represented in utf8mb3 and are replaced
with U+FFFD, the "Replacement Character".

In utf8mb4_general_ci, the Replacement Character compares as equal to any
character that's not in BMP. Because of this, the constructed lookup value
will find all index records that would be considered equal by the original
condition (MB3-4-CMP).

Approved-by: Monty <monty@mariadb.org>

Author: Sergey Petrunia
Author Date: 2023-09-19 15:22:49 UTC

MDEV-32113: utf8mb3_key_col=utf8mb4_value cannot be used for ref

(Variant#3: Allow cross-charset comparisons, use a special
CHARSET_INFO to create lookup keys. Review input addressed.)

Equalities that compare utf8mb{3,4}_general_ci strings, like:

  WHERE ... utf8mb3_key_col=utf8mb4_value (MB3-4-CMP)

can now be used to construct ref[const] access and also participate
in multiple-equalities.
This means that utf8mb3_key_col can be used for key-lookups when
compared with an utf8mb4 constant, field or expression using '=' or
'<=>' comparison operators.

This is controlled by optimizer_switch='cset_narrowing=on', which is
OFF by default.

IMPLEMENTATION
Item value comparison in (MB3-4-CMP) is done using utf8mb4_general_ci.
This is valid as any utf8mb3 value is also an utf8mb4 value.

When making index lookup value for utf8mb3_key_col, we do "Charset
Narrowing": characters that are in the Basic Multilingual Plane (=BMP) are
copied as-is, as they can be represented in utf8mb3. Characters that are
outside the BMP cannot be represented in utf8mb3 and are replaced
with U+FFFD, the "Replacement Character".

In utf8mb4_general_ci, the Replacement Character compares as equal to any
character that's not in BMP. Because of this, the constructed lookup value
will find all index records that would be considered equal by the original
condition (MB3-4-CMP).

Approved-by: Monty <monty@mariadb.org>

Author: Monty
Author Date: 2023-10-19 13:55:14 UTC

MDEV-32518 Test failure: ./mtr --no-reorder main.log_slow_debug main.subselect

There where two errors left from the previous fix.

- subselect.test assumes that mysql.slow_log is empty. This was not
  enforced.
- subselect.test dropped a file that does not exists (for safety).
  This was fixed by ensuring we don't get a warning if the file does
  not exist.

Author: Monty
Author Date: 2023-10-19 09:57:30 UTC

MDEV-32476 LeakSanitizer errors in get_quick_select or Assertion ...

Problem was that JOIN_TAB::cleanup() was not run because
JOIN::top_join_tab_count was not set in case of early errors.

Fixed by setting JOIN::tab_join_tab_count when JOIN_TAB's are allocated.

Something that should eventually be fixed:
- Cleaning up JOIN_TAB's is now done in 3 different loops.
  JOIN_TAB::cleanup() is only doing a partial cleanup. Other cleanups
  are done outside of JOIN_TAB::cleanup().

The above should be fixed so that JOIN_TAB::cleanup() is freeing
everything related to it's own memory, including all its sub JOIN_ TAB's.
JOIN::cleanup() should only loop over all it's top JOIN_TAB's and call
JOIN_TAB::cleanup() on these.
This will greatly simplify and speedup the current code (as we now do some
cleanup's twice).

Author: Marko Mäkelä
Author Date: 2023-10-19 12:48:57 UTC

fixup! 64b77bc664a5885babd53fc0c3b85427d1a2b256

Actually this is a fixup for d1696901b5ea17ede03c45840ac4986fb2917b54
which refactored the code further.

Author: Alexander Barkov
Author Date: 2023-04-07 08:15:41 UTC

MDEV-30048 Prefix keys for CHAR work differently for MyISAM vs InnoDB

Also fixes: MDEV-30050 Inconsistent results of DISTINCT with NOPAD

Problem:

Key segments for CHAR columns where compared using strnncollsp()
for engines MyISAM and Aria.

This did not work correct in case if the engine applyied trailing
space compression.

Fix:

Replacing ha_compare_text() calls to:

- ha_compare_char_varying() - a new function
- ha_compare_char_fixed() - a new function
- cs->coll->strnncollsp()
- cs->coll->strnncollsp_nchars()

The code branch corresponding to comparison of CHAR column keys
(HA_KEYTYPE_TEXT segment type) now uses ha_compare_char_fixed()
which calls strnncollsp_nchars().

This patch does not change the behavior for the rest of the code:
- comparison of VARCHAR/TEXT column keys
  (HA_KEYTYPE_VARTEXT1, HA_KEYTYPE_VARTEXT2 segments types)
- comparison in the fulltext code

Author: Marko Mäkelä
Author Date: 2023-10-19 11:50:42 UTC

Merge 10.10 into 10.11

Author: Vlad Lesin
Author Date: 2023-10-18 13:26:15 UTC

MDEV-32512 log_page_corruption.test fails on windows build

Some calls of extend_space() perl function miss the last parameter. Some
cases in the test check corrupted pages info absence in the output
file if the table is dropped. We should also test if the
corruption presents before testing its absence.

Reviewed by: Vladislav Vaintroub

Author: Marko Mäkelä
Author Date: 2023-10-19 11:35:31 UTC

Merge 10.6 into 10.10

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-10-19 10:09:44 UTC

MDEV-31851 After crash recovery, undo tablespace fails to open

srv_all_undo_tablespaces_open(): While opening the extra unused
undo tablespaces, InnoDB should use ULINT_UNDEFINED instead of
SRV_SPACE_ID_UPPER_BOUND.

Author: VladislavVaintroub
Author Date: 2023-10-18 10:34:04 UTC

MDEV-20471 Assertion during cleanup of failed CREATE TABLE LIKE <sequence>

While cleaning up a failed CREATE TABLE LIKE <sequence>, `mysql_rm_table_no_locks`
erroneously attempted to remove all tables involved in the query, including
the source table (sequence).

Fix to temporarily modify `table_list` to ensure that only the intended
table is removed during the cleanup.

Author: Yuchen Pei
Author Date: 2023-10-19 07:37:53 UTC

MDEV-32515 Use $MYSQLD_LAST_CMD in spider/bugfix.mdev_30370

$MYSQLD_CMD uses .1 as the defaults-group-suffix, which could cause
the use of the default port (3306) or socket, which will fail in
environment where these defaults are already in use by another server.

Adding an extra --defaults-group-suffix=.1.1 does not help, because
the first flag wins.

So we use $MYSQLD_LAST_CMD instead, which uses the correct suffix.

Author: Yuchen Pei
Author Date: 2023-10-19 07:37:53 UTC

MDEV-32515 Use $MYSQLD_LAST_CMD in spider/bugfix.mdev_30370

$MYSQLD_CMD uses .1 as the defaults-group-suffix, which could cause
the use of the default port (3306) or socket, which will fail in
environment where these defaults are already in use by another server.

Adding an extra --defaults-group-suffix=.1.1 does not help, because
the first flag wins.

So we use $MYSQLD_LAST_CMD instead, which uses the correct suffix.

Author: Marko Mäkelä
Author Date: 2023-10-19 05:26:16 UTC

Merge 11.0 into 11.1

Author: Marko Mäkelä
Author Date: 2023-10-19 05:12:16 UTC

Merge 10.11 into 11.0

Author: Yuchen Pei
Author Date: 2023-10-19 04:42:43 UTC

MDEV-32515 [experiment] Add --bootstrap to the $MYSQLD_CMD invocation in mdev_30370

After all, spider/bugfix.mdev_22979 passes, and the only difference
that may matter is the --bootstrap flag

Author: Yuchen Pei
Author Date: 2023-10-19 02:26:43 UTC

MDEV-32515 Use the mtr cnf in the spider/bugfix.mdev_30370 mysqld invocation

This makes sure the $MYSQLD_CMD invocation uses the same port, and
does not use the default port which may already be in use.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-10-19 00:42:55 UTC

MDEV-32453 Bulk insert fails to apply when trigger does insert operation

- InnoDB fails to apply the bulk operation when trigger initiated
another statement on the operated table which is same as
associated table. InnoDB should apply all bulk operation
before ending the bulk insert operation on the operated table.

Author: Yuchen Pei
Author Date: 2023-10-19 00:01:29 UTC

[fixup] Spider fixup of the merge

Fixing the following
- Some commented out code were added in the merge
- Some changes from the following was reverted

b5d317197c6 MDEV-29676 refactored and documented spider_get_share() and friends

Author: Rex Johnston
Author Date: 2023-10-18 19:22:15 UTC

MDEV-32514 Slow log entries causing main.subselect mtr failures

In MDEV-7445, a test required an empty table as a data source.
mysql.slow_log was chosen as this table, but some CI testers
enable this in a worker as a command line option, meaning that
this table may no longer be empty. If the table returns more
than 1 row, the query fails, causing test failure.
This will clearly be a random failure, depending on OS workload.
Removed mysql.slow_log as data source.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-10-18 18:39:48 UTC

Merge branch 'zlib v1.3' into 10.4

Author: Marko Mäkelä
Author Date: 2023-10-18 15:48:42 UTC

Merge 10.6

Author: Brandon Nesterenko
Author Date: 2023-10-18 14:32:28 UTC

MDEV-31949: rpl_xa_prepare_gtid_fail deterministic paths

happy_xac is where the XA COMMIT completes before noticing the
error signalled by the prior XAP

sad_xac is where the XA COMMIT notices the error signalled by
the prior XAC and rolls back, leaving a dangling XAP.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-10-18 13:27:51 UTC

MDEV-32308 Server crash on cleanup of non-fully-constructed-due-to-an-error CTE

Check and do not try to access With_element::rec_result on cleanup
if it is not assigned.

Author: Brandon Nesterenko
Author Date: 2023-10-17 21:48:03 UTC

MDEV-31949: Fix rpl_xa_prepare_gtid_fail

rpl.rpl_xa_prepare_gtid_fail would sporadically fail, as an XA COMMIT,
running concurrently with a failed prepare, could sometimes complete
succesfully, or be rolled back due to failure of prior commit. The
test expected the completion case, but if it failed, the xa transaction
would be left in a prepared state with a lock on its table. This then
created a lock time-out during test cleanup, as it tries to drop that
table.

The fix is to extend the test to check if the transaction is still
prepared, and silently commit it if so, thus releasing the locks.
The non-determinism is fine (i.e. DEBUG_SYNC isn't needed to force
one path), as the verification performed by the test has already
completed. This is just for cleanup.

Author: Marko Mäkelä
Author Date: 2023-10-18 10:06:04 UTC

WIP merge 10.11 into 11.0

FIXME: Lots of assertion failures
sql/sql_statistics.cc:4483: bool is_eits_usable(Field*): Assertion field->orig_table->stats_is_read failed.

Failing test(s): main.subselect_firstmatch main.select_debug main.join_nested_jcl6 main.analyze_format_json_timings main.join_outer_jcl6 main.subselect_sj2_jcl6 optimizer_unfixed_bugs.bug49129 main.selectivity_no_engine main.limit_rows_examined main.subselect3_jcl6 main.subselect_innodb main.subselect_mat main.subselect_sj_mat main.subselect_sj2_mat main.subselect_sj main.subselect_sj_jcl6 main.select_jcl6 main.innodb_ext_key main.join_cache main.derived_opt main.join_outer main.selectivity_innodb

Author: Oleg Smirnov
Author Date: 2023-10-18 09:27:56 UTC

MDEV-32475 test_if_skip_sort_order() should catch the join types JT_EQ_REF, JT_CONST and JT_SYSTEM and skip sort order for these

Add logging of test_if_skip_sort_order to optimizer trace

Author: Yuchen Pei
Author Date: 2023-10-18 07:18:24 UTC

MDEV-32486 Fix duplicate spider_bulk_malloc ID

The id 38 is already used elsewhere, we use 44 instead, which hasn't
been used

Author: Yuchen Pei
Author Date: 2023-10-18 07:18:24 UTC

MDEV-32486 Fix duplicate spider_bulk_malloc ID

The id 38 is already used elsewhere, we use 44 instead, which hasn't
been used

Author: Yuchen Pei
Author Date: 2023-10-18 06:44:54 UTC

MDEV-26247 Re-implement spider gbh query rewrite of tables

Spider GBH's query rewrite of table joins is overly complex and
error-prone. We replace it with something closer to what
dbug_print() (more specifically, print_join()) does, but catered to
spider.

More specifically, we replace the body of
spider_db_mbase_util::append_from_and_tables() with a call to
spider_db_mbase_util::append_join(), and remove downstream append_X
functions.

We make it handle const tables by rewriting them as (select 1). This
fixes the main issue in MDEV-26247.

We also ban semijoin from spider gbh, which fixes MDEV-31645 and
MDEV-30392, as semi-join is an "internal" join, and "semi join" does
not parse, and it is different from "join" in that it deduplicates the
right hand side

Not all queries passed to a group by handler are valid (MDEV-32273),
for example, a join on expr may refer outer fields not in the current
context. We detect this during the handler creation when walking the
join. See also gbh_outer_fields_in_join.test.

It also skips eliminated tables, which fixes MDEV-26193.

Author: Dmitry Shulga
Author Date: 2023-10-18 04:15:16 UTC

MDEV-32369: Memory leak when executing PS for query with IN subquery

The memory allocated for an instance of the class Item_direct_ref_to_item
was leaked on second execution of a query run as a prepared statement and
involving conversion of strings with different character sets.

The reason of leaking the memory was caused by the fact that a statement
arena could be already set by the moment the method
Type_std_attributes::agg_item_set_converter() is called.

Author: Dmitry Shulga
Author Date: 2023-10-18 04:15:16 UTC

MDEV-32369: Memory leak when executing PS for query with IN subquery

The memory allocated for an instance of the class Item_direct_ref_to_item
was leaked on second execution of a query run as a prepared statement and
involving conversion of strings with different character sets.

The reason of leaking the memory was caused by the fact that a statement
arena could be already set by the moment the method
Type_std_attributes::agg_item_set_converter() is called.

Author: Dmitry Shulga
Author Date: 2023-10-18 04:15:16 UTC

MDEV-32369: Memory leak when executing PS for query with IN subquery

The memory allocated for an instance of the class Item_direct_ref_to_item
was leaked on second execution of a query run as a prepared statement and
involving conversion of strings with different character sets.

The reason of leaking the memory was caused by the fact that a statement
arena could be already set by the moment the method
Type_std_attributes::agg_item_set_converter() is called.

Author: Yuchen Pei
Author Date: 2023-10-17 04:02:12 UTC

MDEV-32485 Fix Spider upgrade failure caused by duplication in mysql.func

Author: Igor Babaev
Author Date: 2023-10-18 03:52:34 UTC

MDEV-32351 Significant slowdown for query with many outer joins

This is a preliminary patch. To be used just for testing.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-09-04 11:04:06 UTC

MDEV-29167 new db-level SHOW CREATE ROUTINE privilege

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-10-16 17:43:18 UTC

ibuf_delete_recs() doesn't remove all entries of page id

ibuf_delete_recs() should use the counter field for searching
the tuple when we're reopening the cursor.

Author: Sergei Golubchik
Author Date: 2023-10-15 14:03:14 UTC

MDEV-32473 --disable-ssl doesn't disable it

Author: Monty
Author Date: 2023-10-16 14:14:24 UTC

Fixed memory leak when using histograms

This was introduced in last merge with 10.6
The reason is that 10.6 does not need anything special to free histograms
as everything is allocated on a memroot.
In 10.10 histograms is using the vector class, which has some problems:
- No automatic free
- No memory usage accounting
(we should at some point remove vector usage because of the above problem)

Fixed by expliciting freeing histograms when freeing TABLE_STATISTICS
objects.

Author: Julius Goryavsky
Author Date: 2023-10-17 01:27:11 UTC

MDEV-31467: wsrep_sst_mariabackup not working on FreeBSD

Due to the different command line format of the timeout
utility on FreeBSD and Linux, SST scripts for mariabackup
may not work on FreeBSD. This commit fixes this problem
by adding a different command to test options on FreeBSD
and adding proper formatting for the utility options.

Author: VladislavVaintroub
Author Date: 2023-10-17 09:58:20 UTC

Innodb cleanup - log_t::append_prepare

- make the function non-static
- pass lsn to it, and do log_write_up_to to wait for this lsn, rather than
  always wait for the current one.

Author: Alexander Barkov
Author Date: 2023-10-17 08:46:31 UTC

MDEV-32244 Wrong bit encoding using COALESCE

When aggregating pairs BIT+NULL and NULL+BIT for result, e.g.
in COALESCE(), preserve the BIT data type (ignore explicit NULLs).

The same fix applied to YEAR.

Author: Alexander Barkov
Author Date: 2023-06-23 09:24:02 UTC

In progress: MDEV-31531 Remove my_casedn_str()

Author: Yuchen Pei
Author Date: 2023-10-17 07:14:05 UTC

Merge branch '10.10' into 10.11

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-10-17 06:42:24 UTC

MDEV-31851 After crash recovery, undo tablespace fails to open

Problem:
========
- InnoDB fails to open undo tablespace when page0 is corrupted
and fails to throw error.

Solution:
=========
- InnoDB throws DB_CORRUPTION error when InnoDB encounters
page0 corruption of undo tablespace.

- InnoDB restores the page0 of undo tablespace from
doublewrite buffer if it encounters page corruption

- Moved Datafile::restore_from_doublewrite() to
recv_dblwr_t::restore_first_page(). So that undo
tablespace and system tablespace can use this function
instead of duplicating the code

srv_undo_tablespace_open(): Returns 0 if file doesn't exist
or SRV_SPACE_ID_UPPER_BOUND if page0 is corrupted.

Author: Yuchen Pei
Author Date: 2023-10-17 04:02:12 UTC

MDEV-32485 Fix Spider upgrade failure caused by duplication in mysql.func

Author: Yuchen Pei
Author Date: 2023-10-17 04:28:16 UTC

[fixup] Spider fixup after merge

- Remove some references to dead macros

Author: Yuchen Pei
Author Date: 2023-10-11 05:40:30 UTC

MDEV-31996 Create connection on demand in spider_db_delete_all_rows

When spider_db_delete_all_rows() is called, the supplied spider->conns
may have already been freed. The existing mechanism has spider_trx own
the connections in trx_conn_hash and it may free a conn during the
cleanup after a query. When running a delete query and if the table is
in the table cache, ha_spider::open() would not be called which would
recreate the conn. So we recreate the conn when necessary during
delete by calling spider_check_trx_and_get_conn().

We also reduce code duplication as delete_all_rows() and truncate()
has almost identical code, and there's no need to assign
wide_handler->sql_command in these functions because it has already
been correctly assigned.

Author: Sergey Petrunia
Author Date: 2023-10-16 15:47:24 UTC

Revert accidentally pushed: commit e8c9cdc2f85d3d5c096f8af48216488fd304bc07

Author: Sergei Petrunia <sergey@mariadb.com>
Date: Wed Oct 11 19:02:25 2023 +0300

    MDEV-32301: Server crashes at Arg_comparator::compare_row

    In Item_bool_rowready_func2::build_clone(): if we're setting
      clone->cmp.comparators=0
    also set
      const_item_cache=0
    as the Item is currently in a state where one cannot compute it.

Author: Sergey Petrunia
Author Date: 2023-10-12 11:19:08 UTC

MDEV-32324: Server crashes inside filesort at my_decimal::to_binary

A subquery in form "(SELECT not_null_value LIMIT 1 OFFSET 1)" will
produce no rows which will translate into scalar SQL NULL value.

The code in Item_singlerow_subselect::fix_length_and_dec() failed to
take the LIMIT/OFFSET clause into account and used to set
item_subselect->maybe_null=0, despite that SQL NULL will be produced.

If such subselect was used in ORDER BY, this would cause a crash in
filesort() code when it would get a NULL value for a not-nullable item.

also made subselect_engine::no_tables() const function.

Author: Marko Mäkelä
Author Date: 2023-10-16 10:26:06 UTC

rebase fixup

Author: Tuukka Pasanen
Author Date: 2023-10-10 06:45:48 UTC

MDEV-30953: Add package for Galera server for RPM

Add new package mariadb-server-galera to RPM build
recude mariadb-server package dependencies.
Galera cluster scripts and libraries are currently
in package mariadb-server. Move Galera cluster
scripts and libraries to new package mariadb-server-galera
as most of the people does not need them and
it allows to slim down mariadb-server package

Author: Sergey Petrunia
Author Date: 2023-10-13 09:17:25 UTC

MDEV-32320: Server crashes at TABLE::add_tmp_key

The code inside Item_subselect::fix_fields() could fail to check
that left expression had an Item_row, like this:

  (('x', 1.0) ,1) IN (SELECT 'x', 1.23 FROM ... UNION ...)

In order to hit the failure, the first SELECT of the subquery had
to be a degenerate no-tables select. In this case, execution will
not enter into Item_in_subselect::create_row_in_to_exists_cond()
and will not check if left_expr is composed of scalars.

But the subquery is a UNION so as a whole it is not degenerate.
We try to create an expression cache for the subquery.
We create a temp.table from left_expr columns. No field is created
for the Item_row. Then, we crash when trying to add an index over a
non-existent field.

Fixed by moving the left_expr cardinality check to a point in
check_and_do_in_subquery_rewrites() which gets executed for all
cases.
It's better to make the check early so we don't have to care about
subquery rewrite code hitting Item_row in left_expr.

Author: Elena Stepanova
Author Date: 2023-10-14 16:58:51 UTC

The list is not relevant anymore, bugs are fixed

Author: Sergei Golubchik
Author Date: 2023-10-07 08:49:36 UTC

Revert "MDEV-29091: Correct event_name in PFS for wait caused by FOR UPDATE"

This reverts commit 03c9a4ef4a0363ba0ffd1843284332fb9c5fd692.

The fix is wrong. It was doing this: if the uninitialized
wait->m_class has some specific value, then don't initialize it.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-10-13 09:15:14 UTC

Removing MDEV-27871 because it is not a bug

Part 1

Author: Nikita Malyavin
Author Date: 2023-10-11 10:00:11 UTC

Extend XID instead of xid_t

Author: Sergey Petrunia
Author Date: 2023-10-13 09:17:25 UTC

MDEV-32320: Server crashes at TABLE::add_tmp_key

The code inside Item_subselect::fix_fields() could fail to check
that left expression had an Item_row, like this:

  (('x', 1.0) ,1) IN (SELECT 'x', 1.23 FROM ... UNION ...)

In order to hit the failure, the first SELECT of the subquery had
to be a degenerate no-tables select. In this case, execution will
not enter into Item_in_subselect::create_row_in_to_exists_cond()
and will not check if left_expr is composed of scalars.

But the subquery is a UNION so as a whole it is not degenerate.
We try to create an expression cache for the subquery.
We create a temp.table from left_expr columns. No field is created
for the Item_row. Then, we crash when trying to add an index over a
non-existent field.

Fixed by moving the left_expr cardinality check to a point in
check_and_do_in_subquery_rewrites() which gets executed for all
cases.
It's better to make the check early so we don't have to care about
subquery rewrite code hitting Item_row in left_expr.

Author: Yuchen Pei
Author Date: 2023-10-11 01:13:23 UTC

MDEV-30014 Spider should not second guess server when locking / unlocking

This fixes MDEV-30014, MDEV-29456, and MDEV-29667.

Author: Yuchen Pei
Author Date: 2023-10-13 06:16:57 UTC

MDEV-29963 MDEV-31357 Spider should clear its lock lists when locking fails

Spider populates its lock lists (a hash) in store_lock(), and normally
clears them in the actual lock_tables(). However, if lock_tables()
fails, there's no reset_lock() method for storage engine handlers,
which can cause bad things to happen. For example, if one of the table
involved is dropped and recreated, or simply TRUNCATEd, when executing
LOCK TABLES again, the lock lists would be queried again in
store_lock(), which could cause access to freed space associated with
the dropped table.

Author: Yuchen Pei
Author Date: 2023-10-13 06:16:57 UTC

MDEV-29963 MDEV-31357 Spider should clear its lock lists when locking fails

Spider populates its lock lists (a hash) in store_lock(), and normally
clears them in the actual lock_tables(). However, if lock_tables()
fails, there's no reset_lock() method for storage engine handlers,
which can cause bad things to happen. For example, if one of the table
involved is dropped and recreated, or simply TRUNCATEd, when executing
LOCK TABLES again, the lock lists would be queried again in
store_lock(), which could cause access to freed space associated with
the dropped table.

Author: Yuchen Pei
Author Date: 2023-10-13 06:16:57 UTC

MDEV-29963 MDEV-31357 Spider should clear its lock lists when locking fails

Spider populates its lock lists (a hash) in store_lock(), and normally
clears them in the actual lock_tables(). However, if lock_tables()
fails, there's no reset_lock() method for storage engine handlers,
which can cause bad things to happen. For example, if one of the table
involved is dropped and recreated, or simply TRUNCATEd, when executing
LOCK TABLES again, the lock lists would be queried again in
store_lock(), which could cause access to freed space associated with
the dropped table.

Author: Yuchen Pei
Author Date: 2023-10-05 02:25:10 UTC

MDEV-26247 Re-implement spider gbh query rewrite of tables

Spider GBH's query rewrite of table joins is overly complex and
error-prone. We replace it with something closer to what
dbug_print() (more specifically, print_join()) does, but catered to
spider.

More specifically, we replace the body of
spider_db_mbase_util::append_from_and_tables() with a call to
spider_db_mbase_util::append_join(), and remove downstream append_X
functions.

We make it handle const tables by rewriting them as (select 1). This
fixes the main issue in MDEV-26247.

We also ban semijoin from spider gbh, which fixes MDEV-31645 and
MDEV-30392, as semi-join is an "internal" join, and "semi join" does
not parse, and it is different from "join" in that it deduplicates the
right hand side

Not all queries passed to a group by handler are valid (MDEV-32273),
for example, a join on expr may refer outer fields not in the current
context. We detect this during the handler creation when walking the
join. See also gbh_outer_fields_in_join.test.

It also skips eliminated tables, which fixes MDEV-26193.

Author: Yuchen Pei
Author Date: 2023-10-05 02:19:58 UTC

MDEV-26247 Re-implement spider gbh query rewrite of tables

Spider GBH's query rewrite of table joins is overly complex and
error-prone. We replace it with something closer to what
dbug_print() (more specifically, print_join()) does, but catered to
spider.

More specifically, we replace the body of
spider_db_mbase_util::append_from_and_tables() with a call to
spider_db_mbase_util::append_join(), and remove downstream append_X
functions.

We make it handle const tables by rewriting them as (select 1). This
fixes the main issue in MDEV-26247.

We also ban semijoin from spider gbh, which fixes MDEV-31645 and
MDEV-30392, as semi-join is an "internal" join, and "semi join" does
not parse, and it is different from "join" in that it deduplicates the
right hand side

Not all queries passed to a group by handler are valid (MDEV-32273),
for example, a join on expr may refer outer fields not in the current
context. We detect this during the handler creation when walking the
join. See also gbh_outer_fields_in_join.test.

It also skips eliminated tables, which fixes MDEV-26193.

Author: Yuchen Pei
Author Date: 2023-10-13 04:17:54 UTC

MDEV-28998 remove a known reference to a SPIDER_CONN when it is freed

Author: Yuchen Pei
Author Date: 2023-10-05 02:19:58 UTC

MDEV-26247 Re-implement spider gbh query rewrite of tables

Spider GBH's query rewrite of table joins is overly complex and
error-prone. We replace it with something closer to what
dbug_print() (more specifically, print_join()) does, but catered to
spider.

More specifically, we replace the body of
spider_db_mbase_util::append_from_and_tables() with a call to
spider_db_mbase_util::append_join(), and remove downstream append_X
functions.

We make it handle const tables by rewriting them as (select 1). This
fixes the main issue in MDEV-26247.

We also ban semijoin from spider gbh, which fixes MDEV-31645 and
MDEV-30392, as semi-join is an "internal" join, and "semi join" does
not parse, and it is different from "join" in that it deduplicates the
right hand side

Not all queries passed to a group by handler are valid (MDEV-32273),
for example, a join on expr may refer outer fields not in the current
context. We detect this during the handler creation when walking the
join. See also gbh_outer_fields_in_join.test.

It also skips eliminated tables, which fixes MDEV-26193.

Author: Yuchen Pei
Author Date: 2023-10-05 02:19:58 UTC

MDEV-26247 Re-implement spider gbh query rewrite of tables

Spider GBH's query rewrite of table joins is overly complex and
error-prone. We replace it with something closer to what
dbug_print() (more specifically, print_join()) does, but catered to
spider.

More specifically, we replace the body of
spider_db_mbase_util::append_from_and_tables() with a call to
spider_db_mbase_util::append_join(), and remove downstream append_X
functions.

We make it handle const tables by rewriting them as (select 1). This
fixes the main issue in MDEV-26247.

We also ban semijoin from spider gbh, which fixes MDEV-31645 and
MDEV-30392, as semi-join is an "internal" join, and "semi join" does
not parse, and it is different from "join" in that it deduplicates the
right hand side

Not all queries passed to a group by handler are valid (MDEV-32273),
for example, a join on expr may refer outer fields not in the current
context. We detect this during the handler creation when walking the
join. See also gbh_outer_fields_in_join.test.

It also skips eliminated tables, which fixes MDEV-26193.

Author: Yuchen Pei
Author Date: 2023-09-14 05:19:50 UTC

MDEV-29502 Fix some issues with spider direct aggregate

The direct aggregate mechanism sems to be only intended to work when
otherwise a full table scan query will be executed from the spider
node and the aggregation done at the spider node too. Typically this
happens in sub_select(). In the test spider.direct_aggregate_part
direct aggregate allows to send COUNT statements directly to the data
nodes and adds up the results at the spider node, instead of iterating
over the rows one by one at the spider node.

By contrast, the group by handler (GBH) typically sends aggregated
queries directly to data nodes, in which case DA does not improve the
situation here.

That is why we should fix it by disabling DA when GBH is used.

There are other reasons supporting this change. First, the creation of
GBH results in a call to change_to_use_tmp_fields() (as opposed to
setup_copy_fields()) which causes the spider DA function
spider_db_fetch_for_item_sum_funcs() to work on wrong items. Second,
the spider DA function only calls direct_add() on the items, and the
follow-up add() needs to be called by the sql layer code. In
do_select(), after executing the query with the GBH, it seems that the
required add() would not necessarily be called.

Disabling DA when GBH is used does fix the bug. There are a few
other things included in this commit to improve the situation with
spider DA:

1. Add a session variable that allows user to disable DA completely,
this will help as a temporary measure if/when further bugs with DA
emerge.

2. Move the increment of direct_aggregate_count to the spider DA
function. Currently this is done in rather bizarre and random
locations.

3. Fix the spider_db_mbase_row creation so that the last of its row
field (sentinel) is NULL. The code is already doing a null check, but
somehow the sentinel field is on an invalid address, causing the
segfaults. With a correct implementation of the row creation, we can
avoid such segfaults.

Author: Yuchen Pei
Author Date: 2023-07-05 07:55:30 UTC

MDEV-31117 Fix spider connection info parsing

Spider connection string is a comma-separated parameter definitions,
where each definition is of the form "<param_title> <param_value>",
where <param_value> is quote delimited on both ends, with backslashes
acting as an escaping prefix.

Despite the simple syntax, the existing spider connection string
parser was poorly-written, complex, hard to reason and error-prone,
causing issues like the one described in MDEV-31117. For example it
treated param title the same way as param value when assigning, and
have nonsensical fields like delim_title_len and delim_title.

Thus as part of the bugfix, we clean up the spider comment connection
string parsing, including:

- Factoring out some code from the parsing function
- Simplify the struct `st_spider_param_string_parse`
- And any necessary changes caused by the above changes

Author: Yuchen Pei
Author Date: 2023-10-13 01:06:27 UTC

MDEV-27902 Spider check trx and get conn in rnd_next()

Author: Julius Goryavsky
Author Date: 2023-10-10 09:56:19 UTC

MDEV-29893: SST fails when having datadir set to a symlink

SST for mariabackup may not destroy old files if datadir or
other working directory is declared as a symlink due to the lack
of the "-L" option among the find utility options, similarly SST
for rsync in some cases may not transfer data directories if they
are created as symlinks. This fix adds the missing option and
generally unifies the work with find utility options to avoid
failures in the interpretation of directories and regular
expressions.

Author: Rucha Deodhar
Author Date: 2023-10-12 11:41:07 UTC

MDEV-31684: More tests

Author: Sergey Petrunia
Author Date: 2023-10-12 11:19:08 UTC

MDEV-32324: Server crashes inside filesort at my_decimal::to_binary

A subquery in form "(SELECT not_null_value LIMIT 1 OFFSET 1)" will
produce no rows which will translate into scalar SQL NULL value.

The code Item_singlerow_subselect::fix_length_and_dec() failed to
take LIMIT clause into account and set item_subselect->maybe_null=0.

This caused a crash in filesort() code when it got a NULL value for a
not nullable item.

Author: midenok
Author Date: 2023-10-12 10:26:19 UTC

rpl_row fix

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-10-12 09:19:27 UTC

MDEV-32337 Assertion `pos < table->n_def' failed in dict_table_get_nth_col

While checking for altered column in foreign key constraints,
InnoDB fails to ignore virtual columns. This issue caused
by commit 5f09b53bdb4e973e7c7ec2c53a24c98321223f98(MDEV-31086).

Author: Yuchen Pei
Author Date: 2023-10-12 03:39:14 UTC

MDEV-29963 Spider should clear its lock lists when locking fails

Spider populates its lock lists (a hash) in store_lock(), and normally
clears them in the actual lock_tables(). However, if lock_tables()
fails, there's no reset_lock() method for storage engine handlers,
which can cause bad things to happen. For example, if one of the table
involved is dropped and recreated, when executing LOCK TABLES again,
the lock lists would be queried again in store_lock(), which could
cause access to freed stuff.

This also fixes MDEV-31357

Author: Yuchen Pei
Author Date: 2023-10-12 00:54:21 UTC

MDEV-27902 Spider check trx and get conn in rnd_next()

This is an old fix for the MDEV-27902 test case in 10.4, but the
failure is caused by a slightly different issue MDEV-29962.

Author: Yuchen Pei
Author Date: 2023-10-12 00:52:12 UTC

MDEV-27902 MDEV-29962 Spider check trx and get conn before locking

In 10.4 the testcase for MDEV-27902 displays symptoms of MDEV-29962,
which is why we do the check trx and get conn before locking,
different from other MDEV-27902 fixes for higher versions.

Author: Alexander Barkov
Author Date: 2023-09-27 06:54:30 UTC

MDEV-32249 strings/ctype-ucs2.c:2336: my_vsnprintf_utf32: Assertion `(n % 4) == 0' failed in my_vsnprintf_utf32 on INSERT

The crash inside my_vsnprintf_utf32() happened correctly,
because the caller methods:
  Field_string::sql_rpl_type()
  Field_varstring::sql_rpl_type()
mis-used the charset library and sent pure ASCII data to the
virtual function snprintf() of a utf32 CHARSET_INFO.

It was wrong to use Field::charset() in sql_rpl_type().
We're printing the metadata (the data type) here, not the column data.
The string contraining the data type of a CHAR/VARCHAR column
is a pure ASCII string.

Fixing to use res->charset() to print, like all virtual implementations
of sql_type() do.

Review was done by Andrei Elkin.
Thanks to Andrei for proposing MTR test improvents.

Author: Sergey Petrunia
Author Date: 2023-10-11 16:02:25 UTC

MDEV-32301: Server crashes at Arg_comparator::compare_row

In Item_bool_rowready_func2::build_clone(): if we're setting
  clone->cmp.comparators=0
also set
  const_item_cache=0
as the Item is currently in a state where one cannot compute it.

Author: Dmitry Shulga
Author Date: 2023-10-11 07:47:05 UTC

MDEV-32123: require_secure_transport doesn't allow TCP connections

In case the option require_secure_transport is on the user can't
establish a secure ssl connection over TCP protocol. Inability to set up
a ssl session over TCP was caused by the fact that a type of client's
connection was checked before ssl handshake performed (ssl handshake
happens at the function acl_authenticate()). At that moment vio type has
the value VIO_TYPE_TCPIP for client connection that uses TCP transport.
In result, checking for allowable vio type for fails despite the fact
that SSL session being established. To fix the issue move checking of
vio type for allowable values inside the function
  parse_client_handshake_packet()
right after client's capabilities discovered that SSL is not requested
by the client.

Author: Dmitry Shulga
Author Date: 2023-10-11 07:47:05 UTC

MDEV-32123: require_secure_transport doesn't allow TCP connections

In case the option require_secure_transport is on the user can't
establish a secure ssl connection over TCP protocol. Inability to set up
a ssl session over TCP was caused by the fact that a type of client's
connection was checked before ssl handshake performed (ssl handshake
happens at the function acl_authenticate()). At that moment vio type has
the value VIO_TYPE_TCPIP for client connection that uses TCP transport.
In result, checking for allowable vio type for fails despite the fact
that SSL session being established. To fix the issue move checking of
vio type for allowable values inside the function
  parse_client_handshake_packet()
right after client's capabilities discovered that SSL is not requested
by the client.

Author: Yuchen Pei
Author Date: 2023-10-11 07:31:23 UTC

MDEV-27902 Spider check trx and get conn in rnd_next()

Author: Yuchen Pei
Author Date: 2023-10-11 07:04:40 UTC

MDEV-29962 Spider: check trx and get conn before lock_tables()

Same cause as MDEV-31996. This prevents reuse of conns freed
previously, e.g. commit of previous statement.

Author: Lena Startseva
Author Date: 2023-08-30 05:22:07 UTC

MDEV-31933: Make working view-protocol + ps-protocol (running two protocols together)

Now tests from suite main can be run with view-protocol + ps-protocol (together)

Author: Yuchen Pei
Author Date: 2023-10-11 05:59:10 UTC

MDEV-29962 Spider: check trx and get conn before lock_tables()

Same cause as MDEV-31996. This prevents reuse of conns freed
previously, e.g. commit of previous statement.

Author: Yuchen Pei
Author Date: 2023-10-11 05:29:51 UTC

MDEV-31996 Create connection on demand in spider_db_delete_all_rows

When spider_db_delete_all_rows() is called, the supplied spider->conns
may have already been freed. The existing mechanism has spider_trx own
the connections in trx_conn_hash and it may free a conn during the
cleanup after a query. When running a delete query and if the table is
in the table cache, ha_spider::open() would not be called which would
recreate the conn. So we recreate the conn when necessary during
delete by calling spider_check_trx_and_get_conn().

We also reduce code duplication as delete_all_rows() and truncate()
has almost identical code, and there's no need to assign
wide_handler->sql_command in these functions because it has already
been correctly assigned.

Author: Yuchen Pei
Author Date: 2023-10-11 05:40:30 UTC

MDEV-31996 Create connection on demand in spider_db_delete_all_rows

When spider_db_delete_all_rows() is called, the supplied spider->conns
may have already been freed. The existing mechanism has spider_trx own
the connections in trx_conn_hash and it may free a conn during the
cleanup after a query. When running a delete query and if the table is
in the table cache, ha_spider::open() would not be called which would
recreate the conn. So we recreate the conn when necessary during
delete by calling spider_check_trx_and_get_conn().

We also reduce code duplication as delete_all_rows() and truncate()
has almost identical code, and there's no need to assign
wide_handler->sql_command in these functions because it has already
been correctly assigned.

Author: Yuchen Pei
Author Date: 2023-10-11 05:29:51 UTC

MDEV-31996 Create connection on demand in spider_db_delete_all_rows

When spider_db_delete_all_rows() is called, the supplied spider->conns
may have already been freed. The existing mechanism has spider_trx own
the connections in trx_conn_hash and it may free a conn during the
cleanup after a query. When running a delete query and if the table is
in the table cache, ha_spider::open() would not be called which would
recreate the conn. So we recreate the conn when necessary during
delete by calling spider_check_trx_and_get_conn().

We also reduce code duplication as delete_all_rows() and truncate()
has almost identical code, and there's no need to assign
wide_handler->sql_command in these functions because it has already
been correctly assigned.

Author: Yuchen Pei
Author Date: 2023-10-11 05:29:51 UTC

MDEV-31996 Create connection on demand in spider_db_delete_all_rows

When spider_db_delete_all_rows() is called, the supplied spider->conns
may have already been freed. The existing mechanism has spider_trx own
the connections in trx_conn_hash and it may free a conn during the
cleanup after a query. When running a delete query and if the table is
in the table cache, ha_spider::open() would not be called which would
recreate the conn. So we recreate the conn when necessary during
delete by calling spider_check_trx_and_get_conn().

We also reduce code duplication as delete_all_rows() and truncate()
has almost identical code, and there's no need to assign
wide_handler->sql_command in these functions because it has already
been correctly assigned.

Author: Yuchen Pei
Author Date: 2023-10-11 05:29:51 UTC

MDEV-31996 Create connection on demand in spider_db_delete_all_rows

When spider_db_delete_all_rows() is called, the supplied spider->conns
may have already been freed. The existing mechanism has spider_trx own
the connections in trx_conn_hash and it may free a conn during the
cleanup after a query. When running a delete query and if the table is
in the table cache, ha_spider::open() would not be called which would
recreate the conn. So we recreate the conn when necessary during
delete by calling spider_check_trx_and_get_conn().

We also reduce code duplication as delete_all_rows() and truncate()
has almost identical code, and there's no need to assign
wide_handler->sql_command in these functions because it has already
been correctly assigned.

Author: VladislavVaintroub
Author Date: 2023-10-11 04:15:26 UTC

Merge branch 'bb-10.4-wlad' of https://github.com/mariadb/server into bb-10.4-wlad

Author: Yuchen Pei
Author Date: 2023-10-11 04:04:14 UTC

[experiment] is have_innodb.inc the problem?

Author: Yuchen Pei
Author Date: 2023-09-25 05:46:59 UTC

MDEV-32238 Adding a switch to disable the spider group by handler

The system variable spider_disable_group_by_handler, if on, will
disable the spider group by handler (gbh), and such disablement serves
as workaround to bugs caused by gbh, labelled with spider-gbh on jira,
including MDEV-26247, MDEV-28998, MDEV-29163, MDEV-30392, MDEV-31645.
Tests for these tickets are added accordingly with the workaround in
place.