Dumpling: support specifying the character set of the output files

[kennytm]

Feature Request

Is your feature request related to a problem? Please describe:

Currently dumpling assumes the output is always utf8mb4. If a user wants the result in a different character set such as GBK, they will either need iconv or changing @@character_set_results. The former requires 2x the storage space while the latter may cause unexpected transformation like insertion of unwanted \ and ".

Describe the feature you'd like:

Add a flag that specifies the output encoding (naming and options follow that of Lightning):

• --data-character-set=«encoding», should support:
  • utf8mb4
  • gb18030
  • (optional) gbk
  • (optional) latin1
• (optional) --schema-character-set=«encoding», default to --data-character-set if unspecified
• (optional) --data-invalid-char-replace='?', what to do with characters outside of the specified charset.

Both CSV and SQL output format should be supported, and should round-trip with the corresponding Lightning settings (including the cases with custom CSV separator/terminator etc)

Describe alternatives you've considered:

Nothing, document that Dumpling can only output utf8mb4 and require users to use iconv to perform conversion.

Teachability, Documentation, Adoption, Migration Strategy:

[kennytm]

changing @@character_set_results ... may cause unexpected transformation like insertion of unwanted \ and ".

and to clarify, this is the PoC:

CREATE TABLE test_table (
  id INT PRIMARY KEY,
  value VARCHAR(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci
);
INSERT INTO test_table (id, value) VALUES (1, '玕');

tiup dumpling:v7.1.5 \
  --sql 'select * from test.test_table' \
  --params 'character_set_client=gbk,character_set_connection=gbk,character_set_results=gbk' ...

if you check the produced file in GBK encoding you will see the value 玕\ (note the extra backslash) because the GBK encoding of 玕 is AB 5C so --escape-backslash will escape that 5C in the middle of this multi-byte character.