NextStack
EN 中

seatunnel 2.3.3

patch PublishedAug 22, 2023 Sourceapache/seatunnel

seatunnel 2.3.3

2.3.3 Release Note

Bug fix

Core

  • [Translation] [Spark] Fix SeaTunnelRowConvertor fail to convert when schema contains row type (#5170)

Connector

  • [Connector] [Jdbc] Fix the problem of JdbcOutputFormat database connection leak (#4802)
  • [Connector-V2] [AmazonDynamoDB] Fix the problem that all table data cannot be obtained (#5146)
  • [Connector-V2] [CDC-base] Fix cdc connection does not close (#4922)
  • [Connector-V2] [Clickhouse] Fix clickhouse source connector read Nullable() type is not null,example:Nullable(Float64) while value is null the result is 0.0 (#5080)
  • [Connector-V2] [ClickhouseFile] Fix ClickhouseFile write file failed when field value is null (#4937)
  • [Connector-V2] [Doris] Add streamload Content-type for doris URLdecode error (#4880)
  • [Connector-V2] [E2E] Fix maven scope (#4901)
  • [Connector-V2] [File-Base] Fix Hadoop Kerberos authentication related issues (#5171)
  • [Connector-V2] [Flink-runtime] Flink register table Environment The running mode is set tojob.mode (#4826)
  • [Connector-V2] [HbaseSink] Fix default timestamp (#4958)
  • [Connector-V2] [Hive] Support user-defined hive-site.xml (#4965)
  • [Connector-V2] [Jdbc] Support postgresql json type (#5194)
  • [Connector-V2] [Jdbc] Fix XA Transaction bug (#5020)
  • [Connector-V2] [Jdbc] Modify error comment in JdbcSourceConfigFactory (#5088)
  • [Connector-V2] [Jdbc-Oracle] Fix the Oracle number type mapping problem (#5209)
  • [Connector-V2] [Jdbc-Postgres] PostgreSQL versions below 9.5 are compatible use cdc sync problem (#5120)
  • [Connector-V2] [JindoOssFile] Fix plugin-mapping.properties (#5215)
  • [Connector-V2] [LocalFileSink] Fix LocalFile Sink file_format_type (#5118)
  • [Connector-V2] [Mongodb] Fix document error content and remove redundant code (#4982)
  • [Connector-V2] [Mongodb] Compatible with historical parameters (#4997)
  • [Connector-V2] [MongoDB-CDC] Incremental data kind error in snapshot phase (#5184)
  • [Connector-V2] [MongoDB-CDC] Solve startup resume token is negative (#5143)
  • [Connector-V2] [MongoDB-CDC] Refine data format to adapt to universal logic (#5162)
  • [Connector-V2] [MongoDB-CDC] Fix array index anomalies caused by #5057 (#5195)
  • [Connector-V2] [MySql-CDC] Fix jdbc connection leak for mysql (#5037)
  • [Connector-V2] [MySql-CDC] Fix mysql bit column to java byte (#4817)
  • [Connector-V2] [MySql-CDC] Fix listener not released when BinlogClient reuse (#5011)
  • [Connector-V2] [MySql-CDC] ServerId from int to long (#5033) (#5035)
  • [Connector-V2] [Paimon] Bump paimon-bundle version to 0.4.0-incubating (#5219)
  • [Connector-V2] [Redis] Fix redis nodes format error (#4981)
  • [Connector-V2] [StarRocks] Fix StarRocksJsonSerializer will transform array/map/row to string (#5281)
  • [Connector-V2] [StarRocks] Fix code style (#4966)

Zeta(ST-Engine)

  • [Zeta] Resolved the issue causing checkpoints to halt on tolerable-failure=0 (#5263)
  • [Zeta] Fix the issue of two identical IDs appearing when executing seatunnel.sh -l as the job resumes (#5191)
  • [Zeta] Fix SinkFlowLifeCycle without init lastCommitInfo (#5152)
  • [Zeta] Fix MultipleTableJobConfigParser ignore env option (#5067)
  • [Zeta] Fix checkpoint error report without msg (#5137)
  • [Zeta] [REST-API] Fix class not found exception (#5063)
  • [Zeta] [REST-API] Fix browser long type intercept (#5267)
  • [Zeta] Fix task state memory leak (#5139)
  • [Zeta] Fix when execute seatunnel.sh, console may print error (#4984)
  • [Zeta] Fixed the issue of duplicated metrics caused by job fault tolerance or restore (#5214)
  • [Zeta] Avoid Redundant Job Submissions by Checking Job Status (#5229)
  • [Zeta] Fix the serialization issue of GetMetricsOperation during multi-node operation (#5206)
  • [Zeta] Fix after the savepoint job is restored, the checkpoint file cannot be generated #4985 (#5051)
  • [Zeta] Fixed multi-table job data loss and latency issues (#149) (#5031)
  • [Zeta] Fix the HdfsStorage can not delete checkpoint file #5046 (#5054)
  • [Zeta] Fix if the initialization task throws an error and the task can not return to master (#4888)

TEST

  • [Test] Split updated modules integration test for part 5 (#5208)
  • [Test] Add code style check when docs changed (#5183)
  • [Test] Turn on the testSavepoint test (#5199)
  • [Test] Fix the CI Job name error (#5032)
  • [Test] [E2E] Fix testGetErrorInfo case error (#5282)
  • [Test] [Kafka] Kafka e2e error (#5200)
  • [Test] [MongoDB-CDC] Fix mongodb cdc e2e instability (#5128)
  • [Test] Declare files that will always have UNIX line endings on checkout (#5221)
  • [Test] Fixed a minor bug (#5274)
  • [Test] Fixed IMap file storage e2e bug (#5237)
  • [Test] Split updated modules integration test for part 4 (#5028)
  • [Test] Fix zeta ci error (#5254)

Others

  • Update the project description (#4967)
  • Update version to 2.3.3-SNAPSHOT (#5043)
  • Fix com.google.common.base.Preconditions to seatunnel shade one (#5284)
  • Fix style error (#5280)
  • Fix code style (#5092)

Improve

Core

  • [API & Zeta] Using connector custom serializer encode/decode states (#5238)
  • [API]Improve savemode api (#4767)

Connector

  • [CDC Base] Add some split parameters to the optionRule (#5161)
  • [CDC Base] Support exactly-once of cdc and fix the BinlogOffset comparing bug (#5057)
  • [CDC Base] Add a fast sampling method that supports character types (#5179)
  • [CDC Base] Change driver scope to provider (#5002)
  • [CDC Base] Remove driver for cdc connector (#4952)
  • [Connector-V2] [File] Optimize files commit order (#5045)
  • [Connector-V2] [HiveSink] Remove drop partition when abort (#4940)
  • [Connector-V2] [Jdbc] Remove scheduler in JDBC sink #4736 (#5168)
  • [Connector-V2] [Jdbc] Check url not null throw friendly message (#5097)
  • [Connector-V2] [Jdbc] Support for Decimal types as splict keys (#4634)
  • [Connector-V2] [Mongodb] Sink support transaction update/writing (#5034)
  • [Connector-V2] [Mongodb] Optimize reading logic (#5001)
  • [Connector-V2] [Neo4j] Supports neo4j sink batch write and update docs (#4841)
  • [Connector-V2] [OSS-Jindo] Optimize jindo oss connector (#4964)
  • [Connector-V2] [Tablestore sink] Remove scheduler in Tablestore sink (#5272)

Transform

  • [Transform-V2] [SQL] Support use catalogTableName as SQL expression (#5273)

CI

  • [CI/CD] Remove ‘paths-ignore’, enable the code style check for markdown files (#5286)

Zeta(ST-Engine)

  • [Zeta] Upgrade the key log output level (#4993)
  • [Zeta] Don’t trigger handleSaveMode when restore (#5192)
  • [Zeta] Move checkpoint notify complete in checkpoint stage (#5185)
  • [Zeta] Add sleep for Task to reduce CPU cost (#5117)
  • [Zeta] Add an interface for batch retrieval of JobMetrics (#4576)
  • [Zeta] Move imap storage file dependency packages to submodules (#5218)
  • [Zeta] Improve CheckpointCoordinator log error when report error from task (#178) (#5134)
  • [Zeta] Improve CheckpointCoordinator notify complete when restore (#5136)

Feature

Connector

  • [Connector-V2] [CDC-base] Support tables without primary keys (with unique keys) (#163) (#5150)
  • [Connector-V2] [CDC-base] Support disable/enable exactly once for INITIAL (#4921)
  • [Connector-V2] [CDC-base] Change the time zone to the default time zone (#5030)
  • [Connector-V2] [CDC-base] Support string type shard fields (#5147)
  • [Connector-V2] [Clickhouse] Add clickhouse connector time zone key,default system time zone (#5078)
  • [Connector-V2] [Clickhouse] Clickhouse writes with checkpoints (#4999)
  • [Connector-V2] [Doris] Add doris factory type convertor (#5061)
  • [Connector-V2] [File] Add config of ‘file_filter_pattern’, which used for filtering files (#5153)
  • [Connector-V2] [File] Add cos source&sink (#4979)
  • [Connector-V2] [Hive] Hive add option because web need (#5154)
  • [Connector-V2] [Jdbc] Add JDBC Catalog auto create table (#4917)
  • [Connector-V2] [Jdbc] Add oceanbase dialect factory (#4989)
  • [Connector-V2] [Jdbc] JDBC source support string type as partition key (#4947)
  • [Connector-V2] [Jdbc-DM] Add DM upsert support (#5073)
  • [Connector-V2] [Kafka] Support read debezium format message from kafka (#5066)
  • [Connector-V2] [Kafka] Connector-kafka source support data conversion extracted by kafka connect source (#4516)
  • [Connector-V2] [Mongodb] Mongodb support cdc sink (#4833)
  • [Connector-V2] [Mongodb-CDC] Support source mongodb cdc (#4923)
  • [Connector-V2] [Mysql-CDC] Conversion of tinyint(1) to bool is supported (#5105)
  • [Connector-V2] [Redis] Support redis to set expiration time (#4975)

Zeta(ST-Engine)

  • [Zeta] [CDC-base] Support schema evolution framework(DDL) (#5125)
  • [Zeta] [REST-API] Add REST API To Submit Job (#5107)
  • [Zeta] Checkpoint support hdfs ha mode (#4942)
  • [Zeta] The expiration time of a historical Job can be config (#5180)
  • [Zeta] Support history service record job execute error (#5114)
  • [Zeta] Configuration files support user variable replacement (#4969)

Transform

  • [Transform-V2] [SQL] Support ‘select *’ and ‘like’ clause for SQL Transform plugin (#4991)

CI

  • [CI] Update action config to support run CI on fork repo (#5065)
  • [E2E] [FtpFile] Add ftp file e2e test case (#4647)

Docs

  • [Connector-V2][Clickhouse] Reconstruct the clickhouse connector doc (#5085)
  • [Connector-V2][Doris] Reconstruct the Doris connector document (#4903)
  • [Connector-V2][HDFS] Refactor connector-v2 docs using unified format HDFS (#4871)
  • [Connector-V2][Http] Reconstruct the Http connector document (#4962)
  • [Connector-V2][Hudi] Reconstruct the Hudi connector document (#4905)
  • [Connector-V2][Jdbc] Change the line boundary store value type to BigDecimal (#4900)
  • [Connector-V2][Jdbc-DB2、Jdbc-Vertica] Improve DB2 Source Vertica Source & DB2 Sink Vertica Sink document (#5102)
  • [Connector-V2][Jdbc-Oracle] Add JDBC Oracle Documentation (#5239)
  • [Connector-V2][Kafka] Reconstruct the kafka connector document (#4778)
  • [Connector-V2][MyHours] Reconstruct the MyHours connector document (#5129)
  • [Connector-V2][PostgreSQL] Refactor connector-v2 docs using unified format PostgreSQL #4590 (#4757)
  • [Connector-V2][S3File] Improve S3File Source & S3File Sink document (#5101)
  • [Connector-V2][SelectDB-Cloud] Reconstruct the SelectDB-Cloud connector document (#5130)
  • [Connector-V2][StarRocks] Reconstruct the StarRocks connector document (#5132)
  • [Connector-V2][StarRocks] StarRocks nodeUrls property name fix (#4951)
  • [README] Remove useless github workflow, and adjust description of ‘engineering structure’ (#4305)
  • [Zeta] Add savepoint doc (#5081)
  • Update bug-report.yml (#5160)
  • Add Value types in Java to Schema feature (#5087)
  • Documentation and partial word optimization (#4936)
  • Change repository name from incubator-seatunnel to seatunnel (#4868)
  • Update .asf.yaml (#5242)
  • Update the Readme (#4968)
  • Set Up with Kubernetes, dockerfile document error in constructing docker image (#5022)
  • Fix Dockerfile and seatunnel-flink.yaml in Set Up with Kubernetes (#4793)
  • Change the transform website url (#4954)
  • Update quick-start-seatunnel-engine.md (#4943)
  • Remove incubating keyword in document (#5257)

Source apache/seatunnel · 2.3.3 · 2023-08-22 · upstream release

Mirrored verbatim for indexing. All rights belong to the upstream project.

Release notes are published by the upstream project and mirrored here for indexing. All rights belong to their authors.