NextStack
EN 中

seatunnel 2.3.9

patch PublishedJan 19, 2025 Sourceapache/seatunnel

seatunnel 2.3.9

2.3.9 Release Note

Bug fix

  • [Bug][Avro] Fix avro miss convert short type to int (#8459)
  • [Bug][CI] Fix all-connectors and update-modules run simultaneously (#8343)
  • [Bug] [connector-file] When the data source field is less than the target (Hive) field，it will throw null pointer exception#8150 (#8200)
  • [Bug][connectors-v2] fix mongodb bson convert exception (#8044)
  • [Bug] [connectors-v2] The Hadoop Source/Sink fails with Unable to find valid Kerberos Ticket. (#7809)
  • [Bug][Connector-v2] MongoDB CDC Set SeatunnelRow’s tableId (#7935)
  • [Bug][Core] Local mode cancel running JobStatusRunner (#7943)
  • [bugfix][core] Fix the problem of incorrect association between metrics and nodes (#7799)
  • [BUG] Fixed avro format support for storing null (#8424)
  • [BugFix][Excel] Fix read formulas/number cell value of excel (#8316)
  • [Bug][Kafka] kafka reads repeatedly (#8465)
  • [Fix][API] Fix column length can not be long (#8039)
  • [Fix][CI] Fix ci dependency check error (#8262)
  • [Fix][CI] Fix CI error caused by set classloader-cache-mode to true (#7940)
  • [Fix][CI] Fix schedule test be canceled (#8024)
  • [Fix][CI] time-consuming module optimization (#8052)
  • [Fix][Connecotr-V2] Fix clickhouse sink does not support composite primary key (#8021)
  • [Fix][Connector-starrocks] Fix drop column bug for starrocks (#8216)
  • [Fix][Connector][TDEngine] TDEngine support NCHAR type (#8411)
  • [fix][connector-v2][connector-assert] Optimize Assert Sink verification method (#8356)
  • [Fix][Connector-V2][connector-file-base-hadoop] Fixed HdfsFile source load the krb5_path configuration (#7870)
  • [Fix][Connector-V2] Fix AbstractSingleSplitReader lock useless when do checkpoint (#7764)
  • [Fix][Connector-V2] Fix cdc use default value when value is null (#7950)
  • [Fix][Connector-V2] Fixed clickhouse connectors cannot stop under multiple parallelism (#7921)
  • [Fix][Connector-V2] Fix file binary format sync convert directory to file (#7942)
  • [Fix][Connector-V2] Fix hana type loss of precision (#7912)
  • [Fix][Connector-V2] Fix hive krb5 path not work (#8228)
  • [Fix][Connector-V2] Fix kafka format_error_handle_way not work (#7838)
  • [Fix][connector-v2]Fix Paimon table connector Error log information. (#7873)
  • [Fix][Connector-V2] Fix When reading Excel data, string and date type conversion errors (#7796)
  • [Fix][Connector-V2][FTP] Fix FTP connector connection_mode is not effective (#7865)
  • [Fix][Connector-V2]Oceanbase vector database is added as the source server (#7832)
  • [fix][docker] update Dockerfile (#8421)
  • [Fix][Docker] Workflow “push-docker” failed run with error “No space left on device” (#8339)
  • [Fix][Doris] Fix catalog not closed (#8415)
  • [fix] fix get job metrics get NPE (#8093)
  • [Fix][Flink] Fix NPE when get null row from upstream transform (#8469)
  • [Fix] [Flink/Spark] Fix transform stream choose wrong when not configure source_table_name (#7907)
  • [Fix][Kafka] Fix in kafka streaming mode can not read incremental data (#7871)
  • [fix][log]Fix log returns interface issue and adds test code (#8387)
  • [fix] move docker build plugin into dist module (#8338)
  • [Fix] Set all snappy dependency use one version (#8423)
  • fix transforms-v2 nlpmodel typo (#8430)
  • [Fix][Zeta] Fix apply resource again for another pipeline when restore one pipeline (#7965)
  • [Fix][Zeta] Fix job client print error msg even job finished (#7981)
  • [Fix][Zeta] Fix worker node metrics acquisition (#7862)
  • [Fix][Zeta] Introduce SeaTunnel CompletableFuture to prevent ForkJoinPool thread shortage (#8445)
  • [Fix][Zeta] Remove duplicate LIFECYCLE_WRITER_CLOSE events (#7984)
  • [Fix][Zeta] Set AsyncLogger.ThreadNameStrategy=UNCACHED to avoid thread name cache (#8215)
  • [Hotfix][CDC] Fix occasional database connection leak when read snapshot split (#7918)
  • [Hotfix][Config] Fix configuration key sort disorder (#7893)
  • [hotfix] [connector-cdc-oracle ] support read partition table (#8265)
  • [hotfix][connector-elasticsearch-sink] Convert index to lowercase (#8429)
  • [hotfix][connector-v2][hive]Fix hive3 e2e dependencies (#8045)
  • [Hotfix][Connector-V2][kafka] fix kafka sink config exactly-once exception (#7857)
  • [Hotfix][Core][Flink] SeaTunnel flink engine support application mode on yarn (#7762)
  • [Hotfix][Docs]Correct errors in the FAQ doc (#8048)
  • [Hotfix][E2E] Fix OracleCDCIT compile error (#8023)
  • [HotFix]Fix Dead Link In ObsFile.md (#8251)
  • [hotfix] fix exceptions caused by operator priority in connector-clickhouse when using sharding_key (#8162)
  • [Hotfix] Fix redis sink NPE (#8171)
  • [hotfix][gbase] Fix gbase driver link (#8110)
  • [Hotfix][Transform][Sql] function concat_ws handle array type， avoid write [Ljava.lang.String. (#8369)
  • [Hotfix][Transforms-v2] DynamicCompile Plugin compatibility fix (#8057)
  • [Hotfix][Zeta] Fix the dependency conflict between the guava in hadoop-aws and hive-exec (#7986)
  • [Hotfix][Zeta][Log] Fix zeta parallelStream log trace for job (#8456)

Docs

  • [Chore] delete chinese desc in code (#8306)
  • [Core][Flink] Resolve streaming source high CPU usage (#8354)
  • [doc][connector][hive] Add hive chinese doc (#8169)
  • [Doc][Format] Add protobuf doc (#7989)
  • [Doc][hive] Rename result_table_name to plugin_output in hive doc (#8203)
  • [Docs] [Elasticsearch] fix the doc about index name replacement (#8392)
  • [Doc][Transform-v2] Update doc for Array function (#8419)
  • [Fix][Doc] Correct hive-jdbc config useKerberos to use_kerberos (#7896)
  • [Fix][Doc] fix backup-count config (#8147)
  • [Fix][Doc] fix doc tip style and add document format specification (#8378)
  • [Fix][Doc] Fixed unreachable links (#8131)
  • [Fix][Doc] Fixing errors in the documentation and adding information (#8353)
  • [Fix][Doc] Fix jetty doc error (#7883)
  • [Fix][Doc] Fix key description error in redis zh doc (#7993)
  • [Fix][Doc] Fix klaviyo doc 404 link (#7969)
  • [Fix][Doc] Fix outdated plugin path in oracle documentation (#8403)
  • [Fix][Doc] Fix rest-api missing content (#8384)
  • [Fix][Doc] Fix wrong link in about/index.html (#8486)
  • [Fix][Doc] Fix zh doc build error (#7882)
  • [Fix][Doc] Modify error name SEATNUNNEL to SEATUNNEL (#8257)
  • [Fix][Doc] Modify the example of enable-dynamic-port in the documentation. (#8331)
  • [Fix][docs] fix wrong config in setup.md (#8483)
  • [Fix][Doc] Update Elasticsearch Sink and Oracle CDC Source Sample Configuration (#8425)
  • [Improve][doc] Add doc name check (#8140)
  • [Improve][Doc] Add enable_header_write parameter in File doc (#8397)
  • [Improve][Doc] Add file_filter_pattern example to doc (#7922)
  • [Improve][Doc] Add some connector icons (#7944)
  • [Improve][Doc] Add the way of configure jvm options in client (#8091)
  • [Improve][Doc] Correcting typo errors in logging (#7874)
  • [Improve][Doc] Correction to docker doc for get container ip (#7801)
  • [Improve][Doc] Fix chinese doc miss grafana dashborad config (#7955)
  • [Improve][Doc] Improve the connector-v2 develop doc (#8190)
  • [Improve][Doc] Move seatunnel engine deployment doc (#8239)
  • [Improve][Docs] Add driver reference to doris/starrocks docs (#8414)
  • [Improve][Docs] Edit the description about startup.mode for Opengauss-CDC (#7815)
  • [Improve][Doc] Update connector v2 contribute guide (#8375)
  • [Improve][Doc] Updated telemetry doc (#7827)

Feature

  • [Feature][API] Support timestamp with timezone offset (#8367)
  • [Feature][CDC] Add ‘schema-changes.enabled’ options (#8252)
  • [Feature][CDC] Add ‘schema-changes.enabled’ options (#8285)
  • [Feature]Check Chinese comments in the code (#8319)
  • [Feature][ClickhouseFile] Support add publicKey to identity (#8351)
  • [Feature][Clickhouse] Support sink savemode (#8086)
  • [Feature][connector-elasticsearch] elasticsearch support nested type (#8462)
  • [Feature] [connector-milvus] update milvus connector to support dynamic schema, failed retry, etc. (#7885)
  • [Feature] [Connector-Redis] Redis connector support delete data (#7994)
  • [Feature][Connectors] LocalFile Support reading gz (#8025)
  • [Feature][Connector-V2] Add prometheus source and sink (#7265)
  • [Feature][Connector-V2[Doris]Support sink ddl (#8250)
  • [Feature][Connector-V2]Ftp file source support multiple table (#7795)
  • [Feature][Connector-V2]Jdbc chunk split add snapshotSplitColumn config #7794 (#7840)
  • [Feature][Connector-V2] Jdbc DB2 support upsert SQL (#7879)
  • [Feature][Connector-V2] Piamon Sink supports changelog-procuder is lookup and full-compaction mode (#7834)
  • [Feature][Connector-V2]Sftp file source support multiple table (#7824)
  • [Feature][Connector-V2] Starrocks implements multi table sink (#8467)
  • [Feature][Connector-V2] StarRocks-sink support schema evolution (#8082)
  • [Feature][Connector-V2]Support Doris Fe Node HA (#8311)
  • [Feature][Connector-V2] Support MaxCompute save mode (#8277)
  • [Feature][connector-v2]Support opengauss jdbc connnector using opengauss driver. (#7622)
  • [Feature][Connector-v2] Support S3 filesystem of paimon connector (#8036)
  • [Feature][Connector-v2] Support schema evolution for Oracle connector (#7908)
  • [Feature][Connector-v2] Support schema evolution for paimon sink (#8211)
  • [Feature][Connector-V2] Support the jdbc connector for highgo db (#8282)
  • [Feature][Connector-V2] Support use EasyExcel as read excel engine (#8064)
  • [Feature][Connector-V2] Support write cdc changelog event into hudi sink (#7845)
  • [Feature][Core] Rename result_table_name/source_table_name to plugin_input/plugin_output (#8072)
  • [Feature][Core] Support cdc task ddl restore for zeta (#7463)
  • [Feature][Core] Support read arrow data (#8137)
  • [Feature][Doc]Paimon Souce changelog update (#8187)
  • [Feature][Doris] Support multi-table source read (#7895)
  • [Feature][E2E] Add E2E test case time analysis (#8028)
  • [Feature][E2E] Add hive3 e2e test case (#8003)
  • [Feature][E2E][Kerberos] Support for Kerberos in e2e (#8108)
  • [Feature][Elasticsearch] Support sink ddl (#8412)
  • [Feature][Engine UI] partial pages of the engine (#7602)
  • [Feature][File] Support config null format for text file read (#8109)
  • [Feature][Iceberg] Support custom delete sql for sink savemode (#8094)
  • [Feature][Jdbc] Add Jdbc default dialect for all jdbc series database without dialect (#8132)
  • [Feature][JDBC source] pg support char types (#8420)
  • [Feature][Jdbc] Support sink ddl for postgresql (#8276)
  • [Feature][K8S deploy] add helm chart (#8291)
  • [Feature][Kafka] Add debezium_record_table_filter and fix error (#8391)
  • [Feature][Mongodb-CDC] Support multi-table read (#8029)
  • [Feature][MySQL-CDC] Support database/table wildcards scan read (#8323)
  • [Feature][Redis] Flush data when the time reaches checkpoint.interval (#8198)
  • [Feature][Redis] Flush data when the time reaches checkpoint.interval and update test case (#8308)
  • [Feature][Restapi] Allow metrics information to be associated to logical plan nodes (#7786)
  • [Feature] [rest-api] Support Rest Api to upload file and submit task (#8442)
  • [Feature][RestAPI] Support submit job with seatunnel style hocon format config (#8000)
  • [Feature][Rest api] Use jetty to replace Hazelcast native rest api (#7647)
  • [Feature][Transform-SQL]Support sql transform to generate UUID (#7881)
  • [Feature][Transform] Sql transform support array and split function (#8016)
  • [Feature][Transform] Support merge shading-tables to one table (#8360)
  • [Feature][Transform] Support rename table/column (#8170)
  • [Feature][transforms-v2] Support append only stream from cdc source (#7763)
  • [Feature][transform] transform support explode (#7928)
  • [Feature][Transform-v2] Add metadata transform (#7899)
  • [Feature][Transform-V2] Spark support transform with multi-table (#8340)
  • [Feature][Transform-V2] Support transform with multi-table (#7628)
  • [Feature][Zeta] Add COS support for checkpoint storage (#7931)
  • [Feature][Zeta]Add jar path precheck when job submit on master (#7976)
  • [Feature][Zeta][Core] Support rest api get logs (#7818)
  • [Feature] [Zeta] Optimize CoordinatorService ThreadPool Configuration to Prevent Potential OOM (#8241)
  • [Feature][Zeta] Submit job scheduling support pending (#7693)
  • [Feature][Zeta] Support delete logs regularly (#7787)
  • [Imporve][ClickhouseFile] Directly connect to each shard node to obtain the corresponding path (#8449)
  • [Imporve][ClickhouseFile] Improve clickhousefile generation parameter configuration (#8293)
  • [Improve] Allow nullable for SeaTunnelRowType convert to Column (#8098)
  • [Improve][api] Improve inheritance relationship of AlterTableColumnEvent (#7825)
  • [Improve][API] Move AlterTableNameEvent parent (#7869)
  • [improve][API] support metrics interface in rest v2 (#8312)
  • [Improve][Api] Supports simultaneous config of conditions and optional options (#7788)
  • [Improve][API] Unified tables_configs and table_list (#8100)
  • [Improve][CI]add code specification check (#8209)
  • [Improve][CI] Fix module runs CI repeatedly (#8284)
  • [Improve][CI] Improve oracle schema evolution e2e case (#8224)
  • [Improve][CI] Optimizing CI time cost (#7997)
  • [Improve][CI]skip ui module, improve module dependent (#8225)
  • [Improve][ClickhouseFile] Added attach sql log for better debugging (#8315)
  • [Improve][ClickhouseFile] ClickhouseFile Connector’s rsync transmission supports specifying users (#8236)
  • [Improve][ClickhouseFile] Clickhouse supports option configuration when connecting to shard nodes (#8297)
  • [Improve][ClickhouseFile Connector] Unified specifying clickhouse file generation path (#8302)
  • [Improve][ClickhouseFile] Improve rsync log output (#8332)
  • [Improve][connector-e2e][Hive]Optimize script and method names (#8249)
  • [Improve][Connector-file-base] Improved file allocation algorithm for subtasks. (#8453)
  • [Improve][Connector][Hive] skip temporary hidden directories (#8402)
  • [Improve][Connector-V2] Add doris/starrocks create table with comment (#7847)
  • [Improve][Connector-V2] Add pre-check for table enable cdc (#8152)
  • [Improve][Connector-V2] Add pre-check starrocks version before exeucte alter table field name (#8237)
  • [Improve][Connector-V2] Add some debug log when create dir in (S)FTP (#8286)
  • [Improve][Connector-V2] Add table info when multi table sink error (#8163)
  • [Improve][Connector-V2] Change File Read/WriteStrategy setSeaTunnelRowTypeInfo to setCatalogTable (#7829)
  • [Improve][Connector-V2] Fix SqlServer cdc memory leak (#8083)
  • [Improve][Connector-V2][FTP] delete duplicated code (#7915)
  • [Improve][Connector-V2] Improve jdbc merge table from path and query when type is decimal (#7917)
  • [Improve][Connector-V2] Improve schema evolution on column insert after for mysql-jdbc (#8017)
  • [improve][connector-v2][kafka-connector] Add e2e code for Kafka Kerberos authentication (#8159)
  • [Improve][Connector-V2] Optimized DefaultSaveModeHandlerconstruct method (#8179)
  • [Improve][Connector-V2] Redis support custom key and value (#7888)
  • [Improve][Connector-V2] Reduce the create times of iceberg sink writer (#8155)
  • [Improve][Connector-V2] Reduce the request times of iceberg load table (#8149)
  • [Improve][Connector-V2] RocketMQ Sink add message tag config (#7996)
  • [Improve][Connector-V2] Save data savemode operation when schema savemode created new table (#8178)
  • [Improve][Connector-v2] Support checkpoint in batch mode for paimon sink (#8333)
  • [Improve][Connector-V2] Support read comment when jdbc dialect without catalog (#8196)
  • [Improve][Connector-V2] The interface supports jdbc respects the target database field type (#8031)
  • [Improve][Connector-v2] Use checkpointId as the commit’s identifier instead of the hash for streaming write of paimon sink (#7835)
  • [Improve][Connector-v2] Use regex to match filedName placeholders in jdbc sink (#8222)
  • [Improve][Core] Add protobuf transform test case (#7914)
  • [Improve][Core] Add test case to check deprecated result_table_name and source_table_name (#8153)
  • [Improve][Core] Refactor common options of column/row (#7911)
  • [Improve][dist]add shade check rule (#8136)
  • [Improve][dist]add UT class name check (#8182)
  • [Improve][E2E] improve doris initialize (#8446)
  • [Improve][E2E] Improve Druid E2E Case (#8077)
  • [Improve][E2E] improve kafka e2e (#8295)
  • [Improve][E2E] improve oracle e2e (#8292)
  • [Improve][E2E] modify the method of obtaining JobId (#7880)
  • [Improve][Elasticsearch] Truncate the exception message body for request errors (#8263)
  • [Improve][Example] Add log config for job mdc (#7876)
  • [Improve][Example] Improve zeta local/cluster example (#7877)
  • [Improve][Examples]Enable Flink Example WebUI (#8355)
  • [Improve][Excel] Support read blank string & auto type-cast (#8111)
  • [Improve][Fake] Improve memory usage when split size is large (#7821)
  • [Improve][Formats] Support not primary-key table for debezium format (#7836)
  • [Improve][Format] Support complex data type parse of debezium_json (#8330)
  • [Improve][Iceberg] Filter catalog table primaryKey is empty (#8413)
  • [Improve][Iceberg] Support table comment for catalog (#7936)
  • [Improve] Improve AlterTableSchemaEventHandler (#8050)
  • [improve] improve hazelcast joiner, lite node can’t be election as master (#8261)
  • [Improve][Jdbc] Improve ddl write validate (#8158)
  • [Improve][Jdbc] Optimize index name conflicts when create table for postgresql (#7875)
  • [Improve][Jdbc] Refactor ddl change (#8134)
  • [Improve][Jdbc] Support nvarchar in dm (#8270)
  • [Improve][Jdbc] Support postgresql inet type. (#7820)
  • [Improve][Jdbc] Support save mode for the sink of jdbc-dm (#7814)
  • [Improve][Kafka] Support custom topic for debezium compatible format (#8145)
  • [Improve][POM] Bump the version of maven-shade-plugin to 3.4.1 (#8067)
  • [Improve][PostgreSQL CDC]-PostgresSourceOptions description error (#7813)
  • [Improve] Remove unused code (#8167)
  • [Improve][Rest api]add verifyHtmlResponse test case (#7919)
  • [Improve][rest api] improve rest-api doc (#7804)
  • [Improve][RestApi] Unify the code logic of REST API V1 and V2 (#7920)
  • [Improve][RestApi] Update context-path to / as default path (#7800)
  • [Improve][Sls] Add sls sink connector、e2e、doc (#7830)
  • [Improve][Test] Enable some spark test case on jdk 11 (#8238)
  • [Improve][Transform] Auto mark column length with jsonpath transform (#7636)
  • [Improve][Transform] gz support excel (#8181)
  • [Improve][Transform] Optimize transform structure (#8071)
  • [Improve][Transform] Rename sql transform table name from ‘fake’ to ‘dual’ (#8298)
  • [Improve][transform-v2] Support dynamic types for array function (#8139)
  • [Improve] update clickhouse connector, use factory to create source/sink (#7946)
  • [Improve] Update snapshot version to 2.3.9 (#7841)
  • [Improve] Update snapshot version to 2.3.9 (#7849)
  • [Improve][Zeta] Fixed issues related to log deletion during ci (#8076)
  • [Improve][Zeta] Jetty server enable dynamic port (#7903)
  • [Improve][Zeta] Make sure the local mode CompletableFuture behavior same as server mode (#8476)
  • [Improve][Zeta] Make the default value of classloader-cache-mode is true (#7938)
  • [Improve][Zeta] Remove default jvm memory option (#8068)
  • [Improve][Zeta] Split classloader in config parse phase (#8193)
  • [Improvment][Build] Provide option to disable source jar creation. (#7680)
  • [Zeta][Log] Fix trace parallel stream (#8206)

Source apache/seatunnel · 2.3.9 · 2025-01-19 · upstream release

Mirrored verbatim for indexing. All rights belong to the upstream project.

Release notes are published by the upstream project and mirrored here for indexing. All rights belong to their authors.