[#3515] feat(flink-connector): Support flink iceberg catalog #5914

sunxiaojian · 2024-12-19T07:05:30Z

What changes were proposed in this pull request?

Support flink iceberg catalog

Why are the changes needed?

Fix: #3515

Does this PR introduce any user-facing change?

no

How was this patch tested?

FlinkIcebergCatalogIT
FlinkIcebergHiveCatalogIT

sunxiaojian · 2024-12-19T09:57:27Z

@FANNG1 @coolderli PTAL

FANNG1 · 2024-12-19T11:05:45Z

Cool!, I'll review this PR, but may need some time, :)

sunxiaojian · 2024-12-19T11:28:37Z

Cool!, I'll review this PR, but may need some time, :)

ok, thanks

tengqm · 2024-12-22T04:24:42Z

docs/flink-connector/flink-catalog-iceberg.md

+The Apache Gravitino Flink connector offers the capability to read and write Iceberg tables, with the metadata managed by the Gravitino server. To enable the use of the Iceberg catalog within the Flink connector, you must download the Iceberg Flink runtime JAR to the Flink classpath.
+


Suggested change

The Apache Gravitino Flink connector offers the capability to read and write Iceberg tables, with the metadata managed by the Gravitino server. To enable the use of the Iceberg catalog within the Flink connector, you must download the Iceberg Flink runtime JAR to the Flink classpath.

The Apache Gravitino Flink connector can be used to read and write Iceberg tables, with the metadata managed by the Gravitino server.

To enable the Flink connector, you must download the Iceberg Flink runtime JAR and place it in the Flink classpath.

tengqm · 2024-12-22T04:25:13Z

docs/flink-connector/flink-catalog-iceberg.md

+- `INSERT INTO & OVERWRITE`
+- `SELECT`
+
+#### Not supported operations:


Suggested change

#### Not supported operations:

#### Operations Not Supported:

tengqm · 2024-12-22T04:26:49Z

docs/flink-connector/flink-catalog-iceberg.md

+
+## Catalog properties
+
+Gravitino Flink connector will transform the following property names defined in catalog properties to Flink Iceberg connector configuration.


Suggested change

Gravitino Flink connector will transform the following property names defined in catalog properties to Flink Iceberg connector configuration.

The Gravitino Flink connector transforms the following properties in a catalog to Flink connector configuration.

tengqm · 2024-12-22T04:28:02Z

docs/flink-connector/flink-catalog-iceberg.md

+
+### S3
+
+You need to add s3 secret to the Flink configuration using `s3.access-key-id` and `s3.secret-access-key`. Additionally, download the [Iceberg AWS bundle](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-aws-bundle) and place it in the classpath of Flink.


Suggested change

You need to add s3 secret to the Flink configuration using `s3.access-key-id` and `s3.secret-access-key`. Additionally, download the [Iceberg AWS bundle](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-aws-bundle) and place it in the classpath of Flink.

You need to add an S3 secret to the Flink configuration using `s3.access-key-id` and `s3.secret-access-key`.

Additionally, you need to download the [Iceberg AWS bundle](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-aws-bundle)

and place it in the Flink classpath.

tengqm · 2024-12-22T04:28:48Z

docs/flink-connector/flink-catalog-iceberg.md

+
+### OSS
+
+You need to add OSS secret key to the Flink configuration using `client.access-key-id` and `client.access-key-secret`. Additionally, download the [Aliyun OSS SDK](https://gosspublic.alicdn.com/sdks/java/aliyun_java_sdk_3.10.2.zip) and copy `aliyun-sdk-oss-3.10.2.jar`, `hamcrest-core-1.1.jar`, `jdom2-2.0.6.jar` in the classpath of Flink.


Suggested change

You need to add OSS secret key to the Flink configuration using `client.access-key-id` and `client.access-key-secret`. Additionally, download the [Aliyun OSS SDK](https://gosspublic.alicdn.com/sdks/java/aliyun_java_sdk_3.10.2.zip) and copy `aliyun-sdk-oss-3.10.2.jar`, `hamcrest-core-1.1.jar`, `jdom2-2.0.6.jar` in the classpath of Flink.

You need to add an OSS secret key to the Flink configuration using `client.access-key-id` and `client.access-key-secret`.

Additionally, you need download the [Aliyun OSS SDK](https://gosspublic.alicdn.com/sdks/java/aliyun_java_sdk_3.10.2.zip),

and copy `aliyun-sdk-oss-3.10.2.jar`, `hamcrest-core-1.1.jar`, `jdom2-2.0.6.jar` to the Flink classpath.

tengqm · 2024-12-22T04:29:49Z

docs/flink-connector/flink-catalog-iceberg.md

+
+### GCS
+
+No extra configuration is needed. Please make sure the credential file is accessible by Flink, like using `export GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json`, and download [Iceberg GCP bundle](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-gcp-bundle) and place it to the classpath of Flink.


Suggested change

No extra configuration is needed. Please make sure the credential file is accessible by Flink, like using `export GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json`, and download [Iceberg GCP bundle](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-gcp-bundle) and place it to the classpath of Flink.

No extra configuration is needed. Please make sure the credential file is accessible by Flink.

For example, `export GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json`.

You need to download [Iceberg GCP bundle](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-gcp-bundle) and place it in the Flink classpath.

...c/main/java/org/apache/gravitino/flink/connector/iceberg/GravitinoIcebergCatalogFactory.java

...k/src/main/java/org/apache/gravitino/flink/connector/iceberg/IcebergPropertiesConstants.java

...k/src/main/java/org/apache/gravitino/flink/connector/iceberg/IcebergPropertiesConverter.java

docs/flink-connector/flink-catalog-iceberg.md

FANNG1 · 2024-12-26T13:57:17Z

Hi, @sunxiaojian , Sorry for the delay, I'm working on the issues to release 0.8, may doesn't have enough time to review this PR these days.

xunliu · 2024-12-28T06:49:05Z

hi @sunxiaojian Thank you for your contributions.
Can you send an email to me([email protected]), I have something need to discuss with you.

sunxiaojian · 2024-12-29T14:32:38Z

Hi, @sunxiaojian , Sorry for the delay, I'm working on the issues to release 0.8, may doesn't have enough time to review this PR these days.

@FANNG1 ok , I will also handle the comments above as soon as possible

sunxiaojian · 2024-12-29T14:40:46Z

hi @sunxiaojian Thank you for your contributions. Can you send an email to me([email protected]), I have something need to discuss with you.

@xunliu The email has been sent

docs/flink-connector/flink-catalog-iceberg.md

...k/src/main/java/org/apache/gravitino/flink/connector/iceberg/IcebergPropertiesConverter.java

...org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergHiveCatalogIT.java

...k/src/main/java/org/apache/gravitino/flink/connector/iceberg/IcebergPropertiesConstants.java

...flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/FlinkCommonIT.java

FANNG1 · 2025-01-14T03:39:18Z

LGTM except for minor comments, could you fix it?

sunxiaojian · 2025-01-14T15:45:22Z

LGTM except for minor comments, could you fix it?

@FANNG1 Fixed all，PTAL again, thanks

docs/flink-connector/flink-catalog-iceberg.md

FANNG1 · 2025-01-15T09:50:08Z

@tengqm @coolderli any other comments?

sunxiaojian · 2025-01-15T12:13:57Z

After fixed conflict. This error seems to indicate that the Hive image cannot be accessed

FANNG1 · 2025-01-15T12:17:27Z

After fixed conflict. This error seems to indicate that the Hive image cannot be accessed

maybe incidental error, trigger rerun

FANNG1 · 2025-01-16T01:51:09Z

After fixed conflict. This error seems to indicate that the Hive image cannot be accessed

maybe incidental error, trigger rerun

odd problem :(

sunxiaojian · 2025-01-16T03:56:18Z

@FANNG1 fixed

### What changes were proposed in this pull request? Support flink iceberg catalog ### Why are the changes needed? Fix: [#3515](#3515) ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? FlinkIcebergCatalogIT FlinkIcebergHiveCatalogIT

FANNG1 · 2025-01-16T07:57:52Z

@sunxiaojian, merged to main and cherry pick to branch-0.8, thanks for your contribution!

sunxiaojian changed the title ~~[#3515]feat(flink-connector)Support flink iceberg catalog~~ [#3515] feat(flink-connector): Support flink iceberg catalog Dec 19, 2024

sunxiaojian force-pushed the support-flink-iceberg-catalog branch 4 times, most recently from 24cd6b8 to ef294ba Compare December 19, 2024 07:37

tengqm reviewed Dec 22, 2024

View reviewed changes

coolderli reviewed Dec 23, 2024

View reviewed changes

...k/src/main/java/org/apache/gravitino/flink/connector/iceberg/IcebergPropertiesConverter.java Outdated Show resolved Hide resolved

FANNG1 reviewed Dec 26, 2024

View reviewed changes

docs/flink-connector/flink-catalog-iceberg.md Outdated Show resolved Hide resolved

sunxiaojian force-pushed the support-flink-iceberg-catalog branch 4 times, most recently from fc59502 to 5acadb8 Compare January 4, 2025 03:33

FANNG1 reviewed Jan 10, 2025

View reviewed changes

sunxiaojian force-pushed the support-flink-iceberg-catalog branch 2 times, most recently from 22fadd3 to 9ec6317 Compare January 13, 2025 16:29

FANNG1 reviewed Jan 14, 2025

View reviewed changes

...k/src/main/java/org/apache/gravitino/flink/connector/iceberg/IcebergPropertiesConstants.java Outdated Show resolved Hide resolved

FANNG1 reviewed Jan 14, 2025

View reviewed changes

...flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/FlinkCommonIT.java Outdated Show resolved Hide resolved

sunxiaojian force-pushed the support-flink-iceberg-catalog branch 2 times, most recently from 92b8cd2 to 26d9ead Compare January 14, 2025 15:44

FANNG1 reviewed Jan 15, 2025

View reviewed changes

docs/flink-connector/flink-catalog-iceberg.md Show resolved Hide resolved

FANNG1 assigned sunxiaojian Jan 15, 2025

FANNG1 added the branch-0.8 Automatically cherry-pick commit to branch-0.8 label Jan 15, 2025

Support flink iceberg catalog apache#3515

f46ea4b

sunxiaojian force-pushed the support-flink-iceberg-catalog branch 2 times, most recently from f2f1891 to 97f3ff9 Compare January 15, 2025 11:41

sunxiaojian force-pushed the support-flink-iceberg-catalog branch 2 times, most recently from 989161b to 76138a5 Compare January 15, 2025 16:53

fixed

f043f14

sunxiaojian force-pushed the support-flink-iceberg-catalog branch from 76138a5 to f043f14 Compare January 16, 2025 03:29

FANNG1 approved these changes Jan 16, 2025

View reviewed changes

FANNG1 merged commit fd26b56 into apache:main Jan 16, 2025
25 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#3515] feat(flink-connector): Support flink iceberg catalog #5914

[#3515] feat(flink-connector): Support flink iceberg catalog #5914

sunxiaojian commented Dec 19, 2024

sunxiaojian commented Dec 19, 2024

FANNG1 commented Dec 19, 2024

sunxiaojian commented Dec 19, 2024

tengqm Dec 22, 2024

tengqm Dec 22, 2024

tengqm Dec 22, 2024

tengqm Dec 22, 2024

tengqm Dec 22, 2024

tengqm Dec 22, 2024

FANNG1 commented Dec 26, 2024

xunliu commented Dec 28, 2024

sunxiaojian commented Dec 29, 2024

sunxiaojian commented Dec 29, 2024 •

edited

Loading

FANNG1 commented Jan 14, 2025

sunxiaojian commented Jan 14, 2025

FANNG1 commented Jan 15, 2025

sunxiaojian commented Jan 15, 2025 •

edited

Loading

FANNG1 commented Jan 15, 2025 •

edited

Loading

FANNG1 commented Jan 16, 2025

sunxiaojian commented Jan 16, 2025

FANNG1 commented Jan 16, 2025

		The Apache Gravitino Flink connector offers the capability to read and write Iceberg tables, with the metadata managed by the Gravitino server. To enable the use of the Iceberg catalog within the Flink connector, you must download the Iceberg Flink runtime JAR to the Flink classpath.

-The Apache Gravitino Flink connector offers the capability to read and write Iceberg tables, with the metadata managed by the Gravitino server. To enable the use of the Iceberg catalog within the Flink connector, you must download the Iceberg Flink runtime JAR to the Flink classpath.
+The Apache Gravitino Flink connector can be used to read and write Iceberg tables, with the metadata managed by the Gravitino server.
+To enable the Flink connector, you must download the Iceberg Flink runtime JAR and place it in the Flink classpath.

	#### Not supported operations:
	#### Operations Not Supported:


		## Catalog properties

		Gravitino Flink connector will transform the following property names defined in catalog properties to Flink Iceberg connector configuration.

	Gravitino Flink connector will transform the following property names defined in catalog properties to Flink Iceberg connector configuration.
	The Gravitino Flink connector transforms the following properties in a catalog to Flink connector configuration.


		### S3

		You need to add s3 secret to the Flink configuration using `s3.access-key-id` and `s3.secret-access-key`. Additionally, download the [Iceberg AWS bundle](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-aws-bundle) and place it in the classpath of Flink.


		### OSS

		You need to add OSS secret key to the Flink configuration using `client.access-key-id` and `client.access-key-secret`. Additionally, download the [Aliyun OSS SDK](https://gosspublic.alicdn.com/sdks/java/aliyun_java_sdk_3.10.2.zip) and copy `aliyun-sdk-oss-3.10.2.jar`, `hamcrest-core-1.1.jar`, `jdom2-2.0.6.jar` in the classpath of Flink.

-You need to add OSS secret key to the Flink configuration using `client.access-key-id` and `client.access-key-secret`. Additionally, download the [Aliyun OSS SDK](https://gosspublic.alicdn.com/sdks/java/aliyun_java_sdk_3.10.2.zip) and copy `aliyun-sdk-oss-3.10.2.jar`, `hamcrest-core-1.1.jar`, `jdom2-2.0.6.jar` in the classpath of Flink.
+You need to add an OSS secret key to the Flink configuration using `client.access-key-id` and `client.access-key-secret`.
+Additionally, you need download the [Aliyun OSS SDK](https://gosspublic.alicdn.com/sdks/java/aliyun_java_sdk_3.10.2.zip),
+and copy `aliyun-sdk-oss-3.10.2.jar`, `hamcrest-core-1.1.jar`, `jdom2-2.0.6.jar` to the Flink classpath.


		### GCS

		No extra configuration is needed. Please make sure the credential file is accessible by Flink, like using `export GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json`, and download [Iceberg GCP bundle](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-gcp-bundle) and place it to the classpath of Flink.

-No extra configuration is needed. Please make sure the credential file is accessible by Flink, like using `export GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json`, and download [Iceberg GCP bundle](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-gcp-bundle) and place it to the classpath of Flink.
+No extra configuration is needed. Please make sure the credential file is accessible by Flink.
+For example, `export GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json`.
+You need to download [Iceberg GCP bundle](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-gcp-bundle) and place it in the Flink classpath.

[#3515] feat(flink-connector): Support flink iceberg catalog #5914

[#3515] feat(flink-connector): Support flink iceberg catalog #5914

Conversation

sunxiaojian commented Dec 19, 2024

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

sunxiaojian commented Dec 19, 2024

FANNG1 commented Dec 19, 2024

sunxiaojian commented Dec 19, 2024

tengqm Dec 22, 2024

Choose a reason for hiding this comment

tengqm Dec 22, 2024

Choose a reason for hiding this comment

tengqm Dec 22, 2024

Choose a reason for hiding this comment

tengqm Dec 22, 2024

Choose a reason for hiding this comment

tengqm Dec 22, 2024

Choose a reason for hiding this comment

tengqm Dec 22, 2024

Choose a reason for hiding this comment

FANNG1 commented Dec 26, 2024

xunliu commented Dec 28, 2024

sunxiaojian commented Dec 29, 2024

sunxiaojian commented Dec 29, 2024 • edited Loading

FANNG1 commented Jan 14, 2025

sunxiaojian commented Jan 14, 2025

FANNG1 commented Jan 15, 2025

sunxiaojian commented Jan 15, 2025 • edited Loading

FANNG1 commented Jan 15, 2025 • edited Loading

FANNG1 commented Jan 16, 2025

sunxiaojian commented Jan 16, 2025

FANNG1 commented Jan 16, 2025

sunxiaojian commented Dec 29, 2024 •

edited

Loading

sunxiaojian commented Jan 15, 2025 •

edited

Loading

FANNG1 commented Jan 15, 2025 •

edited

Loading