Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MINOR] docs: Polish Gravitino Flink connector document #6315

Merged
merged 3 commits into from
Jan 17, 2025
Merged

[MINOR] docs: Polish Gravitino Flink connector document #6315

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
886f9fa
polish flink document
FANNG1 Jan 17, 2025
eb457f7
polish flink document
FANNG1 Jan 17, 2025
3fd5cd3
polish flink document
FANNG1 Jan 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/flink-connector/flink-catalog-hive.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Supports most DDL and DML operations in Flink SQL, except such operations:
```sql

// Suppose hive_a is the Hive catalog name managed by Gravitino
USE hive_a;
USE CATALOG hive_a;

CREATE DATABASE IF NOT EXISTS mydatabase;
USE mydatabase;
Expand Down
21 changes: 11 additions & 10 deletions docs/flink-connector/flink-catalog-iceberg.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,12 @@ To enable the Flink connector, you must download the Iceberg Flink runtime JAR a
- `CREATE TABLE LIKE` clause

## SQL example

```sql

-- Suppose iceberg_a is the Iceberg catalog name managed by Gravitino

USE iceberg_a;
USE CATALOG iceberg_a;

CREATE DATABASE IF NOT EXISTS mydatabase;
USE mydatabase;
Expand All @@ -59,15 +60,15 @@ SELECT * FROM sample WHERE data = 'B';
The Gravitino Flink connector transforms the following properties in a catalog to Flink connector configuration.


| Gravitino catalog property name | Flink Iceberg connector configuration | Description | Since Version |
|---------------------------------|---------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------|
| `catalog-backend` | `catalog-type` | Catalog backend type, currently, only `Hive` Catalog is supported, `JDBC` and `Rest` in Continuous Validation | 0.8.0-incubating |
| `uri` | `uri` | Catalog backend URI | 0.8.0-incubating |
| `warehouse` | `warehouse` | Catalog backend warehouse | 0.8.0-incubating |
| `io-impl` | `io-impl` | The IO implementation for `FileIO` in Iceberg. | 0.8.0-incubating |
| `oss-endpoint` | `oss.endpoint` | The endpoint of Aliyun OSS service. | 0.8.0-incubating |
| `oss-access-key-id` | `client.access-key-id` | The static access key ID used to access OSS data. | 0.8.0-incubating |
| `oss-secret-access-key` | `client.access-key-secret` | The static secret access key used to access OSS data. | 0.8.0-incubating |
| Gravitino catalog property name | Flink Iceberg connector configuration | Description | Since Version |
|---------------------------------|---------------------------------------|---------------------------------------------------------------------------------------------------------------|------------------|
| `catalog-backend` | `catalog-type` | Catalog backend type, currently, only `Hive` Catalog is supported, `JDBC` and `Rest` in Continuous Validation | 0.8.0-incubating |
| `uri` | `uri` | Catalog backend URI | 0.8.0-incubating |
| `warehouse` | `warehouse` | Catalog backend warehouse | 0.8.0-incubating |
| `io-impl` | `io-impl` | The IO implementation for `FileIO` in Iceberg. | 0.8.0-incubating |
| `oss-endpoint` | `oss.endpoint` | The endpoint of Aliyun OSS service. | 0.8.0-incubating |
| `oss-access-key-id` | `client.access-key-id` | The static access key ID used to access OSS data. | 0.8.0-incubating |
| `oss-secret-access-key` | `client.access-key-secret` | The static secret access key used to access OSS data. | 0.8.0-incubating |

Gravitino catalog property names with the prefix `flink.bypass.` are passed to Flink iceberg connector. For example, using `flink.bypass.clients` to pass the `clients` to the Flink iceberg connector.

Expand Down
28 changes: 14 additions & 14 deletions docs/flink-connector/flink-catalog-paimon.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ license: "This software is licensed under the Apache License version 2."
---

This document provides a comprehensive guide on configuring and using Apache Gravitino Flink connector to access the Paimon catalog managed by the Gravitino server.

## Capabilities

### Supported Paimon Table Types
Expand All @@ -32,27 +33,26 @@ Supports most DDL and DML operations in Flink SQL, except such operations:

* Paimon 0.8

Higher version like 0.9 or above may also supported but have not been tested fully.
Higher version like 0.9 or above may also support but have not been tested fully.

## Getting Started

### Prerequisites

Place the following JAR files in the lib directory of your Flink installation:

* paimon-flink-1.18-0.8.2.jar

* gravitino-flink-connector-runtime-\${flinkMajorVersion}_$scalaVersion.jar
- `paimon-flink-1.18-${paimon-version}.jar`
- `gravitino-flink-connector-runtime-1.18_2.12-${gravitino-version}.jar`

### SQL Example

```sql

-- Suppose paimon_catalog is the Paimon catalog name managed by Gravitino
use catalog paimon_catalog;
USE CATALOG paimon_catalog;
-- Execute statement succeed.

show databases;
SHOW DATABASES;
-- +---------------------+
-- | database name |
-- +---------------------+
Expand All @@ -71,23 +71,23 @@ CREATE TABLE paimon_tabla_a (
bb BIGINT
);

show tables;
SHOW TABLES;
-- +----------------+
-- | table name |
-- +----------------+
-- | paimon_table_a |
-- +----------------+


select * from paimon_table_a;
SELECT * FROM paimon_table_a;
-- Empty set

insert into paimon_table_a(aa,bb) values(1,2);
INSERT INTO paimon_table_a(aa,bb) VALUES(1,2);
-- [INFO] Submitting SQL update statement to the cluster...
-- [INFO] SQL update statement has been successfully submitted to the cluster:
-- Job ID: 74c0c678124f7b452daf08c399d0fee2

select * from paimon_table_a;
SELECT * FROM paimon_table_a;
-- +----+----+
-- | aa | bb |
-- +----+----+
Expand All @@ -100,9 +100,9 @@ select * from paimon_table_a;

Gravitino Flink connector will transform below property names which are defined in catalog properties to Flink Paimon connector configuration.

| Gravitino catalog property name | Flink Paimon connector configuration | Description | Since Version |
|---------------------------------|----------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|
| `catalog-backend` | `metastore` | Catalog backend of Gravitino Paimon catalog. Supports `filesystem`. | 0.8.0-incubating |
| `warehouse` | `warehouse` | Warehouse directory of catalog. `file:///user/hive/warehouse-paimon/` for local fs, `hdfs://namespace/hdfs/path` for HDFS , `s3://{bucket-name}/path/` for S3 or `oss://{bucket-name}/path` for Aliyun OSS | 0.8.0-incubating |
| Gravitino catalog property name | Flink Paimon connector configuration | Description | Since Version |
|---------------------------------|--------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|
| `catalog-backend` | `metastore` | Catalog backend of Gravitino Paimon catalog. Supports `filesystem`. | 0.8.0-incubating |
| `warehouse` | `warehouse` | Warehouse directory of catalog. `file:///user/hive/warehouse-paimon/` for local fs, `hdfs://namespace/hdfs/path` for HDFS , `s3://{bucket-name}/path/` for S3 or `oss://{bucket-name}/path` for Aliyun OSS | 0.8.0-incubating |

Gravitino catalog property names with the prefix `flink.bypass.` are passed to Flink Paimon connector. For example, using `flink.bypass.clients` to pass the `clients` to the Flink Paimon connector.
2 changes: 2 additions & 0 deletions docs/flink-connector/flink-connector.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ This capability allows users to perform federation queries, accessing data from
## Capabilities

1. Supports [Hive catalog](flink-catalog-hive.md)
1. Supports [Iceberg catalog](flink-catalog-iceberg.md)
1. Supports [Paimon catalog](flink-catalog-paimon.md)
2. Supports most DDL and DML SQLs.

## Requirement
Expand Down
Loading