Minio s3 support

[gesaleh]

I have installed gravitino to support mysql, postgres and other database i want to connect to s3 also using the icebeg connection but it looks like empty where i'm doing it wong

# THE CONFIGURATION FOR AUXILIARY SERVICE
# Auxiliary service names, separate by ','
gravitino.auxService.names = iceberg-rest
# Iceberg REST service classpath
gravitino.auxService.iceberg-rest.classpath = iceberg-rest-server/libs, iceberg-rest-server/conf
# Iceberg REST service host
gravitino.auxService.iceberg-rest.host = 0.0.0.0
# Iceberg REST service http port
gravitino.auxService.iceberg-rest.httpPort = 9001
gravitino.auxService.iceberg-rest.catalog-backend = jdbc
gravitino.auxService.iceberg-rest.uri = jdbc:mysql://mysql:3306/db
gravitino.auxService.iceberg-rest.warehouse = hdfs://hive:9000/user/iceberg/warehouse/
gravitino.auxService.iceberg-rest.jdbc.user = mysql
gravitino.auxService.iceberg-rest.jdbc.password = mysql
gravitino.auxService.iceberg-rest.jdbc-driver = com.mysql.cj.jdbc.Driver
gravitino.iceberg-rest.io-impl = org.apache.iceberg.aws.s3.S3FileIO
gravitino.iceberg-rest.credential-provider-type = s3-secret-key
gravitino.iceberg-rest.s3-access-key-id = xxxxxxxxxxxxxxx
gravitino.iceberg-rest.s3-secret-access-key = xxxxxxxxxxxxyyyyyyyyyyyzzzzzzzzzz
gravitino.iceberg-rest.s3-endpoint = minio:9000

how i can test and see schema of bucket of files in my s3 ?

[gesaleh]

to catalog s3 should i use iceberg or hadoop fileset ? both i see no metadata from the s3 local (minio)

[FANNG1]

I'm a little confused about your scenarios, could you provide more context?
noticed you add configuration for Gravitino Iceberg REST server which could show Iceberg data stored in S3, gravitino.auxService.iceberg-rest.warehouse should start with s3 schema like s3://xx and the iceberg table metadata should stored in your MySQL tables (Iceberg JDBC catalog backend). You could use spark, flink, trino to explore Iceberg data, please refer to https://gravitino.apache.org/docs/0.7.0-incubating/iceberg-rest-service
I you want to explore s3 raw files, suggest using Hadoop fileset which manages a mapping for the logical fileset to physical S3 location. Please refer to https://gravitino.apache.org/docs/0.7.0-incubating/how-to-use-gvfs.

[gesaleh]

thanks for the clarifications
so yes what i need is the fileset option as you mention to explore s3 raw files, suggest using Hadoop fileset which manages a mapping for the logical fileset to physical S3 location.
i did the configuration with th ui for haddop I'm using the playground repo, so hadoop is not part of it ,
if my understanding is correct, to have the fileset part of the metalake i need hadoop, and conifurations as mentions in the url,

in the ui of gravitino there is an option to create a new catalog type fileset (apache hadoop) i added the following option

location
s3-endpoint
s3-access-key-id 
s3-secret-access-key 
filesystem-providers

this doesn't connect to s3 ? fileset
I also saw that one jar file need to be downloaded is it the case also for the playground ?

i used this url https://gravitino.apache.org/docs/0.7.0-incubating/manage-fileset-metadata-using-gravitino/

# create a S3 catalog
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "catalog",
  "type": "FILESET",
  "comment": "comment",
  "provider": "hadoop",
  "properties": {
    "location": "s3a://bucket/root",
    "s3-access-key-id": "access_key",
    "s3-secret-access-key": "secret_key",
    "s3-endpoint": "http://s3.ap-northeast-1.amazonaws.com",
    "filesystem-providers": "s3"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs

[yuqi1129]

Yeah, if you encounter any issues, please provide detailed information like the stack trace here.

[gesaleh]

Hi @yuqi1129 is it possbile to have access and events without moving files around, let me explain if i have datalake that is based on s3 minio and other paths that are not in the s3 can gravitino, trino manage these too ? how to scrap the files using gvfs ? how to listen to add,update delete ?
what happened if a file is replaced or updated ?