Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support the equivalent of session.createDataset(data, data.class) #308

Open
pberkland opened this issue Aug 17, 2016 · 1 comment
Open

Comments

@pberkland
Copy link
Collaborator

The spark programming style of session.createDataset(javaObject, javaObject.class) is much more common in Spark 2.0. We need some kind of JS equivalent.

We should be support something like createDataset(jsObject, { fieldName : "String", fieldName2:"Integer"});

This will probably require creating a JSON encoder which is not derived from the RowEncoder.

@pberkland
Copy link
Collaborator Author

Doing a real JSON encoder which is not derived from RowEncoder would be difficult, because encoder implementations generate code under the covers. This is done thru internal classes. Trying to do this for JSON would require good understanding of the Encoder internals.

Possible future versions of spark will make it easier to create custom encoders.

The 'createDatasetFromJSON' api seems to be working for current needs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant