Plan on adaption to Spark 2.x

[dongx-psu]

I suggest a total rewrite for this particular task. Open a new empty branch and adding things back into the structure.

• Find a way to hack through user defined types. For Spark 2.1, it is in a private scope because of this ticket:
  https://issues.apache.org/jira/browse/SPARK-14155.
  Maybe Encoder is the correct direction to go?
• Dive deep into DataSet abstraction, tailor our current design to this new abstraction.
• Reconstruct the whole project structure for better adoption to the current Spark SQL architecture.

Will keep update to this ticket.

[merlintang]

Any updating on this, for the scope issue, there are some discussion. [Spark Namespace]: Expanding Spark ML under Different Namespace?

Lots of people are running this issue to extend spark with personal functions.

one guy said "What I tend to do is keep my own code in its package and try to do as think a bridge over to it from the [private] scope. It's also important to name things obviously, say, org.apache.spark.microsoft , so stack traces in bug reports can be dealt with more easily"

Thus, we can use the apache.spark.sql.simba as well.

[dongx-psu]

Can you forward me the link on this discussion?

[merlintang]

http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-Namespace-Expanding-Spark-ML-under-Different-Namespace-td21051.html

…

On Sun, Feb 26, 2017 at 6:11 PM, Dong Xie ***@***.***> wrote: Can you forward me the link on this discussion? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#82 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABXY-clazBs0UECohrzuOAkEtjO9HN82ks5rgjC5gaJpZM4L6fRv> .

[dongx-psu]

Finish first migration to Spark 2.1 at branch standalone-2.1. It would be really nice if you can help us test it.

[geoHeil]

@Skyprophet I would like to test it but have some troubles getting started with Simba as outlined #84

[MixalisV]

Is there any guide that describes how to install Simba on Spark 2.1?
The only developing guide that I found, is here
(https://gitlab.com/InitialDLab/simba/blob/master/INSTALL.md) but it refers to spark 1.6

[dongx-psu]

On the standalone branch, simply sbt package will work.

Then, you can import the package to Spark through spark-submit options.