Spark version and jaccard_at_thresholds distance

[juracyjr]

hello guys, first thanks to the developers for this great package for linkage.

i have been using splink for linkage with duckdb , now i will try spark , and i have some questions and problems

1 - is there a specific version of spark to use? which one? 3? 2?

2 - using the same script that worked with pandas I changed it to the spark backend, at the time of the distance it is showing this here

AnalysisException: Undefined function: 'JACCARD'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 28 pos 11

example of fields

#import splink.duckdb.duckdb_comparison_library as cl
import splink.spark.spark_comparison_library as sf

settings2 = {
    "probability_two_random_records_match": 9/50_000,
    "link_type": "dedupe_only",
    "blocking_rules_to_generate_predictions": [
        "l.first_name = r.first_name  and l.dt_DT_NASC = r.dt_DT_NASC",
    ],
    "comparisons": [
        sf.jaccard_at_thresholds("first_name"),
        sf.exact_match("sexo_prep",  term_frequency_adjustments=False)
    ],
    "retain_matching_columns": True,
    "retain_intermediate_calculation_columns": True,
    "max_iterations": 20,
    "em_convergence": 0.01
}

linker.initialise_settings(settings2)


linker.estimate_u_using_random_sampling(target_rows=5e6) <-  show erro

thanks in advance

[ericmanning]

I think you need to register the Jaccard UDF:

from pyspark.context import SparkContext, SparkConf
from pyspark.sql import SparkSession, types

conf = SparkConf()
conf.set("spark.jars", path_to_udf_jar)

sc = SparkContext.getOrCreate(conf=conf)
spark = SparkSession(sc)

spark.udf.registerJavaFunction("jaccard", "uk.gov.moj.dash.linkage.JaccardSimilarity", types.DoubleType())

The UDF jar for Spark is in splink/files/spark_jars here.

[juracyjr]

thanks, it worked, I didn't know I had to register these functions, I added the path directly in
spark-defaults.conf

spark.jars -> path
works!