db adapter and the db_profile setup

[arybach]

Hihi,

"fal credentials must have a db_profile property set"
Can you please clarify somewhere in docs how to setup db adapter and the db_profile for fal properly?
I couldn't find any info on that.

before fal & dbt-fall installation

dbt --version returns
Core:

• installed: 1.3.1
• latest: 1.3.1 - Up to date!

Plugins:

• postgres: 1.3.1 - Up to date!

after:
Encountered an error:
fal credentials must have a db_profile property set

Thx vm.

alex

[mederka]

Hi @arybach, thanks for raising this! We will definitely clarify it in our docs. In the meantime have you looked through the quickstart guide to dbt-fal? And if so, I would be curious if you updated your profiles.yml the way it's suggested in our guide.

[mederka]

I see. Sorry for the confusion, I'll open a PR addressing this later today. In order to use dbt-fal, you need to define another output like this:

fal_test:
  target: dev_fal
  outputs:
    dev_fal:
      type: fal
      db_profiles: dev
    dev:
      type: postgres
      host: localhost
      user: postgres
      password: postgres
      port: 5432
      dbname: postgres
      schema: public
      threads: 4

[mederka]

@arybach, let me know if this makes the quickstart clearer: https://github.com/fal-ai/fal/pull/708/files .

[arybach]

Great. Thx vm.

since i specified outputs for the dbt project earlier - adding fal_test under
outputs:
dev: ...
prod: ...
fal_test:
type: fal
db_profile: dev

and then setting db_profile to an existing entry under outputs (dev) works fine as well.

[arybach]

all worked fine.
I had to comment out lines 37-39 in module_check.py of module_generator
#assert (
# len(all_wtm_calls) > 0
#), "There must be at least one write_to_model call in the Python Model"
as it throws an error when i use a method of an instance:
from fal import FalDbt
faldbt = FalDbt(profiles_dir="./", project_dir="./")
faldbt.write_to_model(final_df,"fal_test")
best..

[mederka]

That is interesting. Are you using fal flow run with fal Python models? Do you have pre- or post-hooks?

[arybach]

under project.yml
vars:
source-paths: ["models"]
fal-models-paths: ["fal_models"]

fal-models/schema.yml
models:
- name: stg_test
meta:
fal:
scripts:
- fal_models/stg_test.py

It batch-wirtes to stg_test table by joining existing data + newly transformed data (identifies and fixes broken urls).
I have post-hooks defined for jinja macros - nothing for python and not for stg_test model though, so there shouldn't be any conflicts.
I tried and then deleted compiled dbt's native python model's test (as it is not executed locally i couldn't pass my aws and api credentials. So fal is a big help where dbt's own implementation is lacking, thank you).

[mederka]

Can stg_test.py be written as a dbt Python model? Something similar to this:

import ...

def model(dbt, session):

    my_sql_model_df = dbt.ref("my_sql_model")

    final_df = ...  # stuff you can't write in SQL!

    return final_df

If so, you can just put it in your models directory and discard the fal_models directory along with schema.yml inside of of it. If dbt-fal is installed, you should be able to do dbt run and it will run the dbt dag along with stg_test.py. If go with this option, you have to make sure your Python models follow the guidelines provided by dbt: https://docs.getdbt.com/docs/build/python-models , especially with regards to using dbt.ref instead of ref.

This month we will prioritize clearer documentation as well as better communication via our blog!