Can I use multiple source URLs?

[danwhitford]

I have a scenario where I am downloading two different JSON files and processing them into a single JSON. I am currently using rx.js in an Action but I would like to try to convert the process to Flat.

Is it possible to have multiple source URLs? My ideal would be to set two http_url params and then have the filenames available as Deno.args[0] and Deno.args[1] in the post processing script. I don't think this is possible right now, I don't see it in the docs anywhere.

As a workaround I am thinking I can have two Flat actions, one to just download file A, and one to download file B with a post-processing script that has both of the filenames hardcoded. It should work but feels less tidy.

If I manage to create an example I will post a follow up here!

[irealva]

Hi there, could you try your second idea around using two Flat actions? That's exactly what we have done in our own repos to combine multiple downloaded files in one repository.

Here's our example covid-dashboard repo on that use case. And here's the yaml file specifically. You'll see it chains 3 subsequent Flat Action steps.

Let us know if that works out for you.

[danwhitford]

Thanks for the great answer and link to the example. I will try out that approach when I have a weekend spare for my side projects!

[irealva]

Also, once downloaded file A is in the repo (for example from the first Flat Action, and let's say you name the file "fileA.json") you can just refer to it in the postprocessing script with the path ./fileA.json.

Deno.args[0] is really also just a shorthand for the filename you specify in the downloaded_filename parameter. See this line here for an example.