Data size & limitations

[luisquintanilla]

Hi,

Just learned about Flat and I had a few questions.

1. Are there limits on the size of data you can load?
2. How / where are the files saved in your repository?
3. Do the saved files count towards the limits on the size of your GitHub repo?

[idan]

Hey @luisquintanilla!

So, there's a difference between the size of the intermediate data and the size of data committed to the repo at the end.

The hosted Actions runners from Github come with 14GB of disk space, some of which is used by the operating system. In principle, your action can fill up a few gigabytes of space on disk! However, you can't then commit all that data to a repo.

Per the documentation regarding repo limits, individual files cannot be larger than 100mb and repos should not be larger than 1gb. In practice, if you're getting towards these limits, your performance is going to be quite bad. It's probably a sign that you should be looking at other tools for your workflow!

Files are saved in your repository wherever you decide to save them! You can specify a pathname in the workflow. That pathname is relative to the root of your repo.

Yes, the committed files are a part of your repo, and they count towards your repo. There's no magic! 😂

[JustinGrote]

I'm glad @luisquintanilla asked this question, as it seems to fly in the face of the mission statement:
https://octo.github.com/projects/flat-data#why-flat-data

"A million rows aren't cool. You know what's cool? A billion rows. Distributed data systems that slip the surly bonds of any one machine to touch the face of scale."

I would then add:

"Unless you want to actually use Github for this, in which case you should be looking at other tools for your workflow"

@idan how do you reconcile these two discordant statements?

[idan]

That part of the intro is talking about how everyone like to talk about their billions of rows, their data lakes and refineries, their evented architectures. The Flat Data project stands in contrast to that kind of language, and to those kinds of tools.

Flat is boring. It doesn't try to replace other tools, but to complement them with an easier, less-powerful alternative. It does not have any theorems. It works on small, pedestrian datasets, since any given file can't be >100mb. It's just an easier way to think about a certain class of problems!

It sounds like that viewpoint didn't present itself clearly to you! I'll think about how to clarify the copy, @JustinGrote

[JustinGrote]

@idan you are correct in that I misread that section and took the point too narrowly. Still, this seems like an extremely high faluten concept of "a script that saves some data to git/azure storage/amazon s3/whatever", is this really considered so revolutionary to data scientists who are so deeply navel gazing into the thoeretical possibilities?

That said, thanks for providing a simple tool and workflow for people who may not be super-versed in pulling data from a Rest API or SQL database.

[mskyttner]

Regarding limits and the mission statement, I wonder if the githubactions based workflow would allow also for steps to commit data into github container repository "data only" containers? Limits for those containers are higher, those could hold billions of rows and be versioned. Another possibility for upping the limits perhaps would be to use a step to commit data to github "releases" with versioning like here https://github.com/ropensci/piggyback (which limits data file sizes to 2GB)?

[RandomFractals]

well, @idan if you can't support billions of rows you should probably remove that eye brows raising intro line. I just called them 'large' data sets in my Data Preview extension hoping most devs will get that 50-100 Mb flat files should be fine, and we can't do it with gigs of data without proper data streaming solution. Totally agree with you on that part.