Enable multi-file data ingest over commandline, including for dynamic…

… config (#86)

* Add _build/ dirs to .gitignore

* Update required dependencies
** rake {dev only)   ~> 10.0
** asciidoctor       ~> 2.0
** asciidoctor-pdf   = 1.5.3
** jekyll            ~> 4.0
** jekyll-asciidoc   ~> 3.9

* Add multi-file ingest for dynamic config operations

* Add proc get_payload & method add_payload
** Abstracts liquify operations shared by parse acton and dynamic config-parsing
** Translates DataFiles objects to ingestible (scoped) data for feeding templates
** get_payload reads files, scopes variables; add_payload bulk merges them into data_obj

* Commandline args accommodate appending data to config (--data/-d file1.yml,file2.json)
** Properly maps comma-delimited filepaths to DataFiles object
** passes this object as @data_files/data_files along during --config builds
** Enables multi-datasource template processing via commandline
** Ex: bundle exec liquidoc -d data/file1.yml,data/file2.xml -t template.html -o out.html

* Reorganizes add_data! methods so data object is required, scope optional (default: "")

* Document new functionality in Basic Parsing & Dynamic Config topics

* Ensure CLI functionality for basic parsing

* Bump verson to 0.12.0-rc1

* Remove test output

* Bump version to 0.12.0-rc2

* Fix missing style setting for PDF builds

* Bump version to 0.12.0.rc3

.gitignore

@@ -4,3 +4,5 @@ pkg
 *.html
 *.pdf
 .DS_Store
+_build
+docs/_build

docs/topics/config_dynamic.adoc

@@ -172,3 +172,43 @@ In Liquid, loops can only iterate through arrays.
 Comma-delimited lists can be converted to arrays using the *split* filter to divide its contents into items.
 The `| split: ","` notation here tells Liquid we wish to apply this filter so
 the variable `portals` can become an array.
+
+== Ingesting data files into configs
+
+Just as a dynamic config can <<config-variables,accept variables>> at build time, it can also be passed whole data files, including complex objects, also at build time.
+Just assign data files to the command using the `--data` flag, with multiple files separated by commas.
+
+.Example ingested data objects command
+ bundle exec liquidoc -c _configs/build.yml -d data/products.yml,data/volumes.yml
+
+This command tells LiquiDoc to pass the `products.yml` and `volumes.yml` files into the dynamic build config, where they can be referenced as variables in Liquid using these object names.
+
+.Example config loop using ingested data objects
+[source,yaml]
+----
+{% assign prods = products.products %}
+{% assign vols = global.volumes %}
+{% assign manuals = vols | where: 'type','manual' %}
+{% for prod in prods %}
+- action: parse
+  data: data/manifests/{{ prod.slug }}.yml
+  builds:
+    - output: _build/includes/topics-meta_{{ prod.slug }}_portal.adoc
+      template: _templates/liquid/topics-meta.asciidoc
+  {% assign prod_vols = vols | where: "prod",prod.slug | where: "type","manual" %}
+  {% for vol in prod_vols %}
+    - output: _build/content/{{ prod.slug }}/{{ vol.slug }}-index.adoc
+      template: _templates/liquid/manual-index.asciidoc
+      variables:
+        title: "{{ vol.title }}"
+  {% endfor %}
+{% endfor %}
+----
+
+Just as with multi-file parse actions, top-level scopes are named after the file from which they were ingested.
+This means we must tell LiquiDoc which files to use.
+Since we are assigning files to the build routine itself, we attach them on the commandline.
+
+.Example commandline with --data option
+[source,shell]
+ bundle exec liquidoc -c build-config.yml -d data/global.yml,data/products.yml

docs/topics/parsing_basic.adoc

@@ -12,3 +12,22 @@ This single-action invocation of LiquiDoc ingests data from YAML file `sample.ym
 [TIP]
 Add `--verbose` to any `liquidoc` command to see the steps the utility is
 taking.
+
+To ingest data from multiple files, pass multiple paths to the `-d`/`--data` option, separated by commas.
+
+.Example -- Use multiple datasources to parse a template
+----
+bundle exec liquidoc -d _data/source1.yml,_data/source2.json -t my_template.html -o my_artifact.html
+----
+
+In this example, data from `source1.yml` will be passed to the template in an object called `source1`, and `source2.json` will ingested as the `source2` object.
+
+.Example my_template.html -- Liquid expression of data from multiple file ingest
+[source,liquid]
+----
+{{ source1.param1 }}
+
+{% for item in source2 %}
+{{ item.name }}
+{% endfor %}
+----

lib/liquidoc.rb

@@ -40,7 +40,7 @@
 @configs_dir = @base_dir + '_configs'
 @templates_dir = @base_dir + '_templates/'
 @data_dir = @base_dir + '_data/'
-@data_file = nil
+@data_files = nil
 @attributes_file_def = '_data/asciidoctor.yml'
 @attributes_file = @attributes_file_def
 @pdf_theme_file = 'theme/pdf-theme.yml'
@@ -75,17 +75,21 @@
 # ===
 
 # Establish source, template, index, etc details for build jobs from a config file
-def config_build config_file, config_vars={}, parse=false
+def config_build config_file, config_vars={}, data_files=nil, parse=false
   @logger.debug "Using config file #{config_file}."
   validate_file_input(config_file, "config")
-  if config_vars.length > 0 or parse or contains_liquid(config_file)
+  if config_vars.length > 0 or data_files or parse or contains_liquid(config_file)
     @logger.debug "Config_vars: #{config_vars.length}"
   # If config variables are passed on the CLI, we want to parse the config file
   # and use the parsed version for the rest fo this routine
     config_out = "#{@build_dir}/pre/#{File.basename(config_file)}"
-    vars = DataObj.new()
-    vars.add_data!("vars", config_vars)
-    liquify(vars, config_file, config_out)
+    data_obj = DataObj.new()
+    if data_files
+      payload = get_payload(data_files)
+      data_obj.add_payload!(payload)
+    end
+    data_obj.add_data!(config_vars, "vars")
+    liquify(data_obj, config_file, config_out)
     config_file = config_out
     @logger.debug "Config parsed! Using #{config_out} for build."
     validate_file_input(config_file, "config")
@@ -131,41 +135,23 @@ def iterate_build cfg
       data_obj = DataObj.new()
       if step.data
         data_files = DataFiles.new(step.data)
-        data_files.sources.each do |src|
-          begin
-            data = ingest_data(src) # Extract data from file
-          rescue Exception => ex
-            @logger.error "#{ex.class}: #{ex.message}"
-            raise "DataFileReadFail (#{src.file})"
-          end
-          begin # Create build.data
-            if data_files.sources.size == 1
-              data_obj.add_data!("", data) if data.is_a? Hash
-              # Insert arrays into the data. scope, and for backward compatibility, hashes as well
-              data_obj.add_data!("data", data)
-            else
-              data_obj.add_data!(src.name, data) # Insert object under self-named scope
-            end
-          rescue Exception => ex
-            @logger.error "#{ex.class}: #{ex.message}"
-            raise "DataIngestFail (#{src.file})"
-          end
-        end
+        payload = get_payload(data_files)
+        data_obj.add_payload!(payload)
       end
       builds.each do |bld|
         build = Build.new(bld, type, data_obj) # create an instance of the Build class; Build.new accepts a 'bld' hash & action 'type'
         if build.template
-          # Prep & perform a Liquid-parsed build build
+          # Prep & perform a Liquid-parsed build
           @explainer.info build.message
-          build.add_data!("vars", build.variables) if build.variables
+          build.add_data!(build.variables, "vars") if build.variables
           liquify(build.data, build.template, build.output) # perform the liquify operation
         else # Prep & perform a direct conversion
           # Delete nested data and vars objects
           build.data.remove_scope("data")
           build.data.remove_scope("vars")
           # Add vars from CLI or config args
-          build.data.add_data!("", build.variables) unless build.variables.empty?
-          build.data.add_data!("", @passed_vars) unless @passed_vars.empty?
+          build.data.add_data!(build.variables) unless build.variables.empty?
+          build.data.add_data!(@passed_vars) unless @passed_vars.empty?
           regurgidata(build.data, build.output)
         end
       end
@@ -479,8 +465,8 @@ def data
     @data unless @data.nil?
   end
 
-  def add_data! obj, scope
-    @data.add_data!(obj, scope)
+  def add_data! data, scope=""
+    @data.add_data!(data, scope)
   end
 
   # def vars
@@ -602,7 +588,7 @@ def validate
     when "render"
       reqs = ["output"]
     end
-    for req in required
+    for req in reqs
       if (defined?(req)).nil?
         raise "ActionSettingMissing"
       end
@@ -612,6 +598,7 @@ def validate
 end # class Build
 
 class DataSrc
+  # Organizes metadata about an ingestible data source
   # initialization means establishing a proper hash for the 'data' param
   def initialize sources
     @datasrc = {}
@@ -679,9 +666,9 @@ def pattern
 # DataFiles
 class DataFiles
   # Accepts a single String, Hash, or Array
-  # String must be a filename
-  # Hash must contain :file and optionally :type and :pattern
-  # Array must contain filenames as strings
+  # String must be a path/filename
+  # Hash must contain file: and optionally type: and pattern:
+  # Array must contain path/filenames as strings
   # Returns array of DataSrc objects
   def initialize data_sources
     @data_sources = []
@@ -696,6 +683,7 @@ def initialize data_sources
   end
 
   def sources
+    # An Array of DataSrc objects
     @data_sources
   end
 
@@ -714,20 +702,35 @@ def initialize
     @data = {"vars" => {}}
   end
 
-  def add_data! scope="", data
+  def add_data! data, scope=""
     # Merges data into existing scope or creates a new scope
     if scope.empty? # store new object at root of this object
       self.data.merge!data
     else # store new object as a subordinate, named object
-      if self.data.key?(scope) # merge into existing key
-        self.data[scope].merge!data
+      if self.data.key?(scope) # merge/append into existing object
+        self.data[scope].merge!data if self.data[scope].is_a? Hash
+        self.data[scope] << data if self.data[scope].is_a? Array
       else # create a new key named after the scope
         scoped_hash = { scope => data }
         self.data.merge!scoped_hash
       end
     end
   end
 
+  def add_payload! payload
+    # Expects an Array of Hashes ([{name=>String, data=>Object},...])
+    if payload.size == 1
+      # If payload is a single Hash, store it at the root level (no scope)
+      self.add_data!(payload[0]['data']) if payload[0]['data'].is_a? Hash
+      # Insert arrays into the data. scope, and for backward compatibility, hashes as well
+      self.add_data!(payload[0]['data'], "data")
+    end
+    # For ALL payloads, create a self-named obj scope
+    payload.each do |obj|
+      self.add_data!(obj['data'], obj['name']) # Insert object under self-named scope
+    end
+  end
+
   def data
     @data
   end
@@ -770,7 +773,25 @@ def type
 # PARSE-type build procs
 # ===
 
-# Pull in a semi-structured data file, converting contents to a Ruby hash
+def get_payload data_files
+  # data_files: a proper DataFile object
+  payload = []
+  data_files.sources.each do |src|
+    obj = {}
+    begin
+      data = ingest_data(src) # Extract data from file
+    rescue Exception => ex
+      @logger.error "#{ex.class}: #{ex.message}"
+      raise "DataFileReadFail (#{src.file})"
+    end
+    obj['name'] = src.name
+    obj['data'] = data
+    payload << obj
+  end
+  return payload
+end
+
+# Pull in a semi-structured data file, converting contents to a Ruby object
 def ingest_data datasrc
   raise "InvalidDataSrcObject" unless datasrc.is_a? DataSrc
   case datasrc.type
@@ -862,21 +883,20 @@ def liquify data_obj, template_file, output
   end
 end
 
-def cli_liquify data_file=nil, template_file=nil, output_file=nil, passed_vars
+def cli_liquify data_files=nil, template_file=nil, output_file=nil, passed_vars
   # converts command-line options into liquify or regurgidata inputs
   data_obj = DataObj.new()
-  if data_file
-    df = DataFiles.new(data_file)
-    ingested = ingest_data(df.sources[0])
-    data_obj.add_data!("", ingested)
+  if data_files
+    payload = get_payload(data_files)
+    data_obj.add_payload!(payload)
   end
   if template_file
-    data_obj.add_data!("data", ingested) if df
-    data_obj.add_data!("vars", passed_vars) if passed_vars
+    # data_obj.add_data!(ingested, "data") if df
+    data_obj.add_data!(passed_vars, "vars") if passed_vars
     liquify(data_obj, template_file, output_file)
   else
     data_obj.remove_scope("vars")
-    data_obj.add_data!("", passed_vars) if passed_vars
+    data_obj.add_data!(passed_vars) if passed_vars
     regurgidata(data_obj, output_file)
   end
 end
@@ -1042,6 +1062,7 @@ def asciidocify doc, build
   # Perform the aciidoctor convert
   if build.backend == "pdf"
     @logger.info "Generating PDF. This can take some time..."
+    attrs.merge!({"pdf-theme"=>build.style}) if build.style
   end
   Asciidoctor.convert_file(
     doc.index,
@@ -1053,7 +1074,7 @@ def asciidocify doc, build
     safe: "unsafe",
     sourcemap: true,
     verbose: @verbose,
-    mkdirs: true
+    mkdirs: true,
   )
   @logger.info "Rendered file #{to_file}."
 end
@@ -1303,8 +1324,10 @@ def regexreplace input, regex, replacement=''
     @config_file = @base_dir + n
   end
 
-  opts.on("-d PATH", "--data=PATH", "Semi-structured data source (input) path. Ex. path/to/data.yml. Required unless --config is called." ) do |n|
-    @data_file = @base_dir + n
+  opts.on("-d PATH[,PATH]", "--data=PATH[,PATH]", "Semi-structured data source (input) path or paths. Ex. path/to/data.yml or data/file1.yml,data/file2.json. Required unless --config is called; optional with config." ) do |n|
+    data_files = n.split(',')
+    data_files = data_files.map! {|file| @base_dir + file}
+    @data_files = DataFiles.new(data_files)
   end
 
   opts.on("-f PATH", "--from=PATH", "Directory to copy assets from." ) do |n|
@@ -1396,13 +1419,13 @@ def regexreplace input, regex, replacement=''
 
 unless @config_file
   @logger.debug "Executing config-free build based on API/CLI arguments alone."
-  if @data_file
-    cli_liquify(@data_file, @template_file, @output_file, @passed_vars)
+  if @data_files
+    cli_liquify(@data_files, @template_file, @output_file, @passed_vars)
   end
   if @index_file
     @logger.warn "Rendering via command line arguments is not yet implemented. Use a config file."
   end
 else
   @logger.debug "Executing... config_build"
-  config_build(@config_file, @passed_vars, @parseconfig)
+  config_build(@config_file, @passed_vars, @data_files, @parseconfig)
 end

lib/liquidoc/version.rb

@@ -1,3 +1,3 @@
 module Liquidoc
-  VERSION = "0.11.0"
+  VERSION = "0.12.0-rc3"
 end

liquidoc.gemspec

@@ -32,15 +32,15 @@ Gem::Specification.new do |spec|
   spec.required_rubygems_version = ">= 2.7.0"
 
   spec.add_development_dependency "bundler", ">= 1.15"
-  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "rake", ">= 12.3.3"
 
-  spec.add_runtime_dependency "asciidoctor", "~>1.5"
+  spec.add_runtime_dependency "asciidoctor", "~>2.0"
   spec.add_runtime_dependency "json", "~>2.2"
   spec.add_runtime_dependency "liquid", "~>4.0"
-  spec.add_runtime_dependency "asciidoctor-pdf", "~>1.5.0.alpha.16"
+  spec.add_runtime_dependency "asciidoctor-pdf", "=1.5.3"
   spec.add_runtime_dependency "logger", "~>1.3"
   spec.add_runtime_dependency "crack", "~>0.4"
-  spec.add_runtime_dependency "jekyll", "~>3.0"
-  spec.add_runtime_dependency "jekyll-asciidoc", "~>2.1"
+  spec.add_runtime_dependency "jekyll", "~>4.0"
+  spec.add_runtime_dependency "jekyll-asciidoc", "~>3.0"
   spec.add_runtime_dependency "highline", "~>2.0"
 end