From f593bf24eb77c84d2934a7b1258baea97190335b Mon Sep 17 00:00:00 2001 From: Brian Dominick Date: Tue, 28 Aug 2018 23:47:42 -0400 Subject: [PATCH] Parser upgrades alt -- redo 0.9.0 (#51) * Revert "Add dynamic config & allow vars for non-config builds (#49)" This reverts commit 006f16f9c81b64a4ea17ae51b9d0046da02fef16. We want to reapply these changes and reconcile the conflicts without overwriting anything. * Merge --cvar into --var * bugfix: handle adding variables to emptyset * Bump verion to 0.9.1 --- README.adoc | 105 +++++++++++++++++++++++++++++++++++----- lib/liquidoc.rb | 94 ++++++++++++++++++++++++++--------- lib/liquidoc/version.rb | 2 +- 3 files changed, 164 insertions(+), 37 deletions(-) diff --git a/README.adoc b/README.adoc index 9bf2a6b..3abaa49 100644 --- a/README.adoc +++ b/README.adoc @@ -1,4 +1,6 @@ = LiquiDoc +:toc: preamble + // tag::overview[] LiquiDoc is a documentation build utility for true single-sourcing of technical content and data. It is especially suited for documentation projects with various required output formats, but it is intended for any project with complex, versioned input data for use in docs, user interfaces, and even back-end code. @@ -876,8 +878,12 @@ See <>. === Deploy Operations -It's not clear how deeply we will delve into deploy operations, since other build systems (such as rake) would seem far more suitable. -For testing purposes, however, spinning up a local webserver with the same stroke that you build a site is pretty rewarding and time saving. +Mainstream deployment platforms are probebly better suited to tying all your operations together, but we plan to bake a few common operations in to help you get started. +For true build-and-deployment control, consider build tools such as Make, Rake, and Gradle, or deployment tools like Travis CI, CircleCI, and Jenkins. + +==== Jekyll Serve + +For testing purposes, however, spinning up a local webserver with the same stroke that you build a site is pretty rewarding and time saving, so we'll start there. For now, this functionality is limited to adding a `--deploy` flag to your `liquidoc` command. This will attempt to serve files from the *destination* set for the associated Jekyll build. @@ -885,8 +891,69 @@ This will attempt to serve files from the *destination* set for the associated J [WARNING] Deployment of Jekyll sites is both limited and untested under nonstandard conditions. +==== Algolia Search Indexing for Jekyll + +If you're using Jekyll to build sites, LiquiDoc makes indexing your files with the Algolia cloud search service a matter of configuration. +The heavy lifting is performed by the jekyll-algolia plugin, but LiquiDoc can handle indexing even a complex site by using the same configuration that built your HTML content (which is what Algolia actually indexes). + [NOTE] -The <> has moved to the <>. +You will need a free community (or premium) link:https://www.algolia.com/users/sign_up/hacker[Algolia account] to take advantage of Algolia's indexing service and REST API. +Simply create a named index, then visit the API Keys to collect the rest of the info you'll need to get going. + +Two hard-coding steps are required to prep your source to handle Algolia index pushes. + +. Add a block to your main Jekyll configuration file. ++ +.Example Jekyll Algolia configuration +[source,yaml] +---- +algolia: + application_id: 'your-application-id' # <1> + search_only_api_key: 'your-search-only-api-key' # <2> + extensions_to_index: [adoc] # <3> +---- ++ +<1> From the top bar of your Algolia interface. +<2> From the API Keys screen of your Algolia interface. +<3> List as many extensions as apply, separated by commas. + +. Add a block to your build config. ++ +[source,yaml] +---- + - action: render + data: globals.yml + builds: + - backend: jekyll + properties: + files: + - _configs/jekyll-global.yml + - _configs/jekyll-portal-1.yml + arguments: + destination: build/site/user-basic + attributes: + portal_term: Guide + search: + index: 'portal-1' +---- ++ +The `index:` parameter is for the name of the index you are pushing to. +(An Algolia “app” can have multiple “indices”.) +If you have + +Now you can call your same LiquiDoc build command with the `--search-index-push` or `--search-index-dry` flags along with the `--search-api-key='your-admin-api-key-here'` argument in order to invoke the indexing operation. +The `--search-index-dry` flag merely tests content packaging, whereas `--search-index-push` connects to the Algolia REST API and attempt to push your content for indexing and storage. + +.Example Jekyll Algolia deployment +[source,shell] +---- +bundle exec liquidoc -c _configs/build-docs.yml --search-index-push --search-index-api-key='90f556qaa456abh6j3w7e8c10t48c2i57' +---- + +This operation performs a complete build, including each render operation, before the Algolia plugin processes content and pushes each build to the indexing service, in turn. + +[TIP] +To add modern site search for your users, add link:https://community.algolia.com/instantsearch.js/[Algolia's InstantSearch functionality] to your front end! == Configuring a LiquiDoc Build @@ -894,6 +961,9 @@ In order to seriously explore and instruct this tool's ability to single-source That is, I'm using LiquiDoc to document LiquiDoc in a separate repository -- one which treats the LiquiDoc gem repository (_this_ repo) as a Git submodule. That is, for purposes of coding, _all_ of the *liquidoc-gem* repo's code is accesible by way of an alias path, `products/liquidoc-gem`, from the base of the *liquidoc-docs* project. +[NOTE] +The <> has moved to the <>. + === Gem Repo as Submodule I am on record as prefering to _keep docs source in the same codebase as that of the product they reference_. @@ -988,8 +1058,8 @@ That is, your config files can contain variables, conditionals, and iterative lo All you have to do is (1) add Liquid tags to your YAML configuration file and (2) either (a) pass at least one _config variable_ to it when running your `liquidoc` command or (b) pass it the `--parseconfig` flag. Let's explore that second requirement. -If the Liquid markup in your config file expects variables, pass those variables on the `liquidoc` CLI using `--configvar key=value`. -Otherwise, if you are not passing variables to your config, instruct LiquiDoc to parse the config file using the `parseconfig` CLI option. +If the Liquid markup in your config file expects variables, pass those variables on the `liquidoc` CLI using `--var key=value`. +Otherwise, if you are not passing variables to your config, instruct LiquiDoc to parse the config file using the `--parseconfig` CLI flag. For example, this might be the case if your config merely contains some simple looping functionality to process lots of files. [[config-variables]] @@ -1011,16 +1081,19 @@ Let's first take a look at a sample dynamic configuration to see if we can under This config file wants to build a product datasheet for a specific product, which it expects to be indicated by a config variable called `product_slug`. -Config variables are passed using the `--configvar varname='var val'` format, where `varname` is any key that exists as a Liquid variable in your config file, and `'var val'` is its value, wrapped in single quotes. +Config variables are passed using the `--var varname='var val'` format, where `varname` is any key that exists as a Liquid variable in your config file, and `'var val'` is its value, wrapped in single quotes. Let's say in this case, we want to generate the datasheet for the Windows Enterprise edition of our product. [source,shell] ---- -bundle exec liquidoc -c _configs/build-config.yml --configvar product_slug=win-ent +bundle exec liquidoc -c _configs/build-config.yml -v product_slug=win-ent ---- -This will cause our dynamic configuration to look for a data block formatted like so: `data/products.yml:product_win-ent`. -So long as our `products.yml` file contains a top-level data structure called `product_win-ent`, we're off to the races. +[NOTE] +The `-v` option is an alias for `--var`. + +This will cause our dynamic configuration to look for a data block formatted like so: `data/products.yml:win-ent`. +So long as our `products.yml` file contains a top-level data structure called `win-ent`, we're off to the races. ==== Eliminating Config Variables @@ -1080,7 +1153,7 @@ LiquiDoc {{ vars.product.edition }} ---- -To set the values of `vars.edition` and `vars.env` in the config file, add for instance `--configvar edition=basic --configvar env=staging` +To set the values of `vars.edition` and `vars.env` in the config file, add for instance `--var edition=basic --var env=staging` ==== Constraining Build Options with Dynamic Configuration @@ -1113,7 +1186,7 @@ For instance, {% endif %} ---- -With a build config like this, optionally invoking `--configvar recipe=nopdf`, for instance, will suppress the PDF substep during the build routine. +With a build config like this, optionally invoking `--var recipe=nopdf`, for instance, will suppress the PDF substep during the build routine. ==== Generating Starter Files with Dynamic Configs @@ -1125,7 +1198,7 @@ Once you're comfortable with the concept of < records} + output = records rescue Exception => ex @logger.error "Something went wrong trying to parse the free-form file. #{ex.class} thrown. #{ex.message}" raise "Freeform parse error" @@ -911,8 +922,8 @@ def generate_site doc, build when "jekyll" attrs = doc.attributes build.add_config_file("_config.yml") unless build.prop_files_array - jekyll_config = YAML.load_file(build.prop_files_array[0]) # load the first Jekyll config file locally - attrs.merge! ({"base_dir" => jekyll_config['source']}) # Sets default Asciidoctor base_dir to == Jekyll root + jekyll = load_jekyll_data(build) # load the first Jekyll config file locally + attrs.merge! ({"base_dir" => jekyll['source']}) # Sets default Asciidoctor base_dir to == Jekyll root # write all AsciiDoc attributes to a config file for Jekyll to ingest attrs.merge!(build.attributes) if build.attributes attrs = {"asciidoctor" => {"attributes" => attrs} } @@ -925,14 +936,31 @@ def generate_site doc, build if build.props['arguments'] opts_args = build.props['arguments'].to_opts_args end - command = "bundle exec jekyll build --config #{config_list} #{opts_args} #{quiet}" + base_args = "--config #{config_list} #{opts_args}" + command = "bundle exec jekyll build #{base_args} #{quiet}" + if @search_index + # TODO enable config-based admin api key ingest once config is dynamic + command = algolia_index_cmd(build, @search_api_key, base_args) + @logger.warn "Search indexing failed." unless command + end + end + if command + @logger.info "Running #{command}" + @logger.debug "AsciiDoc attributes: #{doc.attributes.to_yaml} " + system command end - @logger.info "Running #{command}" - @logger.debug "AsciiDoc attributes: #{doc.attributes.to_yaml} " - system command jekyll_serve(build) if @jekyll_serve end +def load_jekyll_data build + data = {} + build.prop_files_array.each do |file| + settings = YAML.load_file(file) + data.merge!settings if settings + end + return data +end + # === # DEPLOY procs # === @@ -948,6 +976,20 @@ def jekyll_serve build system command end +def algolia_index_cmd build, apikey=nil, args + unless build.search and build.search['index'] + @logger.warn "No index configuration found for build; jekyll-algolia operation skipped for this build." + return false + else + unless apikey + @logger.warn "No Algolia admin API key passed; skipping jekyll-algolia operation for this build." + return false + else + return "ALGOLIA_INDEX_NAME='#{build.search['index']}' ALGOLIA_API_KEY='#{apikey}' bundle exec jekyll algolia #{@search_index_dry} #{args} " + end + end +end + # === # Text manipulation Classes, Modules, procs, etc # === @@ -1115,7 +1157,7 @@ def regexreplace input, regex, replacement='' @quiet = true end - opts.on("--explicit", "Log explicit step descriptions to console as build progresses. (Otherwise writes to file at #{@build_dir}/pre/config-explainer.adoc .)") do |n| + opts.on("--explain", "Log explicit step descriptions to console as build progresses. (Otherwise writes to file at #{@build_dir}/pre/config-explainer.adoc .)") do |n| explainer_init("STDOUT") @explainer.level = Logger::INFO @logger.level = Logger::WARN # Suppress all those INFO-level messages @@ -1130,21 +1172,27 @@ def regexreplace input, regex, replacement='' @jekyll_serve = true end - opts.on("--var KEY=VALUE", "For passing variables directly to the 'vars.' scope template via command line, for non-config builds only.") do |n| - pair = {} - k,v = n.split('=') - pair[k] = v - @passed_vars.merge!pair + opts.on("--search-index-push", "Runs any search indexing configured in the build step and pushes to Algolia.") do + @search_index = true + end + + opts.on("--search-index-dry", "Runs any search indexing configured in the build step but does NOT push to Algolia.") do + @search_index = true + @search_index_dry = "--dry-run" + end + + opts.on("--search-api-key=STRING", "Passes Algolia Admin API key (which you should keep out of Git).") do |n| + @search_api_key = n end - opts.on("-x", "--cvar KEY=VALUE", "For sending variables to the 'vars.' scope of the config file and triggering Liquid parsing of config.") do |n| + opts.on("-v", "--var KEY=VALUE", "For passing variables directly to the 'vars.' scope of a template; for dynamic configs, too.") do |n| pair = {} k,v = n.split('=') pair[k] = v - @passed_configvars.merge!pair + @passed_vars.merge!pair end - opts.on("--parse-config", "Preprocess the designated configuration file as a Liquid template. Superfluous when passing -x/--cvar arguments.") do + opts.on("--parse-config", "Preprocess the designated configuration file as a Liquid template. Superfluous when passing -v/--var arguments.") do @parseconfig = true end @@ -1176,5 +1224,5 @@ def regexreplace input, regex, replacement='' end else @logger.debug "Executing... config_build" - config_build(@config_file, @passed_configvars, @parseconfig) + config_build(@config_file, @passed_vars, @parseconfig) end diff --git a/lib/liquidoc/version.rb b/lib/liquidoc/version.rb index 67de968..618c97f 100644 --- a/lib/liquidoc/version.rb +++ b/lib/liquidoc/version.rb @@ -1,3 +1,3 @@ module Liquidoc - VERSION = "0.9.0" + VERSION = "0.9.1" end