Repository Template

Highly automated, up-to-date, and well-documented repository template.

Checks for common problems, Markdown, YAML, Bash, formats, lints, and tests before committing or pushing so you don't have any surprises at CI or when releasing your code to GitLab and GitHub!

The project is under active development.

The Purpose

The purpose is to have a template repository and to have it well-manageable and well-maintainable by both human beings and automation tools.

The rationale behind this is that taking care of tens or hundreds of repositories while keeping them working, tidy, consistent, and up-to-date, might be a daunting task.

The way how to achieve the desired state of manageability and maintainability is to unify and automate workflow to allow frequent small changes for multiple projects at scale.

Objectives:

• Simple and easy environment check and setup
• Fast and unified code change contribution
• Automated and reliable code change propagation (build, testing, integration, publication or deployment, and release)

Strategies and tactics to achieve objectives:

1. Automate
2. Automate
3. Automate

Table of Contents

• Features
  • Releases
  • Hooks
  • Tests
    • Test Set
  • Templates
  • Images
• Installation and Configuration
  • Local Environment
  • GitLab Project
    • GitLab - GitHub Synchronization
    • GitLab CI Settings
    • GitLab CI Nightly Pipeline
• Usage
  • Usage Examples
• Contributing
  • Testing
    • Test at Docker Container
• To-Do list
• Roadmap
• Credits and Acknowledgments
• Copyright and Licensing
• Changelog and News
• Notes and References
  • Dependencies
  • Recommendations
  • Suggestions
  • Further Reading

Features

Optimized for GitHub flow, easily adjustable to GitLab flow or any other workflow.

• Automated workflow using git hooks, and GitLab CI
• Git commit scans committed codebase change, git push scans pushed codebase change, and GitLab CI scans the whole codebase, and the hooks are applied
  • Commit messages are checked using gitlint, commit message should follow Conventional Commits
• Git commit (both regular and merge) is normalized, checked, and tested:
  • Runs hooks
  • Runs fast test set
  • Lints the commit message
• Git push is checked, and tested:
  • Runs hooks
  • Runs reduced test set
  • Create merge request directly by git push options, see https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-merge-requests
• GitLab CI run is checked, and tested:
  • Lints the latest commit message (except release commits)
  • Runs hooks
  • Runs full test set on non-scheduled pipeline runs
  • Runs nightly test set on scheduled pipeline runs
  • Skip GitLab CI run if commit message contains [ci skip] or [skip ci], using any capitalization, or pass git push option ci.skip (git push -o ci.skip git >= 2.17, git push --push-option=ci.skip git >= 2.10)
• Hooks could be skipped by setting SKIP variable to a comma-separated list of skipped hooks, for example, SKIP=forbid-new-submodules,gitlab-ci-linter git commit
• Hooks could be run manually pre-commit run -a --hook-stage manual
• When feat or fix commit is present, GitLab CI job release publishes release using semantic-release/semantic-release
  • When merged to the maintenance release branch (N.N.x or N.x.x or N.x with N being a number), publish maintenance release
  • When merged to next, next-major, beta, or alpha branch, publish pre-release release
  • When merged to the main branch publish release
• Included scripts for your convenience in a fashion of The GitHub Blog: Scripts to Rule Them All
  • scripts/setup setups commit-msg, pre-commit, pre-merge-commit, and pre-push hooks
  • scripts/bootstrap installs dependencies
  • scripts/test runs tests, as arguments accepts test files or test sets (see *.bats or *.set files at the tests directory)
  • scripts/update updates used dependencies
• EditorConfig

Releases

Release branches must match regex ^(((0|[1-9]\d*)\.)(((0|[1-9]\d*|x)\.)?x)|main|next(-major)?|beta|alpha)$, see https://regex101.com/r/gH9dCG/2/.

When feat or fix commit is present in the merge (to be more precise since the last release tag) to main, next, next-major, beta, alpha, major.x, or major.minor.x branch, the publish release:

• Determines major, minor, or patch version bump using semantic-release/commit-analyzer
• Generates release notes using semantic-release/release-notes-generator
• Generates changelog using semantic-release/changelog
• Commits changelog and new version using semantic-release/git
• Publishes new version by creating a tag using semantic-release/gitlab
• Publishes new version by creating a tag using semantic-release/github
• Semantic-release skips release if commit contains [skip release] or [release skip] in the commit message

Hooks

Pre-commit is by default configured to run these hooks:

• Lints git commit message using gitlint
  • Prevents messages not following Conventional Commits
  • Prevents words wip, todo, or tbd at the commit message title
• Enforces max file size to 1024 kB using pre-commit/pre-commit-hooks: check-added-large-files
• Prevents case insensitive filename conflict using pre-commit/pre-commit-hooks: check-case-conflict
• Enforces executables have shebangs using pre-commit/pre-commit-hooks: check-executables-have-shebangs
• Loads JSON to validate it using pre-commit/pre-commit-hooks: check-json
• Prevents merge conflict strings using pre-commit/pre-commit-hooks: check-merge-conflict
• Prevents stale symlinks using pre-commit/pre-commit-hooks: check-symlinks
• Prevents non-permanent GitHub links using pre-commit/pre-commit-hooks: check-vcs-permalinks
• Prevents destroyed symlinks using pre-commit/pre-commit-hooks: destroyed-symlinks
• Prevents the existence of private keys using pre-commit/pre-commit-hooks: detect-private-key
• Enforces files end with empty newline using pre-commit/pre-commit-hooks: end-of-file-fixer
• Prevents UTF8 byte order marker using pre-commit/pre-commit-hooks: fix-byte-order-marker
• Prevents new git submodules pre-commit/pre-commit-hooks: forbid-new-submodules
• Converts line endings to LF using pre-commit/pre-commit-hooks: mixed-line-ending
• Prevents commits to protected branches using pre-commit/pre-commit-hooks: no-commit-to-branch
• Enforces pretty JSON format using pre-commit/pre-commit-hooks: pretty-format-json
• Prevents trailing whitespace characters using pre-commit/pre-commit-hooks: trailing-whitespace
• Prevents botched name/email translations in git history using jumanjihouse/pre-commit-hooks: check-mailmap
• Prevents binary files from being added by accident using jumanjihouse/pre-commit-hooks: forbid-binary
• Prevents git conflict markers and whitespace errors by using jumanjihouse/pre-commit-hooks: git-check
• Prevents modified, staged, or untracked by using jumanjihouse/pre-commit-hooks: git-dirty
• Prevents foxtrot merges by using jumanjihouse/pre-commit-hooks: protect-first-parent
• Enforces executable scripts have no extension using jumanjihouse/pre-commit-hooks: script-must-not-have-extension
• Enforces non-executable script libraries have extension using jumanjihouse/pre-commit-hooks: script-must-have-extension
• When manually run, prevents hooks not applied to any file in the repository using pre-commit/pre-commit-hooks: meta hook check-hooks-apply
• Prevents useless pre-commit hook exclude directives using pre-commit/pre-commit-hooks: meta hook check-useless-excludes
• Checks spelling using GitHub - codespell-project/codespell: check code for common misspellings
• Lints Markdown using igorshubovych/markdownlint-cli: MarkdownLint Command Line Interface (except CHANGELOG.md)
• Lints YAML using adrienverge/yamllint
• Lints .gitlab-ci.yml file using devopshq/gitlab-ci-linter when GL_TOKEN environment variable is set to GitLab Personal Token
• When MR (merge request) is created or updated, or when manually run, prevents presence of words wip, todo, or tbd preceded with # at a text file, checked by regex (?i)#\s*\b(wip|todo|tbd)\b, see https://regex101.com/r/mKueFx/1
• Lints shell scripts formatting using mvdan/sh: A shell parser, formatter, and interpreter with bash support; includes shfmt
• Checks shell scripts using koalaman/shellcheck: ShellCheck, a static analysis tool for shell scripts
• Detects hardcoded secrets like passwords, api keys, and tokens in git repos using GitHub - gitleaks/gitleaks: Scan git repos (or files) for secrets using regex and entropy key
• For other formats and rules see pre-commit: Supported hooks, there are many for .NET, Ansible, AWS, C, CMake, CSV, C++, Chef, Dart, Docker, Flutter, git, GitHub, GitLab, Go, HTML, INI, Java, JavaScript, Jenkins, Jinja, JSON, Kotlin, Lisp, Lua, Mac, Markdown, Node.js, Perl, PHP, Prometheus, Protocol Buffers, Puppet, Python, R, Ruby, Rust, Shell, Swift, Terraform, TOML, Typescript, XML, YAML, ... or create new using regular expressions.

Tests

Tests are written using BATS - GitHub - bats-core/bats-core: Bash Automated Testing System, GitHub - bats-core/bats-support: Supporting library for Bats test helpers, GitHub - bats-core/bats-assert: Common assertions for Bats, and GitHub - bats-core/bats-file: Common filesystem assertions for Bats and organized in test sets.

Test Set

Test set is a simple text file format. Each line must begin or end without leading or trailing whitespace. Each line should contain included test sets (*.set), test files to be run (*.bats), comments starting with # as the first character on the line, or empty lines. File paths are relative to the test set file.

Example:

# Commented and empty lines are ignored

another.set

script1.bats
script2.bats
script3.bats

Templates

• Readme Template
• Licenses
  • MIT License Template
  • NO License Template

Images

• Git workflow examples & template
• Example of the full workflow
• Example of a release workflow
• Example of a feature workflow
• Example of a bugfix workflow

Installation and Configuration

Local Environment

Clone the project with --recursive option, run scripts/bootstrap as root to install dependencies, scripts/setup for a complete setup, and adjust to Your needs. Make sure GL_TOKEN: GitLab Personal Access Token with scope api is present, otherwise gitlab-ci-linter is skipped. To load secrets you can use shell extension like direnv, encryption like SOPS, or secrets manager HashiCorp Vault, please make sure you won't commit your secrets.

Example:

git clone --recursive [email protected]:xebis/repository-template.git
cd repository-template
sudo scripts/bootstrap
scripts/setup

Run scripts/update from time to time to update repository dependencies.

GitLab Project

GitLab - GitHub Synchronization

To create working GitLab to GitHub repository synchronization:

• Prepare GitHub token, let's call it GitLab GitHub Sync, with scopes:
  • repo (and repo:status, repo_deployment, public_repo, repo:invite, security_events)
  • workflow
  • write:packages (and read:packages)
  • delete:packages
• Have or create a GitHub repository
• Set up GitLab GitHub synchronization: Settings
  • Repository
    • Mirroring repositories, Expand
      • Add new mirror:
        • Git repository URL: https://[email protected]/org/repo.git, please replace user, org, and repo
        • Mirror direction: Push
        • Password: GitLab GitHub Sync token
        • Keep divergent refs: On or Off
        • Mirror only protected branches: On (in that case all release, maintenance, and pre-release branches should be set as protected, otherwise GitHub release would fail on non-existent branch) or Off

GitLab CI Settings

Set up release and GitLab CI Linter tokens as the GitLab group or the GitLab project variable:

• GL_TOKEN: GitLab Personal Access Token with scopes api and write_repository.

  • If the variable is protected, GitLab CI job lint is skipped on non-protected branches.

• GH_TOKEN: GitHub Personal Access Token with at least scopes repo for a private repository or public_repo for a public repository.

  • If the variable is protected, then releasing to GitHub works only from protected branches.

• Settings

  • CI/CD
    • Variables, Expand
      • Add Variable:
        • Key: GL_TOKEN or GH_TOKEN
        • Value: token
        • Flags:
          • Protect variable: On (GitLab & GitHub Releases and GitLab CI Linter will work only on protected branches) or Off (insecure - accessible to anybody, who can create a commit in GitLab)

GitLab CI Nightly Pipeline

Set up the GitLab scheduled pipeline:

• CI/CD
  • Schedules
    • New schedule
      • Fill and Save pipeline schedule

Usage

Simply fork the repository at GitLab or GitHub, delete all git tags, and tag the last commit to the desired starting version, e.g. v0.0.0. Clone the repository with --recursive option, run sudo scripts/bootstrap, scripts/setup, scripts/update, at .pre-commit-config.yaml replace gitlab-ci-linter project with your project, and enjoy!

• git commit, or git merge runs checks on changed files and runs fast test set
• git push runs checks on all files and runs reduced test set
• GitLab push, merge request runs checks on all files and runs full test set
• GitLab merge to main runs checks on all files, runs full test set, and publishes a new version release
• GitLab schedule runs checks on all files, runs nightly test set
• Run scripts/update manually from time to time to update repository dependencies

Usage Examples

For usage examples, you might take a look at:

• GitHub - xebis/shellib: Simple Bash scripting library. - example of version bumping and creating deb package
• GitHub - xebis/infrastructure-template: Template for automated GitOps and IaC in a cloud. GitLab CI handles static and dynamic environments. Environments are created, updated, and destroyed by Terraform, then configured by cloud-init and Ansible. - example of GitOps (IaC + MRs + CI/CD) and multiple environments orchestration
• GitHub - xebis/xebis-ansible-collection: A collection of Xebis shared Ansible roles.

Contributing

Please read CONTRIBUTING for details on our code of conduct, and the process for submitting merge requests to us.

Testing

• Git hooks check a lot of things for you, including running automated tests scripts/test full

• Make sure all scripts/*, git hooks, and GitLab pipelines work as expected, testing checklist:

• scripts/* scripts - covered by unit tests tests/*

  • scripts/bootstrap
  • scripts/pre-commit
  • scripts/pre-push
  • scripts/setup
  • scripts/test
  • scripts/update

• Local working directory

  • git commit runs pre-commit hook-type commit-msg and scripts/pre-commit
  • git merge
    • Fast-forward shouldn't run any hooks or scripts
    • Automatically resolved merge commit runs pre-commit hook-type commit-msg and scripts/pre-commit
    • Manually resolved merge commit runs pre-commit hook-type commit-msg and scripts/pre-commit
  • git push runs scripts/pre-push
  • pre-commit run -a --hook-stage manual runs all hooks and check-hooks-apply hook fails on check-symlinks and forbid-binary

• GitLab CI

  • Commit in non-main branch runs validate:lint and validate:test-*-full
  • Merge to main branch runs validate:lint, validate:test-*-full, and release:release
    • With a new feat or fix commit releases a new version in GitHub and GitLab
      • GitHub - xebis/repository-template ∙ Releases
      • GitLab - xebis/repository-template ∙ Releases
    • Without a new feature or fix commit does not release a new version
  • Scheduled (nightly) pipeline runs validate:lint and validate:test-*-nightly

Test at Docker Container

To test your changes in a different environment, you might try to run a Docker container and test it from there.

Run a disposal Docker container:

• sudo docker run -it --rm -v "$(pwd)":/repository-template alpine:latest
• sudo docker run -it --rm -v "$(pwd)":/repository-template --entrypoint bash node:latest

In the container:

cd repository-template
# Set variables GL_TOKEN and GH_TOKEN when needed
# Put here commands from .gitlab-ci.yml job:before_script and job:script
# For example job test-full:
apk -U upgrade
apk add bats
bats tests
# Result is similar to:
# 1..1
# ok 1 dummy test

To-Do list

• Fix workaround for pre-commit local hook shellcheck - shellcheck has duplicated parameters from .shellcheckrc, because these are not taken into account

Roadmap

• Find a satisfactory way how to manage (list, install, update) dependencies across various distributions and package managers
• Add jumanjihouse/pre-commit-hooks hook protect-first-parent
• Speed up CI/CD by preparing a set of Docker images with pre-installed dependencies for each CI/CD stage, or by cache for apk, pip, and npm

Credits and Acknowledgments

• Martin Bružina - Author

Copyright and Licensing

• MIT License
• Copyright © 2021 Martin Bružina

Changelog and News

• Changelog

Notes and References

Dependencies

• git
• GitLab: The complete DevOps platform
  • GitLab: Continuous Integration (CI) with GitLab
  • GitLab: GitLab Runner
• Docker Hub - Alpine
• Docker Hub - Node
• GitHub - semantic-release/semantic-release: Fully automated version management and package publishing
• pre-commit: A framework for managing and maintaining multi-language pre-commit hooks
  • pre-commit: Supported hooks
• GitHub - pre-commit/pre-commit-hooks: Some out-of-the-box hooks for pre-commit
• GitHub - jumanjihouse/pre-commit-hooks: git pre-commit hooks that work with http://pre-commit.com/
• GitHub - jorisroovers/gitlint: Linting for your git commit messages
• GitHub - codespell-project/codespell: check code for common misspellings
• GitHub - igorshubovych/markdownlint-cli: MarkdownLint Command Line Interface
• GitHub - adrienverge/yamllint: A linter for YAML files.
• GitLab - devopshq/gitlab-ci-linter
• GitHub - mvdan/sh: A shell parser, formatter, and interpreter with bash support; includes shfmt
• GitHub - koalaman/shellcheck: ShellCheck, a static analysis tool for shell scripts
• GitHub - gitleaks/gitleaks: Scan git repos (or files) for secrets using regex and entropy key
• For scripts and hooks:
  • Tools standard in any Linux (Bash, Coreutils, Grep)
  • GitHub - xebis/shellib: Simple Bash scripting library.
  • GitHub - bats-core/bats-core: Bash Automated Testing System
    • GitHub - bats-core/bats-support: Supporting library for Bats test helpers
    • GitHub - bats-core/bats-assert: Common assertions for Bats
    • GitHub - bats-core/bats-file: Common filesystem assertions for Bats

Recommendations

• Commitizen
• readme.so: Easiest Way to Create A README
• GitHub - matiassingers/awesome-readme: A curated list of awesome READMEs
• Grammarly or Hemingway Editor
• EditorConfig
• direnv, SOPS, or HashiCorp Vault

Suggestions

• Shields.io: Quality metadata badges for open source projects
• Visual Studio Code with Extensions for Visual Studio Code:
  • DX & UX
    • One Dark Pro++ (TPack) - includes One Dark Pro, Material Icon Theme, and FiraCode Font
    • DX Enhancer Pack (EPack) - includes Project Manager, Better Comments, Markdown Preview Enhanced, file-size, Error Lens, and Path Intellisense
    • EditorConfig for VS Code
    • Gremlins tracker for Visual Studio Code
  • English, and grammar:
    • Code Spell Checker
    • Grammarly (unofficial)
  • Git, and GitLab:
    • Git Extension Pack (GPack) - includes GitLens — Git supercharged, Git File History, Conventional Commits, and Checkpoints
    • gitignore
    • Git Graph
    • Git History
    • GitLab Workflow
  • Markdown:
    • Markdown All in One - includes Github Markdown Preview, Markdown Checkboxes, Markdown Emoji, Markdown PDF, and markdownlint
    • Paste Image
    • Markdown yaml Preamble
    • HTTP/s and relative link checker
  • Bash or shell:
    • Bash IDE
    • ShellCheck
    • shell-format
    • Bats (Bash Automated Testing System)

Further Reading

• Git - Git Hooks
• Conventional Commits - Conventional Commits 1.0.0
• Semantic Versioning - Semantic Versioning 2.0.0, GitHub - romversioning/romver: Romantic Versioning Specification, or Sentimental Versioning
• Wikipedia: README
• Make a README: Because no one can read your mind (yet)
• GitHub - PurpleBooth/a-good-readme-template: A template to make good README.md
• Wikipedia: Contributing guidelines
• Wikipedia: Code of conduct
• Contributor Covenant: A Code of Conduct for Open Source Communities
• Programster's Blog: Git Workflows
  • GitHub Guides: Understanding the GitHub flow
  • GitLab Docs: Introduction to GitLab Flow
  • GitHub Docs: Introduction to GitHub Actions - contains CI/CD workflow terminology
• The GitHub Blog: Scripts to Rule Them All