Json placeholder string explained

When you write a new benchmark specification in a json file, you may want to specify the same value in different fields. You may also want to specify create temporary placeholders between different commands. FAI-PEP provides a mechanism to replace a string surrounded with curly braces ({}) with a replacement string. It provides flexibility and modularity.

Due to implementation restriction, the placeholder names cannot contain dot '.'.

Reference a redefined field

Some patten strings are predefined and can be referenced in any field of the json file. Here is a list of them:

• {HOSTDIR}: a temporary directory generated on the host platform.
• {TGTDIR}: a directory on the target platform.
• {FAIPEPROOT}: the root directory of the github clone of FAI-PEP.
• {INDEX}: if the json file has a field repeat within model, the test is repeated repeat times. {INDEX} is the index for the current iteration within repeat, in the range of 0 to repeat-1.

Reference a field parsed from the command line

If you want to pass some values to the benchmark, you can specify a json string using flag --string_map. An example of the command line is:

./benchmarking/run_bench.py -b specifications/models/caffe2/squeezenet/squeezenet_accuracy_imagenet.json --string_map '{"imagenet_dir": "~/caffe2/imagenet"}'

The {imagenet_dir} in json file specifications/models/caffe2/squeezenet/squeezenet_accuracy_imagenet.json is replaced with ~/caffe2/imagenet.

Reference a file generated during build

In the build, one main benchmark runtime is generated. That binary can be referenced as {program} in the commands.

In addition, it is also possible to build multiple binaries in the build step. Besides the main runtime, other binaries can be placed in the same directory as the main binary (needs to be manually done in the build step). an example can be found here.

Those binaries can be referenced as the name of the binary with curly braces surrounded ({<binary_name>}).

Reference a field in the json file

The strings in the commands field can reference other fields in the json file. The harness first checks the name of the keys in the same test, if not found, it checks the fields in the model field. When referencing the value of an inner field, you need to specify all the names joined with .. Here is one example:

{
  "model": {
    "name": "SqueezeNet",
    "model_name": "squeezenet"
  },
  "tests": [
   {
      "name": "test_name",
      "files": {
        "data": {
          "name": "data"
        }
      },
      "commands": [
        "test name is {name}, model name is {model_name}, and file name is {files.data.name}"
      ]
    }
  ]
}

Here {name} is replaced with test_name since it is the root field of the test. The {name} in model is shadowed by the {name} in test so that field cannot be referenced. {model_name} is replaced with squeezenet since there is no field model_name under the test but field model_name can be found within model. {files.data.name} is replaced with data since it is an inner field under test.