Variable substitution

CHANGELOG.md

@@ -11,6 +11,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Builds are now reproducible.
 - Provide checksums of prebuild artifacts.
+- `import.var-substitution=true` to enable substitution of environment variables or system properties. (default: false)
+- Multiple file formats could be detected by file ending
 
 ### Changed
 

README.md

@@ -17,6 +17,15 @@ But keep your files as small as possible. Remove all UUIDs and all stuff which i
 [moped.json](./contrib/example-config/moped.json) is a full working example file you can consider.
 Other examples are located in the [test resources](./src/test/resources/import-files).
 
+### Variable Substitution
+
+keycloak-config-cli supports variable substitution of config files. This could be enabled by `import.var-substitution=true` (disabled by default).
+Use variables like `${sys:name-of-system-property}` or `${env:NAME_OF_ENVIRONMENT}` to replace the values with java system properties or environment variables.
+
+The variable substitution is running before the json parser gets executed. This allows json structures or complex values.
+
+See: [Apache Common StringSubstitutor](https://commons.apache.org/proper/commons-text/apidocs/org/apache/commons/text/StringSubstitutor.html) for more information and advanced usage.
+
 ## Supported features
 
 See: [docs/FEATURES.md](./docs/FEATURES.md)
@@ -111,10 +120,11 @@ Checkout helm docs about [chart dependencies](https://helm.sh/docs/topics/charts
 | keycloak.availability-check.enabled | Wait until keycloak is available                                                  | `false`     |
 | keycloak.availability-check.timeout | Wait timeout for keycloak availability check                                      | `120s`      |
 | import.path                         | Location of config files (if location is a directory, all files will be imported) | `/config`   |
+| import.substitution                 | Enable variable substitution config files                                         | `false`     |
 | import.force                        | Enable force import of realm config                                               | `false`     |
 | import.cache-key                    | Cache key for importing config.                                                   | `default`   |
 | import.state                        | Enable state management. Purge only resources managed by kecloak-config-cli.      | `true`      |
-| import.file-type                    | Format of the configuration import file. Allowed values: JSON/YAML                | `json`      |
+| import.file-type                    | Format of the configuration import file. Allowed values: AUTO,JSON,YAML           | `auto`      |
 | import.parallel                     | Enable parallel import of certain resources                                       | `false`     |
 
 See [application.properties](src/main/resources/application.properties) for all available settings.

contrib/native/test-with-import-files.sh

@@ -8,6 +8,14 @@ export SPRING_PROFILES_ACTIVE=dev
 ./target/keycloak-config-cli-native
 ./target/keycloak-config-cli-native --import.force=true
 
+./target/keycloak-config-cli-native \
+  -Dkcc.junit.display-name=DISPLAYNAME \
+  -Dkcc.junit.verify-email=true \
+  -Dkcc.junit.not-before=1200 \
+  -Dkcc.junit.browser-security-headers="{\"xRobotsTag\":\"noindex\"}" \
+  --import.path="src/test/resources/import-files/realm-substitution/0_create_realm.json" \
+  --import.var-substitution=true
+
 while read -r file; do
   ./target/keycloak-config-cli-native --import.path="${file}"
 done < <(
@@ -17,6 +25,11 @@ done < <(
     ! -path '*/cli/*' \
     -and ! -path '*exported-realm*' \
     -and ! -path '*parallel*' \
+    -and ! -path '*realm-substitution*' \
+    -and ! -path '*realm-file-type/yaml*' \
+    -and ! -path '*realm-file-type/json*' \
+    -and ! -path '*realm-file-type/invalid*' \
+    -and ! -path '*realm-file-type/syntax-error*' \
     -and ! -name '*invalid*' \
     -and ! -name '*try*' | sort -n
 )

docs/FEATURES.md

@@ -58,20 +58,24 @@
 `authenticationFlowBindingOverrides` on client is configured by Keycloak like this,
 
 ```json
-"authenticationFlowBindingOverrides": {
-    "browser": "ad7d518c-4129-483a-8351-e1223cb8eead"
-},
+{
+    "authenticationFlowBindingOverrides": {
+        "browser": "ad7d518c-4129-483a-8351-e1223cb8eead"
+    }
+}
 ```
 
 In order to be able to configure this in `keycloak-config-cli`, we use authentication flow alias instead of `id` (which is not known)
 
-`keycloak-config-cli` will automatically resolves the alias reference to its ids.
+`keycloak-config-cli` will automatically resolve the alias reference to its ids.
 
 So if you need this, you have to configure it like :
 
 ```json
-"authenticationFlowBindingOverrides": {
-    "browser": "my awesome browser flow"
-},
+{
+    "authenticationFlowBindingOverrides": {
+        "browser": "my awesome browser flow"
+    }
+}
 ```
 

pom.xml

@@ -74,6 +74,7 @@
 
         <maven-replacer.version>1.5.3</maven-replacer.version>
         <junit5-system-exit.version>1.0.0</junit5-system-exit.version>
+        <junit-pioneer.version>0.9.1</junit-pioneer.version>
 
         <pmd.version>3.13.0</pmd.version>
         <spotbugs-plugin.version>4.0.4</spotbugs-plugin.version>
@@ -179,6 +180,12 @@
             <artifactId>commons-codec</artifactId>
         </dependency>
 
+        <dependency>
+            <groupId>org.apache.commons</groupId>
+            <artifactId>commons-text</artifactId>
+            <version>1.9</version>
+        </dependency>
+
         <dependency>
             <groupId>net.jodah</groupId>
             <artifactId>failsafe</artifactId>
@@ -227,6 +234,12 @@
             <version>${junit5-system-exit.version}</version>
             <scope>test</scope>
         </dependency>
+        <dependency>
+            <groupId>org.junit-pioneer</groupId>
+            <artifactId>junit-pioneer</artifactId>
+            <version>${junit-pioneer.version}</version>
+            <scope>test</scope>
+        </dependency>
     </dependencies>
 
     <build>

src/main/java/de/adorsys/keycloak/config/properties/ImportConfigProperties.java

@@ -37,6 +37,9 @@ public class ImportConfigProperties {
     @NotBlank
     private final String path;
 
+    @NotNull
+    private final boolean varSubstitution;
+
     @NotNull
     private final boolean force;
 
@@ -54,8 +57,9 @@ public class ImportConfigProperties {
 
     private final ImportManagedProperties managed;
 
-    public ImportConfigProperties(String path, boolean force, String cacheKey, boolean state, ImportFileType fileType, boolean parallel, ImportManagedProperties managed) {
+    public ImportConfigProperties(String path, boolean varSubstitution, boolean force, String cacheKey, boolean state, ImportFileType fileType, boolean parallel, ImportManagedProperties managed) {
         this.path = path;
+        this.varSubstitution = varSubstitution;
         this.force = force;
         this.cacheKey = cacheKey;
         this.state = state;
@@ -72,6 +76,10 @@ public boolean isForce() {
         return force;
     }
 
+    public boolean isVarSubstitution() {
+        return varSubstitution;
+    }
+
     public String getCacheKey() {
         return cacheKey;
     }
@@ -93,6 +101,7 @@ public boolean isParallel() {
     }
 
     public enum ImportFileType {
+        AUTO,
         JSON,
         YAML
     }

src/main/java/de/adorsys/keycloak/config/provider/KeycloakImportProvider.java

@@ -28,12 +28,15 @@
 import de.adorsys.keycloak.config.model.RealmImport;
 import de.adorsys.keycloak.config.properties.ImportConfigProperties;
 import de.adorsys.keycloak.config.util.ChecksumUtil;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.commons.text.StringSubstitutor;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import org.springframework.stereotype.Component;
 
 import java.io.File;
 import java.io.IOException;
+import java.nio.charset.StandardCharsets;
 import java.nio.file.Files;
 import java.util.*;
 import java.util.stream.Collectors;
@@ -42,6 +45,10 @@
 public class KeycloakImportProvider {
     private static final Logger logger = LoggerFactory.getLogger(KeycloakImportProvider.class);
 
+    private static final StringSubstitutor interpolator = StringSubstitutor.createInterpolator()
+            .setEnableSubstitutionInVariables(true)
+            .setEnableUndefinedVariableException(true);
+
     private final ImportConfigProperties importConfigProperties;
 
     public KeycloakImportProvider(
@@ -84,7 +91,7 @@ public KeycloakImport readRealmImportsFromDirectory(File importFilesDirectory) {
         return new KeycloakImport(realmImports);
     }
 
-    private KeycloakImport readRealmImportFromFile(File importFile) {
+    public KeycloakImport readRealmImportFromFile(File importFile) {
         Map<String, RealmImport> realmImports = new HashMap<>();
 
         RealmImport realmImport = readRealmImport(importFile);
@@ -105,31 +112,49 @@ private RealmImport readRealmImport(File importFile) {
     }
 
     private RealmImport readToRealmImport(File importFile) {
-        RealmImport realmImport;
-
         ImportConfigProperties.ImportFileType fileType = importConfigProperties.getFileType();
 
         ObjectMapper objectMapper;
+
         switch (fileType) {
             case YAML:
                 objectMapper = new ObjectMapper(new YAMLFactory());
                 break;
             case JSON:
                 objectMapper = new ObjectMapper();
                 break;
+            case AUTO:
+                String fileExt = FilenameUtils.getExtension(importFile.getName());
+                switch (fileExt) {
+                    case "yaml":
+                    case "yml":
+                        objectMapper = new ObjectMapper(new YAMLFactory());
+                        break;
+                    case "json":
+                        objectMapper = new ObjectMapper();
+                        break;
+                    default:
+                        throw new InvalidImportException("Unknown file extension: " + fileExt);
+                }
+                break;
             default:
-                throw new InvalidImportException("Unknown import file type :" + fileType.toString());
+                throw new InvalidImportException("Unknown import file type: " + fileType.toString());
         }
 
         objectMapper.enable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);
 
+        byte[] importFileInBytes = readRealmImportToBytes(importFile);
+        String importConfig = new String(importFileInBytes, StandardCharsets.UTF_8);
+
+        if (importConfigProperties.isVarSubstitution()) {
+            importConfig = interpolator.replace(importConfig);
+        }
+
         try {
-            realmImport = objectMapper.readValue(importFile, RealmImport.class);
+            return objectMapper.readValue(importConfig, RealmImport.class);
         } catch (IOException e) {
             throw new InvalidImportException(e);
         }
-
-        return realmImport;
     }
 
     private String calculateChecksum(File importFile) {

src/main/resources/application.properties

@@ -16,9 +16,10 @@ keycloak.availability-check.enabled=false
 keycloak.availability-check.timeout=120s
 keycloak.availability-check.retry-delay=2s
 import.cache-key=default
+import.var-substitution=false
 import.force=false
 import.state=true
-import.file-type=json
+import.file-type=auto
 import.parallel=false
 import.managed.authentication-flow=full
 import.managed.group=full

src/test/java/de/adorsys/keycloak/config/AbstractImportTest.java

@@ -21,17 +21,14 @@
 package de.adorsys.keycloak.config;
 
 import de.adorsys.keycloak.config.configuration.TestConfiguration;
-import de.adorsys.keycloak.config.exception.InvalidImportException;
-import de.adorsys.keycloak.config.model.KeycloakImport;
 import de.adorsys.keycloak.config.model.RealmImport;
 import de.adorsys.keycloak.config.provider.KeycloakImportProvider;
 import de.adorsys.keycloak.config.provider.KeycloakProvider;
 import de.adorsys.keycloak.config.service.RealmImportService;
 import de.adorsys.keycloak.config.test.util.KeycloakAuthentication;
 import de.adorsys.keycloak.config.test.util.KeycloakRepository;
 import de.adorsys.keycloak.config.test.util.ResourceLoader;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.BeforeEach;
+import org.junit.After;
 import org.junit.jupiter.api.MethodOrderer.OrderAnnotation;
 import org.junit.jupiter.api.TestMethodOrder;
 import org.junit.jupiter.api.extension.ExtendWith;
@@ -46,7 +43,6 @@
 
 import java.io.File;
 import java.time.Duration;
-import java.util.Map;
 
 @ExtendWith(SpringExtension.class)
 @ContextConfiguration(
@@ -97,38 +93,25 @@ abstract public class AbstractImportTest {
     @Autowired
     public KeycloakAuthentication keycloakAuthentication;
 
-    public KeycloakImport keycloakImport;
-
     public String resourcePath;
 
-    @BeforeEach
-    public void setup() {
-        if (this.resourcePath != null) {
-            File configsFolder = ResourceLoader.loadResource(this.resourcePath);
-            this.keycloakImport = keycloakImportProvider.readRealmImportsFromDirectory(configsFolder);
-        }
-    }
-
-    @AfterEach
+    @After
     public void cleanup() {
         keycloakProvider.close();
     }
 
-    public void doImport(String realmImport) {
-        RealmImport foundImport = getImport(realmImport);
-        realmImportService.doImport(foundImport);
-    }
-
-    public RealmImport getImport(String importName) {
-        Map<String, RealmImport> realmImports = keycloakImport.getRealmImports();
+    public void doImport(String fileName) {
+        RealmImport realmImport = getImport(fileName);
 
-        return realmImports.entrySet()
-                .stream()
-                .filter(e -> e.getKey().equals(importName))
-                .map(Map.Entry::getValue)
-                .findFirst()
-                .orElseThrow(() -> new InvalidImportException("Cannot find '" + importName + "'"));
+        realmImportService.doImport(realmImport);
     }
 
+    public RealmImport getImport(String fileName) {
+        File realmImportFile = ResourceLoader.loadResource(this.resourcePath + File.separator + fileName);
 
+        return keycloakImportProvider
+                .readRealmImportFromFile(realmImportFile)
+                .getRealmImports()
+                .get(fileName);
+    }
 }

src/test/java/de/adorsys/keycloak/config/properties/ImportConfigPropertiesTest.java

@@ -38,6 +38,7 @@
 @TestPropertySource(properties = {
         "spring.main.log-startup-info=false",
         "import.cache-key=custom",
+        "import.var-substitution=true",
         "import.force=true",
         "import.path=other",
         "import.state=false",
@@ -61,6 +62,7 @@ class ImportConfigPropertiesTest {
     @Test
     void shouldPopulateConfigurationProperties() {
         assertThat(properties.getPath(), is("other"));
+        assertThat(properties.isVarSubstitution(), is(true));
         assertThat(properties.isForce(), is(true));
         assertThat(properties.getCacheKey(), is("custom"));
         assertThat(properties.isState(), is(false));

src/test/java/de/adorsys/keycloak/config/provider/KeycloakProviderIT.java

@@ -30,17 +30,10 @@
 import static org.hamcrest.Matchers.matchesPattern;
 import static org.junit.jupiter.api.Assertions.assertThrows;
 
-class KeycloakProviderIT extends AbstractImportTest {
-    @Override
-    @SuppressWarnings("unused")
-    public void setup() {
-    }
-}
-
 @TestPropertySource(properties = {
         "keycloak.url=https://@^",
 })
-class KeycloakProviderInvalidUrlIT extends KeycloakProviderIT {
+class KeycloakProviderInvalidUrlIT extends AbstractImportTest {
     @Test
     void testInvalidUrlException() {
         KeycloakProviderException thrown = assertThrows(KeycloakProviderException.class, keycloakProvider::get);
@@ -55,7 +48,7 @@ void testInvalidUrlException() {
         "keycloak.availability-check.timeout=300ms",
         "keycloak.availability-check.retry-delay=100ms",
 })
-class KeycloakProviderTimeoutIT extends KeycloakProviderIT {
+class KeycloakProviderTimeoutIT extends AbstractImportTest {
     @Test
     void testTimeout() {
         KeycloakProviderException thrown = assertThrows(KeycloakProviderException.class, keycloakProvider::get);