As of this writing, aerospike-operator allows for performing Aerospike version upgrades in the Aerospike clusters it manages. Version upgrades are performed by the administrator by changing the .spec.version field of the corresponding AerospikeCluster resource. Upon being requested, the transition between the source and the target versions is validated by aerospike-operator against a whitelist of supported transitions. If the transition is valid and supported, the upgrade process is performed as described in the Upgrades design document (i.e. by deleting and re-creating each pod in the Aerospike cluster using the target version). However, and as described in this same document, there have historically been both major and minor versions of Aerospike that did require some special upgrade steps. For example, an upgrade from pre-4.2.0.2 to 4.2.0.2 or newer requires storage devices or files to be completely erased before re-starting each node using the new Aerospike version (as a result of changes made to Aerospike’s internal storage format). Supporting such a transition in aerospike-operator would require the current upgrade mechanism to be enhanced in order to support special upgrade steps. Introducing these enhancements to the upgrade mechanism at this point will allow for greater flexibility and reduced complexity in the future, when new versions of Aerospike requiring special steps are introduced.
-
Provide support for performing special upgrade steps when upgrading an Aerospike cluster, based on the specified source and target versions.
In order to introduce support for special upgrade steps to the current upgrade mechanism, the concept of upgrade strategy is introduced. An upgrade strategy is an extensible set of properties that are passed to the reconciler and used to make certain decisions and trigger certain behaviours inside the reconcile loop. An example of such an upgrade strategy is the following:
type UpgradeStrategy struct {
// RecreatePersistentVolumeClaims indicates whether the
// existing PVCs should be deleted and re-created before
// re-creating a pod.
RecreatePersistentVolumeClaims bool
}Introducing such a set of properties will allow aerospike-operator to represent and adequately support the abovementioned transition between pre-4.2.0.2 and 4.2.0.2 (or newer) by using the following upgrade strategy:
To42XYStrategy := UpgradeStrategy{RecreatePersistentVolumeClaims: true}For all other transitions where no special action is required, the following upgrade strategy can be used:
DefaultStrategy := UpgradeStrategy{RecreatePersistentVolumeClaims: false}From the point this extensible set of properties is introduced, upgrading an Aerospike cluster to a later version becomes a matter of:
-
Deciding what upgrade strategy to use based on the source and target versions.
-
Modifying the reconciler’s behaviour in order to properly observe the upgrade strategy.
aerospike-operator currently validates transitions between a source and target versions using the VersionUpgrade struct. A new GetStrategy method will be added to this struct that will return the adequate upgrade strategy for the transition. For instance, the upgrade strategy for a transition between 4.0.0.4 and 4.2.0.2 will be obtained by:
upgrade := versioning.VersionUpgrade{sourceVersion, targetVersion}
us, err := upgrade.GetStrategy()After executing the code above, us.RecreatePersistentVolumeClaims will be true, and err will be nil. Similarly, the upgrade strategy for a transition between 4.0.0.4 to 4.0.0.5 will be obtained using the same code (with a different value for targetVersion) and, in this case, us.RecreatePersistentVolumeClaims will be false. In a scenario where an upgrade between sourceVersion and targetVersion is not allowed, us will be nil and err will contain an error. In order to keep things simple and reduce the amount of code to a minimum, a default upgrade strategy will be implemented and will be returned whenever a given transition does not require special upgrade steps. This default upgrade strategy will correspond to the current behaviour of the reconciler (i.e. the behaviour as of 0.7.0).
The choice of the upgrade strategy for a given transition will be made according to a set of rules such as the following:
// when upgrading from a pre-4.2.X.Y version to 4.2.X.Y or newer existing
// data must be erased, so we delete and re-create existing persistent
// volume claims.
// https://www.aerospike.com/docs/operations/upgrade/storage_to_4_2
if vu.Target.Major == 4 && vu.Source.Minor <= 1 && vu.Target.Minor >= 2 {
return To42XYStrategy, nil
}
return DefaultStrategy, nilThis allows for covering more version transitions with less code. Once the two mentioned upgrade strategies are defined, the reconcile loop will be modified in order to support them.
The current method for provisioning PV/PVCs will be slightly refactored in order to allow the proper implementation of a method for re-creating new PVCs. This may include using a different naming strategy for the PVCs of each pod so that, instead of deleting and re-creating each PVC, aerospike-operator can simply create new ones and leave the deletion of the old ones to the garbage collector [1]. The set of labels that is added to each PVC will also be revisited as part of this effort. Regardless of the actual implementation details, aerospike-operator will ensure that the new pods will never be started using the old PVCs.
The introduction of such a mechanism and behaviour at this point naturally paves the way for extensibility in the future. New versions of Aerospike requiring special, different upgrade steps will be supported in the same fashion by later versions of aerospike-operator to be released as required. For an analysis of an alternative approach that was considered, one can have a look at the Alternatives Considered section.
Instead of using a GetStrategy method as described above, a matrix containing supported version transitions and the associated upgrade strategies was considered. A representation of such a matrix could be the following:
| Source\Target | 4.0.0.4 | 4.0.0.5 | 4.1.0.1 | 4.2.0.3 | 4.2.0.4 |
|---|---|---|---|---|---|
4.0.0.4 |
- |
S0 |
S0 |
S1 |
S1 |
4.0.0.5 |
- |
- |
S0 |
S1 |
S1 |
4.1.0.1 |
- |
- |
- |
S1 |
S1 |
4.2.0.3 |
- |
- |
- |
- |
S0 |
4.2.0.4 |
- |
- |
- |
- |
- |