GDPR Compliance

[sebastianstucke87]

Hello everyone,

In A pragmatic introduction to Event Sourcing (49:09) on YouTube, @frankdejonge answered, that there are two possible way to deal with personal data in a GDPR compliant system:

1. modify the event-stream (re-write history)
2. or save personal data separate from normal events and only store references to it in the event-stream

I'm very interested in the 2nd option and I'm wondering, what the "best" implementation of such a solution would look like.

---

Default architecture w/out GDPR:

Adding a GDPR PHP-attribute to mark properties that contain personal data:

namespace MarkupV1 {

    #[Attribute(Attribute::TARGET_PROPERTY)]
    final class GDPR
    {
        // NOP
    }


    final class AccountWasUpdatedEvent implements \EventSauce\EventSourcing\Serialization\SerializablePayload
    {
        public function __construct(
            public readonly string $myFavoriteColor,
            #[GDPR]
            public readonly string $mySocialSecurityNumber,
        ) {}

        public function toPayload(): array
        {
            return (array)$this;
        }

        public static function fromPayload(array $payload): static
        {
            return new static(
                $payload['myFavoriteColor'],
                $payload['mySocalSecurityNumber'],
            );
        }
    }
}

Using a custom MessageSerializer to redact personal data that would otherwise be stored in the event-stream. The event-dispatcher will still get the unchanged event to update projections correctly:

namespace MarkupV1 {

    /*
     * based on \EventSauce\EventSourcing\Serialization\ConstructingMessageSerializer
     */
    final class GdprMessageSerializer implements \EventSauce\EventSourcing\Serialization\MessageSerializer
    {

        // (...)

        public function serializeMessage(Message $message): array
        {
            /** @var SerializablePayload $event */
            $event = $message->payload();

            // GDPR
            $personalData = [];
            $hasEventPersonalData = false;

            // simplefied for readablity: this can not resolve nested objects or iterables.
            foreach (get_object_vars($event) as $property => $value) {
                $reflect = new ReflectionProperty($event::class, $property);
                $attributes = $reflect->getAttributes(GDPR::class);

                if ($attributes !== []) {
                    $hasEventPersonalData = true;
                    $personalData[$property] = '[REDACTED]';
                }
            }

            if ($hasEventPersonalData) {
                $redactedEvent = $event->fromPayload(array_merge($event->toPayload(), $personalData));

                // persist personal data
                $this->personalDataRepository->save($headers[Header::EVENT_ID], $personalData);
            }

            // (...)
        }
    }
}

And to reconstitute an aggregate:

namespace MarkupV1 {

    /*
     * based on \EventSauce\EventSourcing\Serialization\ConstructingMessageSerializer
     */
    final class GdprMessageSerializer implements \EventSauce\EventSourcing\Serialization\MessageSerializer
    {

        // (...)

        public function unserializePayload(array $payload): Message
        {
           // (...)

            try {
                $personalData = $this->personalDataRepository->get($payload['headers'][Header::EVENT_ID]);
                $eventAsArray = array_merge($event->toPayload(), $personalData);
                $event = $event->fromPayload($eventAsArray);
            } catch (PersonalDataNotFound) {
                // NOP
            }

            return new Message($event, $payload['headers']);
        }
    }
}

This mockup reveals multiple problems:

1. the GdprMessageSerializer is mostly copy & paste from \EventSauce\EventSourcing\Serialization\ConstructingMessageSerializer; maybe wrapping it would be a better approach?
2. resolving nested objects/iterables is very error prone and should be done with a library
3. redacting/restoring personal data from/to events can be tricky with nested objects/iterables
4. the event-id is perfectly fine as a reference; no additional IDs needed to reference the personal data per event
5. for actual GDPR compliance, additional projections can be added to help with requesting/updating/deleting personal data:

• e.g. account-id, [ { event-id: {<personal-data>} } ]
• any requests can then be handled by simply referring to an account-id (or similar).

Is this a good idea? Could this be done within the library itself?

[sebastianstucke87]

Update

I wrapped the MessageSerializer like this:

namespace MockupV2
{
    final class GdprMessageSerializer implements MessageSerializer
    {
        public function __construct(
            private readonly ConstructingMessageSerializer $eventSauceMessageSerializer,
            private readonly GdprRepository $gdprRepository,
            private readonly GdprAttributeReader $gdprAttributeReader,
        ){}

        public function serializeMessage(Message $message): array
        {
            $payload = $this->eventSauceMessageSerializer->serializeMessage($message);
            $payload['headers'][Header::EVENT_ID] ??= Uuid::uuid4()->toString();

            $messageAnalyseResult = $this->gdprAttributeReader->analyse($message);

            if ($messageAnalyseResult->hasPersonalData()) {
                // persist personal data
                $personalData = $messageAnalyseResult->personalData();
                $eventId = $payload['headers'][Header::EVENT_ID];
                $this->gdprRepository->save($eventId, $personalData);

                // redact payload
                $redactedData = $messageAnalyseResult->redactedData();

                // TODO
                $payloadAsJson = json_encode($payload['payload'], flags: JSON_THROW_ON_ERROR);
                $payloadAsArray = json_decode($payloadAsJson, true);

                $payload['payload'] = array_replace_recursive($payloadAsArray, $redactedData);
            }

            return $payload;
        }

        public function unserializePayload(array $payloadAsArray): Message
        {
            // get personal data or empty array
            $eventId = (string)@$payloadAsArray['headers'][Header::EVENT_ID];
            $personalData =  $this->gdprRepository->find($eventId);

            // restore personal data
            $originalPayload = $payloadAsArray['payload'];
            $payloadAsArray['payload'] = array_replace_recursive($originalPayload, $personalData);

            return $this->eventSauceMessageSerializer->unserializePayload($payloadAsArray);
        }
    }
}

And added a MessageAnalyseResult-DTO:

namespace MockupV2
{
    final class MessageAnalyseResult
    {
        public function __construct(
            private readonly bool $hasPersonalData,
            private readonly array $personalData,
            private readonly array $redactedData,
        ){}

        public function hasPersonalData(): bool
        {
            return $this->hasPersonalData;
        }

        public function personalData(): array
        {
            return $this->personalData;
        }

        public function redactedData(): array
        {
            return $this->redactedData;
        }
    }
}

Then I tried to resolve all (nested) properties and check, if they have the GDPR-attribute. I'm not proud of this one, but it somewhat works... As I stated above, this should be done with a proper library:

namespace MockupV2
{
    final class GdprAttributeReader
    {
        public function analyse(Message $message): MessageAnalyseResult
        {
            /** @var SerializablePayload $event */
            $event = $message->payload();

            [$redactedData, $personalData] = $this->readPropertiesWithGdprAttribute($event);

            $hasPersonalData = $personalData !== [];

            return new MessageAnalyseResult($hasPersonalData, $personalData, $redactedData);
        }

        // oh god...
        private function readPropertiesWithGdprAttribute(
            $object,
            ?string $parentProperty = null,
            array $redactedData = [],
            array $personalData = [],
            bool $isIterable = false,
            mixed $key = null,
        ): array
        {
            if (is_iterable($object)) {
                foreach ($object as $key => $item) {
                    // recursive
                    [$redactedData, $personalData] = $this->readPropertiesWithGdprAttribute(
                        object: $item,
                        parentProperty: $parentProperty,
                        redactedData: $redactedData,
                        personalData: $personalData,
                        isIterable: true,
                        key: $key
                    );
                }
            }

            if (!is_object($object)) {
                return [$redactedData, $personalData];
            }

            foreach (get_object_vars($object) as $property => $value) {
                $reflect = new ReflectionProperty($object::class, $property);
                $attributes = $reflect->getAttributes(GDPR::class);

                if ($attributes !== []) {
                    if (!$isIterable) {
                        $redactedData[$parentProperty][$property] = '[REDACTED]';
                        $personalData[$parentProperty][$property] = $value;
                    } else {
                        $redactedData[$parentProperty][$key][$property] = '[REDACTED]';
                        $personalData[$parentProperty][$key][$property] = $value;
                    }
                }

                // recursive
                [$redactedData, $personalData] = $this->readPropertiesWithGdprAttribute(
                    object: $value,
                    parentProperty: $property,
                    redactedData: $redactedData,
                    personalData: $personalData,
                    isIterable: $personalData
                );
            }

            return [$redactedData, $personalData];
        }
    }
}

[frankdejonge]

An implementation option can be to embrace GDPR in the domain itself rather than hiding it away in technical concerns (repositories serializers, etc). This would consist of passing the GDPR compliant storage to the actions where PII is collected, received, or generated.

In a bit of pseudo-code, this would look like this:

$aggregateRoot = $repository->retrieve($id);

$aggregateRoot->performAction($gdrpComplicantStorage);

function performAction(GdprStorage $gdprStorage, PersonalIdentifiableInformation $pii)
{
    $storageId = $gdprStorage->storePii([
        'key' => $pii->value(),
    ]);

    $this->recordThat(new PiiWasCollected($storageId));
}

This leaves the PII out of the information flow of events. The domain can deal with storage, retrieving, an having forgotten PIIl.

[sebastianstucke87]

Thank you, that is an interesting approach! Does this mean you would have a one $storageId per aggregate? Saving only the current value per key (and overwriting any previous values)?

In your example, how would the applyPiiWasCollected-method look like?

---

I tried to flesh it out a bit - this is the storage interface:

interface GdprStorageInterface
{
    public function save(PersonalIdentifiableInformation $pii);

    public function getOrCreate(string $aggregateId): PersonalIdentifiableInformation;

    public function delete(string $aggregateId): void;
}

A simple DTO for the moment:

final class PersonalIdentifiableInformation
{
    public function __construct(
        public readonly string $aggregateId,
        public ?string         $socialSecurityNumber = null,
        public ?string         $emailAddress = null,
    ){}
}

The aggregate:

final class UserAggregate
{
    private string $aggregateId;

    private bool $isSubscribedToNewsletter = false;

    public function subscribeToNewsletter(GdprStorageInterface $storage, string $emailAddress): void
    {
        // ensure user is currently not subscribed
        assert(!$this->isSubscribedToNewsletter);

        // update PII
        $pii = $storage->getOrCreate($this->aggregateId);
        $pii->emailAddress = $emailAddress;
        $storage->save($pii);

        $this->recordThat(new UserWasSubscribedToNewsletter($emailAddress));
    }

    public function applyUserWasSubscribedToNewsletter(UserWasSubscribedToNewsletter $event): void
    {
        $this->isSubscribedToNewsletter = true;
    }
}

Deleting PII looks like this:

final class UserAggregate
{
  
    // (...)
    
    public function deleteEmailAddress(GdprStorageInterface $storage): void
    {
        // update PII
        $pii = $storage->getOrCreate($this->aggregateId);
        $emailAddress = $pii->emailAddress;
        $pii->emailAddress = null;
        $storage->save($pii);

        $this->recordThat(new UserEmailAddressWasDeleted($emailAddress));
    }

    public function applyUserEmailAddressWasDeleted(UserEmailAddressWasDeleted $event): void
    {
        $this->isSubscribedToNewsletter = false;
    }

    public function deleteAccount(GdprStorageInterface $storage): void
    {
        // delete PII
        $storage->delete($this->aggregateId);

        $this->recordThat(new UserAccountWasScheduledForDeletion());
    }

    public function applyUserAccountWasScheduledForDeletion(UserAccountWasScheduledForDeletion $event): void
    {
        $this->isSubscribedToNewsletter = false;
    }
}

I think, most methods would require the GdprStorage. Here an example for reads... 😬:

final class UserAggregate
{

    // (...)

    public function abusingWriteModelForReads(GdprStorageInterface $storage): ExampleUserViewModel
    {
        $pii = $storage->getOrCreate($this->aggregateId);

        return new ExampleUserViewModel(
            // $this
            $this->aggregateId,
            $this->isSubscribedToNewsletter,

            // PII
            $pii->emailAddress,
            $pii->socialSecurityNumber
        );
    }
}

[sebastianstucke87]

To answer this question:

Yes, GDPR-compliance can be archived with one of these options:

1. Directly modify the event-stream (YouTube)
   • certainly doable - especially when using a DB with native JSON-fields - but generally not recommended.
2. Mark personal identifiable information on a per-event-basis and redact before persisting:
   • pros:
     • once the implementation is in place, processing PII is as simple as adding the "GDPR" PHP-attribute (or something similar)
     • full history (name changes etc.)
   • cons:
     • technical implementation is more on the heavy side
3. Save PPI separately from aggregate:
   • pros:
     • dead simple implementation (basically saving individual fields in a separate DB)
   • cons:
     • no full history (but should be doable with extra steps)
     • dealing with infrastructure concerns in aggregate
       • aggregate depends heavily on (infrastructure) service
       • bad for testing

Summary

GDPR	"Modify event-stream"	"Redact events"	"Store PPI separately"
Right of access (Art. 15)	easy (projection)	easy	easy
Right to rectification (Art. 16)	difficult	easy	easy
Right to erasure (Art. 17)	difficult	easy	easy

Additional considerations

• Encryption: On top of storing PII in a separate DB, it could also be encrypted. But this should only be done for the
  event-stream DB, not the projections (performance, full-text search etc.)
• Backups: When restoring the event-stream from a backup, deleted PPI must never be recovered.

Misc.	"Modify event-stream"	"Redact events"	"Store PPI separately"
+ has full history (good)?	yes (projection)	yes	maybe
+ deleted PII can NOT be recovered by backups (good)?	yes (migrations)	yes	yes

---

(I'll mark this question as "solved" just to get it out of the way.
If anyone wants to contribute, even ten years from now, please do! I'm still very interested in any idea/solution regarding this question! thx 😄)