Exploring change data capture techniques

23/05/2024

Change Data Capture

Change Data Capture represents the changes in salesforce records. It acts as a real-time notification system for changes in Salesforce, allowing external databases to stay up-to-date. Receive near-real-time changes of Salesforce records, and synchronize corresponding records in an external data store. Change may be like create, update, delete and undelete of the record.

CDC are available in salesforce classic, Lightning Experience and also available in Enterprise, Performance, Unlimited, and Developer editions.

The CDC Salesforce securely sends real-time updates (events) to other systems, even for millions of changes daily. These systems can access past updates for 3 days. Everything is encrypted to keep your data safe.

Select Objects for Change Notifications in the User Interface

To receive notification on the default standard channel for record change choose the custom object or standard object for the change data capture.

Go to setup, In the Quick find box enter the Change Data Capture and Click on Change Data Capture.

Here will show object lists. From the Available Entities List select the object according to a requirement (Note : You can select up to 5 entities including custom and standard objects. To enable more entities, contact your Salesforce Account Representative to purchase an add-on license).

Here is LWC example:

changeDataCapture.html


        <template>
            <lightning-card title="Employees">
                <div class="slds-p-around_small">
                    <div style="height: 300px;">
                        <lightning-datatable key-field="id" data={employeeData} columns={columns}></lightning-datatable>
                    </div>
                </div>
            </lightning-card>
        </template>

changeDataCapture.html.js


            import { LightningElement, api, wire, track } from 'lwc';
            import { subscribe, unsubscribe, onError} from 'lightning/empApi';
            import { ShowToastEvent } from "lightning/platformShowToastEvent";
            import { notifyRecordUpdateAvailable } from 'lightning/uiRecordApi';
            import { refreshApex } from '@salesforce/apex';
            import getEmployees from '@salesforce/apex/ChangeDataCaptureController.getEmployees';
            

            const COLUMNS = [
                { label: 'Name', fieldName: 'Name' },
                { label: 'FirstName', fieldName: 'First_Name__c'},
                { label: 'LastName', fieldName: 'Last_Name__c'},
                { label: 'Tenure', fieldName: 'Tenure__c'}
            ]
            export default class ChangeDataCapture extends LightningElement {
               
                @api recordId;
                columns = COLUMNS;
                @track employeeData;
                channelName = '/data/Employee__ChangeEvent';
                subscription = {};
            

                connectedCallback() {
                    this.registerErrorListener();
                    this.handleSubscribe();
                }
            

                employeesActivity;
                @wire(getEmployees)
                wiredGetEmployees(value){
                    this.employeesActivity = value;
                    const {data, error} = value;
                    if(data) {
                        this.employeeData = data;
                    } else if(error) {
                        this.toastMessage("Error", error.body.message, "error");
                    }
                }
            

                handleSubscribe() {
                    const messageCallback = (response) => {
                        console.log('New message received: ');
                        this.handleCaptureDataChange(response);
                    };
            

                    subscribe(this.channelName, -1, messageCallback).then((response) => {
                        console.log('Subscription request sent to: ',JSON.stringify(response.channel));
                        this.subscription = response;
                    });
                }
            

                handleUnsubscribe() {
                    unsubscribe(this.subscription, (response) => {
                        console.log('unsubscribe() response: ', JSON.stringify(response));
                    });
                }
            

                handleCaptureDataChange(response) {
                    console.log('----handleCaptureDataChange----');
                    if(response.hasOwnProperty("data")){
                        if(response.data.hasOwnProperty("payload")){
                            const payload = response.data.payload;
                            //Refresh the data in own system and also in external system
                            if(payload.ChangeEventHeader.changeType == 'UPDATE'){
                                 refreshApex(this.employeesActivity);
                                 this.toastMessage("Success", "Record is updated", "success");
                            }
                            console.log(`${response.data.payload.Name}, ${response.data.payload.First_Name__c}, ${response.data.payload.Last_Name__c}, ${response.data.payload.Tenure__c}`);
                        }
                    }
                }    
            

                registerErrorListener() {
                    onError((errors) => {
                        console.log('Received error from server: ', JSON.stringify(errors));
                        this.toastMessage("Error", errors.error, "error");
                    });
                }
            

                toastMessage(title, message, variant) {
                    this.dispatchEvent(
                        new ShowToastEvent({
                          title: title,
                          message: message,
                          variant: variant,
                          mode: 'dismissible'
                        }),
                    );
                }
               
                disconnectedCallback() {
                    this.handleUnsubscribe();
                }
            

            }

changeDataCapture.js-meta.xml


        <?xml version="1.0" encoding="UTF-8"?>
        <LightningComponentBundle xmlns="http://soap.sforce.com/2006/04/metadata">
            <apiVersion>59.0</apiVersion>
            <isExposed>true</isExposed>
            <targets>
                <target>lightning__RecordPage</target>
            </targets>
        </LightningComponentBundle>

Here is a sample of subscribed data receive message :


        {
          "data": {
            "schema": "8p3YM3UaAJcr6hsjct71Yw",
            "payload": {
              "LastModifiedDate": "2024-05-07T08:19:00Z",
              "ChangeEventHeader": {
                "commitNumber": 1277335655194,
                "commitUser": "0052w00000GHr0WAAT",
                "sequenceNumber": 1,
                "entityName": "Employee__c",
                "changeType": "UPDATE",
                "changedFields": [
                  "Name",
                  "LastModifiedDate",
                  "First_Name__c",
                  "Last_Name__c",
                  "Tenure__c"
                ],
                "changeOrigin": "com/salesforce/api/soap/60.0;client=SfdcInternalAPI/",
                "transactionKey": "000a4cbc-55bd-9461-abca-3d79fe804e6a",
                "commitTimestamp": 1715069940000,
                "recordIds": [
                  "a062w00000kFFPhAAO"
                ]
              },
              "First_Name__c": "Daniel",
              "Tenure__c": 1,
              "Name": "Daniel Smith",
              "Last_Name__c": "Smith"
            },
            "event": {
              "replayId": 17346826
            }
          },
          "channel": "/data/Employee__ChangeEvent"
        }

Standard Channel for All Selected Entities /data/ChangeEvents Single-Entity Channel for a Standard Object /data/ChangeEvent For example, the channel to subscribe to change events for Account records is: /data/AccountChangeEvent Single-Entity Channel for a Custom Object /data/__ChangeEvent For example, the channel to subscribe to change events for Employee__c custom object records is: /data/Employee__ChangeEvent

Change events, like platform events, are stored for a short time (3 days) in a temporary holding area. You can access these messages later if needed. Each message has a unique ID that lets you replay the stream of events starting from that specific point.

Subscription Channel:

A subscription channel is stream of change events that corresponds one or more entities. You can subscribe the channel to receive change event notification for record create, update, delete and undelete operations. Change data capture provides pre-defined standard channels and you can create own custom channel. The channel name is case-sensitve.

Standard channel:

Standard channel for all selected entities:

/data/ChangeEvents

Single-Entity channel for standard object:

/data/AccountChangeEvent

Single-Entity channel for custom object:

/data/Employee__ChangeEvent

changeDataCaptureController.cls


        public with sharing class ChangeDataCaptureController {
           
            @AuraEnabled(cacheable=true)
            public static List<Employee__c> getEmployees(){
                try {
                    return [SELECT Id, Name, First_Name__c, Last_Name__c, Tenure__c FROM Employee__c];
                } catch (Exception e) {
                    throw new AuraHandledException(e.getMessage());
                }
            }
        }

When to Use Change Data Capture?

Real-time data synchronization.
Initial (day 0) copy of the entire data set to the external system.
Reconciliation of duplicate data between the two systems.
Continuous synchronization of new and updated data to the external system.
Keep external systems in sync with Salesforce data.
Receive notifications of Salesforce record changes, including create, update, delete, and undelete operations.
Subscribe using CometD, Pub/Sub API, or Apex triggers.
Capture field changes for all records.
Get broad access to all data regardless of sharing rules.
Deliver only the fields a user has access to based on field-level security.
Encrypt change event fields at rest.
Get information about the change in the event header, such as the origin of the change, which allows ignoring changes that your client generates.
Perform data updates using transaction boundaries.
Use a versioned event schema.
Subscribe to mass changes in a scalable way.
Get access to retained events for up to three days.

When not to use Change Data Capture?

Perform audit trails based on record and field changes.
Update the UI for many users in apps subscribed with CometD or Pub/Sub API. Change Data Capture is intended to keep downstream systems in sync but not individual users. If many users are subscribed with CometD or Pub/Sub API clients, the concurrent client limit can be hit. For more information, see Change Data Capture Allocations.

Overall, CDC offers a versatile approach to data management, promoting real-time insights, efficient workflows, and robust data governance. However, it's important to consider potential drawbacks like resource consumption and implementation complexity when making a decision. Change Data Capture (CDC) in Salesforce is like having a watchful eye over your data, alerting you whenever something changes. It's a feature that tracks alterations in your Salesforce data and then sends out notifications about those changes. Think of it as having a guardian that keeps you informed about any updates, additions, or deletions in your data, helping you stay on top of things and make informed decisions.