How to: Bulk load and concurrent processing
This template supports data bulk load with high efficient parallel processing.
To adjust the max number of concurrent processing, use the property
maxConcurrency. This property is defined in each environment’s configuration YAML file. It is recommended that the value of maxConcurrency no more than 4, if the worker size is 0.2 vCore or lower.
Sequence of the load
Due to hierarchical relationship among different entities, the data load needs to follow a specific sequence. Before loading the Employee data, please make sure that the Organization and Location data is loaded.
To perform the initial bulk load and subsequent loads
This template queries Workday for all existing locations in the scheduled run. Delta load processing is not covered in this template as it is not supported by Workday's getLocation SOAP service.
In Workday, timezone is available only for business site locations. Other locations, such as Floor, Space, etc., do not have a timezone assigned, which means it is not possible to populate the Timezone field for all locations during the migration. Therefore, before each data load run, it is required to disable the Timezone validation rule in Salesforce. After the location Migration is completed, this Timezone validation rule could be enabled again, if necessary.
The following is the step to disable the Timezone validation rule:
- Login the work.com
- Go to Setup -> Object Manager -> Select Location -> Select Validation Rules -> Select Edit the Timezone_Required_Field
In some situations, the definition of location type and time zone may be different between Workday and Salesforce, so two csv files (locationmapping.csv and timezonemapping.csv located inside the scr/main/resources) are used for the mapping conversion. The first row represents Salesforce and the second row represents Workday. The below provides an example:
Once the location data is loaded, timzeone can be associated with a location by following the below steps:
- Open the Item, Locations.
- Under Information, click the Time Zone field.
- Select the correct time zone.
- Click Save.
Associating the Time Zone with the current Location will apply the Time Zone to any Employee records using the updated Location.
Sequence of the location data load
Location data retrieved from Workday is not organized in a hierarchical manner. This hierarchical relationship must be established using the ParentLocationId that is defined as a “Hierarchy” data type in Salesforce.
To achieve this, the Mule application will process the data from Workday in two steps:
- Stage one: Once the location data is fetched from Workday in a batch, load location information into Salesforce without ParentLocationId and store Location's Name and Location’s ParentLocationId in Anypoint Object Store with a unique key. Repeat this for each batch of results from Workday.
- Stage two: After loading location data without ParentLocationId into Salesforce successfully, retrieve key-value pairs from Object Store, which contains the Location's Name and Location’s ParentLocationId. Fetch ParentLocationId in Salesforce using Parent Location’s Id and update ParentLocationId for each location using the bulk upsert.
During the process of location data loading, the address information would be created or updated in the Address object of the Salesforce, then it uses the VisitorAddressId to associate the relationship between location and address.
Reminder: Both Organization and Location data load must be completed before Employee data load. Employee data load process builds the relationship between employee and location/organization.