D365 FO: Integration with DMF using REST API

on

 


This blog describes the method to interact with the Data Management framework using REST API to export the delta changes of an entity. The package API lets third party applications to integrate by using data packages.

Use case scenario:

The vendor changes are tracked using the “Change Tracker” functionality available on D365FO at Entity Level. Using the Change Tracker on Vendor V2 entity, the DMF Incremental export job can export the Entities which are modified from last export execution. Using the REST API, third part applications will initiate the export Job. The following image describes on high level the data flow.

D365FO: Integrating with DMF using REST API

Introduction to the terms

Data Management Framework: DMF is the new all-in-one concept introduced by Microsoft in Dynamics 365 for Finance and Operations. It supports and manages all core data management related tasks. This enables asynchronous and high-performing data insertion and extraction scenarios. Here are some examples: Interactive file-based import/export, Recurring integrations (file, queue, and so on)

Data Entity: A data entity in D365 is an abstraction from the physical implementation of database tables. A data entity is a simplified de-normalized representation of underlying tables. A data entity represents a common data concept or functionality, (e.g. Vendors V2  where the details are stored in normalized relational tables) but all details are represented in one flat view in Vendor Details data entity.

Data Package: Data Package is a simple .zip file that contains the source (import) or target data(export) itself . The zip file contains three files. The data file and the manifest files which contain metadata information of the Data Entity and the processing instructions for DMF.

Implementation Details

The integration involves the following steps

  • Enable change tracking
  • Creation of the Data Export DMF Project
  • Authentication against Azure AD
  • Interact with DMF using REST API
D365 FO Integrating with DMF using REST API Implementation Details

Enable change tracking

Change tracking enables incremental export of data from Finance and Operations by using Data management. In an incremental export, only records that have changed are exported. To enable incremental export, you must enable change tracking on entities. If you don’t enable change tracking on an entity, you can only enable a full export each time.

The vendor changes are tracked using the “Change Tracker” functionality available on Entity Level. Using the Change Tracker on Vendor V2 entity, the DMF Incremental export job can export the Entities which are modified from last export execution. The following steps are used to enable Change Tracking in D365 FO

  1. Go to Data Management work space-> Data Entities.
  2. Select the “Vendors V2” Entity for which you want to enable Change Tracking.
  3. In the Action Pane, go to Change Tracking. Select the following option:
    • Enable entire entity – It will enable tracking for all writable Data sources used in the entities. It would result in a negative performance impact on the system.
    • Enable primary table
    • Enable Custom query
D365 FO Integrating with DMF using REST API Entity

Creation of the DMF Data Project

In D365FO a batch job should be created in the Data Management Framework to export the Vendor Changes. The export will be provided with the changes happened to Vendor Records from previous successful export. Below are the steps to import or export data.

  1. Create an import or export job (more on this can be found here)
    • Define the project category (Export/Import): Export
    • Identify the entities to import or export: “Vendors V2″
    • Set the data format for the job: Excel, CSV, XML etc.
    • Determine whether to use staging tables : No
    • Group Name: VendorIncrementalChanges
    • Default refresh type: Incremental Push Only
  2. Validate that the source data and target data are mapped correctly.
D365 FO Integrating with DMF using REST API Enable change tracking

Azure AD Authentication

In order to call the D365 F&O APIs, it is necessary to authenticate with a valid access token. The token can be retrieved from Azure Active Directory using a valid Application Id and secret key, which has access to the D365FO environment. The application ID and secret key are created by registering an application in Azure Active directory.

Pre-requisite :

  1. Register an application in Azure AD and Grant access to D365FO. The detailed steps are described here. Instead of Dynamics CRM  select Dynamics ERP 
  2. Register the AAD application in D365FO
    • System administration > Setup > Azure Active Directory applications
    • Click “New” -> Enter APP-ID(created as part of the previous step), Meaningful name and User ID (the permission you would like to assign).
D365 FO Integrating with DMF using REST API Authentication
  1. The client application authenticates to the Azure AD token issuance endpoint and requests an access token.
  2. The Azure AD token issuance endpoint issues the access token.
  3. The access token is used to authenticate to the D365FO DMF and initiate DMF Job.
  4. Data from the DMF is returned to the third-party application.

HTTP Request Message to retrieve Access Token

1
2
3
4
5
6
Http Method: POST
Request URL: https://login.microsoftonline.com//oauth2/token
Parameters : grant_type: client_credentials [Specifies the requested grant type. In a Client Credentials Grant flow, the value must be client_credentials.]
client_id: Registered App ID of the AAD Application
client_secret: Enter a key of the registered application in AAD.
Resource: Enter the URL of the D365FO Url (e.g. https://dev-d365-fo-ultdeabc5b35da4fe25devaos.cloudax.dynamics.com)

The Resource URL should not have “/” in the end, othwerise you would always get access denied while accessing the target resource

C# Code to retrieve Access Token from Azure AD

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
//Azure AAD Application settings
//The Tenant URL (use friendlyname or the TenantID
static string aadTenant = "https://login.windows.net/dev.onmicrosoft.com";
//The URL of the resource you would be accessing using the access token.Please ensure / is not there in the end of the URL
static string aadResource = "https://dev-testdevaos.sandbox.ax.dynamics.com";
//APplication ID . Store them securely / Encrypted config file or secure store
static string aadClientAppId = "GUID Of the Azure application";
//Application secret. Store them securely / Encrypted config file or secure store
static string aadClientAppSecret = "Secret of the Azure application";
///
/// Retrieves an authentication header from the service.The authentication header for the Web API call.        private static string GetAuthenticationHeader()
        {
            //using Microsoft.IdentityModel.Clients.ActiveDirectory;
            AuthenticationContext authenticationContext = new AuthenticationContext(aadTenant);
            var creadential = new ClientCredential(aadClientAppId, aadClientAppSecret);
            AuthenticationResult authenticationResult = authenticationContext.AcquireTokenAsync(aadResource, creadential).Result;
            return authenticationResult.AccessToken;
        }

Interaction using REST API

The high level interaction of API calls to get the delta package via REST API is shown below.

D365 FO Integrating with DMF using REST API steps

Step 1: Export to Package:

The Export to Package API is used to initiate an export of a data package.
Request Message to invoke Export to Packge API

1
2
3
4
5
6
7
8
9
POST /data/DataManagementDefinitionGroups/Microsoft.Dynamics.DataEntities.ExportToPackage
Message Body
{
    "definitionGroupId":" The name of the data project for export.",
    "packageName":" The name of the exported data package.",
    "executionId":" The ID to use for the job. If an empty ID is assigned, a new execution ID will be created.",
    "reExecute”: True,
    "legalEntityId":" The legal entity for the data import."
}

C# Code to invoke Export to Packge API

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
string authHeader = GetAuthenticationHeader();
 HttpClient client = new HttpClient();
 client.BaseAddress = new Uri(aadResource);
 client.DefaultRequestHeaders.Clear();       
 client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", authHeader);
//Initiate the Export
 string execytionID = Guid.NewGuid().ToString();
 var payload = new DMFExport()
 {
 DefinitionGroupId = jobName,
 PackageName = packageName,
 ExecutionId =execytionID,
 ReExecute = true,
 LegalEntityId =legalEntity
m};
 var stringPayload = JsonConvert.SerializeObject(payload);
var httpContent = new StringContent(stringPayload, Encoding.UTF8, "application/json");
 var result = client.PostAsync("/data/DataManagementDefinitionGroups/Microsoft.Dynamics.DataEntities.ExportToPackage", httpContent).Result;        
 string resultContent = await result.Content.ReadAsStringAsync();
 JObject joResponse = JObject.Parse(resultContent);
 string outPut = string.Empty;
 if (result.StatusCode == System.Net.HttpStatusCode.OK)
 {
 // Successs
 }
 else
{
// failure
}

Step 2: GetExecutionSummaryStatus

The GetExecutionSummaryStatus API is used for both import jobs and export jobs. It is used to check the status of a data project execution job. The following values are possible values for the Execution Status:
Unknown / NotRun / Executing / Succeeded / PartiallySucceeded / Failed / Canceled
If the status is ‘Executing’, then submit the request after 60 second until the response gets completion status in success or failure response.

HTTP Request message to GetExecutionSummaryStatus

1
2
3
POST /data/DataManagementDefinitionGroups/Microsoft.Dynamics.DataEntities.GetExecutionSummaryStatus
BODY
{"executionId":"Execution Id Provided to the Previous Request"}

C# to invoke GetExecutionSummaryStatus API

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
int maxLoop = 15;
do
{
//"Waiting for package to execution to complete"
Thread.Sleep(5000);
maxLoop--;
if (maxLoop <= 0)
{
break;
}
//("Checking status...");
stringPayload = JsonConvert.SerializeObject(new DMFExportSummary() { ExecutionId = execytionID });
httpContent = new StringContent(stringPayload, Encoding.UTF8, "application/json");
result = client.PostAsync("/data/DataManagementDefinitionGroups/Microsoft.Dynamics.DataEntities.GetExecutionSummaryStatus", httpContent).Result;
resultContent = await result.Content.ReadAsStringAsync();
outPut = JObject.Parse(resultContent).GetValue("value").ToString();
//"Status of export is "+ outPut
}
while (outPut == "NotRun" || outPut == "Executing");

Step 3 GetExportedPackageUrl

The GetExportedPackageUrl API is used to get the URL of the data package that was exported by a call to Export Package.

Step 4 Download package File

The file can be downloaded using HTTP GET request using the URL provided in the response of the previous request. The downloaded files will be a zip file. The file needs to be extracted.  The extracted zip file contains three files but “Vendors V2.xml” would be the only file be used for processing.

Complete C# code to Interact with DMF using REST API

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
using Microsoft.Azure.Storage.Blob;
using Microsoft.IdentityModel.Clients.ActiveDirectory;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System;
using System.IO;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Threading;
 
namespace Dev.DMF.Interface
{
    public class DMFExport
    {
        [JsonProperty("definitionGroupId")]
        public string DefinitionGroupId { get; set; }
 
        [JsonProperty("packageName")]
        public string PackageName { get; set; }
 
        [JsonProperty("executionId")]
        public string ExecutionId { get; set; }
 
        [JsonProperty("reExecute")]
        public bool ReExecute { get; set; }
 
        [JsonProperty("legalEntityId")]
        public string LegalEntityId { get; set; }
    }
 
    public class DMFExportSummary
    {     
        [JsonProperty("executionId")]
        public string ExecutionId { get; set; }    
    }
    internal class DMFManager
    {
       
        static string downloadUrl = string.Empty;
 
        //Azure AAD Application settings
        //The Tenant URL (use friendlyname or the TenantID
        static string aadTenant = "https://login.windows.net/dev.onmicrosoft.com";
        //The URL of the resource you would be accessing using the access token
        //Please ensure / is not there in the end of the URL
        static string aadResource = "https://dev-testdevaos.sandbox.ax.dynamics.com";
        //APplication ID . Store them securely / Encrypted config file or secure store
        static string aadClientAppId = "GUID Of the Azure application";
        //Application secret . Store them securely / Encrypted config file or secure store
        static string aadClientAppSecret = "Secret of the Azure application";     
 
        /// <summary>
        /// Retrieves an authentication header from the service.
        /// </summary>
        /// <returns>The authentication header for the Web API call.</returns>
        private static string GetAuthenticationHeader()
        {
            //using Microsoft.IdentityModel.Clients.ActiveDirectory;
            AuthenticationContext authenticationContext = new AuthenticationContext(aadTenant);
            var creadential = new ClientCredential(aadClientAppId, aadClientAppSecret);
            AuthenticationResult authenticationResult = authenticationContext.AcquireTokenAsync(aadResource, creadential).Result;
            return authenticationResult.AccessToken;
        }
 
        // Setup Step
        // - Create an export project within Dynamics called ExportVendors in company USMF before you run the following code
        // - It can of any data format XML and can include any number of data entities
        // 1. Initiate export of a data project to create a data package within Dynamics 365 for Operations
 
        private static async void Export(string jobName, string packageName, string legalEntity, string filePath, string fileName)
        {
            string authHeader = GetAuthenticationHeader();
            HttpClient client = new HttpClient();
            client.BaseAddress = new Uri(aadResource);
            client.DefaultRequestHeaders.Clear();       
            client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", authHeader);
 
            //Initiate the Export
            string execytionID = Guid.NewGuid().ToString();
            var payload = new DMFExport()
            {
                DefinitionGroupId = jobName,
                PackageName = packageName,
                ExecutionId =execytionID,
                ReExecute = true,
                LegalEntityId =legalEntity
            };
            Console.WriteLine("Initiating export of a data project...");
            var stringPayload = JsonConvert.SerializeObject(payload);
            var httpContent = new StringContent(stringPayload, Encoding.UTF8, "application/json");
            var result = client.PostAsync("/data/DataManagementDefinitionGroups/Microsoft.Dynamics.DataEntities.ExportToPackage", httpContent).Result;        
            string resultContent = await result.Content.ReadAsStringAsync();
            JObject joResponse = JObject.Parse(resultContent);
            string outPut = string.Empty;
            if (result.StatusCode == System.Net.HttpStatusCode.OK)
            {
            
                Console.WriteLine("Initiating export of a data project...Complete");
                int maxLoop = 15;
                do
                {
                    Console.WriteLine("Waiting for package to execution to complete");
 
                    Thread.Sleep(5000);
                    maxLoop--;
 
                    if (maxLoop <= 0)
                    {
                        break;
                    }
 
                    Console.WriteLine("Checking status...");
 
                    stringPayload = JsonConvert.SerializeObject(new DMFExportSummary() { ExecutionId = execytionID });
                    httpContent = new StringContent(stringPayload, Encoding.UTF8, "application/json");
 
                     result = client.PostAsync("/data/DataManagementDefinitionGroups/Microsoft.Dynamics.DataEntities.GetExecutionSummaryStatus", httpContent).Result;
                     resultContent = await result.Content.ReadAsStringAsync();
                     outPut = JObject.Parse(resultContent).GetValue("value").ToString();
                   
                    Console.WriteLine("Status of export is "+ outPut);
 
                }
                while (outPut == "NotRun" || outPut == "Executing");
 
                if (outPut != "Succeeded" && outPut != "PartiallySucceeded")
                {
                    throw new Exception("Operation Failed");
                }
                else
                {
                    // 3. Get downloable Url to download the package   
                    //    POST / data / DataManagementDefinitionGroups / Microsoft.Dynamics.DataEntities.GetExportedPackageUrl
                    stringPayload = JsonConvert.SerializeObject(new DMFExportSummary() { ExecutionId = execytionID });
                    httpContent = new StringContent(stringPayload, Encoding.UTF8, "application/json");
 
                    result = client.PostAsync("/data/DataManagementDefinitionGroups/Microsoft.Dynamics.DataEntities.GetExportedPackageUrl", httpContent).Result;
                    resultContent = await result.Content.ReadAsStringAsync();
                    downloadUrl = JObject.Parse(resultContent).GetValue("value").ToString();
                }
 
                // 4. Download the file from Url to a local folder
                Console.WriteLine("Downloading the file ...");
                var blob = new CloudBlockBlob(new Uri(downloadUrl));
                blob.DownloadToFile(Path.Combine(filePath, fileName + ".zip"), System.IO.FileMode.Create);
                Console.WriteLine("Downloading the file ...Complete");
 
 
            }
            else
            {
                Console.WriteLine("Initiating export of a data project...Failed");
            }
        }      
    }
}

Comments

Post a Comment