To protect Finance and Operations service from outages and performance degradation due to high incoming API traffic or long running requests against the server, Microsoft has applied limits to API calls.
Service protection API limits are only available in finance & operations online service for tier2 and above environments. This feature is not available for on-premises and development environments.
There are two types of Service protection APU limits Resource-based and User-based.
Resource-based service protection API limits enforces threshold of resource consumption of environment. It is enabled from application version 10.0.19 (PU43). When the utilization of resources reaches to the level that threaten the finance & operations service performance and availability, the error message ‘This request could not be processed at this time due to system experiencing high resource utilization. ‘ along with API response 429 (Too Many Requests) is returned.
We can set the priority of throttling for Integrations in D365 F&O so that low priority integrations get throttled first then Medium and finally high priority integrations. You must be assigned System administrator or Integration priority manager role to access this Priority mapping form. The navigation path of this form in F&O is System Administration -> Setup -> Throttling priority mapping.
Service protection API limits are applied only for finance and operations OData and custom service API endpoints. Services that don’t use these endpoints aren’t subject to throttling.
User-based service protection API limits enforces below three limits per user, per application Id, per web server to ensure system performance and availability
If any of the above limit threshold is surpassed, the related error message will be returned along with 429 ‘Too Many Requests’ response.
Error Messages
Number of requests exceeded the limit of 6000 over the time window of 300 seconds.
Combined execution time of incoming requests exceeded the limit of 1200 seconds over the time window of 300 seconds. Decrease the number of concurrent requests or reduce the duration of requests and try again later.
Number of concurrent requests exceeded the limit of 52.
As shown below, User-based API limit feature is optional in Version 10.0.28 (PU53) and 10.0.29 (PU54) but it will be mandatory from 10.0.33 not 10.0.32 as per Microsoft Learn weblink provided in the end of this article. Therefore, it will be essential to test all your relevant Integrations in Test environment before moving your production to 10.0.33.
429 ‘Too Many Requests’ Response can be handled:
- By Implementing retry logic for Retry-After intervals.
- For non-interactive applications, wait for the duration of Retry-After interval before sending the request again.
- For interactive applications, display a message that server is busy, provide option to cancel the operation, don’t allow user to submit additional requests until previous request is completed.
- Decreased the total number of records that can be selected and combine up to 5000 selected operations into smaller batches.
Implementing retry logic for Retry-After intervals
After 429 error is received, you can see server calculated duration in Retry-After value which indicates the time you should wait before processing new requests.
The below C# code is to retrieve this Retry-After duration and use it to inform our system to wait for that time before processing next request.
if (!response.IsSuccessStatusCode)
{
if ((int)response.StatusCode == 429)
{
int seconds = 30;
//Try to use the Retry-After header value if it is returned.
if (response.Headers.Contains(“Retry-After”))
{
seconds = int.Parse(response.Headers.GetValues(“Retry-After”).FirstOrDefault());
}
Thread.Sleep(TimeSpan.FromSeconds(seconds));
// Retry sending the request.
}
}
The service protection API limits don’t apply to all Microsoft services. The following services are temporarily exempted from them:
- Document Routing Agent (DRA)
- Warehouse Management mobile app (WHSMobile)
- Retail Server API
- Office Integration
- Data Import/Export Framework (DIXF)
- Recurring integrations
- Data Integrator
- Dual-write
- Power Platform virtual tables for finance and operations apps
- Finance and operations apps Connector
Strategies to Maximize Throughput of your API Integrations
| Strategy | Description |
| Let the server tell you when to retry the request | As discussed above, use the Retry-After interval provided by the server. |
| Use multiple threads | Depending on the data, for quick operations, increasing the number of threads can help achieving optimum throughput. |
| Distribute workloads across multiple service principals | The service protection API limits can be reached quickly if your integration is using a single service principal to perform large bulk data operations, or if it’s an interactive client application that uses a single service principal to send all user requests to the server. Therefore, distribute workload using smaller batches across multiple service principals. |
| Avoid large batches | Start with a small batch size and then increase concurrency until you start to receive service protection API limit errors that indicate that the operation must be retried. In finance and operations apps, batch size is limited to 5,000 operations |
| Remove the affinity cookie | Azure service affinity cookie ensure request are routed to same server so cached data can be reused. Request can be routed to any eligible server when the affinity cookie is removed. |
| Optimize your connection | You can optimize the connection using the C# code provided in below link. https://learn.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/data-entities/service-protection-maximizing-api-throughput |
Comments
Post a Comment