SugarCRM: Bulk API Explained

Recently I have found myself in various conversations revolving around the topic of the Bulk API and its capabilities. These conversations have helped highlight various misunderstandings relating to its functionality, which I believe are the result of a lack of documentation on the subject. Hopefully this post will help clarify some of the more common themes.

To begin with, let us discuss what the Bulk API is not.

It is not a separate interface to Sugar. It is merely another endpoint of the existing REST v10 API. In short, the moniker Bulk API is a misnomer, as the term is just referring to a component of the REST v10 API, not an alternative to it.

Quite regularly I also hear of attempts to use the Bulk API to perform mass insertions of data. For example, migrating hundreds of thousands of records from another system and into Sugar. While the Bulk API can help expedite the process, it is not specifically designed for that task and its benefits for that type of work are limited. 

What is the purpose of the Bulk API endpoint?

The Bulk API endpoint aims at helping developers reduce the quantity of web requests sent to the REST v10 API. By reducing the number of requests necessary to complete a programmatic task, the time to completion can also be reduced.

The following example helps demonstrate its value quite well.



Suppose you are developing code that requires you to create 10 new records in a Sugar module. Typically, your code would make one request per record, along the lines of:

POST /rest/v10/<SomeModule>

BODY: {JSON Payload}

Under this model, as the volume of data increases, so do the number of required web requests.

Leveraging the Bulk API allows us to reduce the number of required web requests to complete the same task by allowing us to bundle multiple requests into one. The following JSON payload extracted from the Sugar API Help system demonstrates this very point:

{"requests":
[
  {
    "url": "/v10/Accounts", "method": "POST", "data": "{\"name\": \"Acme\"}"
  },
  {
    "url": "/v10/Accounts", "method": "GET"
  }
]
}

This entire payload actually instructs Sugar to perform 2 different tasks, but through a single web request. As a result, an action that would normally take 2 web requests to complete is reduced to a single web request. If we apply this same principle to the previous example, we could potentially create all 10 records via a single request, instead of 10.

Also note that the type of requests in the payload can vary. In this example, we see a POST and a GET request being executed by the same request to the Bulk API.

Lastly, it is important to note that the Bulk API does not perform any sort of throttling. 

This is another important point to bear in mind if your intention is to use it to perform mass data inserts. The size of the body in the request is subject to limits controlled primarily by your web server. As such, they can vary from server to server, and in all cases the Bulk API assumes that the payload you are providing does not exceed the limit. If you do POST a request that exceeds the limit, you should expect to see an error and failure to complete the request.

The Bulk API endpoint can be accessed by sending a POST request to /rest/v10/bulk and is available in Sugar 7.5 and higher.