Have you ever had to move a lot data into Salesforce really fast?

Well, unfortunately, I think a lot of us have been in that situation and the store in the past wasn't all that great.

When I first started working with Salesforce, I wrote an API tool that would replicate the entire Salesforce object structure on SQL Server with picklists.  It was a pretty cool application.  I later extended it to bring down all the data as well, but was really disappointed when I realized how long it takes to pull down several hundred thousand records.  So, I ended up leveraging Cast Iron and the Data Loader to solve some of my integration problems. 

The Data Loader was a real life saver, but it had some severe limitations.  First, it was slow.  One thing I used to do was split my loads into 3 to 5 batches and run each batch in a seperate Data Loader instance at the same time.  While this works, it can get kind of crazy when there are a lot of files to load and splitting them up is cumbersome.

Later I leveraged integrations on CastIron, but there are a whole different set of challenges in an integration than a migration.

With the introduction of the Bulk API, Salesforce has given us a whole new set of tools that move data up to 10 times faster than before.  However, with great speed comes great brakes.  The brakes in this case have to do with governor limits that are imposed on batch sizes, number of batches per day, etc. 

The new Data Loader v17 release takes advantage of the new Bulk API and having seen some demos, I'm very impressed with the speet improvements.  What's more, the same functionality that existed in the earlier versions exists in this version.  For instance failures are logged in the result file and can be used for reprocessing later.  This is important because the Bulk API will load the entire file up to Salesforce before it begins proccessing and then it will batch the records in batches up to 10,000 at a time.  If a batch fails the rest of the batches may continued to process until all of the batches have had a chance to run or a governor limit is reached on the number of batches allowed.

By the way, each batch must finish in a given time frame, I believe I heard 10 minutes, or it will fail so if you've got triggers, etc, just be aware that you may need to reduce your batch size to work with this.

In any case, the changes with the Batch API are a significant new feature and have definately helped with data migration using the Data Loader.