Bypassing plugins in Dataverse: faster, simpler data imports

Data migration can be one of the most time consuming parts of deploying a new CRM system, and all that complex business logic in your shiny new plugins just complicates matters. At best they just slow down the import, or worse they change your historical data according to your latest rules.

Traditionally you’ve had to disable your plugins in the Plugin Registration Tool to avoid these problems, but this has been a manual, time-consuming and error prone process.

Today though, Microsoft have released a new option for you to bypass plugins for your imports while keeping them enabled for everything else.

First up, a quick refresh on how plugins interact with Dataverse. Each user request (creating a record, querying data, executing a custom API etc.) is represented as a message, with request and response parameters. The message flows through a pipeline of handlers:

Without any customization, the pipeline validates the values provided in the request, and then the request will be executed. For example, for an Update request, validation will check each attribute is the correct type and in the right range. If everything is OK then the execution step will actually update the record.

As a developer you can register plugins to run at various steps of the pipeline to control what happens. You could add extra validation logic, change the input or output values or call out to other external systems.

The same pipeline applies to custom API and custom actions too, except you also provide the core “execute” step. Custom actions have their synchronous workflow step, while custom APIs can select a plugin.

Because all these customizations are run within the core Dataverse platform it’s a great way to make sure that the same logic is applied regardless of whether you’re using a model driven app, canvas app, JavaScript or the SDK. Unfortunately they also add cost in terms of processing time, which isn’t great if you’re doing a large data migration.

So how can I bypass the plugins?

Instead of having to disable the plugins, run the data import and then remember to re-enable the plugins again, we can now select to bypass all plugins for particular requests. This is an all-or-nothing option though – you can’t select to bypass particular plugins but still invoke other ones. If you want to do this you’ll still need to manually disable those that you don’t want to be invoked.

At a low level you can add the new BypassCustomPluginExecution parameter to any message. For an SDK app this looks like:

var req = new CreateRequest
{
  Target = new Entity("contact")
  {
    ["firstname"] = "Mark",
    ["lastname"] = "Carrington"
  }
};

req.Parameters["BypassCustomPluginExecution"] = true;

svc.Execute(req);

Using Web API you accomplish the same thing by adding the MSCRM.BypassCustomPluginExecution header:

const contact = {
  firstname: "Mark",
  lastname: "Carrington"
};
await fetch("/api/data/v9.0/contacts",
  method: "POST",
  headers: {
    "MSCRM.BypassCustomPluginExecution": "true"
  },
  body: JSON.stringify(contact)
);

Using this approach means you would have to switch from using the common Create, Update and Delete methods on IOrganizationService to creating and executing CreateRequest, UpdateRequest and DeleteRequest messages respectively. To make this easier though you can also use the new BypassPluginExecution property on CrmServiceClient:

var svc = new CrmServiceClient("connection-string");
svc.BypassPluginExecution = true;

svc.Create(new Entity("contact")
{
  ["firstname"] = "Mark",
  ["lastname"] = "Carrington"
});

There was a bug with this in previous versions of the Microsoft.CrmSdk.XrmTooling.CoreAssembly package – make sure you’re using at least version 9.1.0.68 or you’ll still find your plugins are executed even when BypassPluginExecution is true.

Sync vs. Async plugins

This BypassPluginExecution option will bypass synchronous plugins. Any async plugins will continue to run, as these should not impact the performance of your data load. You may still want to disable those plugins though if they’re not necessary for your import as it can lead to delays getting other important system jobs processed.

Is it secure?

To be able to use the new parameter you need to have the new prvByassCustomPlugins privilege. The security role editor doesn’t show it, but the System Administrator role automatically includes it. So long as you’re not giving that role out to users that don’t really need it you should be good. And you don’t do that, right? Right?

If a user that doesn’t have this privilege tries this their request will be rejected with a standard security error:

System.ServiceModel.FaultException`1: 'Principal user (Id=<guid>, type=8, roleCount=1, privilegeCount=409, accessMode=4), is missing prvBypassCustomPlugins privilege (Id=148a9eaf-d0c4-4196-9852-c3a38e35f6a1) on OTC=0 for entity ''. context.Caller=<guid>'

With System Administrator access you could bypass the plugins anyway, simply by disabling them, so this doesn’t reduce security.

You can give other security roles the Bypass Custom Plugins privilege, but not with the standard security role editor. You can use a tool like Role Updater in XrmToolBox to add it, or there are examples in the docs on how to add it programmatically. This makes it possible to set up an appropriate security role to an application you’re using for data imports/integrations without giving the full System Administrator role.

How about custom actions & APIs?

This option skips all custom code. For custom actions this includes the core “execute” step that’s implemented as a synchronous workflow. So if you use this option on a custom action the system will validate that your input parameters are valid, then give you the output values set to their default null values without doing anything really useful.

Custom APIs however will still execute their core plugin, so another great reason to move your actions to APIs!

How does it cascade?

Sometimes, the operation you invoke can start other operations as well. For example, if you delete a record that’s linked to other records, the related records might also be deleted or updated.

If you start an operation with the BypassCustomPluginExecution parameter set, any related operations that are triggered as a result will also have their plugins bypassed.

CrmServiceClient considerations

I mentioned earlier that you can use the BypassPluginExecution property on CrmServiceClient to automatically apply this option to an entire connection, which is very helpful. However, there are a few things you need to be aware of if you use this:

1. It’s not preserved when using .Clone(). If you clone a connection (e.g. if you’re multi-threading requests) the clone will have this property set to false, so you need to set it back to true again on the clone
2. As I mentioned earlier, version 9.1.0.64 of the package contained a bug which caused plugins to still run, so make sure you’re using 9.1.0.68 or later.

How much difference does it make?

This is the crucial question of course – the entire point of bypassing plugins is to make our data import faster. The actual difference will of course change depending on how long your plugins would normally take to execute, but to demonstrate I’ve set up a few tests.

I’m going to test importing 100 contacts. In my first scenario I’ll do this without any plugins registered. Then I’ll try again with a plugin registered in the pre-operation step. My test plugin simply sleeps for 100ms before continuing to simulate doing some real work. Finally I’ll try again with this plugin still registered but with the BypassCustomPluginExecution parameter set. For each scenario I’ll also test the effect of multi-threading to see how plugins affect the platform scaling.

As expected, bypassing plugins and having no plugins registered gives very similar performance. I did wonder if there might be some overhead in the platform checking for plugins that need executing that can be bypassed at the same time, but there’s no material difference.

I get about double the throughput when I move from 1 to 2 threads, but then it levels off. It might well be hitting some rate limiting here and you might see further performance gains when running against a production system with higher limits.

As always when looking at performance, test, test, test! Your situation will be different to mine and so will the results you’ll see.