Dataverse Search API v2

Microsoft have just announced an updated version of the API for Dataverse Search.

The original version of this API allowed you to perform searches and get autocomplete suggestions, but could only be used from the Web API. This prevented SDK applications and plugins from using this powerful search feature.

With the new version, not only are there some new features available to get information about the search index such as its size and how recently the data for each entity type was indexed, but it’s also all now available to use from the SDK! That opens up Dataverse Search to be used from plugins and any external integrations too.

There’s some great examples available on Microsoft Learn and GitHub on how to use the new SDK messages. They’re a little more complicated to use than most as you have to handle JSON serialization for the request and response parameters, but that’s all taken care of in the sample code.

The samples use Newtonsoft to handle the JSON serialization, so if you’re using that in a plugin you’ll need to use the new Plugin Package system rather than a single assembly. Alternatively you might want to use the DataContractJsonSerializer class instead to avoid additional dependencies:

public class SearchPlugin : IPlugin
{
    public void Execute(IServiceProvider serviceProvider)
    {
        var orgFactory = (IOrganizationServiceFactory)serviceProvider.GetService(typeof(IOrganizationServiceFactory));
        var org = orgFactory.CreateOrganizationService(null);

        var req = new OrganizationRequest("searchquery")
        {
            Parameters = new ParameterCollection
            {
                ["search"] = "data8",
                ["count"] = true,
                ["entities"] = Serialize(new[] {
                    new SearchEntity
                    {
                        Name = "account",
                        SelectColumns = new List<string> { "accountid", "name", "ownerid" }
                    }
                }),
                ["filter"] = "statecode eq 0",
                ["orderby"] = Serialize(new[] { "createdon desc" })
            }
        };

        var resp = org.Execute(req);
        var results = Deserialize<SearchQueryResults>((string)resp.Results["response"]);

        // TODO: Use the results
    }

    private string Serialize<T>(T obj)
    {
        var serializer = new DataContractJsonSerializer(typeof(T));

        using (var stream = new MemoryStream())
        {
            serializer.WriteObject(stream, obj);

            return Encoding.UTF8.GetString(stream.ToArray());
        }
    }

    private T Deserialize<T>(string json)
    {
        var serializer = new DataContractJsonSerializer(typeof(T));

        using (var stream = new MemoryStream(Encoding.UTF8.GetBytes(json)))
        {
            return (T)serializer.ReadObject(stream);
        }
    }
}

One thing that’s worth calling out is the request limiting. This API has a much lower limit on the number of requests compared to the normal Dataverse API. You can only make one request per user per second, and 150 per organization per minute.

The documentation indicates you’ll get an HTTP 429 error if you go over this limit. In terms of an SDK app though, this will surface as a FaultException:

try
{
  svc.Execute(new searchquery
  {
    // ...
  });
}
catch (FaultException<OrganizationServiceFault> ex)
{
  if (ex.Detail.ErrorDetails.TryGetValue("ApiExceptionHttpStatusCode", out var details) &&
      details is int statusCode &&
      statusCode == 429)
  {
    // We hit the rate limit. The Retry-After header isn't available directly so we need to parse
    // it from the error message
    var retryAfterPos = ex.Message.IndexOf("Retry-After: ");
    if (retryAfterPos != -1)
    {
      var retryAfter = TimeSpan.Parse(ex.Message.Substring(retryAfterPos));
      // TODO: Retry the operation after this delay
    }
    else
    {
      // Couldn't find the suggested retry delay
      // TODO: Retry the operation after a fixed period
    }
  }
  else
  {
    // Something else went wrong...
  }
}