Camunda external tasks - a powerful tool for building applications with resilient and scalable architecture

image

In Tinkoff, we use the Camunda + Spring framework to develop business process automation systems . We describe business processes using BPMN (Business Process Management Notation) in the form of flowcharts.

The most commonly used element in our diagrams is service tasks (gear rectangle). Camunda supports two ways to perform service tasks :

  1. Using a synchronous call to java code.
  2. Creating an external task.

The second method allows you to perform tasks using external systems - for example, if you need to call one camunda application from another or even delegate work to any external system.

image
Example BPMN Diagram

This is useful when you intend to reuse logic in multiple applications. Or when you want to stick to microservice architecture. For example, separating services that deal with business processes from services that implement technical tasks, such as generating reports or mailing.

Along with this feature, an external task provides a scalable, resilient architecture. To understand why this happens, you first need to understand how the external task works at the BPMN and application levels.

External task in BPMN


External task involves creating a task that can be performed by an external handler. The essence of the external task pattern is that:

  1. , «» , «».
  2. camunda , , .
  3. camunda (/).

In the diagram above, I described a fictional process in which we want to get a list of users, send them an advertisement and after 2 hours calculate the number of applications after the marketing newsletter. And, if there are more than 10 applications, increase the selection for the next mailing.

I want my camunda application to be responsible only for business processes, and any other application should be responsible for emailing. In this case, the external task pattern works fine for me . In my process, I will simply create an email newsletter task and wait for it to be completed by some external handler.

To create an external task in the diagram , you must:

  1. Create a regular task .
  2. Change its type to service task .
  3. Set implementation to external .
  4. Specify the value of the Topic field .

image

Topic is the name of the queue into which tasks of one type will be added and to which an external handler will subscribe.

Now that there is an external task in the process , you can start it, but it will not be executed, since no one is processing it.

External tasks worker


The external task pattern is good in that it allows you to implement task processing in any language, using any tools that can perform HTTP requests .

The following is an example from the camunda blog . The example implements an external javascript handler that, every 20 seconds, requests a list of processing tasks from camunda . If there are tasks, then it sends them out and notifies camunda about the completion of the task.

const baseUrl = 'http://localhost:8080/my-app/rest';
const workerSettings = {
 workerId: 'worker01', // some unique name for the current worker instance
 maxTasks: 5,
 topics: [
   {
     topicName: 'sendEmail',
     lockDuration: 10000, // How much time the worker thinks he needs to process the task
     variables: ['video'] // Which variables should be returned in the response (to avoid additional REST calls to read data)
   }]};
const requestParams = {method: 'POST', headers: {contentType: 'application/json'}};

function pollExternalTasks() {
 return fetch(`${baseUrl}/external-task/fetchAndLock`, {
   ...requestParams,
   body: JSON.stringify(workerSettings)
 })
}

function processExternalTask(result = []) {
 return Promise.all(result.map(externalTask => {
   sendEmail(externalTask); // Here the actual work would be done

   return fetch(`${baseUrl}/external-task/${externalTask.id}/complete`, {
     ...requestParams,
     body: JSON.stringify({workerId: workerSettings.workerId}),
   })
 }));
}

setInterval(() => {
 pollExternalTasks().then(processExternalTask)
}, 20000);

As you can see from the code above, the key methods for handling external tasks are fetchAndLock and complete . The first method requests a list of tasks and secures their implementation, and the second informs about the completion of the task. In addition to these two methods, there are others, you can read about them in the official documentation .

Camunda external task client


image

To implement external tasks processing, camunda provided Javascript and Java clients that allow you to create external task handlers in just a few lines. There is also a detailed guide that describes the basic principles of processing external tasks - again with examples in Javascript and Java .

An example implementation of an external handler using ExternalTaskClient :

public class App {
   public static void main(String... args) {
       // bootstrap the client
       ExternalTaskClient client = ExternalTaskClient.create()
           .baseUrl("http://localhost:8080/engine-rest")
           .asyncResponseTimeout(1000)
           .build();

       // subscribe to the topic
       client.subscribe("sendEmail").handler((externalTask, externalTaskService) -> {
           try {
               String result = sendEmail(externalTask)
               Map<String, Object> variables = new HashMap<>();

               variables.put("result", result);
               externalTaskService.complete(externalTask, variables);
               System.out.println("The External Task " + externalTask.getId() + " has been completed!");
           } catch (e: Exception) {
               externalTaskService.handleFailure(externalTask, e.message, e.stackTrace.toString())
           }
       }).open();
   }
}

If your task requires not just performing some synchronous action, but starting the whole process, then you can do it, for example, by starting the process through RuntimeService :

@Service
class EmailWorker(
   private val runtimeService: RuntimeService
) {
   val builder = ExternalTaskClientBuilderImpl().baseUrl("http://localhost:8080").workerId("myWorker")
   val taskClient = builder.build()
   val engineClient = (builder as ExternalTaskClientBuilderImpl).engineClient

   @PostConstruct
   fun init() {
       taskClient
           .subscribe("sendEmail")
           .lockDuration(10000)
           .handler { externalTask, externalService ->
               runtimeService.startProcessInstanceByKey(
                   "SendEmailProcess",
                   externalTask.getVariable("emailId"),
                   mapOf(
                       "text" to externalTask.getVariable("text"),
                       "email" to externalTask.getVariable("email")
                   )
               )
           }
           .open()
   }


   @PreDestroy
   fun destroy() {
       taskClient.stop()
   }
}

// Delegate from SendEmailProcess process
@Component
class EmailResultDelegate(private val emailWorker: EmailWorker) {
   fun doExecute(execution: DelegateExecution) {
       emailWorker.engineClient.complete(
           execution.readVar(EXTERNAL_TASK_ID),
           mapOf("result" to "Success")
       )
   }
}

In this example, the external tasks handler ( EmailWorker ), when the task is received, starts the SendEmailProcess process .

Imagine that this process performs some actions necessary to send a newsletter, and at the end calls EmailResultDelegate , which, in turn, completes the external task .

Architectural Benefits of External Task


It is worth noting that there is a way to start the process in another camunda application in a simpler way: POST: / rest / process-definition / key / $ {id} / start

When you use REST , you do not have any transaction guarantees. But after all, we also work with external task through REST , what is the difference then?

The difference is that we do not call the external service directly, but only publish tasks that can be processed. Consider an example:

image

Some external handler takes a task that is now assigned to it, but when a response is received, the connection is disconnected. Now on the side of camundaa task that will not be processed is blocked because the external handler did not receive a response. But this is not scary: in camunda for external tasks there is a timeout by which the task will return to the queue again and someone else can handle it.

Now let's look at the case when an external handler received a task, completed it, and called the complete method , which failed because of network problems. Now you will not be able to understand whether the task was successfully completed in camunda or not. You can try again, but there is a chance that network problems will continue.

In this case, the best solution would be to ignore the problem. If the task was successfully completed, then everything is in order. If not, after the timeout the task will again be available for processing. But this means that your external handler must be idempotent or contain logic to deduplicate tasks.

A similar problem can happen when starting a new process, so before that you should check existing instances with the same data, for example businessKey .

In addition to high fault tolerance external taskallows you to easily scale external handlers and implement them in any programming language. At the same time, the pattern helps to implement microservices so that they influence each other as little as possible, thereby increasing their stability.

More about external task :
https://docs.camunda.org/manual/latest/user-guide/process-engine/external-tasks/
https://docs.camunda.org/manual/latest/reference/rest/external -task /
https://docs.camunda.org/manual/latest/user-guide/ext-client/

All Articles