Queue processor FAQ
A queue processor rule is an internal background process that you configure for queue management and
asynchronous message processing. Use standard queue processor rules for simple queue management or
low-throughput scenarios, or use dedicated queue processor rules for higher-scaling throughput, and
customized or delayed processing of messages.
Best practice is to configure queue processors instead of using standard agents to manage message
queuing in Pega Platform. For details, see Background processing best practices.
Frequently Asked Questions about queue processors include the following:
How do I queue a message to a queue processor?
You can queue a message to a queue processor by performing either of the following actions:
Use the Queue-for-Processing activity method.
Use the Run in Background SmartShape when you create a case.
For more information, see Queue-For-Processing method and Running a queued background process in a
case.
Can I queue a message to be processed at a scheduled time?
Yes. When you define a queue processor, set the When to process parameter to Delayed. Then, when you queue
the message, you can specify the time to process the message.
For more information, see Queue-For-Processing method.
Do I have to create a queue processor to use this feature?
No. By default, Pega Platform includes standard queue processor rules. For more information, see Queue
processor rules.
How does a queue processor work, and where do I find
information about how it runs?
The queue processor automatically generates a stream data set and a corresponding data flow. The stream
data set sends messages to, and receives messages from, the stream service. The data flow manages the
subscription of messages to ensure message processing.
Queue processor logical flow
You can view data flows that correspond with the queue processor rules in your system on the
QueueProcessors landing page in Admin Studio. For more information see, Managing queue processors.
How do I configure the scaling requirements for my queue
processor rules?
Queue processor rules can scale both horizontally and vertically. You can achieve horizontal scaling of
queue processor rules by adding more nodes, and vertical scaling of queue processor rules by increasing
the number of threads per node (on the queue processor rule form).
For more information, see Creating a queue processor rule.
Why is my new queue processor not working?
If your queue processor does not work, consider the following:
Queue processor rules rely on the DSM service. In Pega Cloud services and client-managed cloud
environments, a stream node is automatically configured and ready to use. For on-premises
environments, ensure that you define at least one node in a cluster to be a Stream node using the
setting: -DNodeType=Stream. For more information, see Stream node configurations for a queue processor
rule.
One problem may be that Pega Platform is not be able to properly resolve your queue processor rule.
For more information on how queue processor rules work, see System Runtime Context and
Automating the runtime context management of background processes.
If further troubleshooting is necessary, see Troubleshooting background processing.
What happens when I queue messages with no stream node
running?
If a stream node is not available when the system queues a message, the system saves the message in a
database, and then pushes the message to Kafka when the stream node becomes available again.
For more information, see Stream node configurations for a queue processor rule.
When I create a queue processor, what happens if I enter a
large number of threads per node?
The maximum number of Kafka partitions is 6 for one Pega Platform cluster, which means that the
maximum number of queue processor threads that can process messages across all nodes in a cluster is 6.
This is not the global limit, but it is the limit for each queue processor.
How do I monitor and diagnose issues with queue processor
rules?
In Admin Studio, the QueueProcessors landing page lists the queue processor rules that currently run on
your system. From this landing page, you can open the DataFlow associated with every queue processor
rule, with which you can monitor and diagnose the background processes. Also on this landing page, you
can trace, enable, and disable your queue processor rules, or you can perform these same functions using
REST APIs.
For more information, see Tracing a queue processor.
How do I increase the performance of my queue processor
rules?
Performance has two dimensions: the time to process a message and the total message throughput. You
can increase performance by completing one or both of the following actions:
The time to process a message depends on the amount of work done by the processing activity.
Optimize the activity to reduce the time needed to process a message.
Enhance total message throughput in the following ways:
Scale out by increasing the number of processing nodes. This setting applies only to on-premises
users. For Pega Cloud, a larger sandbox is required.
Scale up by increasing the number of threads per node, up to 6 threads per cluster. If you need
more threads (partitions) per cluster, you can increase the number of partitions. For more
information, see Sizing queue processors individually and Changing the default number of
partitions per topic.
For more information, see Queue processor rules.
How do I enable diagnostic logging for queue processor rules?
You can use background processing log categories to identify log entries that are relevant to and specific to
background processing: job scheduler, queue processor, and agent log entries. For more information, see
Details provided in background processing logs.
How does queue processing handle large message exceptions?
Queue processing automatically handles large messages that exceed the Stream Service default file size
threshold. Automatic handling prevents oversize messages from breaking.
A large message could contain text for an HR email notification, for example, that runs as a background
process activity. A message is the data, or input, to a queue processor. It is also referred to as a queued
item.
For more information, see Large message exception handling in queue processing.
If the number of threads for a queue processor is adjusted,
does it need a restart to take effect?
No. Just save the queue processor rule, and the change will take effect immediately.
