Design Document for Zero-Copy via Loaned Messages

[mjcarroll]

Design document for using loaned messages to provide support for zero-copy middleware implementations.

This is from notes that @Karsten1987 and @wjwwood composed, reformatted and expanded.

[clalancette]

I just added some typo fixes. I haven't reviewed the whole thing yet, particularly the proposed API.

[Karsten1987]

fyi @michael-poehnl

[ivanpauno]

I think that the idea of loaned messages is the best solution for zero-copy intraprocess communication.
Currently, most of the middleware vendors we're supporting doesn't have a loaned message API.
So, it may be a good idea to also push development to add this in some of them.

Fixes #251.
That issue have some comments in line with this proposal.
Also, there is a similar previous discussion in #239.

[ivanpauno]

I guess, that no modification in RCLCPP subscription would be needed.
Maybe we would need to add a new subscription signature more convenient for loaned messages, but not more than that. Probably, similar changes would be needed in rcl/rmw.

[Karsten1987]

I think we talked about this, but I think we didn't come to an conclusion.

When we talk about taking sequences for messages which can be loaned, this implies that there has to be some sort of adjustment/enhancement on for a rmw_take_loaned_message_sequence function.
Right now I envision something like:

rmw_ret_t
rmw_take_loaned_message_sequence(rmw_message_sequence_t * message_sequence, size_t n);

Where n is the amount of messages being fetched and stored in the message sequence.

If I bring this all the way up to a possible implementation in the rclcpp::Executor::execute_subscription function, this would like as such:

void
Executor::execute_subscription(
  rclcpp::SubscriptionBase::SharedPtr subscription)
{
  rmw_message_info_t message_info;
  message_info.from_intra_process = false;

  // handle serialized messages
  if (subscription->is_serialized()) {
    auto serialized_msg = subscription->create_serialized_message();
    auto ret = rcl_take_serialized_message(
      subscription->get_subscription_handle().get(),
      serialized_msg.get(), &message_info, nullptr);
    if (RCL_RET_OK == ret) {
      auto void_serialized_msg = std::static_pointer_cast<void>(serialized_msg);
      subscription->handle_message(void_serialized_msg, message_info);
    } else if (RCL_RET_SUBSCRIPTION_TAKE_FAILED != ret) {
      RCUTILS_LOG_ERROR_NAMED(
        "rclcpp",
        "take_serialized failed for subscription on topic '%s': %s",
        subscription->get_topic_name(), rcl_get_error_string().str);
      rcl_reset_error();
    }
    subscription->return_serialized_message(serialized_msg);
    return;
  }

  // if a middleware can provide loaned messages, we always take them as such
  if (subscription->can_loan_messages()) {
    size_t n = 1;  // how many messages are we fetch at a time
    // Could be part of the subscription API in case of polling
    // Maybe take a parameter n for reserving space in the loaned message sequence
    auto LoanedMessageSequence lms = subscription->create_loaned_sequence(n);
    auto ret = rcl_take_loaned_message_sequence(lms.get_sequence_handle(), n);
    if (RCL_RET_OK == ret) {
      subscription->handle_message(lms.msg_at(0), lms.msg_info_at(0));
    } else if (RCL_RET_TAKE_FAILED != ret) {
      RCUTILS_LOG_ERROR_NAMED(
        "rclcpp",
        "could not take loaned message sequence on topic '%s': %s",
        subscription->get_topic_name(), rcl_get_error_string().str);
      rcl_reset_error();
    }
    subscription->return_loaned_sequence(lms);
    return;
  }

  // if the middleware can't loan message, we create our own instances and take them
  std::shared_ptr<void> message = subscription->create_message();
  auto ret = rcl_take(
    subscription->get_subscription_handle().get(),
    message.get(), &message_info, nullptr);
  if (RCL_RET_OK == ret) {
    subscription->handle_message(message, message_info);
  } else if (RCL_RET_SUBSCRIPTION_TAKE_FAILED != ret) {
    RCUTILS_LOG_ERROR_NAMED(
      "rclcpp",
      "could not deserialize serialized message on topic '%s': %s",
      subscription->get_topic_name(), rcl_get_error_string().str);
    rcl_reset_error();
  }
  subscription->return_message(message);
}

Looking at the scribble above, you can see an inconsistency between serialized/non-loaned messages and loaned messages. The first two types are always fetch one-by-one, whereas loaned messages are always fetched in a sequence - main reason for this being a strong requirements for RTI Connext Micro in order to enable zero-copy.

The question which arises now is whether any rmw_take_* should be modified in a way that it always takes n instances or if we restrict that to loaned messages exclusively.
If we did change rmw_take to always take multiple, then we have to push this consequently to rmw_serialized_messages as well as ROS messages allocated by the rclcpp::MessageMemoryStrategy and implement C-based container structures for it.

My thinking here is that in a context of an executor, we probably would get around implementing the message sequences for serialized and non-loaned messages as we handle one callback at a time.
But thinking this further and having the ability to use the API outside of the scope of an executor (by polling the waitset or having different implementations of an executor), I personally believe it would be the right thing to do and extend the sequencing also to these types. As the rmw_take_multiple functions can be implemented next to the existing rmw_take, we can still use the latter inside the executor.

Opinions?

[allenh1]

it may refuse to loan memory to the client library, and in that case, a user provided allocator should be used.

this makes a lot of sense.

This allows the user to have control over the allocation of the message when the middleware might otherwise use the standard allocator (new/malloc)

This seems to contradict that prior statement, though. Maybe I'm not reading this quite right?

I think this should read something more along the lines of "If the middleware does not have specific protocols for handling preallocations of messages, it should use the rcutils allocator."

I could be way off base in my interpretation, though.

[jacobperron]

I agree it could be more clear.

"If the middleware does not have specific protocols for handling preallocations of messages, it should use the rcutils allocator."

I think this is almost right, but I don't think the user is limited to the rcutils allocator. They could provide their own.

[allenh1]

I don't think the user is limited to the rcutils allocator. They could provide their own.

Yeah, I like that better. 👍

[mjcarroll]

So does this need additional clarification? I believe that what is there summarizes the correct behavior?

[jacobperron]

Re-reading what's in the document, it makes sense to me. I don't have a solid suggestion for changing it.

[allenh1]

Has the message already been loaned? If it is planned to be loaned, maybe this should be called rcl_allocate_loaner_message?

[Karsten1987]

connected to ros2/ros2#785

[jacobperron]

I agree it could be more clear.

"If the middleware does not have specific protocols for handling preallocations of messages, it should use the rcutils allocator."

I think this is almost right, but I don't think the user is limited to the rcutils allocator. They could provide their own.

[jacobperron]

A few more minor nitpicks. Besides that there are still some unresolved comments that would be good to address before merging this.

[jacobperron]

One sentence per line.

[jacobperron]

The question which arises now is whether any rmw_take_* should be modified in a way that it always takes n instances or if we restrict that to loaned messages exclusively.

It makes sense to me to also extend the "take N" functionality to non-loaned messages, but we probably don't need to include this detail/discussion in this document since it is not exactly related to loaned messages. Probably worth discussing as part of the implementation ticket or a new issue.

[mjcarroll]

Probably worth discussing as part of the implementation ticket or a new issue.

Agreed.

[jacobperron]

LGTM

[ros-discourse]

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/ros-2-galactic-default-middleware-announced/18064/1