Add support for local pipeline executions when server uses Enterprise Gateway

[kevin-bates]

This pull request adds support for running local pipelines against a server configured to use Enterprise Gateway for its kernel management. These changes include the following:

Autodetection of Enterprise Gateway

The local pipeline processor, because it's running in the notebook server, uses the server's configuration information to detect whether or not a gateway server is configured. If a gateway is configured, it invokes Papermill indicating that the HTTPKernelManager class be used to manage the underlying NBClient kernel.

ElyraEngine

A new Papermill engine has been added named ElyraEngine. This engine passes the Notebook node's env and cwd values to the kernel (regardless of the kernel's location). This engine derives directly from NBClientEngine and is essentially the same engine used today, with some additional support for conveying parameters to the underlying kernel manager.

HTTPKernelManager

HTTPKernelManager is a kernel manager (derived from AsyncKernelManager) that forwards all kernel manager requests to the associated gateway server (when configured). Because it leverages the GatewayClient class and gateway_request function built into the notebook server, HTTPKernelManager cannot be invoked outside of the server process. This limitation can be mitigated if deemed necessary.

HTTPKernelClient

This kernel client class (derived from AsyncKernelClient) is responsible for emulating kernel message requests and responses. Each of the 5 kernel channels is associated with a ChannelQueue class that derives from the built-in Queue class. The channel send() methods forward the message to the underlying websocket shared across the channels. The message is processed on the gateway server and its response is returned back to the local server where the response is routed to the appropriate channel queue. The channel's get_msg() method, then pulls responses from its corresponding queue - thereby completing the message sequence.

When channels are started a response router thread is started prior to their instantiation. This thread's sole purpose is to receive messages (responses) from the websocket opened against the gateway server, routing each response to the appropriate channel queue based on the response's 'channel' indicator. The response router thread is terminated when the channels are stopped following the kernel's shutdown.

Resolves #961, #164

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

[elyra-bot]

Thanks for making a pull request to Elyra!

To try out this branch on binder, follow this link: