• Dependencies
• Jupyter REPL
  • Inspection
  • Associating other buffers with a REPL
• Overview of implementation
  • jupyter-kernel-client
    • Initializing a connection
    • Channel subprocess
    • Starting/stopping channels
    • Client local variables
  • jupyter-kernel-manager
    • Kernelspecs
    • Starting kernels
    • Making clients connected to a kernel
  • Making requests to a kernel
  • Receiving messages from a kernel
    • jupyter-kernel-client handler methods
    • jupyter-request callbacks
    • Channel hooks
    • Blocking until certain messages are received
    • Preventing handler methods from running
• Extending the jupyter-kernel-client class
• Known problems
• Wish list

Dependencies

libzmq

https://github.com/zeromq/libzmq

• With the DRAFT API (for zmq_poller)

emacs-zmq

http://github.com/dzop/emacs-zmq

jupyter

http://jupyter.org/install

TODO Jupyter REPL

The jupyter-repl-client class implements a REPL client for a Jupyter kernel. Its main features at the moment are:

• Completion
• Inspection

Inspection

Associating other buffers with a REPL

After starting a REPL for a language, it is possible to associate the REPL with other buffers if they pass certain criteria. Namely the buffer must have the major-mode that corresponds to the REPL kernel's language. To associate a buffer with a REPL you can run the command jupyter-repl-associate-buffer.

jupyter-repl-associate-buffer will ask you for the kernel you would like to associate with the current-buffer and enable the minor mode jupyter-repl-interaction-mode. This minor mode populates the following keybindings for interacting with the REPL:

Key binding	Command
C-c C-c	jupyter-repl-eval-line-or-region
C-c C-f	jupyter-repl-inspect-at-point
C-c C-z	jupyter-repl-pop-to-buffer
C-c C-i	jupyter-repl-interrupt-kernel
C-c C-r	jupyter-repl-restart-kernel

Overview of implementation

jupyter-kernel-client

Initializing a connection

Channel subprocess

When a jupyter-kernel-client is connected to a kernel, a subprocess is created that listens for messages from the kernel to send to the parent emacs process and does all of the message passing between the kernel and the client. The parent emacs process is responsible for making requests and handling the messages in response to each request.

Starting/stopping channels

jupyter-start-channels

Starts the previously configured channels of the client.

jupyter-stop-channels

Stops all live channels of the client.

jupyter-channels-runngin

Returns non-nil if at least one channel is live.

Client local variables

Some variables which are used internally by jupyter-kernel-client have client local values. For example the variable jupyter-include-other-output which tells a jupyter-kernel-client to handle IOPub messages originating from a different client can be set to t on a per client basis. Its default value is nil.

(jupyter-set client 'jupyter-include-other-output t)

This just sets the buffer local value of jupyter-include-other-output in a private buffer used by the client. When an IOPub message is received which does not originate from our client, the client local value of jupyter-include-other-output is examined to determine if it should be sent to the handlers.

(jupyter-get client 'jupyter-include-other-output)

You may also use the macro with-jupyter-client-buffer to examine the client local variables of a client:

(with-jupyter-client-buffer client
  (message "jupyter-include-other-output: %s" jupyter-include-other-output))

jupyter-kernel-manager

Kernelspecs

Starting kernels

Making clients connected to a kernel

Making requests to a kernel

This implementation follows closely the base Jupyter client implementation. Sending and receiving messages is centered around the jupyter-kernel-client class. Each message sent or received has a corresponding method in jupyter-kernel-client. For request messages the method names have the form jupyter-<msg-type> where <msg-type> is the request message type as defined in the Jupyter messaging protocol with underscores replaced by hyphens. So an execute_request message has the corresponding method jupyter-execute-request.

(jupyter-execute-request client :code "1 + 2") ; Returns a `jupyter-request'

All requests sent to a kernel return a jupyter-request object which encapsulates the current state of the request with the kernel and how the jupyter-kernel-client should handle messages received from the kernel in response to the request.

Receiving messages from a kernel

jupyter-kernel-client handler methods

The receiving message types have method names of the form jupyter-handle-<msg-type>, e.g. an execute_reply has the method name jupyter-handle-execute-reply. The handler methods are intended to be overridden by subclasses of jupyter-kernel-client since the default methods of jupyter-kernel-client do nothing. The exception to this is jupyter-handle-input-reply which will ask for input to send to the kernel. The method signature of the handlers has the form

(cl-defmethod jupyter-handle-<msg-type> ((client jupyter-kernel-client) req arg1 arg2 ...)

where req is the jupyter-request that generated the message and arg1, arg2, … will be the unwrapped message contents passed to the handler and will depend on <msg-type>. See the help for each handler type for the order of the arguments.

jupyter-request callbacks

There is also the possibility of capturing received messages through request callbacks instead of subclassing jupyter-kernel-client. Or both methods can be used in parallel. Note if you are using both callbacks and handler methods, the callbacks will run before the handler methods.

To add callbacks to a request, you would call jupyter-add-callback, passing the request object as the first argument and any callbacks as the remaining arguments. See the documentation of jupyter-add-callback for more details.

(jupyter-add-callback (jupyter-execute-request client :code "1 + 2")
  :execute-result (lambda (msg)
                    (let ((result (jupyter-message-data msg :text/plain)))
                      (message "1 + 2 = %s" result))))

The main entry point to attach callbacks to a request is through jupyter-add-callback which takes a message type, a jupyter-request object, and a callback function as arguments. The callback is registered with the request object to run whenever a message is received that has the same message type as the one passed to jupyter-add-callback. For example, to do something with the client's kernel info you would do the following:

(jupyter-add-callback (jupyter-kernel-info-request client)
  :kernel-info-reply (lambda (msg)
                       (let ((info (jupyter-message-content msg)))
                         BODY)))

To print out the results of an execute request:

(jupyter-add-callback (jupyter-execute-request client :code "1 + 2")
  :execute-result (lambda (msg)
                    (message (jupyter-message-data msg :text/plain))))

To add multiple callbacks to a request:

(jupyter-add-callback (jupyter-execute-request client :code "1 + 2")
  :execute-result (lambda (msg)
                    (message (jupyter-message-data msg :text/plain)))
  :status (lambda (msg)
            (when (jupyter-message-status-idle-p msg)
              (message "DONE!"))))

There is also the possibility of running the same handler for different message types:

(jupyter-add-callback (jupyter-execute-request client :code "1 + 2")
  '(:status :execute-result :execute-reply)
  (lambda (msg)
    (pcase (jupyter-message-type msg)
      ("status" ...)
      ("execute_reply" ...)
      ("execute_result" ...))))

Channel hooks

There are also hook variables for each channel: jupyter-iopub-message-hook, jupyter-stdin-message-hook, and jupyter-shell-message-hook. These hooks are run just before passing a message to the message type handler and they accept a single argument, a message received on the channel that corresponds to the hook. Note that if handlers are inhibited for a request, these hooks will be prevented from running for that request as well.

Note that you can either specify a global hook that will be run for all messages and for all clients. But in reality you will probably only want to specify hooks local to a particular client. This can be accomplished using jupyter-add-hook.

(jupyter-add-hook
 client 'jupyter-iopub-message-hook
 (lambda (msg)
   (when (jupyter-message-status-idle-p msg)
     (message "Kernel idle."))))

There is also the function jupyter-remove-hook to remove a hook local to a client.

Blocking until certain messages are received

All message sending and receiving happens asynchronously, therefore we need primitives which will block until certain conditions have been met on the received messages for a request.

The following functions all wait for different conditions to be met on the received messages of a request and return the message that caused the function to stop waiting or nil if no message was received within a timeout period. Note that if the timeout argument is nil, the timeout will default to 1 second.

To wait until an idle message is received for a request:

(let ((timeout 4))
  (jupyter-wait-until-idle
   (jupyter-execute-request
    client :code "import time\ntime.sleep(3)")
   timeout))

To wait until a message of a specific type is received for a request:

(jupyter-wait-until-received :execute-reply
  (jupyter-execute-request client :code "[i*10 for i in range(100000)]"))

The most general form of the blocking functions is jupyter-wait-until which takes an arbitrary function as the last argument. The function must take a single argument, a message with the same message type supplied to jupyter-wait-until as its first argument, and the function should return non-nil if jupyter-wait-until should return from waiting:

(defun stream-prints-50-p (msg)
  (let ((text (jupyter-message-get msg :text)))
    (cl-loop for line in (split-string text "\n")
             thereis (equal line "50"))))

(let ((timeout 2))
  (jupyter-wait-until
      (jupyter-execute-request client :code "[print(i) for i in range(100)]")
      :stream #'stream-prints-50-p
    timeout))

The above code runs stream-prints-50-p for every stream message received from a kernel (here assumed to be a python kernel) for an execute request that prints the numbers 0 to 99 and waits until the kernel has printed the number 50 before returning from the jupyter-wait-until call. If the number 50 is not printed before the two second timeout, jupyter-wait-until returns nil. Otherwise it returns the non-nil value which caused it to stop waiting. In this case, the t returned from cl-loop in stream-prints-50-p.

Preventing handler methods from running

To prevent a client from handling any messages received from the kernel for a request, e.g. if you would like the jupyter-repl-client to prevent printing error messages to the REPL buffer for a request, set jupyter-inhibit-handlers to t before making the request. Note that if you set the global value of this variable to t, all new requests will prevent the handlers from running. The less intrusive way to prevent handlers from running for individual requests is to locally bind jupyter-inhibit-handlers:

(let ((jupyter-inhibit-handlers t))
  (jupyter-execute-request client :code "print(\"foo\")\n1 + 2"))

Not you can also set the jupyter-request-run-handlers-p slot of a jupyter-request to nil.

Is a request inhibits the client handler methods, the only callbacks associated with the request will run.

Extending the jupyter-kernel-client class

To hook into the entire message receiving machinery, subclass jupyter-kernel-client and override the default message handlers. For example to capture a kernel_info_reply on a client you can do the following:

(defclass my-kernel-client (jupyter-kernel-client)
  (kernel-info))

(cl-defmethod jupyter-handle-kernel-info-reply ((client my-kernel-client)
                                                protocol-version
                                                implementation
                                                implementation-version
                                                language-info
                                                banner
                                                help-links)
  (oset client kernel-info
        (list :protocol-version protocol-version
              :implementation implementation
              :implementation-version implementation-version
              :language-info language-info
              :banner banner
              :help-links help-links)))

But the above is more easily achieved with callbacks:

(setq kernel-info (jupyter-message-content
                   (jupyter-wait-until-received :kernel-info-reply
                     (jupyter-kernel-info-request client))))

Or you could have interactivity between org-mode and a Jupyter kernel:

(defclass jupyter-org-client (jupyer-kernel-client)
  (src-block-marker))


(cl-defmethod jupyter-execute-request ((client jupyter-org-client))

  )

(cl-defmethod jupyter-handle-execute-result ((client jupyter-org-client)
                                             execution-count
                                             data
                                             metadata)
  )

Known problems

• Reading messages from the IOLoop is a bottleneck. If many messages are coming in, it seems its hard to get messages sent.

TODO Wish list

• Notebook interface

• Integration with Jupyter widgets

  • One solution would be to show a browser and inject the necessary javascript using skewer-mode. Look