* Dependencies - zmq - ipython or jupyter * Overview of implementation This implementation follows closely the base Jupyter client implementation. Message sending and receiving messages is centered around the =jupyter-kernel-client= class. Every message type that is a request to the kernel has a method =jupyter-= where == is the type of the message as defined in the Jupyter messaging protocol with underscores replaced by hyphens. So an =execute_request= message has the corresponding method =jupyter-execute-request=. All methods that send requests to the kernel return a =jupyter-request= object which encapsulates the current state of the request with the kernel and how the =jupyter-kernel-client= should handle messages received from the kernel in response to the request. When a message is received from the kernel it is passed to the client's =jupyter-handle-message= method which runs the callbacks for the message and does all of the bookkeeping for =jupyter-request= objects. Depending on the value of =jupyter-request-run-handlers-p=, the message is then passed to the =jupyter-handle-message= method of the =jupyter-channel= the message was received on. For channels, =jupyter-handle-message= just dispatches the message to one of the client's handler methods based on the message type. All of the handler methods of a =jupyter-kernel-client= are named =jupyter-handle-=. So an =execute_reply= message has the corresponding method =jupyter-handle-execute-reply=. When a =jupyter-kernel-client= is connected to a kernel, a subprocess is created that listens for messages from the kernel to send to the parent =emacs= process and does all of the message passing between the kernel and the client. The parent =emacs= process is responsible for making requests and handling the messages in response to each request. * =jupyter-kernel-client= class ** Starting/stopping channels - jupyter-start-channels :: Starts the previously configured channels of the client. - jupyter-stop-channels :: Stops all live channels of the client. - jupyter-channels-runngin :: Returns non-nil if at least one channel is live. * TODO Creating a kernel client :PROPERTIES: :Effort: 30 :END: * Sending requests to the kernel :PROPERTIES: :Effort: 30 :END: :LOGBOOK: CLOCK: [2018-01-01 Mon 12:40]--[2018-01-01 Mon 12:50] => 0:10 :END: All messages which can be sent to the kernel follow the naming convention =jupyer-= where == is one of the message request types found in the [[http://jupyter-client.readthedocs.io/en/latest/messaging.html][Jupyter messaging protocol]] (with the =_= characters replaced by =-=). So an =execute_request= to the kernel can be made by calling the method =jupyter-execute-request=: #+BEGIN_SRC elisp (jupyter-execute-request client :code "1 + 2") ; Returns a `jupyter-request' object #+END_SRC All request messages sent to the kernel return a =jupyter-request= object. A =jupyter-request= object encapsulates the current status of the request and how the client handles the messages it receives from the kernel in response to the request. To add callbacks to a request you would call =jupyter-add-callback= like so: #+BEGIN_SRC elisp (jupyter-add-callback 'execute-result (jupyter-execute-request client :code "1 + 2") (lambda (msg) (let ((res (jupyter-message-data msg :text/plain))) (message "1 + 2 = %s" res)))) #+END_SRC For more information on adding callbacks to a request, see [[id:A94C8391-C918-404D-82F9-C257CC2A1809][Request callbacks]]. * Receiving messages from the kernel There are two ways of running code when a certain type of message is received from the kernel. 1. Define a new subclass of =jupyter-kernel-client= and override the default message handling methods. 2. Register callbacks to run when receiving messages associated with a request. Both ways work in parallel. If a client subclass has a handler for a particular message type and a callback is registered for messages of the same type for a particular request, the callback will run first and the handler second. ** TODO Client handlers :PROPERTIES: :Effort: 30 :END: Sub-classing =jupyter-kernel-client= and overriding the message handling methods is intended to be used for updating user-interface elements for the particular client application. This is how =jupyter-repl-client= updates the REPL buffer. ** Request callbacks :PROPERTIES: :ID: A94C8391-C918-404D-82F9-C257CC2A1809 :END: The main entry point to attach callbacks to a request is through =jupyter-add-callback= which takes a message type, a =jupyter-request= object, and a callback function as arguments. The callback is registered with the request object to run whenever a message is received that has the same message type as the one passed to =jupyter-add-callback=. For example, to do something with the client's kernel info you would do the following: #+BEGIN_SRC elisp (jupyter-add-callback 'kernel-info-reply (jupyter-kernel-info-request client) (lambda (msg) (let ((info (jupyter-message-content msg))) ...))) #+END_SRC To print out the results of an execute request: #+BEGIN_SRC elisp (jupyter-add-callback 'execute-result (jupyter-execute-request client :code "1 + 2") (lambda (msg) (message (jupyter-message-data msg :text/plain)))) #+END_SRC You can also run the same callback for different message types: #+BEGIN_SRC elisp (jupyter-add-callback '(status execute-result execute-reply) (jupyter-execute-request client :code "1 + 2") (lambda (msg) (pcase (jupyter-message-type msg) ("status" ...) ("execute_reply" ...) ("execute_result" ...)))) #+END_SRC *** Blocking until certain messages are received All message sending and receiving happens asynchronously, therefore we need primitives which will block until certain conditions have been met on the received messages for a request. The following functions all wait for different conditions to be met on the received messages of a request and return the message that caused the function to stop waiting or =nil= if no message was received within a timeout period. Note that if the timeout argument is =nil=, the timeout will default to 1 second. To wait until an idle message is received for a request: #+BEGIN_SRC elisp (let ((timeout 4)) (jupyter-wait-until-idle (jupyter-execute-request client :code "import time\ntime.sleep(3)") timeout)) #+END_SRC To wait until a message of a specific type is received for a request: #+BEGIN_SRC elisp (jupyter-wait-until-received 'execute-reply (jupyter-execute-request client :code "[i*10 for i in range(100000)]")) #+END_SRC The most general form of the blocking functions is =jupyter-wait-until= which takes an arbitrary function as the last argument. The function must take a single argument, a message with the same message type supplied to =jupyter-wait-until= as its first argument, and the function should return non-nil if =jupyter-wait-until= should return from waiting: #+BEGIN_SRC elisp (defun stream-prints-50-p (msg) (let ((text (jupyter-message-get msg :text))) (cl-loop for line in (split-string text "\n") thereis (equal line "50")))) (let ((timeout 2)) (jupyter-wait-until 'stream (jupyter-execute-request client :code "[print(i) for i in range(100)]") timeout #'stream-prints-50-p)) #+END_SRC The above code runs =stream-prints-50-p= for every =stream= message received from a kernel (here assumed to be a python kernel) for an execute request that prints the numbers 0 to 99 and waits until the kernel has printed the number 50 before returning from the =jupyter-wait-until= call. If the number 50 is not printed before the two second timeout, =jupyter-wait-until= returns =nil=. Otherwise it returns the non-nil value which caused it to stop waiting. In this case, the =t= returned from =cl-loop= in =stream-callback=. *** Suspending client handlers for individual requests Since the client handlers are intended to be used for updating user-interface elements, you may run into situations where you would like to make a request to the kernel and only handle the received messages for the request using callbacks without using the client handlers. For example, you may want to synchronize the execution count of a =jupyter-kernel-client= subclass with the current execution count of the kernel without allowing the client's handlers to process any of the messages which result from the request. To do this you can either set the =jupyter-request-run-handlers-p= field of a =jupyter-request= object to =nil= or wrap your requests with the convenience function =jupyter-request=inhibit-handlers= which does this for you: #+BEGIN_SRC elisp (jupyter-add-callback 'execute-reply (jupyter-request-inhibit-handlers (jupyter-execute-request client :code "" :silent t)) (apply-partially (lambda (client msg) (oset client execution-count (jupyter-message-get msg :execution_count))) client)) #+END_SRC * Connecting to kernels - =jupyer-kernel-client-from-connection-file= - =jupyter-start-kernel= * Extending the =jupyter-kernel-client= class To hook into the entire message receiving machinery, subclass =jupyter-kernel-client= and override the default message handlers. For example to capture a =kernel_info_reply= on a client you can do the following: #+BEGIN_SRC elisp (defclass my-kernel-client (jupyter-kernel-client) (kernel-info)) (cl-defmethod jupyter-handle-kernel-info-reply ((client my-kernel-client) protocol-version implementation implementation-version language-info banner help-links) (oset client kernel-info (list :protocol-version protocol-version :implementation implementation :implementation-version implementation-version :language-info language-info :banner banner :help-links help-links))) #+END_SRC Or you could have interactivity between =org-mode= and a =Jupyter= kernel #+BEGIN_SRC elisp (defclass jupyter-org-kernel-client (jupyer-kernel-client) (src-block-marker)) (cl-defmethod jupyter-handle-execute ((client jupyter-org-kernel-client) execution-count user-expressions payload) ) (cl-defmethod jupyter-handle-execute-input ((client jupyter-org-kernel-client) code execution-count) ) (cl-defmethod jupyter-handle-execute-result ((client jupyter-org-kernel-client) execution-count data metadata) ) #+END_SRC * TODO Wish list - Make a default client front-end, see how =jupyter console= does this - How could I make =org-mode= the default front end? - =org-edit-src-buffer= connected to a kernel. multiple clients can connect to the src code block and edit it.