emacs-jupyter/README.org

* Dependencies

- libzmq :: https://github.com/zeromq/libzmq
  - With the DRAFT API (for =zmq_poller=)
- emacs-zmq :: http://github.com/dzop/emacs-zmq
- jupyter :: http://jupyter.org/install

* TODO Jupyter REPL

The =jupyter-repl-client= class implements a REPL client for a Jupyter kernel.
Its main features at the moment are:

- Completion
- Inspection

** Inspection


** Associating other buffers with a REPL

After starting a REPL for a language, it is possible to associate the REPL with
other buffers if they pass certain criteria. Namely the buffer must have the
=major-mode= that corresponds to the REPL kernel's language. To associate a
buffer with a REPL you can run the command =jupyter-repl-associate-buffer=.

=jupyter-repl-associate-buffer= will ask you for the kernel you would like to
associate with the =current-buffer= and enable the minor mode
=jupyter-repl-interaction-mode=. This minor mode populates the following
keybindings for interacting with the REPL:

| Key binding | Command                          |
|-------------+----------------------------------|
| C-c C-c     | jupyter-repl-eval-line-or-region |
| C-c C-f     | jupyter-repl-inspect-at-point    |
| C-c C-z     | jupyter-repl-pop-to-buffer       |
| C-c C-i     | jupyter-repl-interrupt-kernel    |
| C-c C-r     | jupyter-repl-restart-kernel      |

* Overview of implementation
** =jupyter-kernel-client=
*** Initializing a connection
*** Channel subprocess

When a =jupyter-kernel-client= is connected to a kernel, a subprocess is
created that listens for messages from the kernel to send to the parent =emacs=
process and does all of the message passing between the kernel and the client.
The parent =emacs= process is responsible for making requests and handling the
messages in response to each request.
*** Starting/stopping channels

- jupyter-start-channels :: Starts the previously configured channels of the
     client.
- jupyter-stop-channels :: Stops all live channels of the client.
- jupyter-channels-runngin :: Returns non-nil if at least one channel is live.

*** Client local variables

Some variables which are used internally by =jupyter-kernel-client= have client
local values. For example the variable =jupyter-include-other-output= which
tells a =jupyter-kernel-client= to handle IOPub messages originating from a
different client can be set to t on a per client basis. Its default value is
nil.

#+BEGIN_SRC elisp
(jupyter-set client 'jupyter-include-other-output t)
#+END_SRC

This just sets the buffer local value of =jupyter-include-other-output= in a
private buffer used by the client. When an IOPub message is received which does
not originate from our client, the client local value of
=jupyter-include-other-output= is examined to determine if it should be sent to
the handlers.

#+BEGIN_SRC elisp
(jupyter-get client 'jupyter-include-other-output)
#+END_SRC

You may also use the macro =with-jupyter-client-buffer= to examine the client
local variables of a client:

#+BEGIN_SRC elisp
(with-jupyter-client-buffer client
  (message "jupyter-include-other-output: %s" jupyter-include-other-output))
#+END_SRC

** =jupyter-kernel-manager=
*** Kernelspecs
*** Starting kernels
*** Making clients connected to a kernel
** Making requests to a kernel

This implementation follows closely the base Jupyter client implementation.
Sending and receiving messages is centered around the =jupyter-kernel-client=
class. Each message sent or received has a corresponding method in
=jupyter-kernel-client=. For request messages the method names have the form
=jupyter-<msg-type>= where =<msg-type>= is the request message type as defined
in the Jupyter messaging protocol with underscores replaced by hyphens. So an
=execute_request= message has the corresponding method
=jupyter-execute-request=.

#+BEGIN_SRC elisp
(jupyter-execute-request client :code "1 + 2") ; Returns a `jupyter-request'
#+END_SRC

All requests sent to a kernel return a =jupyter-request= object which
encapsulates the current state of the request with the kernel and how the
=jupyter-kernel-client= should handle messages received from the kernel in
response to the request.

** Receiving messages from a kernel
*** =jupyter-kernel-client= handler methods

The receiving message types have method names of the form
=jupyter-handle-<msg-type>=, e.g. an =execute_reply= has the method name
=jupyter-handle-execute-reply=. The handler methods are intended to be
overridden by subclasses of =jupyter-kernel-client= since the default methods
of =jupyter-kernel-client= do nothing. The exception to this is
=jupyter-handle-input-reply= which will ask for input to send to the kernel.
The method signature of the handlers has the form

#+BEGIN_SRC elisp
(cl-defmethod jupyter-handle-<msg-type> ((client jupyter-kernel-client) req arg1 arg2 ...)
#+END_SRC

where =req= is the =jupyter-request= that generated the message and =arg1=,
=arg2=, ... will be the unwrapped message contents passed to the handler and
will depend on =<msg-type>=. See the help for each handler type for the order
of the arguments.

*** =jupyter-request= callbacks

There is also the possibility of capturing received messages through request
callbacks instead of subclassing =jupyter-kernel-client=. Or both methods can
be used in parallel. Note if you are using both callbacks and handler methods,
the callbacks will run /before/ the handler methods.

To add callbacks to a request, you would call =jupyter-add-callback=, passing
the request object as the first argument and any callbacks as the remaining
arguments. See the documentation of =jupyter-add-callback= for more details.

#+BEGIN_SRC elisp
(jupyter-add-callback (jupyter-execute-request client :code "1 + 2")
  :execute-result (lambda (msg)
                    (let ((result (jupyter-message-data msg :text/plain)))
                      (message "1 + 2 = %s" result))))
#+END_SRC

The main entry point to attach callbacks to a request is through
=jupyter-add-callback= which takes a message type, a =jupyter-request= object,
and a callback function as arguments. The callback is registered with the
request object to run whenever a message is received that has the same message
type as the one passed to =jupyter-add-callback=. For example, to do something
with the client's kernel info you would do the following:

#+BEGIN_SRC elisp
(jupyter-add-callback (jupyter-kernel-info-request client)
  :kernel-info-reply (lambda (msg)
                       (let ((info (jupyter-message-content msg)))
                         BODY)))
#+END_SRC

To print out the results of an execute request:

#+BEGIN_SRC elisp
(jupyter-add-callback (jupyter-execute-request client :code "1 + 2")
  :execute-result (lambda (msg)
                    (message (jupyter-message-data msg :text/plain))))
#+END_SRC

To add multiple callbacks to a request:

#+BEGIN_SRC elisp
(jupyter-add-callback (jupyter-execute-request client :code "1 + 2")
  :execute-result (lambda (msg)
                    (message (jupyter-message-data msg :text/plain)))
  :status (lambda (msg)
            (when (jupyter-message-status-idle-p msg)
              (message "DONE!"))))
#+END_SRC

There is also the possibility of running the same handler for different message
types:

#+BEGIN_SRC elisp
(jupyter-add-callback (jupyter-execute-request client :code "1 + 2")
  '(:status :execute-result :execute-reply)
  (lambda (msg)
    (pcase (jupyter-message-type msg)
      ("status" ...)
      ("execute_reply" ...)
      ("execute_result" ...))))
#+END_SRC

*** Channel hooks

There are also hook variables for each channel: =jupyter-iopub-message-hook=,
=jupyter-stdin-message-hook=, and =jupyter-shell-message-hook=. These hooks are
run just before passing a message to the message type handler and they accept a
single argument, a message received on the channel that corresponds to the
hook. Note that if handlers are inhibited for a request, these hooks will be
prevented from running for that request as well.

Note that you can either specify a global hook that will be run for all
messages and for all clients. But in reality you will probably only want to
specify hooks local to a particular client. This can be accomplished using
=jupyter-add-hook=.

#+BEGIN_SRC elisp
(jupyter-add-hook
 client 'jupyter-iopub-message-hook
 (lambda (msg)
   (when (jupyter-message-status-idle-p msg)
     (message "Kernel idle."))))
#+END_SRC

There is also the function =jupyter-remove-hook= to remove a hook local to a
client.

*** Blocking until certain messages are received

All message sending and receiving happens asynchronously, therefore we need
primitives which will block until certain conditions have been met on the
received messages for a request.

The following functions all wait for different conditions to be met on the
received messages of a request and return the message that caused the function
to stop waiting or =nil= if no message was received within a timeout period.
Note that if the timeout argument is =nil=, the timeout will default to 1
second.

To wait until an idle message is received for a request:

#+BEGIN_SRC elisp
(let ((timeout 4))
  (jupyter-wait-until-idle
   (jupyter-execute-request
    client :code "import time\ntime.sleep(3)")
   timeout))
#+END_SRC

To wait until a message of a specific type is received for a request:

#+BEGIN_SRC elisp
(jupyter-wait-until-received :execute-reply
  (jupyter-execute-request client :code "[i*10 for i in range(100000)]"))
#+END_SRC

The most general form of the blocking functions is =jupyter-wait-until= which
takes an arbitrary function as the last argument. The function must take a
single argument, a message with the same message type supplied to
=jupyter-wait-until= as its first argument, and the function should return
non-nil if =jupyter-wait-until= should return from waiting:

#+BEGIN_SRC elisp
(defun stream-prints-50-p (msg)
  (let ((text (jupyter-message-get msg :text)))
    (cl-loop for line in (split-string text "\n")
             thereis (equal line "50"))))

(let ((timeout 2))
  (jupyter-wait-until
      (jupyter-execute-request client :code "[print(i) for i in range(100)]")
      :stream #'stream-prints-50-p
    timeout))
#+END_SRC

The above code runs =stream-prints-50-p= for every =stream= message received
from a kernel (here assumed to be a python kernel) for an execute request that
prints the numbers 0 to 99 and waits until the kernel has printed the number 50
before returning from the =jupyter-wait-until= call. If the number 50 is not
printed before the two second timeout, =jupyter-wait-until= returns =nil=.
Otherwise it returns the non-nil value which caused it to stop waiting. In this
case, the =t= returned from =cl-loop= in =stream-prints-50-p=.

*** Preventing handler methods from running

To prevent a client from handling any messages received from the kernel for a
request, e.g. if you would like the =jupyter-repl-client= to prevent printing
error messages to the REPL buffer for a request, set =jupyter-inhibit-handlers=
to t before making the request. Note that if you set the global value of this
variable to t, all new requests will prevent the handlers from running. The
less intrusive way to prevent handlers from running for individual requests is
to locally bind =jupyter-inhibit-handlers=:

#+BEGIN_SRC elisp
(let ((jupyter-inhibit-handlers t))
  (jupyter-execute-request client :code "print(\"foo\")\n1 + 2"))
#+END_SRC

Not you can also set the =jupyter-request-run-handlers-p= slot of a
=jupyter-request= to nil.

Is a request inhibits the client handler methods, the only callbacks associated
with the request will run.

* Extending the =jupyter-kernel-client= class

To hook into the entire message receiving machinery, subclass
=jupyter-kernel-client= and override the default message handlers. For example
to capture a =kernel_info_reply= on a client you can do the following:

#+BEGIN_SRC elisp
(defclass my-kernel-client (jupyter-kernel-client)
  (kernel-info))

(cl-defmethod jupyter-handle-kernel-info-reply ((client my-kernel-client)
                                                protocol-version
                                                implementation
                                                implementation-version
                                                language-info
                                                banner
                                                help-links)
  (oset client kernel-info
        (list :protocol-version protocol-version
              :implementation implementation
              :implementation-version implementation-version
              :language-info language-info
              :banner banner
              :help-links help-links)))
#+END_SRC

But the above is more easily achieved with callbacks:

#+BEGIN_SRC elisp
(setq kernel-info (jupyter-message-content
                   (jupyter-wait-until-received :kernel-info-reply
                     (jupyter-kernel-info-request client))))
#+END_SRC

Or you could have interactivity between =org-mode= and a =Jupyter= kernel:

#+BEGIN_SRC elisp
(defclass jupyter-org-client (jupyer-kernel-client)
  (src-block-marker))


(cl-defmethod jupyter-execute-request ((client jupyter-org-client))

  )

(cl-defmethod jupyter-handle-execute-result ((client jupyter-org-client)
                                             execution-count
                                             data
                                             metadata)
  )
#+END_SRC

* Known problems

- Reading messages from the IOLoop is a bottleneck. If many messages are coming
  in, it seems its hard to get messages sent.

* TODO Wish list

- Notebook interface
- Integration with Jupyter widgets
  - One solution would be to show a browser and inject the necessary javascript
    using ~skewer-mode~. Look