Previously it was hard to step through the code whenever messages were
coming in live from a kernel due to the asynchronous nature of
handling process output in Emacs.
but first...
* jupyter-base.el: ...idle-received-p -> idle-p for jupyter-request objects
* jupyter-client.el: ...then, do it.
* test/jupyter-test.el: Do it.
These will be used in future commits.
* jupyter-client.el (jupyter-idle-sync)
(jupyter-add-idle-sync-hook): New functions.
* test/jupyter-test.el (jupyter-idle-sync): New test.
The following changes are made:
jupyter--run-handler-p -> jupyter--request-allows-handler-p
jupyter--handler-dispatch -> jupyter--run-handler
* jupyter-client.el: Do it.
* test/jupyter-test.el: Do it.
`jupyter-handle-*` methods now only take an argument list like
(client req msg)
instead of each method being passed an expanded list of arguments
unpacked from a message contents. Not sure why the unpacking was done
in the first place. The strategy now is to use
`jupyter-with-message-contents` when accessing the contents of a
message.
* jupyter-client.el
(define-jupyter-client-handler)
(define--jupyter-client-sender): New macros.
(jupyter-send-execute-request)
(jupyter-send-inspect-request)
(jupyter-send-complete-request)
(jupyter-send-history-request)
(jupyter-send-is-complete-request)
(jupyter-send-comm-info-request)
(jupyter-send-comm-open)
(jupyter-send-comm-msg)
(jupyter-send-comm-close)
(jupyter-send-kernel-info-request)
(jupyter-send-shutdown-request): Use `define--jupyter-client-sender`.
(jupyter-handle-input-request)
(jupyter-handle-execute-reply)
(jupyter-handle-inspect-reply)
(jupyter-handle-complete-reply)
(jupyter-handle-history-reply)
(jupyter-handle-is-complete-reply)
(jupyter-handle-comm-info-reply)
(jupyter-handle-kernel-info-reply)
(jupyter-handle-shutdown-reply)
(jupyter-handle-comm-open)
(jupyter-handle-comm-close)
(jupyter-handle-stream)
(jupyter-handle-execute-input)
(jupyter-handle-execute-result)
(jupyter-handle-error)
(jupyter-handle-status)
(jupyter-handle-clear-output)
(jupyter-handle-display-data)
(jupyter-handle-update-display-data): Use
`define-jupyter-client-handler`.
* jupyter-org-client.el
(jupyter-handle-stream)
(jupyter-handle-error)
(jupyter-handle-execute-result)
(jupyter-handle-display-data)
(jupyter-handle-execute-reply): Update to take into account changes in
the argument lists of `jupyter-handle-*` methods.
* jupyter-python.el
(jupyter-handle-error): Ditto.
* jupyter-repl.el
(jupyter-handle-execute-reply)
(jupyter-handle-execute-result)
(jupyter-handle-clear-output)
(jupyter-handle-display-data)
(jupyter-handle-update-display-data)
(jupyter-handle-status)
(jupyter-handle-stream)
(jupyter-handle-error):
(jupyter-handle-history-reply)
(jupyter-handle-shutdown-reply): Ditto.
* jupyter-widget-client.el
(jupyter-handle-comm-open)
(jupyter-handle-comm-close)
(jupyter-handle-comm-msg): Ditto.
Reduce the number of arguments `jupyter-handle-message` takes, by just
passing the message property list directly. Expanding out the
arguments just creates unnecessary work. Make the
`jupyter-kernel-client' implementation of `jupyter-handle-message`
more readable. Remove the unnecessary and confusing handler method
dispatching. Document `jupyter-handle-message`.
* README.org: Update reference to `jupyter-handle-*` method.
* jupyter-client.el
(jupyter-dispatch-messages-cases): Remove.
(jupyter-run-hook-with-args-until-success): Remove.
(jupyter-handle-message-p): New function. Similar to the above except
its return value is negated.
(jupyter--client-handlers): New variable. Holds the table of message
handlers.
(jupyter--handler-dispatch): Use them.
(jupyter--run-callbacks): Change to `defsubst`
(jupyter--run-handler-maybe): Remove. Expand out at call sites, replace
`jupyter-handle-message` with `jupyter--handler-dispatch` at the sites.
(jupyter--handler-dispatch): New function. Unifies all the
`jupyter-handle-message` implementations for channels.
(jupyter--update-execution-state, jupyter--message-completes-request-p): New functions.
(jupyter-handle-message): Remove channel implementations. Use the new
functions in the client implementation.
The following changes are made:
`jupyter-initialize-connection` -> `jupyter-comm-initialize`
`jupyter-connect-client` -> `jupyter-comm-add-handler`
`jupyter-disconnect-client` -> `jupyter-comm-remove-handler`
`jupyter-comm-client-loop` -> `jupyter-comm-handler-loop`
* README.org: Do it.
* jupyter-channel-ioloop-comm: Do it.
* jupyter-client.el: Do it.
* jupyter-comm-layer.el: Do it.
(jupyter-comm-layer): Rename `clients` slot to `handlers`. Update all uses.
* jupyter-kernel-process-manager.el: Do it.
* jupyter-repl.el: Do it.
* jupyter-server.el: Do it.
* jupyter-zmq-channel-comm.el: Do it.
* test/jupyter-server-test.el: Do it.
* test/jupyter-test.el: Do it.
* README.org: Add notes on behavior when a kernel's language name has spaces
and uppercase characters.
* jupyter-base.el (jupyter-canonicalize-language-string): New function.
* jupyter-client.el (jupyter-kernel-info): Use it.
* jupyter-kernelspec.el (jupyter-available-kernelspecs): Use it.
* ob-jupyter.el (org-babel-jupyter-aliases-from-kernelspecs): Use it.
* test/jupyter-test.el (jupyter-canonicalize-language-string): New test.
`jupyter-wait-until-idle` can return immediately if an idle message has already
been received, but the function `jupyter-wait-until` will eventually raise an
error in that case. Avoid the error by checking
`jupyter-request-idle-received-p` before calling `jupyter-wait-until`.
* jupyter-client.el (jupyter-wait-until-idle): Do it.
* ob-jupyter.el (org-babel-execute:jupyter): Account for changes in
`jupyter-wait-until-idle`.
Move the whitespace that was originally concatenated to the prefix variable `jupyter-eval-overlay-prefix` to the variable itself, so that it can be easily customized. For instance, in the case where one does not want any prefix, with the changes, it can be easily removed by setting the prefix variable to the empty string. Originally one would have to also modify the function `jupyter-eval-ov--propertize` to achieve the same result.
* jupyter-base.el: Don't add `jupyter-server-mode--unset-client-soon` to
`server-switch-hook` at toplevel
(jupyter-server-mode-set-client): Add `jupyter-server-mode--unset-client-soon`
to `server-switch-hook` if necessary.
* jupyter-client.el: (jupyter--run-callbacks): Use `when-let`.
(jupyter--set-callback): Remove.
(jupyter--add-callback): Use `add-function`.
* jupyter-ioloop.el (jupyter-ioloop-add-arg-type): Use `setf` along with `alist-get`.
(jupyter-ioloop--replace-args): Use `pcase`.
(jupyter-ioloop--event-dispatcher): Extract out event cases into variable `user-events`.
(jupyter-ioloop--body): New function.
(jupyter-ioloop--function): Use it.
* jupyter-kernel-process-manager.el
(jupyter--kernel-died-process-sentinel)
(jupyter-stop-channels): Use `when-let`.
* jupyter-org-client.el (jupyter-org-interaction-mode): Use `cl-callf2?`.
* jupyter-server.el: Add TODO.
* jupyter-tramp.el: Remove unused function declaration.
* README.org: Add section on `jupyter-eval-use-overlays`, minor formatting fix
* jupyter-client.el (jupyter-eval-overlay): New face.
(jupyter-eval-use-overlays, jupyter-eval-overlay-prefix): New custom variables.
(jupyter--display-eval-result): Remove function.
(jupyter-eval): Use `jupyter-eval-string`.
(jupyter-eval-result-callbacks): New function.
(jupyter-eval-add-callbacks): Result callbacks now obtained from
`jupyter-eval-result-callbacks`, only add callbacks for non-result message
types. Allow `beg` and `end` arguments remove `result-cb` argument, update all
callers.
(jupyter-eval-string): Allow `beg` and `end` arguments remove `cb` argument,
update all callers.
(jupyter-eval-string-command): Remove `cb` argument, update all callers.
(jupyter-eval-region): Ditto.
(jupyter-eval-line-or-region): Refactor.
(jupyter-eval-overlay-keymap): New keymap.
(jupyter-eval-ov--delete, jupyter-eval-ov--remove-all)
(jupyter-eval-ov--propertize, jupyter-eval-ov--fold-boundary)
(jupyter-eval-ov--expand-string, jupyter-eval-ov--make)
(jupyter-eval-ov--expand, jupyter-eval-ov--fold)
(jupyter-eval-toggle-overlay, jupyter-eval-remove-overlays)
(jupyter-eval-display-overlay, jupyter-eval-display-with-overlay-p): New
functions.
* jupyter-repl.el (jupyter-eval-string): Ensure callbacks are added in the
original (non-REPL) buffer the command was called from. So that
`jupyter-eval-display-with-overlay-p`, indirectly called by
`jupyter-eval-add-callbacks`, works.
(jupyter-repl-interaction-mode-map): Set `C-c C-o` binding to
`jupyter-eval-remove-overlays`.
Fixes#137
* jupyter-base.el (jupyter-kernelspec): Un-require.
(jupyter-command, jupyter-locate-python)
(jupyter-runtime-directory): Move to new file `jupyter-env.el`
(jupyter-include-other-output, jupyter-iopub-message-hook)
(jupyter-shell-message-hook)
(jupyter-stdin-message-hook): Move to `jupyter-client.el`
(jupyter-sha256, jupyter-hmac-sha256):
(jupytern-new-uuid): Move to `jupyter-messages.el`. Add declaration of
`jupyter-new-uuid` to account for its removal.
(jupyter-create-connection-info)
(jupyter-write-connection-file): Move to `jupyter-kernel-manager.el`
(jupyter-connect-endpoint, jupyter-connect-channel): Move to `jupyter-channels.el`
* jupyter-channels.el: Accept moved functions.
* jupyter-client.el: Accept moved variables.
* jupyter-kernel-manager.el: Accept moved functions.
(jupyter-env, jupyter-kernelspec): New requires.
* jupyter-kernelspec (jupyter-env): New require.
(jupyter-command): Remove declaration.
(jupyter-read-plist-from-string): New declaration.
* jupyter-messages.el: Accept moved functions.
(hmac-def, json): New requires.
* jupyter-org-extensions.el (jupyter-kernelspec): New require.
* jupyter-repl.el (jupyter-kernelspec): New require.
* jupyter-env.el: New file.
* ob-jupyter.el (jupyter-env, jupyter-kernelspec): New requires.
* test/jupyter-test.el (jupyter-env): New require.
Speeds up the line count test associated with
`jupyter-eval-short-result-max-lines` by an order of magnitude for somewhat
large strings. The byte compiled versions are not that different.
This accounts for idle messages being dropped before a request's reply message.
In such a case, the request is dropped from the client's request table before
the handlers/callbacks of the request get a chance to handle the reply message.
The completion at point functions are called several times in the course of a
single completion attempt so we get multiple chances to receive a completion
result. This means there is no need to wait for it to be received once and for
all on the first time completions are requested.
So that client subclasses can provide specialized behavior if needed.
Also adds the function `jupyter-eval-add-callbacks` so that client methods can
more easily obtain the default behavior of `jupyter-eval-string'.
This refactor implements a new class hierarchy to manage the lifetime of a
Jupyter kernel. The first node in this hierarchy is the
`jupyter-kernel-lifetime` class which defines a set of methods to manage the
lifetime of a kernel. An object that inherits from `jupyter-kernel-lifetime` is
stating that it has an association with a kernel and can be used to manage the
lifetime of the associated kernel.
The `jupyter-meta-kernel` class inherits from `jupyter-kernel-lifetime` and
mainly defines a `spec` slot used to hold the `kernelspec` from which a command
can be constructed to start a kernel and a `session` slot used to hold the
`jupyter-session` object that clients can use to establish communication with a
kernel once its live. Concrete classes that actually launch kernels are
intended to inherit from this class and use its slots.
`jupyter-kernel-process` manages the lifetime of a kernel started as a process
using the function `start-file-process`, `jupyter-command-kernel` calls the
`jupyter kernel` shell command to start a kernel, finally `jupyter-spec-kernel`
uses the `spec` slot to construct a shell command to start a kernel.
A `jupyter-kernel-manager` now consists of a `kernel` slot that holds a
`jupyter-meta-kernel` and a `control-channel` slot and inherits from
`jupyter-kernel-lifetime`. The `jupyter-kernel-lifetime` methods of the manager
just defer to those of `kernel` while also taking into account the
`control-channel`.
* jupyter-base.el (jupyter-write-connection-file): New function.
* jupyter-channel-ioloop.el
(jupyter-channel-ioloop-add-start-channel-event): Remove `sleep-for` call.
The startup message is not so important anymore.
* jupyter-client.el (jupyter-wait-until-startup: New function.
* jupyter-kernel-manager.el (jupyter-kernel-lifetime)
(jupyter-kernel, jupyter-kernel-process, jupyter-command-kernel)
(jupyter-spec-kernel): New classes.
(jupyter-kernel-manager): Inherit from jupyter-kernel-lifetime only and
implement its methods.
(jupyter-kernel-manager--cleanup, jupyter-kernel-managers)
(jupyter-delete-all-kernels, jupyter--kernel-sentinel)
(jupyter--start-kernel): Remove and remove related, their functionality has
been generalized in the new classes.
(jupyter-interrupt-kernel, jupyter-shutdown-kernel)
(jupyter-start-channels, jupyter-start-kernel, jupyter-kernel-alive-p)
(jupyter-kill-kernel): Refactor and implement to use the new class hierarchy.
* test/jupyter-test.el: Refactor tests to account for changes.
(jupyter-write-connect-file, jupyter-command-kernel): New tests.
* jupyter-kernelspec.el (jupyter-guess-kernelspec): New function.
The previous mechanism to communicate with a kernel was too low level from the
perspective of a client. The client interfaced directly with the subprocess
abstraction, `jupyter-ioloop`, and had to handle all "events" that occurred in
the `jupyter-ioloop`, e.g. when a channel was started or stopped. But in
reality such events should not be the concern of a client.
A client should only care about events that are directly related to kernel
messages and not events related to the implementation details of *how*
communication occurs.
This commit abstracts out the way in which a client communicates with its
kernel by introducing a new `jupyter-comm-layer` class. The
`jupyter-comm-layer` class takes care of managing the communication channel
between a kernel and its clients as well as sending events to all registered
clients. This way, clients operate solely at the level of events on the
communication layer. All a client does is register itself to receive events on
the communication layer and send events on the layer.
* jupyter-base.el (jupyter-session-endpoints): New function.
* jupyter-client.el (jupyter-kernel-client): Remove ioloop and channels slots.
Add kcomm slot.
(initialize-instance): Unconditionally stop channels.
(jupyter-initialize-connection): Change into a method call.
Call `jupyter-initialize-connection` on the `kcomm` slot.
(jupyter-with-client-buffer): Remove stale comment.
(jupyter-send): Call `jupyter-send` on the `kcomm` slot.
(jupyter-ioloop-handler): Remove all method definitions, replace `sent` and
`message` methods with their `jupyter-event-handler` equivalents.
(jupyter-hb-pause, jupyter-hb-unpause, jupyter-hb-beating):
(jupyter-channel-alive-p, jupyter-start-channel, jupyter-stop-channel):
(jupyter-start-channels, jupyter-stop-channels):
Replace with calls to their equivalents using the `kcomm` slot.
* jupyter-comm-layer.el: New file.
* jupyter-kernel-manager (jupyter-make-client): Set a client's `kcomm` slot to
`jupyter-channel-ioloop-comm`.
* jupyter-messages.el (jupyter-decode-message): Use `list` directly. There
seemed to be issues when using the new `jupyter-sync-channel-comm` due to
using quoted lists.
* test/jupyter-test.el: Add `jupyter-comm-layer` test. Update other tests.
* test/test-helper.el: Add `jupyter-comm-layer` mock objects. Update
`jupyter-echo-client`.
