Ob-ein Improvements

Bring the org offering to feature parity.
This commit is contained in:
dickmao 2019-05-16 15:01:45 -04:00
parent 74e79c7b94
commit e491ac6f1c
24 changed files with 340 additions and 738 deletions

2
.gitignore vendored
View file

@ -29,4 +29,4 @@ _static
.gitattributes
.ecukes*
dist
*.ob-ein.ipynb
.*ein*.ipynb

View file

@ -17,6 +17,7 @@ cache:
- $HOME/.cache/pip
- $HOME/.evm
- $HOME/.emacs.d
- $HOME/Library/Caches/Homebrew
- $HOME/.pyenv
- $HOME/Library/Caches/pip
@ -84,4 +85,4 @@ script:
make test-jupyterhub
fi
- rm -rf $HOME/.matplotlib $HOME/.cache/fontconfig
- make test || ( ( zip -q - log/{testein,testfunc,ecukes}.* 2>/dev/null | uuencode log.zip ) && ( printf "To diagnose, travis logs -i | dos2unix | sed '/^begin 664/,/^end/!d' | uudecode" ) && false)
- make test || ( ( zip -q - log/{testein,testfunc,ecukes}.* 2>/dev/null | uuencode log.zip ) && ( printf "To diagnose, travis logs -i | dos2unix | sed '/^begin 644/,/^end/!d' | uudecode" ) && false)

4
Cask
View file

@ -12,7 +12,7 @@
(depends-on "ert-runner")
(depends-on "ecukes")
(depends-on "espuds")
(depends-on "org-plus-contrib") ;; see https://github.com/cask/cask/issues/119
;; (depends-on "org-plus-contrib") ;; see https://github.com/cask/cask/issues/119
(depends-on "mocker")
(depends-on "skewer-mode")
(depends-on "deferred")
@ -21,6 +21,8 @@
(depends-on "smartrep")
(depends-on "polymode")
(depends-on "markdown-mode")
(depends-on "julia-mode")
(depends-on "ess")
(depends-on "px")
(depends-on "f")
(depends-on "s"))

View file

@ -13,7 +13,7 @@ endif
.DEFAULT_GOAL := test-compile
README.rst: README.in.rst
README.rst: README.in.rst lisp/ein.el
cask eval "(progn \
(add-to-list 'load-path \"./lisp\") \
(load \"ein-notebook\") \

View file

@ -19,7 +19,10 @@
:target: http://melpa-stable.milkbox.net/#/ein
:alt: MELPA stable version
.. _Jupyter: http://jupyter.org
.. _tkf: https://tkf.github.io/emacs-ipython-notebook
.. _Babel: https://orgmode.org/worg/org-contrib/babel/intro.html
.. _Org: https://orgmode.org
.. _[tkf]: http://tkf.github.io
.. _[gregsexton]: https://github.com/gregsexton/ob-ipython
Install
=======
@ -35,6 +38,8 @@ Start EIN using **one** of the following:
Use ``C-u M-x ein:login`` for services such as ``mybinder.org`` requiring cookie authentication.
Alternatively, ob-ein_.
.. _Cask: https://cask.readthedocs.io/en/latest/guide/installation.html
.. _MELPA: http://melpa.org/#/
@ -43,7 +48,7 @@ It doesn't work
EIN is tested on GNU Emacs versions
.. CI VERSION (see Makefile)
and later. Your mileage may vary with the `spacemacs layer`_ and other *emacsen*.
and later. We presently do not recommend the `spacemacs layer`_.
You may also try to self-diagnose:
@ -68,29 +73,41 @@ Enable `polymode`_ via::
M-x customize-group RET ein
Toggle Ein:Polymode
Org-mode Integration
====================
ob-ein
======
EIN provides org-babel functionality similar to ob-ipython_ and scimax_.
Configuration:
*Language* is ``ein``. The ``:session`` header argument is the notebook url, e.g., ``https://localhost:8888/my.ipynb``, or simply ``localhost``, in which case EIN will evaluate org blocks in an anonymous notebook::
::
#BEGIN_SRC ein :session localhost :results raw drawer :image output.png
import matplotlib.pyplot as plt
import numpy as np
M-x customize-group RET org-babel
Org Babel Load Languages:
Insert (ein . t)
For example, '((emacs-lisp . t) (ein . t))
%matplotlib inline
x = np.linspace(0, 1, 100)
y = np.random.rand(100,1)
plt.plot(x,y)
Snippet:
::
#BEGIN_SRC ein-python :session localhost :results raw drawer
import numpy, math, matplotlib.pyplot as plt
%matplotlib inline
x = numpy.linspace(0, 2*math.pi)
plt.plot(x, numpy.sin(x))
#+END_SRC
You may also specify the port, i.e., ``localhost:8889``. See `ob-ein details`_.
The ``:session`` is the notebook url, e.g., ``http://localhost:8888/my.ipynb``, or simply ``localhost``, in which case org evaluates anonymously. A port may also be specified, e.g., ``localhost:8889``.
*Language* can be ``ein-python``, ``ein-r``, or ``ein-julia``. **The relevant** `jupyter kernel`_ **must be installed before use**. Additional languages can be configured via::
M-x customize-group RET ein
Ob Ein Languages
.. _polymode: https://github.com/polymode/polymode
.. _ob-ipython: https://github.com/gregsexton/ob-ipython
.. _scimax: https://github.com/jkitchin/scimax
.. _ob-ein details: http://millejoh.github.io/emacs-ipython-notebook/#org-mode-integration
.. _jupyter kernel: https://github.com/jupyter/jupyter/wiki/Jupyter-kernels
Connected Buffers
=================

View file

@ -4,11 +4,14 @@
--- or **E**\ IN **I**\ s not only for pytho\ **N**\ .
Emacs IPython Notebook (EIN) lets you edit and run Jupyter_ (formerly IPython)
Emacs IPython Notebook (EIN) lets you run Jupyter (formerly IPython)
notebooks within Emacs. It channels all the power of Emacs without the
idiosyncrasies of in-browser editing.
EIN was originally written by tkf_. More `complete documentation`_ is available.
Org_ users please find ob-ein_, a jupyter Babel_ backend.
EIN was originally written by `[tkf]`_. A jupyter Babel_ backend was first
introduced by `[gregsexton]`_.
.. |build-status|
image:: https://secure.travis-ci.org/millejoh/emacs-ipython-notebook.png?branch=master
@ -23,7 +26,10 @@ EIN was originally written by tkf_. More `complete documentation`_ is available
:target: http://melpa-stable.milkbox.net/#/ein
:alt: MELPA stable version
.. _Jupyter: http://jupyter.org
.. _tkf: https://tkf.github.io/emacs-ipython-notebook
.. _Babel: https://orgmode.org/worg/org-contrib/babel/intro.html
.. _Org: https://orgmode.org
.. _[tkf]: http://tkf.github.io
.. _[gregsexton]: https://github.com/gregsexton/ob-ipython
Install
=======
@ -39,6 +45,8 @@ Start EIN using **one** of the following:
Use ``C-u M-x ein:login`` for services such as ``mybinder.org`` requiring cookie authentication.
Alternatively, ob-ein_.
.. _Cask: https://cask.readthedocs.io/en/latest/guide/installation.html
.. _MELPA: http://melpa.org/#/
@ -47,7 +55,7 @@ It doesn't work
EIN is tested on GNU Emacs versions
25.1
and later. Your mileage may vary with the `spacemacs layer`_ and other *emacsen*.
and later. We presently do not recommend the `spacemacs layer`_.
You may also try to self-diagnose:
@ -72,29 +80,41 @@ Enable `polymode`_ via::
M-x customize-group RET ein
Toggle Ein:Polymode
Org-mode Integration
====================
ob-ein
======
EIN provides org-babel functionality similar to ob-ipython_ and scimax_.
Configuration:
*Language* is ``ein``. The ``:session`` header argument is the notebook url, e.g., ``https://localhost:8888/my.ipynb``, or simply ``localhost``, in which case EIN will evaluate org blocks in an anonymous notebook::
::
#BEGIN_SRC ein :session localhost :results raw drawer :image output.png
import matplotlib.pyplot as plt
import numpy as np
M-x customize-group RET org-babel
Org Babel Load Languages:
Insert (ein . t)
For example, '((emacs-lisp . t) (ein . t))
%matplotlib inline
x = np.linspace(0, 1, 100)
y = np.random.rand(100,1)
plt.plot(x,y)
Snippet:
::
#BEGIN_SRC ein-python :session localhost :results raw drawer
import numpy, math, matplotlib.pyplot as plt
%matplotlib inline
x = numpy.linspace(0, 2*math.pi)
plt.plot(x, numpy.sin(x))
#+END_SRC
You may also specify the port, i.e., ``localhost:8889``. See `ob-ein details`_.
The ``:session`` is the notebook url, e.g., ``http://localhost:8888/my.ipynb``, or simply ``localhost``, in which case org evaluates anonymously. A port may also be specified, e.g., ``localhost:8889``.
*Language* can be ``ein-python``, ``ein-r``, or ``ein-julia``. **The relevant** `jupyter kernel`_ **must be installed before use**. Additional languages can be configured via::
M-x customize-group RET ein
Ob Ein Languages
.. _polymode: https://github.com/polymode/polymode
.. _ob-ipython: https://github.com/gregsexton/ob-ipython
.. _scimax: https://github.com/jkitchin/scimax
.. _ob-ein details: http://millejoh.github.io/emacs-ipython-notebook/#org-mode-integration
.. _jupyter kernel: https://github.com/jupyter/jupyter/wiki/Jupyter-kernels
Connected Buffers
=================

View file

@ -160,4 +160,5 @@ Scenario: Smoke test julia
Given new julia notebook
When I type "isapprox(Base.MathConstants.e ^ (pi * im), -1)"
And I wait for cell to execute
Then I should see "true"
Then I should see "true"
And I dump buffer

View file

@ -1,3 +1,48 @@
@memory
Scenario: R and Julia in the same org file
Given I stop the server
When I open temp file "ecukes.org"
And I call "org-mode"
And I type "<s"
And I press "TAB"
And I type "ein-r :session localhost :results raw drawer"
And I press "RET"
And I type "data.frame(x=c("a", "b c", "d"), y=1:3)"
And I ctrl-c-ctrl-c
And I wait for buffer to say " x y\n1 a 1\n2 b c 2\n3 d 3"
And I should not see "[....]"
And I press "M->"
And I type "<s"
And I press "TAB"
And I type "ein-julia :session localhost :results raw drawer"
And I press "RET"
And I type "isapprox(Base.MathConstants.e ^ (pi * im), -1)"
And I ctrl-c-ctrl-c
And I wait for buffer to say "true"
And I dump buffer
@org
Scenario: ein-python can be python2 or python3
Given I stop the server
When I open temp file "ecukes.org"
And I call "org-mode"
And I type "<s"
And I press "TAB"
And I type "ein :session localhost :results raw drawer"
And I press "RET"
And I type "(1 + 5 ** 0.5) / 2"
And I ctrl-c-ctrl-c
And I wait for buffer to say "1.618"
And I should not see "[....]"
And I place the cursor before ":results"
And I type ":kernelspec python5 "
And I ctrl-c-ctrl-c
And I switch to log expr "ein:log-all-buffer-name"
And I wait for buffer to say "ob-ein--initiate-session: switching"
And I switch to buffer like "ecukes.org"
And I wait for buffer to say "1.618"
And I dump buffer
@org
Scenario: Specific port, portless localhost refers to same, concurrent execution
Given I stop the server
@ -11,6 +56,7 @@ Scenario: Specific port, portless localhost refers to same, concurrent execution
And I dump buffer
And I ctrl-c-ctrl-c
And I wait for buffer to say "1.618"
And I should not see "[....]"
And I press "M->"
And I type "<s"
And I press "TAB"
@ -21,6 +67,7 @@ Scenario: Specific port, portless localhost refers to same, concurrent execution
And I clear log expr "ein:log-all-buffer-name"
And I ctrl-c-ctrl-c
And I wait for buffer to say "3.14159"
And I should not see "[....]"
And I switch to log expr "ein:log-all-buffer-name"
Then I should not see "Login to"
And I switch to buffer like "ecukes.org"
@ -42,6 +89,7 @@ Scenario: Specific port, portless localhost refers to same, concurrent execution
And I wait for buffer to say "1.618"
And I dump buffer
And I wait for buffer to say "3.1415"
And I should not see "[....]"
@org
Scenario: portless url with path, image, C-c ' lets you C-c C-c as well
@ -56,6 +104,7 @@ Scenario: portless url with path, image, C-c ' lets you C-c C-c as well
And I type "(1 + 5 ** 0.5) / 2"
And I ctrl-c-ctrl-c
And I wait for buffer to say "1.618"
And I should not see "[....]"
And I press "M->"
And I type "<s"
And I press "TAB"
@ -78,14 +127,3 @@ Scenario: portless url with path, image, C-c ' lets you C-c C-c as well
And I ctrl-c-ctrl-c
And I dump buffer
And I wait for buffer to say "file:ein-image"
And I press "C-c '"
And I switch to buffer like "Org Src"
And I press "C-a"
And I press "C-k"
And I type "import math ; math.e"
And I dump buffer
And I press "C-c C-c"
And I press "C-c C-k"
And I switch to buffer like "path.org"
And I dump buffer
And I wait for buffer to say "2.718"

View file

@ -6,7 +6,7 @@
(lambda (port)
(ein:process-refresh-processes)
(assert (not (ein:process-url-match (ein:url port))))
(When (format "I type \"ein :session localhost:%s :result raw drawer\"" port))))
(When (format "I type \"ein :session localhost:%s :results raw drawer\"" port))))
(When "^I ctrl-c-ctrl-c$"
(lambda ()
@ -152,11 +152,11 @@
(multiple-value-bind (url-or-port token) (ein:jupyter-server-conn-info)
(let (notebook)
(with-current-buffer (ein:notebooklist-get-buffer url-or-port)
(let* ((kslist (mapcar #'car (ein:list-available-kernels url-or-port)))
(found (or (seq-some (lambda (x) (and (search prefix x) x)) kslist)
(error "No kernel %s among %s" prefix kslist)))
(ks (ein:get-kernelspec url-or-port found)))
(ein:and-let* ((kslist (mapcar #'car (ein:list-available-kernels url-or-port)))
(found (seq-some (lambda (x) (and (search prefix x) x)) kslist))
(ks (ein:get-kernelspec url-or-port found)))
(setq notebook (ein:testing-new-notebook url-or-port ks))))
(should notebook)
(let ((buf-name (format ein:notebook-buffer-name-template
(ein:$notebook-url-or-port notebook)
(ein:$notebook-notebook-name notebook))))
@ -336,12 +336,13 @@
(When "^I wait for buffer to say \"\\(.+\\)\"$"
(lambda (bogey)
(ein:testing-wait-until
(lambda () (ein:aif (s-contains? bogey (buffer-string)) it
(when (with-current-buffer ein:log-all-buffer-name
(search "WS closed unexpectedly" (buffer-string)))
(Then "I ctrl-c-ctrl-c")
(And "I clear log expr \"ein:log-all-buffer-name\""))
nil))
(lambda ()
(ein:aif (s-contains? (s-replace "\\n" "\n" bogey) (buffer-string)) it
(when (with-current-buffer ein:log-all-buffer-name
(search "WS closed unexpectedly" (buffer-string)))
(And "I clear log expr \"ein:log-all-buffer-name\"")
(Then "I ctrl-c-ctrl-c"))
nil))
nil 40000 2000)))
(When "^I wait for cell to execute$"

View file

@ -3,6 +3,9 @@
(require 'espuds)
(require 'ert)
(with-eval-after-load "python"
(setq python-indent-guess-indent-offset-verbose nil))
(let* ((support-path (f-dirname load-file-name))
(root-path (f-parent (f-parent support-path))))
(add-to-list 'load-path (concat root-path "/lisp"))
@ -14,6 +17,7 @@
(require 'ein-testing)
(require 'ein-ipynb-mode)
(require 'ein-contents-api)
(require 'poly-ein)
(require 'ob-ein)
(if (member "timestamp" ecukes-include-tags)
@ -23,14 +27,14 @@
(unless (member "jupyterhub" ecukes-include-tags)
(!cons "jupyterhub" ecukes-exclude-tags))
(unless ein:polymode
(!cons "julia" ecukes-exclude-tags))
(when (file-exists-p (concat default-directory "features/support/test-poly.el"))
(load-file (concat default-directory "features/support/test-poly.el")))
(if (eq system-type 'darwin)
(!cons "switch" ecukes-exclude-tags))
(if (> (string-to-number org-version) 9.1) ;; they got rid of easy templates
(!cons "org" ecukes-exclude-tags))
(cond ((not ein:polymode)
(!cons "julia" ecukes-exclude-tags)
(!cons "memory" ecukes-exclude-tags))
((string= (getenv "TRAVIS_OS_NAME") "linux")
(!cons "memory" ecukes-exclude-tags)))
(defvar ein:testing-jupyter-server-root (f-parent (f-dirname load-file-name)))
@ -41,18 +45,18 @@
(loop for notebook in (ein:notebook-opened-notebooks)
for path = (ein:$notebook-notebook-path notebook)
do (ein:notebook-kill-kernel-then-close-command notebook)
do (loop repeat 8
do (loop repeat 16
until (not (ein:notebook-live-p notebook))
do (sleep-for 0 500)
do (sleep-for 0 1000)
finally do (when (ein:notebook-live-p notebook)
(ein:display-warning (format "cannot close %s" path))))
do (when (or (search "Untitled" path) (search "Renamed" path))
(ein:notebooklist-delete-notebook path)
(loop repeat 8
(loop repeat 16
with fullpath = (concat (file-name-as-directory ein:testing-jupyter-server-root) path)
for extant = (file-exists-p fullpath)
until (not extant)
do (sleep-for 0 500)
do (sleep-for 0 1000)
finally do (when extant
(ein:display-warning (format "cannot del %s" path))))))))
(ein:aif (ein:notebook-opened-notebooks)

View file

@ -352,6 +352,19 @@ global setting. For global setting and more information, see
:error (apply-partially #'ein:content-rename-error (ein:$content-path content)))
(ein:content-legacy-rename content new-path callback cbargs)))
(defun ein:session-rename (url-or-port session-id new-path)
(ein:query-singleton-ajax
(list 'session-rename session-id new-path)
(ein:url url-or-port "api/sessions" session-id)
:type "PATCH"
:data (json-encode `((path . ,new-path)))
:complete #'ein:session-rename--complete))
(defun* ein:session-rename--complete (&key data response symbol-status
&allow-other-keys
&aux (resp-string (format "STATUS: %s DATA: %s" (request-response-status-code response) data)))
(ein:log 'debug "ein:session-rename--complete %s" resp-string))
(defun* update-content-path (content callback cbargs &key data &allow-other-keys)
(setf (ein:$content-path content) (plist-get data :path)
(ein:$content-name content) (plist-get data :name)

View file

@ -173,7 +173,8 @@ the log of the running jupyter server."
*ein:last-jupyter-command*
*ein:last-jupyter-directory*
(if (numberp port)
`("--port" ,(format "%s" port))))))
`("--port" ,(format "%s" port)
"--port-retries" "0")))))
(when (eql system-type 'windows-nt)
(accept-process-output proc (/ ein:jupyter-server-run-timeout 1000)))
(loop repeat 30

View file

@ -70,7 +70,6 @@ in these buffer will be synced with the kernel's cwd.")
(defun ein:kernelinfo-update-all (kerinfo)
"Update KERINFO slots by triggering all update functions."
(ein:log 'debug "EIN:KERNELINFO-UPDATE-ALL")
(ein:log 'debug "(ein:kernel-live-p kernel) = %S"
(ein:kernel-live-p (slot-value kerinfo 'kernel)))
(ein:kernelinfo-update-ccwd kerinfo)

View file

@ -69,6 +69,7 @@
(require 'ein-shared-output)
(require 'ein-notebooklist)
(require 'ein-multilang)
(require 'ob-ein)
(require 'poly-ein)
;;; Configuration
@ -219,15 +220,22 @@ Current buffer for these functions is set to the notebook buffer.")
;;; Constructor
(defun ein:notebook-new (url-or-port notebook-path kernelspec &rest args)
(if (or (stringp kernelspec) (symbolp kernelspec))
(setq kernelspec (ein:get-kernelspec url-or-port kernelspec)))
(let ((notebook (apply #'make-ein:$notebook
:url-or-port url-or-port
:kernelspec kernelspec
:notebook-path notebook-path
args)))
notebook))
(defun ein:notebook-new (url-or-port notebook-path pre-kernelspec &rest args)
(let ((kernelspec
(cond ((ein:$kernelspec-p pre-kernelspec) pre-kernelspec)
((consp pre-kernelspec)
(loop for (name ks) on (ein:need-kernelspecs url-or-port) by 'cddr
when (and (ein:$kernelspec-p ks)
(string= (cdr pre-kernelspec)
(cl-struct-slot-value
'ein:$kernelspec (car pre-kernelspec) ks)))
return ks))
(t (ein:get-kernelspec url-or-port pre-kernelspec)))))
(apply #'make-ein:$notebook
:url-or-port url-or-port
:kernelspec kernelspec
:notebook-path notebook-path
args)))
;;; Destructor
@ -328,8 +336,9 @@ will be updated with kernel's cwd."
(pm-select-buffer (pm-innermost-span))
(pop-to-buffer (pm-span-buffer (pm-innermost-span))))
(pop-to-buffer (ein:notebook-buffer notebook*)))))
(when (null (plist-member (ein:$notebook-metadata notebook*)
:kernelspec))
(when (and (not noninteractive)
(null (plist-member (ein:$notebook-metadata notebook*)
:kernelspec)))
(ein:aif (ein:$notebook-kernelspec notebook*)
(progn
(setf (ein:$notebook-metadata notebook*)
@ -409,7 +418,12 @@ where `created' indicates a new notebook or an existing one.
;; Start websocket only after worksheet is rendered
;; because ein:notification-bind-events only gets called after worksheet's
;; buffer local notification widget is instantiated
(ein:kernel-retrieve-session (ein:$notebook-kernel notebook))
(ein:kernel-retrieve-session (ein:$notebook-kernel notebook) nil
(apply-partially (lambda (callback0* name* kernel)
(funcall callback0*)
(ein:log 'info "Notebook %s is ready" name*))
callback0
(ein:$notebook-notebook-name notebook)))
(setf (ein:$notebook-kernelinfo notebook)
(ein:kernelinfo-new (ein:$notebook-kernel notebook)
(cons #'ein:notebook-buffer-list notebook)
@ -418,10 +432,7 @@ where `created' indicates a new notebook or an existing one.
(ein:notebook--check-nbformat (ein:$content-raw-content content))
(setf (ein:$notebook-q-checkpoints notebook) q-checkpoints)
(ein:notebook-enable-autosaves notebook)
(ein:gc-complete-operation)
(ein:log 'info "Notebook %s is ready" (ein:$notebook-notebook-name notebook))
(when callback0
(funcall callback0)))))
(ein:gc-complete-operation))))
(defun ein:notebook-maybe-set-kernelspec (notebook content-metadata)
(ein:aif (plist-get content-metadata :kernelspec)
@ -530,18 +541,19 @@ notebook buffer."
(defsubst ein:notebook-toggle-latex-fragment ()
(interactive)
(if (featurep 'px)
(let ((outline-regexp "$a")
(kill-buffer-hook nil)) ;; outline-regexp never matches to avoid headline
(cl-letf (((symbol-function 'px-remove) #'ignore))
(if ein:%notebook-latex-p%
(progn
(ein:worksheet-render (ein:worksheet--get-ws-or-error))
(setq ein:%notebook-latex-p% nil))
(px-preview)
(setq ein:%notebook-latex-p% t))))
(ein:display-warning "px package not found")))
(cond (ein:polymode (ein:display-warning "ein:notebook-toggle-latex-fragment: delegate to markdown-mode"))
((featurep 'px)
(let ((outline-regexp "$a")
(kill-buffer-hook nil)) ;; outline-regexp never matches to avoid headline
(cl-letf (((symbol-function 'px-remove) #'ignore))
(if ein:%notebook-latex-p%
(progn
(ein:worksheet-render (ein:worksheet--get-ws-or-error))
(setq ein:%notebook-latex-p% nil))
(px-preview)
(setq ein:%notebook-latex-p% t)))))
(t (ein:display-warning "px package not found"))))
;;; Kernel related things
(defun ein:kernelspec-for-nb-metadata (kernelspec)
@ -897,10 +909,11 @@ NAME is any non-empty string that does not contain '/' or '\\'."
(ein:$notebook-notebook-path ein:%notebook%))))
(unless (and (string-match "\\.ipynb" path) (= (match-end 0) (length path)))
(setq path (format "%s.ipynb" path)))
(let ((content (ein:content-from-notebook ein:%notebook%)))
(ein:log 'verbose "Renaming notebook %s" (ein:notebook-url ein:%notebook%))
(let* ((notebook (ein:notebook--get-nb-or-error))
(content (ein:content-from-notebook notebook)))
(ein:log 'verbose "Renaming notebook %s to '%s'" (ein:notebook-url notebook) path)
(ein:content-rename content path #'ein:notebook-rename-success
(list ein:%notebook% content))))
(list notebook content))))
(defun ein:notebook-save-to-command (path)
"Make a copy of the notebook and save it to a new path specified by NAME.
@ -926,6 +939,11 @@ NAME is any non-empty string that does not contain '/' or '\\'.
(mapc #'ein:worksheet-set-buffer-name
(append (ein:$notebook-worksheets notebook)
(ein:$notebook-scratchsheets notebook)))
(ein:and-let* ((kernel (ein:$notebook-kernel notebook)))
(ein:session-rename (ein:$kernel-url-or-port kernel)
(ein:$kernel-session-id kernel)
(ein:$content-path content))
(setf (ein:$kernel-path kernel) (ein:$content-path content)))
(ein:log 'info "Notebook renamed to %s." (ein:$content-name content)))
(defmacro ein:notebook-avoid-recursion (&rest body)
@ -960,7 +978,8 @@ NAME is any non-empty string that does not contain '/' or '\\'.
(defun ein:notebook-ask-save (notebook callback0)
(unless callback0
(setq callback0 #'ignore))
(if (ein:notebook-modified-p notebook)
(if (and (ein:notebook-modified-p notebook)
(not (ob-ein-anonymous-p (ein:$notebook-notebook-path notebook))))
(if (y-or-n-p (format "Save %s?" (ein:$notebook-notebook-name notebook)))
(lexical-let ((success-positive 0))
(add-function :before callback0 (lambda () (setq success-positive 1)))

View file

@ -422,7 +422,7 @@ This function is called via `ein:notebook-after-rename-hook'."
;;;###autoload
(defun ein:notebooklist-new-notebook-with-name
(url-or-port kernelspec name &optional callback no-pop)
"Open new notebook and rename the notebook."
"Upon notebook-open, rename the notebook, then funcall CALLBACK."
(interactive
(let* ((url-or-port (or (ein:get-url-or-port)
(ein:default-url-or-port)))
@ -468,12 +468,6 @@ This function is called via `ein:notebook-after-rename-hook'."
(ein:log 'debug "ein:notebooklist-delete-notebook--complete %s" resp-string)
(when callback (funcall callback)))
;; Because MinRK wants me to suffer (not really, I love MinRK)...
(defun ein:get-actual-path (path)
(ein:aif (cl-position ?/ path :from-end t)
(substring path 0 it)
""))
(defun generate-breadcrumbs (path)
"Given notebooklist path, generate alist of breadcrumps of form (name . path)."
(let* ((paths (split-string path "/" t))
@ -650,10 +644,6 @@ This function is called via `ein:notebook-after-rename-hook'."
for name = (plist-get note :name)
for path = (plist-get note :path)
for last-modified = (plist-get note :last_modified)
;; (cond ((= 2 api-version)
;; (plist-get note :path))
;; ((= 3 api-version)
;; (ein:get-actual-path (plist-get note :path))))
for type = (plist-get note :type)
for opened-notebook-maybe = (ein:notebook-get-opened-notebook url-or-port path)
do (widget-insert " ")

View file

@ -4,7 +4,7 @@
;; Author: John Miller <millejoh at millejoh.com>, Takafumi Arakaki <aka.tkf at gmail.com>
;; URL: http://millejoh.github.io/emacs-ipython-notebook/
;; Keywords: applications, tools
;; Keywords: jupyter, literate programming, reproducible research
;; This file is NOT part of GNU Emacs.
@ -23,11 +23,14 @@
;;; Commentary:
;; Emacs IPython Notebook (EIN) lets you edit and run Jupyter_ (formerly IPython)
;; Emacs IPython Notebook (EIN) lets you run Jupyter (formerly IPython)
;; notebooks within Emacs. It channels all the power of Emacs without the
;; idiosyncrasies of in-browser editing.
;;
;; EIN was originally written by tkf_. More `complete documentation`_ is available.
;; Org_ users please find ob-ein_, a jupyter Babel_ backend.
;;
;; EIN was originally written by `[tkf]`_. A jupyter Babel_ backend was first
;; introduced by `[gregsexton]`_.
;;
;;; Code:

View file

@ -28,12 +28,13 @@
;;; Commentary:
;; Support executing org-babel source blocks using EIN worksheets.
;; Modelled after https://github.com/gregsexton/ob-ipython by Greg Sexton
;; Async support based on work by @khinsen on github in ob-ipython-async: https://github.com/khinsen/ob-ipython-async/blob/master/ob-ipython-async.el
;; which was in turn inspired by the scimax starter kit by @jkitchin: https://github.com/jkitchin/scimax
;;; Credits:
;; Uses code from https://github.com/gregsexton/ob-ipython (MIT License)
;;; Code:
(require 'ob-python)
(require 'org-element)
(require 'ein-utils)
(require 'ein-notebooklist)
(require 'ein-process)
@ -41,17 +42,37 @@
(defvar *ob-ein-sentinel* "[....]"
"Placeholder string replaced after async cell execution")
(defcustom ob-ein-anonymous-path ".ob-ein.ipynb"
"When session header specifies only server, prosecute all ob-ein interactions in this single anonymous notebook."
(defcustom ob-ein-languages
'(("ein" . python)
("ein-python" . python)
("ein-R" . R)
("ein-r" . R)
("ein-julia" . julia)
("ein-hy" . hy))
"ob-ein has knowledge of these (ein-LANG . LANG-MODE) pairs."
:type '(repeat (cons string symbol))
:group 'ein)
(defcustom ob-ein-anonymous-path ".%s.ipynb"
"When session header doesn't specify ipynb, prosecute all interactions for a given language in this throwaway notebook (substitute %s with language)."
:type '(string)
:group 'ein)
(defsubst ob-ein-anonymous-p (path)
"Return t if PATH looks like ob-ein-anonymous-path. Fragile"
(string-match (replace-regexp-in-string "%s" ".+"
(replace-regexp-in-string "\\." "\\\\." ob-ein-anonymous-path))
path))
(defcustom ob-ein-inline-image-directory "ein-images"
"Default directory where to save images generated from ein org-babel source blocks."
"Store ob-ein images here."
:group 'ein
:type '(directory))
(defvar org-babel-default-header-args:ein nil)
(defcustom ob-ein-default-header-args:ein nil
"No documentation."
:group 'ein
:type '(repeat string))
(defun ob-ein--inline-image-info (value)
(let* ((f (md5 value))
@ -107,32 +128,46 @@
(insert (format "#+NAME: %s" id))
id))))
(defun ein:org-register-lang-mode (lang-name lang-mode)
"Define org+ein language LANG-NAME with syntax highlighting from LANG-MODE. Untested.
(defun ob-ein--babelize-lang (lang-name lang-mode)
"Stand-up LANG-NAME as a babelized language with LANG-MODE syntax table.
For example, call (ein:org-register-lang-mode \"ein-R\" 'R) to define a language \"ein-R\" with R syntax highlighting for use with org-babel and ein.
Based on ob-ipython--configure-kernel.
"
Based on ob-ipython--configure-kernel."
(add-to-list 'org-src-lang-modes `(,lang-name . ,lang-mode))
(defvaralias (intern (concat "org-babel-default-header-args:" lang-name))
'org-babel-default-header-args:ein)
(defalias (intern (concat "org-babel-execute:" lang-name))
'org-babel-execute:ein))
'ob-ein-default-header-args:ein)
(fset (intern (concat "org-babel-execute:" lang-name))
`(lambda (body params)
(require (quote ,(intern (format "ob-%s" lang-mode))) nil t)
(if (boundp 'python-indent-guess-indent-offset-verbose)
(setq python-indent-guess-indent-offset-verbose nil))
(let* ((parser
(quote
,(intern
(format "org-babel-variable-assignments:%s" lang-mode))))
(assignments (if (fboundp parser)
(funcall (symbol-function parser) params)
(ein:log 'verbose "%s: No suitable ob-%s module"
(concat "org-babel-execute:" ,lang-name)
(quote ,lang-mode))
nil)))
(ob-ein--execute-body body params assignments)))))
;;;###autoload
(defun org-babel-execute:ein (body params)
"This function is called by `org-babel-execute-src-block'."
(defun ob-ein--execute-body (body params assignments)
(let* ((buffer (current-buffer))
(processed-params (org-babel-process-params params))
(result-params (cdr (assq :result-params params)))
(session (format "%s" (cdr (assoc :session processed-params))))
(kernelspec (or (cdr (assoc :kernelspec processed-params)) "default"))
(lang (nth 0 (org-babel-get-src-block-info)))
(kernelspec (or (cdr (assoc :kernelspec processed-params))
(ein:aif (cdr (assoc lang org-src-lang-modes))
(cons 'language (format "%s" it))
(error "ob-ein--execute-body: %s not among %s"
lang (mapcar #'car org-src-lang-modes)))))
(name (ob-ein--get-name-create (org-babel-get-src-block-info)))
(full-body (org-babel-expand-body:generic
(encode-coding-string body 'utf-8)
params
(org-babel-variable-assignments:python params)))
assignments))
(callback (lambda (notebook)
(ob-ein--execute-async
buffer
@ -144,10 +179,6 @@ Based on ob-ipython--configure-kernel.
(ob-ein--initiate-session session kernelspec callback))
*ob-ein-sentinel*)
;;;###autoload
(defun org-babel-execute:ein-hy (body params)
(org-babel-execute:ein (ein:pytools-wrap-hy-code body) params))
(defsubst ob-ein--execute-async-callback (buffer params result-params name)
"Callback of 1-arity (the shared output cell) to update org buffer when
`ein:shared-output-eval-string' completes."
@ -157,19 +188,26 @@ Based on ob-ipython--configure-kernel.
(ansi-color-apply (ein:join-str "\n" it))
(ob-ein--process-outputs
(ein:oref-safe cell 'outputs) params*)))
(result (org-babel-result-cond result-params*
raw (org-babel-python-table-or-string raw))))
(result
(let ((tmp-file (org-babel-temp-file "ein-")))
(with-temp-file tmp-file raw)
(org-babel-result-cond result-params*
raw (org-babel-import-elisp-from-file tmp-file '(16)))))
(info (org-babel-get-src-block-info 'light)))
(ein:log 'debug "ob-ein--execute-async-callback %s %s" name* result)
(save-excursion
(save-restriction
(with-current-buffer buffer*
(org-babel-goto-named-src-block name*)
(org-babel-remove-result)
(org-babel-insert-result
result
(cdr (assoc :result-params
(third (org-babel-get-src-block-info)))))
(org-redisplay-inline-images))))))
(when (not (stringp (org-babel-goto-named-src-block name*)))
(when info ;; kill #+RESULTS: (no-name)
(setf (nth 4 info) nil)
(org-babel-remove-result info))
(org-babel-remove-result) ;; kill #+RESULTS: name
(org-babel-insert-result
result
(cdr (assoc :result-params
(third (org-babel-get-src-block-info)))))
(org-redisplay-inline-images)))))))
buffer params result-params name))
(defun ob-ein--execute-async (buffer body kernel params result-params name)
@ -189,30 +227,6 @@ one at a time. Further, we do not order the queued up blocks!"
(lambda (_x)
(ein:shared-output-eval-string kernel body nil)))))
(defun ob-ein--edit-ctrl-c-ctrl-c ()
"C-c C-c mapping in ein:connect-mode-map."
(interactive)
(org-edit-src-save)
(when (boundp 'org-src--beg-marker)
(let* ((beg org-src--beg-marker)
(buf (marker-buffer beg)))
(with-current-buffer buf
(save-excursion
(goto-char beg)
(org-ctrl-c-ctrl-c))))))
;;;###autoload
(defun org-babel-edit-prep:ein (babel-info)
"C-c ' enters the lightly tested connect-to-notebook mode."
(let* ((buffer (current-buffer))
(processed-params (org-babel-process-params (third babel-info))))
(ob-ein--initiate-session
(format "%s" (cdr (assoc :session processed-params)))
(or (cdr (assoc :kernelspec processed-params)) "default")
(lambda (notebook)
(ein:connect-buffer-to-notebook notebook buffer t)
(define-key ein:connect-mode-map "\C-c\C-c" 'ob-ein--edit-ctrl-c-ctrl-c)))))
(defun ob-ein--parse-session (session)
(multiple-value-bind (url-or-port _password) (ein:jupyter-server-conn-info)
(let ((tokens (split-string session "/"))
@ -233,10 +247,11 @@ one at a time. Further, we do not order the queued up blocks!"
"Retrieve notebook based on SESSION path and KERNELSPEC, starting jupyter instance
if necessary. Install CALLBACK (i.e., cell execution) upon notebook retrieval."
(let* ((nbpath (ob-ein--parse-session session))
(info (org-babel-get-src-block-info))
(anonymous-path (format ob-ein-anonymous-path (nth 0 info)))
(parsed-url (url-generic-parse-url nbpath))
(slash-path (car (url-path-and-query parsed-url)))
(path (if (string= slash-path "")
ob-ein-anonymous-path
(path (if (string= slash-path "") anonymous-path
(substring slash-path 1)))
(url-or-port (if (string= slash-path "")
nbpath
@ -260,7 +275,28 @@ if necessary. Install CALLBACK (i.e., cell execution) upon notebook retrieval."
(callback-login (lambda (_buffer url-or-port)
(ein:notebook-open url-or-port path kernelspec
callback-nbopen errback-nbopen t))))
(cond (notebook (funcall callback notebook))
(cond ((and notebook
(string= path anonymous-path)
(stringp kernelspec)
(not (equal (ein:$kernelspec-name (ein:$notebook-kernelspec notebook))
kernelspec)))
(ein:log 'debug "ob-ein--initiate-session: switching %s from %s to %s"
path (ein:$kernelspec-name (ein:$notebook-kernelspec notebook))
kernelspec)
(cl-letf (((symbol-function 'y-or-n-p) #'ignore))
(ein:notebook-close notebook)
(ein:query-singleton-ajax
(list 'ob-ein--initiate-session (ein:url url-or-port path))
(ein:notebook-url notebook)
:type "DELETE"))
(loop repeat 8
for extant = (file-exists-p path)
until (not extant)
do (sleep-for 0 500)
finally do (if extant
(ein:display-warning (format "cannot del %s" path))
(ob-ein--initiate-session session kernelspec callback))))
(notebook (funcall callback notebook))
((string= (url-host parsed-url) ein:url-localhost)
(ein:process-refresh-processes)
(ein:aif (ein:process-url-match nbpath)
@ -276,13 +312,7 @@ if necessary. Install CALLBACK (i.e., cell execution) upon notebook retrieval."
(t (url-port parsed-url)))))))
(t (ein:notebooklist-login url-or-port callback-login)))))
;;;###autoload
(with-eval-after-load "python"
(setq python-indent-guess-indent-offset-verbose nil))
;;;###autoload
(add-hook 'org-mode-hook (lambda ()
(add-to-list 'org-src-lang-modes '("ein" . python))
(add-to-list 'org-src-lang-modes '("ein-hy" . hy))))
(loop for (lang . mode) in ob-ein-languages
do (ob-ein--babelize-lang lang mode))
(provide 'ob-ein)

View file

@ -120,7 +120,7 @@ if I call this between links in a deferred chain. Adding a flush-queue."
(and notebook
(ein:aand (ein:$notebook-kernel notebook)
(ein:kernel-live-p it))))
nil 10000 1000)
nil 20000 1000)
notebook)
(error (let ((notice (format "ein:testing-new-notebook: [%s] %s"
url-or-port (error-message-string err))))

View file

@ -5,7 +5,13 @@
;; Test utils
;;; This is the content portion of a response fromt he content API.
(ert-deftest ein:ob-anonymous-p ()
(should (ob-ein-anonymous-p ".ein-python.ipynb"))
(should (ob-ein-anonymous-p ".ein.ipynb"))
(should-not (ob-ein-anonymous-p "ein-python.ipynb"))
(should-not (ob-ein-anonymous-p "Untitled.ipynb")))
;;; This is the content portion of a response from the content API.
(defvar eintest:ob-src-block
"#+BEGIN_SRC ein :session 8888/Untitled.ipynb
import sys

View file

@ -16,4 +16,10 @@ if [ "x$TRAVIS_OS_NAME" = "xlinux" ] ; then
fi
R -e "install.packages('IRkernel', repos='http://cran.mirrors.hoobly.com')"
R -e "IRkernel::installspec()"
elif [ "x$TRAVIS_OS_NAME" = "xosx" ]; then
brew update
brew list r &>/dev/null || brew install r
R -e "install.packages('IRkernel', repos='http://cran.mirrors.hoobly.com')"
R -e "IRkernel::installspec()"
fi
R --version

View file

@ -16,13 +16,6 @@ cask_upgrade_cask_or_reset() {
}
cask_install_or_reset() {
if [ $(cask eval "(princ emacs-major-version)") -gt "25" ]; then
echo "!!!! ALERT WORKAROUND !!!!"
set -x
grep -v "org-plus-contrib" ./Cask > ./Cask.tmp
mv ./Cask.tmp ./Cask
set +x
fi
cask install </dev/null
cask update </dev/null
# travis cache

View file

@ -10,7 +10,7 @@ WORKDIR=${HOME}/local
if [ "x$TRAVIS_OS_NAME" = "xosx" ]; then
brew update
brew list pyenv-virtualenv &>/dev/null || brew install pyenv-virtualenv
brew list pyenv-virtualenv || brew install pyenv-virtualenv
case "${TOXENV}" in
py27)

View file

@ -1,21 +0,0 @@
#!/bin/sh
env="$1"
req="$2"
activate=$env/bin/activate
if [ -z "$env" -o -z "$req" ]; then
echo "Usage:"
echo " $0 ENVIRONMENT REQUIREMENT"
exit 1
fi
if [ -e $activate ]; then
echo "virtualenv $env exists."
else
echo "Creating virtualenv $env."
virtualenv -v $env
fi
. $activate
pip install --requirement $req

View file

@ -1,521 +0,0 @@
#!/usr/bin/env python
"""
Run EIN test suite
"""
import glob
import os
import sys
import re
from subprocess import Popen, PIPE, STDOUT, check_output
EIN_ROOT = os.path.normpath(
os.path.join(os.path.dirname(__file__), os.path.pardir))
def cask_load_path():
try:
path = check_output(['cask','load-path'])
except WindowsError:
path = check_output(['C:/Users/mille/.cask/bin/cask.bat', 'load-path'])
return path.decode().rstrip()
def has_library(emacs, library):
"""
Return True when `emacs` has build-in `library`.
"""
with open(os.devnull, 'w') as devnull:
proc = Popen(
[emacs, '-Q', '-batch', '-l', 'cl',
'--eval', '(assert (locate-library "{0}"))'.format(library)],
stdout=devnull, stderr=devnull)
return proc.wait() == 0
def eindir(*path):
return os.path.join(EIN_ROOT, *path)
def einlispdir(*path):
return eindir('lisp', *path)
def eintestdir(*path):
return eindir('test', *path)
def einlibdir(*path):
return eindir('lib', *path)
def show_nonprinting(string, stream=sys.stdout):
"""Emulate ``cat -v`` (``--show-nonprinting``)."""
stream.writelines(map(chr, convert_nonprinting(string)))
def convert_nonprinting(string):
"""
Convert non-printing characters in `string`.
Output is iterable of int. So for Python 2, you need to
convert it into string using `chr`.
Adapted from: http://stackoverflow.com/a/437542/727827
"""
for b in map(ord, string.decode('utf-8')):
assert 0 <= b < 0x100
if b in (0x09, 0x0a): # '\t\n'
yield b
continue
if b > 0x7f: # not ascii
yield 0x4d # 'M'
yield 0x2d # '-'
b &= 0x7f
if b < 0x20: # control char
yield 0x5e # '^'
b |= 0x40
elif b == 0x7f:
yield 0x5e # '^'
yield 0x3f # '?'
continue
yield b
class BaseRunner(object):
def __init__(self, **kwds):
self.__dict__.update(kwds)
self.batch = self.batch and not self.debug_on_error
def logpath(self, name, ext='log'):
path = os.path.join(
self.log_dir,
"{testname}_{logname}_{modename}_{emacsname}.{ext}".format(
ext=ext,
logname=name,
emacsname=os.path.basename(self.emacs),
testname=os.path.splitext(self.testfile)[0],
modename='batch' if self.batch else 'interactive',
))
path = re.sub(r'\\', '/', path)
return path
@property
def command(self):
raise NotImplementedError
def do_run(self):
raise NotImplementedError
def run(self):
if self.dry_run:
command = self.command
if isinstance(command, str):
print(command)
else:
print((construct_command(command)))
return 0
else:
mkdirp(self.log_dir)
return self.do_run()
class TestRunner(BaseRunner):
def __init__(self, **kwds):
super(TestRunner, self).__init__(**kwds)
fmtdata = self.__dict__.copy()
fmtdata.update(
emacsname=os.path.basename(self.emacs),
testname=os.path.splitext(self.testfile)[0],
modename='batch' if self.batch else 'interactive',
)
quote = '"{0}"'.format
self.logpath_log = self.logpath('log')
self.logpath_messages = self.logpath('messages')
self.logpath_server = self.logpath('server')
self.notebook_dir = os.path.join(EIN_ROOT, "test")
self.lispvars = {
'ein:testing-dump-file-log': quote(self.logpath_log),
'ein:testing-dump-file-server': quote(self.logpath_server),
'ein:testing-dump-file-messages': quote(self.logpath_messages),
'ein:log-level': self.ein_log_level,
'ein:force-sync': "t",
'ein:log-message-level': self.ein_message_level
}
if self.ein_debug:
self.lispvars['ein:debug'] = "'t"
def setq(self, sym, val):
self.lispvars[sym] = val
def bind_lispvars(self):
command = []
for (k, v) in self.lispvars.items():
if v is not None:
command.extend([
'--eval', '(setq {0} {1})'.format(k, v)])
return command
@property
def base_command(self):
command = [self.emacs, '-Q'] + self.bind_lispvars()
if self.batch:
command.append('-batch')
if self.debug_on_error:
command.extend(['-f', 'toggle-debug-on-error'])
# load modules
if self.need_ert():
ertdir = einlibdir('ert', 'lisp', 'emacs-lisp')
command.extend([
'-L', ertdir,
# Load `ert-run-tests-batch-and-exit`:
'-l', os.path.join(ertdir, 'ert-batch.el'),
# Load `ert-run-tests-interactively`:
'-l', os.path.join(ertdir, 'ert-ui.el'),
])
for path in self.load_path:
command.extend(['-L', path])
for path in self.load:
command.extend(['-l', path])
command.extend(['-L', einlispdir(),
'-L', eintestdir(),
'-l', eintestdir(self.testfile)])
# command.extend(['-L', einlispdir(),
# '-L', einlibdir('websocket'),
# '-L', einlibdir('request'),
# '-L', einlibdir('auto-complete'),
# '-L', einlibdir('popup'),
# '-L', eintestdir(),
# '-l', eintestdir(self.testfile)])
return command
@property
def command(self):
command = self.base_command[:]
if self.batch:
command.extend(['-f', 'ert-run-tests-batch-and-exit'])
else:
command.extend(['--eval', "(ert 't)"])
return command
def show_sys_info(self):
print(("*" * 50))
command = self.base_command + [
'-batch', '-l', 'lisp/ein-dev.el', '-f', 'ein:dev-print-sys-info']
proc = Popen(command, stderr=PIPE)
err = proc.stderr.read()
proc.wait()
if proc.returncode != 0:
print(("Error with return code {0} while running {1}".format(
proc.returncode, command)))
print(err)
pass
print(("*" * 50))
def need_ert(self):
if self.load_ert:
return True
if self.auto_ert:
if has_library(self.emacs, 'ert'):
print(("{0} has ERT module.".format(self.emacs)))
return False
else:
print("{0} has no ERT module.".format(self.emacs))
print("ERT is going to be loaded from git submodule.")
return True
return False
def make_process(self):
print("Start test {0}".format(self.testfile))
print("Emacs command {0}".format(self.command))
self.proc = Popen(self.command, stdout=PIPE, stderr=STDOUT)
return self.proc
def report(self):
(stdout, _) = self.proc.communicate()
self.stdout = stdout
self.failed = self.proc.returncode != 0
if self.failed:
print("*" * 50)
print("Showing {0}:".format(self.logpath_log))
print(open(self.logpath_log).read())
print()
print("*" * 50)
print("Showing STDOUT/STDERR:")
show_nonprinting(stdout)
print()
print("{0} failed".format(self.testfile))
else:
print("{0} OK".format(self.testfile))
for line in reversed(stdout.decode('utf-8').splitlines()):
if line.startswith('Ran'):
print(line)
break
return int(self.failed)
def do_run(self):
self.show_sys_info()
self.make_process()
return self.report()
def is_known_failure(self):
"""
Check if failures are known, based on STDOUT from ERT.
"""
import re
lines = iter(self.stdout.splitlines())
for l in lines:
if re.match("[0-9]+ unexpected results:.*", l.decode('utf-8')):
break
else:
return True # no failure
# Check "FAILED <test-name>" lines
for l in lines:
if not l:
break # end with an empty line
for f in self.known_failures:
if re.search(f, l.decode('utf-8')):
break
else:
return False
return True
known_failures = [
"ein:notebook-execute-current-cell-pyout-image$",
]
"""
A list of regexp which matches to test that is known to fail (sometimes).
This is a workaround for ##74.
"""
def mkdirp(path):
"""Do ``mkdir -p {path}``"""
if not os.path.isdir(path):
os.makedirs(path)
def remove_elc():
files = glob.glob(einlispdir("*.elc")) + glob.glob(eintestdir("*.elc"))
list(map(os.remove, files))
print("Removed {0} elc files".format(len(files)))
class ServerRunner(BaseRunner):
port = None
notebook_dir = os.path.join(EIN_ROOT, "test", "notebook")
def __enter__(self):
self.run()
return self.port
def __exit__(self, type, value, traceback):
self.stop()
def do_run(self):
self.clear_notebook_dir()
self.start()
self.get_port()
print("Server running at", self.port)
def clear_notebook_dir(self):
files = glob.glob(os.path.join(self.notebook_dir, '*.ipynb'))
list(map(os.remove, files))
print("Removed {0} ipynb files".format(len(files)))
@staticmethod
def _parse_port_line(line):
if line.find('token'):
port = line.rpartition('/')[0]
port = line.strip().rsplit(':', 1)[-1].strip('/')
return port
def get_port(self):
if self.port is None:
val = self.proc.stdout.readline()
dval = val.decode('utf-8')
self.port = self._parse_port_line(dval)
return self.port
def start(self):
from subprocess import Popen, PIPE, STDOUT
self.proc = Popen(
self.command, stdout=PIPE, stderr=STDOUT, stdin=PIPE,
shell=True)
# Answer "y" to the prompt: Shutdown Notebook Server (y/[n])?
self.proc.stdin.write(b'y\n')
def stop(self):
if self.dry_run:
return
print("Stopping server", self.port)
returncode = self.proc.poll()
if returncode is not None:
logpath = self.logpath('server')
print("Server process was already dead by exit code", returncode)
print("*" * 50)
print("Showing {0}:".format(logpath))
print(open(logpath).read())
print()
return
try:
kill_subprocesses(self.proc.pid, lambda x: 'ipython' in x)
finally:
self.proc.terminate()
@property
def command(self):
fmtdata = dict(
notebook_dir=self.notebook_dir,
ipython=self.ipython,
server_log=self.logpath('server'),
)
return self.command_template.format(**fmtdata)
command_template = r"""{ipython} notebook --notebook-dir {notebook_dir} --no-browser --NotebookApp.token='' --debug 2>&1 | tee {server_log} | grep --line-buffered 'Notebook is running at' | head -n1"""
def kill_subprocesses(pid, include=lambda x: True):
from subprocess import Popen, PIPE
import signal
command = ['ps', '-e', '-o', 'ppid,pid,command']
proc = Popen(command, stdout=PIPE, stderr=PIPE)
(stdout, stderr) = proc.communicate()
if proc.returncode != 0:
raise RuntimeError(
'Command {0} failed with code {1} and following error message:\n'
'{2}'.format(command, proc.returncode, stderr))
for line in map(lambda l: str.strip(str(l)), stdout.decode('utf-8').splitlines()):
(cmd_ppid, cmd_pid, cmd) = line.split(None, 2)
if cmd_ppid == str(pid) and include(cmd):
print("Killing PID={0} COMMAND={1}".format(cmd_pid, cmd))
os.kill(int(cmd_pid), signal.SIGINT)
def construct_command(args):
"""
Construct command as a string given a list of arguments.
"""
command = []
escapes = set(' ()')
for a in args:
if set(a) & escapes:
command.append(repr(str(a))) # hackish way to escape
else:
command.append(a)
return " ".join(command)
def run_ein_test(unit_test, func_test, func_test_max_retries,
no_skip, clean_elc, **kwds):
if clean_elc and not kwds['dry_run']:
remove_elc()
if unit_test:
unit_test_runner = TestRunner(testfile='testein.el', **kwds)
if unit_test_runner.run() != 0:
return 1
if func_test:
for i in range(func_test_max_retries + 1):
func_test_runner = TestRunner(testfile='test-func.el', **kwds)
with ServerRunner(testfile='test-func.el', **kwds) as port:
func_test_runner.setq('ein:testing-port', port)
if func_test_runner.run() == 0:
print("Functional test succeeded after {0} retries."\
.format(i))
return 0
if not no_skip and func_test_runner.is_known_failure():
print("All failures are known. Ending functional test.")
return 0
print("Functional test failed after {0} retries.".format(i))
return 1
return 0
def main():
import sys
import os
from argparse import ArgumentParser
os.environ['EMACSLOADPATH'] = cask_load_path()
os.environ['LC_ALL'] = 'en_us.UTF-8'
parser = ArgumentParser(description=__doc__.splitlines()[1])
parser.add_argument('--emacs', '-e', default='emacs',
help='Emacs executable.')
parser.add_argument('--load-path', '-L', default=[], action='append',
help="add a directory to load-path. "
"can be specified multiple times.")
parser.add_argument('--load', '-l', default=[], action='append',
help="load lisp file before tests. "
"can be specified multiple times.")
parser.add_argument('--load-ert', default=False, action='store_true',
help="load ERT from git submodule. "
"you need to update git submodule manually "
"if ert/ directory does not exist yet.")
parser.add_argument('--no-auto-ert', default=True,
dest='auto_ert', action='store_false',
help="load ERT from git submodule. "
"if this Emacs has no build-in ERT module.")
parser.add_argument('--batch', '-B', default=True,
dest='batch', action='store_false',
help="start interactive session.")
parser.add_argument('--debug-on-error', '-d', default=False,
action='store_true',
help="set debug-on-error to t and start "
"interactive session.")
parser.add_argument('--func-test-max-retries', default=0, type=int,
help="""
Specify number of retries for functional test
before failing with error.
""")
parser.add_argument('--no-skip', default=False, action='store_true',
help="""
Do no skip known failures. Known failures
are implemented as another workaround for the
issue #74.
""")
parser.add_argument('--no-func-test', '-F', default=True,
dest='func_test', action='store_false',
help="do not run functional test.")
parser.add_argument('--no-unit-test', '-U', default=True,
dest='unit_test', action='store_false',
help="do not run unit test.")
parser.add_argument('--clean-elc', '-c', default=False,
action='store_true',
help="remove *.elc files in ein/lisp and "
"ein/test directories.")
parser.add_argument('--dry-run', default=False,
action='store_true',
help="Print commands to be executed.")
parser.add_argument('--ipython', default='ipython',
help="""
ipython executable to use to run notebook server.
""")
parser.add_argument('--ein-log-level', default=40)
parser.add_argument('--ein-message-level', default=30)
parser.add_argument('--ein-debug', default=False, action='store_true',
help="(setq ein:debug t) when given.")
parser.add_argument('--log-dir', default="log",
help="Directory to store log (default: %(default)s)")
args = parser.parse_args()
sys.exit(run_ein_test(**vars(args)))
if __name__ == '__main__':
main()