Appveyor iterating (WIP) (#573)

* ob-ein: Bring back old functionality.

Bring back some old features to babel edit buffers while trying to respect
recent addition of polymode support.

* Override polymode if the user really wants.

Polymode is really for notebook buffers in any case, but this will override
whatever completion backmode a user has configured for python-mode.

* Install cask using python2.

For now python2 is the easiest option for testing on Windows since cask does
not properly support python3 when in Windows.

* Let's throw in the ert-runner, see what happens.

* Can I use my fork of cask?

Work around smartrep weirdness, try to live without command line wildcard
expansion.

* Get the url for the fork right.

* Experiment with python37, use test_script.

* Unstick appveyor, I hope.

* Fix parsing error.

* test_script is not executing. Why?

* Add ert testing.

But why are the test_script commands not executing?

* tasks: Automate building and testing using invoke.

Invoke leverages Python, which I hope will allow us to abstract out differences
in platforms when it comes to building and testing ein.

* Use invoke on appveyor.

* appveyor: Use the environment python.

So we can test versions other than python 2.7.

* Parsing error.

Is
https://packaging.python.org/guides/supporting-windows-using-appveyor/#setting-up
lying to me?

* Quote commands, just in case.

* Get python onto the path.

* Appveyor is catching up to travis.

* Parsing error.

* Update pip, try to get quoted syntax right.

* Still not liking my pip call.

Last try, next step we go to a requirements.txt file.

* Go to using a requirements file for pip.

* ecukes needs bash to work.

* Cleaning up and fiddling.

Seems like the emacs-jupyter guy has his act together - maybe we can take some
inspiration for our appveyor config.

* Syntax error in environment.

* More syntax errros.

* Maybe we need quoting.

* I give up.

* Formatting and cleanup.

* Add customization, yet another syntax error.

New customizable variable `ob-ein-babel-edit-polymode-ignore' to override
keybinding for \C-c\C-c in an org source code edit buffer.

* John learned some Powershell today.

* Fix the executable path.

Sometimes there is more than one curl installed on the system, make sure we can
account for that in testing.

* Handle updating the path inside invoke.

* Report which curl we are using before starting functional tests.

* Enable RDP so we can see error logs.

* Keep the build alive even when it finishes.

* Fix #568.

Apparently we need to specify the user agent when on windows, otherwise tornado
will start throwing 403 responses. Currently using Mozilla/4.0 as the agent, but
might be a good idea to make this value customizable.

* Clean up emacs config.

* Why is appveyor dropping the xsrf token?

* xsrf cookie found, what does the header look like?

* Try different user-agent header, reenable rdp.

* JSON encoding issues on Python side, it appears. Let's try an older Python.

Login works, contents query to get notebooklist works (i.e. GET on
/contents/api), but creating a notebook (i.e. POST on /contents/api) fails with
invalid JSON. ein and emacs-request appear to be generating the proper json, but
jupyter notebook does not see the same thing that is being written. Could be
bytes vs. text issue with modern v3.x python, so let us see how this all works
with Python 2.7.

* Python27 does not have pathlib out of the box.

* Make amends with Python27

* Back to python37.

Tornado/notebook still isn't reading the POST'ed json correctly.

* Do we need to specify content type?

* Must be selective in specifying application/json content.

* Re-enable rdp.

* Let's try a different curl.

* Ensure most recent curl is on path

* Try a different path.

* Try to warn user if suspicious curl detected.

* Remove debugging statements.

* EVM depends on trusty for 26.x

See issue #125 (https://github.com/rejeep/evm/issues/125). Let's hope I got the
travis.yml syntax right.

* Minimal support for ecukes from invoke.

* Cleaner server shutdowns, better ecukes support from invoke.

Use the /api/shutdown REST API call now to shutdown running server. Also support
more command line options for ecukes from invoke.

* Almost full support of ecukes using invoke.

But! Also disabling integration testing for the time being until I understand
why ecukes fails even though everything else is working.

* Just do integration and functional testing on appveyor.

Better than nothing while I work out what is breaking the integration tests.

.appveyor.yml

@@ -22,6 +22,7 @@ image: Visual Studio 2015
 # scripts that are called at very beginning, before repo cloning
 init:
   - git config --global core.autocrlf input
+  - ps: iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/enable-rdp.ps1'))
 
 # clone directory
 clone_folder: c:\projects\myproject
@@ -32,205 +33,66 @@ shallow_clone: true                 # default is "false"
 # set clone depth
 clone_depth: 5                      # clone entire repository history if not defined
 
-# setting up etc\hosts file
-hosts:
-  queue-server: 127.0.0.1
-  db.server.com: 127.0.0.2
-
 # environment variables
 environment:
-  # this is how to set encrypted variable. Go to "Settings" -> "Encrypt YAML" page in account menu to encrypt data.
+  PYTHON: C:\Python37
-  my_secure_var1:
+  MINICONDA: C:\Miniconda3-x64
-    secure: FW3tJ3fMncxvs58/ifSP7w==
+  APPVEYOR_RDP_PASSWORD: ein_t3st1ing_admin
 
-# environment:
-#  global:
-#    connection_string: server=12;password=13;
-#    service_url: https://127.0.0.1:8090
-#
-#  matrix:
-#  - db: mysql
-#    provider: mysql
-#
-#  - db: mssql
-#    provider: mssql
-#    password:
-#      secure: $#(JFDA)jQ@#$
-
-# this is how to allow failing jobs in the matrix
 matrix:
-  fast_finish: true     # set this flag to immediately finish build once one of the jobs fails.
+  fast_finish: true
-  allow_failures:
-    - platform: x86
-      configuration: Debug
-    - platform: x64
-      configuration: Release
-
-# exclude configuration from the matrix. Works similarly to 'allow_failures' but build not even being started for excluded combination.
-  exclude:
-    - platform: x86
-      configuration: Debug
 
 # build cache to preserve files/folders between builds
 cache:
 #  - packages -> **\packages.config  # preserve "packages" directory in the root of build folder but will reset it if packages.config is modified
-  - C:\python27\
+  # - C:\python37\
   - C:\ProgramData\chocolatey\lib
   - C:\ProgramData\chocolatey\bin
-#  - '%LocalAppData%'
-#  - node_modules                    # local npm modules
   - '%LocalAppData%\NuGet\Cache'    # NuGet < v3
   - '%LocalAppData%\NuGet\v3-cache' # NuGet v3
 
-# enable service required for build/tests
-#services:
-#  - mssql2014           # start SQL Server 2014 Express
-#  - mssql2014rs         # start SQL Server 2014 Express and Reporting Services
-#  - mssql2012sp1        # start SQL Server 2012 SP1 Express
-#  - mssql2012sp1rs      # start SQL Server 2012 SP1 Express and Reporting Services
-#  - mssql2008r2sp2      # start SQL Server 2008 R2 SP2 Express
-#  - mssql2008r2sp2rs    # start SQL Server 2008 R2 SP2 Express and Reporting Services
-#  - mysql               # start MySQL 5.6 service
-#  - postgresql          # start PostgreSQL 9.5 service
-#  - iis                 # start IIS
-#  - msmq                # start Queuing services
-#  - mongodb             # start MongoDB
-
 # scripts that run after cloning repository
 install:
-  # to run script as a PowerShell command prepend it with ps:
+  - choco install curl
-  - python -m pip install jupyter
+  # - if not exist %userprofile%\local mkdir %userprofile%\local
+  - if not exist %userprofile%\.emacs.d mkdir %userprofile%\.emacs.d
+  # - copy C:\projects\myproject\tools\appveyor.init.el %userprofile%\.emacs.d\init.el
+  - copy C:\projects\myproject\tools\appveyor.init.el %userprofile%\.emacs
+  - set PATH=C:\ProgramData\chocolatey\bin;%userprofile%\.cask\bin;C:\msys64\usr\bin;%PYTHON%\Scripts;%PATH%
+  - "%PYTHON%\\python.exe -m pip install --upgrade pip"
+  - "%PYTHON%\\python.exe -m pip install pathlib2"
+  - "%PYTHON%\\python.exe -m pip install -r test_requirements.txt"
   - jupyter kernelspec list
   - ipython --version
+  - curl --version
   - choco install emacs64
   - emacs --version
-  - if not exist %userprofile%\local mkdir %userprofile%\local
+  - curl -fsSL https://raw.githubusercontent.com/millejoh/cask/appveyor-build/go | python
-  - if not exist %userprofile%\.emacs.d mkdir %userprofile%\.emacs.d
-  - cd c:\projects\myproject
-  - set PATH=%PATH%;%userprofile%\local\cask\bin;\msys64\usr\bin
-  - copy C:\projects\myproject\tools\appveyor.init.el %userprofile%\.emacs.d\init.el
-  - copy C:\projects\myproject\tools\appveyor.init.el %userprofile%\.emacs
-#  - bash -lc "cd /c/projects/myproject ; bash tools/install-cask.sh"
-#  - choco install gnuwin32-make.portable
 
-# enable patching of AssemblyInfo.* files
-# assembly_info:
-#   patch: true
-#   file: AssemblyInfo.*
-#   assembly_version: "2.2.{build}"
-#   assembly_file_version: "{version}"
-#   assembly_informational_version: "{version}"
-
-
-# Automatically register private account and/or project AppVeyor NuGet feeds.
-#nuget:
-#  account_feed: true
-#  project_feed: true
-#  disable_publish_on_pr: true     # disable publishing of .nupkg artifacts to
-#                                  # account/project feeds for pull request builds
 
 #---------------------------------#
 #       build configuration       #
 #---------------------------------#
 
-# build platform, i.e. x86, x64, Any CPU. This setting is optional.
+build_script:
-#platform: Any CPU
+  - cd %APPVEYOR_BUILD_FOLDER%
-
+  - invoke clean
-# to add several platforms to build matrix:
+  - invoke test-compile
-#platform:
-#  - x86
-#  - Any CPU
-
-# build Configuration, i.e. Debug, Release, etc.
-# configuration: Release
-
-# to add several configurations to build matrix:
-#configuration:
-#  - Debug
-#  - Release
-
-# Build settings, not to be confused with "before_build" and "after_build".
-# "project" is relative to the original build directory and not influenced by directory changes in "before_build".
-#build:
-#  parallel: true                  # enable MSBuild parallel builds
-#  project: MyTestAzureCS.sln      # path to Visual Studio solution or project
-#  publish_wap: true               # package Web Application Projects (WAP) for Web Deploy
-#  publish_wap_xcopy: true         # package Web Application Projects (WAP) for XCopy deployment
-#  publish_azure: true             # package Azure Cloud Service projects and push to artifacts
-#  publish_nuget: true             # package projects with .nuspec files and push to artifacts
-#  publish_nuget_symbols: true     # generate and publish NuGet symbol packages
-#  include_nuget_references: true  # add -IncludeReferencedProjects option while packaging NuGet artifacts
-
-  # MSBuild verbosity level
-#  verbosity: quiet|minimal|normal|detailed
-
-
-# to disable automatic builds
-build: off
 
 #---------------------------------#
 #       tests configuration       #
 #---------------------------------#
 
-# to run tests against only selected assemblies and/or categories
+test_script:
-#test:
+  - cd %APPVEYOR_BUILD_FOLDER%
-#  assemblies:
+  # - ps: $env:PATH = "$(cask exec-path);$env:PATH"
-#    only:
+  # - ps: $env:EMACSLOADPATH = $(cask load-path)
-#      - asm1.dll
+  - invoke test-unit
-#      - asm2.dll
+  - invoke test-func
-#
+  # - invoke test-int
-#  categories:
+  # - bash -lc "cd /c/projects/myproject ; make test"
-#    only:
-#      - UI
-#      - E2E
 
-# to run tests against all except selected assemblies and/or categories
+# Uncomment the following if you need to connect to build after it finishes to
-#test:
+# troubleshoot any issues.
-#  assemblies:
+# on_finish:
-#    except:
+#   - ps: $blockRdp = $true; iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/enable-rdp.ps1'))
-#      - asm1.dll
-#      - asm2.dll
-#
-#  categories:
-#    except:
-#      - UI
-#      - E2E
-
-# to run tests from different categories as separate jobs in parallel
-#test:
-#  categories:
-#    - A            # A category common for all jobs
-#    - [UI]         # 1st job
-#    - [DAL, BL]    # 2nd job
-
-# scripts to run before tests (working directory and environment changes are persisted from the previous steps such as "before_build")
-#before_test:
-
-# to run your custom scripts instead of automatic tests
-#test_script:
-#  - cd c:\projects\myproject
-#  - make quick
-
-# scripts to run after tests
-#after_test:
-
-# to disable automatic tests
-test: off
-
-# to disable deployment
-deploy: off
-
-#---------------------------------#
-#        global handlers          #
-#---------------------------------#
-
-# on successful build
-#on_success:
-
-# on build failure
-#on_failure:
-#  - ps: $blockRdp = $true; iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/enable-rdp.ps1'))
-
-# after build failure or success
-#on_finish:
-#  - ps: $blockRdp = $true; iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/enable-rdp.ps1'))

.travis.yml

@@ -30,10 +30,10 @@ matrix:
     - os: linux
       python: 2.7
       env: EVM_EMACS=emacs-25.1-travis IPYTHON=5.8.0
-    - os: linux
+    - dist: trusty
       python: 3.5
       env: EVM_EMACS=emacs-26.1-travis IPYTHON=6.5.0
-    - os: linux
+    - dist: trusty
       python: 3.6
       env: EVM_EMACS=emacs-26.2-travis IPYTHON=7.5.0
     - os: osx

EIN_Manual.org

@@ -117,8 +117,9 @@ Travis token: 2yP9mWPstlPi9kUoK9DIKw 2yP9mWPstlPi9kUoK9DIKw
 
 ** Notebook Security
 Life gets more complicated with jupyter notebook [[https://blog.jupyter.org/2016/12/21/jupyter-notebook-4-3-1/][v4.3.1]], though the intentions
-are purely altruistic. The notebook server now requires a ~[[http://www.tornadoweb.org/en/stable/guide/security.html#cross-site-request-forgery-protection][xsrf]]~ token via
+are purely altruistic. The notebook server now requires a
-cookies to validate requests.
+~[[http://www.tornadoweb.org/en/stable/guide/security.html#cross-site-request-forgery-protection][xsrf]]~
+token via cookies to validate requests.
 
 If I interpret the documentation correctly, a successful login should set a
 _xsrf cookie, which then needs to be included in the header with all additional
@@ -131,13 +132,16 @@ If we are using the curl backend for request:
 Once a user succesfully authenticates against a running jupyter server via
 either ~ein:notebooklist-login~ curl will store two cookies in the
 ~curl-cookie-jar~ file. The location of this file is set by the variable
-~[[help:request-storage-directory][request-storage-directory]]~. One cookie is for the token authentication, and the
+~[[help:request-storage-directory][request-storage-directory]]~. One cookie is
-other is the xsrf token. [[*Example curl cookie jar file][Below]] is an example of cookies that were set after
+for the token authentication, and the other is the xsrf token. [[*Example curl cookie jar file][Below]] is an
-authenticating against a jupyter notebook server running on my personal aws
+example of cookies that were set after authenticating against a jupyter notebook
-instance. On subsequent calls to the content API request/curl will automatically
+server running on my personal aws instance. On subsequent calls to the content
-supply the correct cookies. The websocket package uses the ~[[info:url#Top][url]]~ package to set
+API request/curl will automatically supply the correct cookies. The websocket
-cookies, and so will not know about the authentication and ~xsrf~ tokens unless
+package uses the ~[[info:url#Top][url]]~ package to set cookies, and so will not
-EIN does some extra work. EIN does this in the ~[[file:c:/Users/mille/Dropbox/Projects/emacs-ipython-notebook/lisp/ein-websocket.el::(defun ein:websocket--prepare-cookies (url)][ein:websocket--prepare-cookies]]~
+know about the authentication and ~xsrf~ tokens unless EIN does some extra work.
+EIN does this in the
+~[[file:c:/Users/mille/Dropbox/Projects/emacs-ipython-notebook/lisp/ein-websocket.el::(defun
+ein:websocket--prepare-cookies (url)][ein:websocket--prepare-cookies]]~
 function. The function does some extra work to only copy over cookies pertaining
 to the host the websocket is connecting on.
 
@@ -350,8 +354,9 @@ Use the [[info:elisp#Abstract%20Display][abstract display]] and [[info:elisp#Ove
 Should take some time to understand how magit manages its buffers. Magic seems to happen
 in ~[[magit-insert-section][magit-insert-section]]~.
 
-Magit's sections are interesting, but I do not think I am smart enough to use them. I really would like to avoid
+Magit's sections are interesting, but I do not think I am smart enough to use
-reinventing the wheel, but maybe I can get by just reinventing the spokes.
+them. I really would like to avoid reinventing the wheel, but maybe I can get by
+just reinventing the spokes.
 
 #+BEGIN_SRC elisp
   (require 'cl-lib)

lisp/ein-cell-edit.el

@@ -239,7 +239,7 @@ appropriate language major mode. Functionality is very similar to
     (if raw-cell-p
         (funcall ein:raw-cell-default-edit-mode)
       (case (ein:get-mode-for-kernel (ein:$notebook-kernelspec notebook))
-	(julia (julia-mode))
+        (julia (julia-mode))
         (python (python-mode))
         (R (R-mode))))))
 

lisp/ein-jupyter.el

@@ -216,32 +216,51 @@ the log of the running jupyter server."
 ;;;###autoload
 (defalias 'ein:stop 'ein:jupyter-server-stop)
 
+(defun ein:shutdown-server (url-or-port)
+  (ein:query-singleton-ajax
+   (list 'content-query-sessions url-or-port)
+   (ein:url url-or-port "api/shutdown")
+   :type "POST"
+   :timeout ein:content-query-timeout
+   :sync t))
+
+(cl-defun ein:shutdown-server-success (url-or-port &rest -ignore-)
+  (ein:log 'info "Notebook server on %s succesfully shut down." url-or-port))
+
+(cl-defun ein:shutdown-server-error (&key symbol-status response &allow-other-keys)
+  (ein:display-warning (format "Signal to shut down server failed with status %s" symbol-status)))
+
+
 ;;;###autoload
 (defun ein:jupyter-server-stop (&optional force log)
   (interactive)
-  (ein:and-let* ((proc (ein:jupyter-server-process))
+  (ein:and-let* ((url-or-port (first (ein:jupyter-server-conn-info)))
                  (_ok (or force (y-or-n-p "Stop server and close notebooks?"))))
     (ein:notebook-close-notebooks t)
     (loop repeat 10
           do (ein:query-running-process-table)
           until (zerop (hash-table-count ein:query-running-process-table))
           do (sleep-for 0 500))
+    (ein:shutdown-server url-or-port)
+    (sleep-for 2.0)
+    (let ((proc (ein:jupyter-server-process)))
+      (when (and proc (not (eql (process-status proc) 'exit)))
+        (ein:display-warning (format "Notebook server (%s) did not exit cleanly." url-or-port))))
     ;; Both (quit-process) and (delete-process) leaked child kernels, so signal
-    (if (eql system-type 'windows-nt)
+    ;; (if (eql system-type 'windows-nt)
-        (delete-process proc)
+    ;;     (delete-process proc)
-      (lexical-let* ((proc proc)
+    ;;   (lexical-let* ((proc proc)
-                     (pid (process-id proc)))
+    ;;                  (pid (process-id proc)))
-        (ein:log 'verbose "Signaled %s with pid %s" proc pid)
+    ;;     (ein:log 'verbose "Signaled %s with pid %s" proc pid)
-        (signal-process pid 15)
+    ;;     (signal-process pid 15)
-        (run-at-time 2 nil (lambda ()
+    ;;     (run-at-time 2 nil (lambda ()
-                             (ein:log 'verbose "Resignaled %s with pid %s" proc pid)
+    ;;                          (ein:log 'verbose "Resignaled %s with pid %s" proc pid)
-                             (signal-process pid 15)))))
+    ;;                          (signal-process pid 15)))))
 
     (ein:log 'info "Stopped Jupyter notebook server.")
 
     ;; `ein:notebooklist-sentinel' frequently does not trigger
-    (multiple-value-bind (url-or-port _password) (ein:jupyter-server-conn-info)
+    (ein:notebooklist-list-remove url-or-port)
-      (ein:notebooklist-list-remove url-or-port))
 
     (when log
       (with-current-buffer ein:jupyter-server-buffer-name

lisp/ein-notebooklist.el

@@ -385,6 +385,7 @@ This function is called via `ein:notebook-after-rename-hook'."
      url
      :type "POST"
      :data (json-encode '((:type . "notebook")))
+     :headers (list (cons "Content-Type" "application/json"))
      :parser #'ein:json-read
      :error (apply-partially #'ein:notebooklist-new-notebook-error
                              url-or-port kernelspec path callback no-pop retry)

lisp/ein-query.el

@@ -1,4 +1,4 @@
-;;; ein-query.el --- jQuery like interface on to of url-retrieve
+;;; ein-query.el --- jQuery like interface on to of url-retrieve -*- lexical-binding: t -*-
 
 ;; Copyright (C) 2012- Takafumi Arakaki
 
@@ -83,6 +83,10 @@ aborts).  Instead you will see Race! in debug messages.
          (cookies (request-cookie-alist host "/" securep))
          (xsrf (or (cdr (assoc-string "_xsrf" cookies))
                    (gethash host ein:query-xsrf-cache))))
+    (ein:log 'debug "EIN:QUERY-PREPARE-HEADER: Found xsrf: %s" xsrf)
+    (setq settings (plist-put settings :headers
+                              (append (plist-get settings :headers)
+                                      (list (cons "User-Agent" "Mozilla/5.0")))))
     (when xsrf
       (setq settings (plist-put settings :headers
                                 (append (plist-get settings :headers)
@@ -100,7 +104,19 @@ variable to a reasonable value you can avoid this situation."
   :group 'ein
   :type 'integer)
 
+(let ((checked-curl-version nil))
+  (defun ein:warn-on-curl-version ()
+    (let ((curl (executable-find request-curl)))
+      (unless checked-curl-version
+        (setq checked-curl-version t)
+        (with-temp-buffer
+          (call-process curl nil t nil "--version")
+          (goto-char (point-min))
+          (when (search-forward "mingw32" nil t)
+            (warn "The current version of curl (%s) may not work with ein. We recommend you install the latest, official version from the curl website: https://curl.haxx.se" (buffer-string))))))))
+
 (defsubst ein:query-enforce-curl ()
+  (ein:warn-on-curl-version)
   (when (not (eq request-backend 'curl))
     (ein:display-warning
      (format "request-backend: %s unsupported" request-backend))

lisp/ein-smartrep.el

@@ -43,11 +43,12 @@
   :type '(repeat (cons string function))
   :group 'ein)
 
-(defmacro ein:smartrep-config (map)
+(when (fboundp 'smartrep-define-key)
-  `(smartrep-define-key
+  (defmacro ein:smartrep-config (map)
-    ,map
+    `(smartrep-define-key
-    "C-c"
+      ,map
-    ein:smartrep-notebook-mode-alist))
+      "C-c"
+      ein:smartrep-notebook-mode-alist)))
 
 (provide 'ein-smartrep)
 

lisp/ob-ein.el

@@ -115,6 +115,7 @@
                   (loop for o in outputs
                         collecting (ob-ein--return-mime-type o file)))))
 
+
 (defun ob-ein--get-name-create (src-block-info)
   "Get the name of a src block or add a uuid as the name."
   (if-let ((name (fifth src-block-info)))
@@ -315,6 +316,53 @@ if necessary.  Install CALLBACK (i.e., cell execution) upon notebook retrieval."
                       (t (url-port parsed-url)))))))
           (t (ein:notebooklist-login url-or-port callback-login)))))
 
+(defun ob-ein--edit-ctrl-c-ctrl-c ()
+  "C-c C-c mapping in ein:connect-mode-map."
+  (interactive)
+  (if (not (org-src-edit-buffer-p))
+      (ein:connect-run-buffer)
+    (org-edit-src-save)
+    (when (boundp 'org-src--beg-marker)
+      (let* ((beg org-src--beg-marker)
+             (buf (marker-buffer beg)))
+        (with-current-buffer buf
+          (save-excursion
+            (goto-char beg)
+            (org-ctrl-c-ctrl-c)))))))
+
+(defcustom ob-ein-babel-edit-polymode-ignore nil
+  "When true override default python mode key mapping for `\C-c\C-c' while inside a babel edit buffer.
+Instead the binding will be to `ob-ein--edit-ctrl-c-ctrl-c', which will execute the code block being edited."
+  :group 'ein
+  :type '(boolean))
+
+(defun org-babel-edit-prep:ein (babel-info)
+  (if (and ein:polymode (not ob-ein-babel-edit-polymode-ignore))
+      (progn
+        (use-local-map (copy-keymap python-mode-map))
+        (local-set-key "\C-c\C-c" 'ob-ein--edit-ctrl-c-ctrl-c))
+    (let* ((buffer (current-buffer))
+           (processed-parameters (nth 2 babel-info))
+           (session (or (ein:aand (cdr (assoc :session processed-parameters))
+                                  (unless (string= "none" it)
+                                    (format "%s" it)))
+                        ein:url-localhost))
+           (lang "ein-python")
+           (kernelspec (or (cdr (assoc :kernelspec processed-parameters))
+                           (ein:aif (cdr (assoc lang org-src-lang-modes))
+                               (cons 'language (format "%s" it))
+                             (error "ob-ein--execute-body: %s not among %s"
+                                    lang (mapcar #'car org-src-lang-modes))))))
+      (ob-ein--initiate-session
+       session
+       kernelspec
+       (lambda (notebook)
+         (ein:connect-buffer-to-notebook notebook buffer t)
+         (define-key ein:connect-mode-map "\C-c\C-c" 'ob-ein--edit-ctrl-c-ctrl-c))))))
+
+(defun org-babel-edit-prep:ein-python (babel-info)
+  (org-babel-edit-prep:ein babel-info))
+
 (loop for (lang . mode) in ob-ein-languages
       do (ob-ein--babelize-lang lang mode))
 

tasks.py

@@ -0,0 +1,170 @@
+import os
+import shutil
+
+from invoke import task
+
+try:
+    from pathlib import Path
+except ImportError:
+    from pathlib2 import Path
+
+
+def find_file(start, name):
+    for relpath, dirs, files in os.walk(start):
+        if name in files:
+            full_path = os.path.join(start, relpath, name)
+            return Path(full_path)
+    return None
+
+
+def generate_cask_env(ctx, curl_path):
+    current_path = os.environ['PATH']
+    execpath = ctx.run("cask exec-path")
+    loadpath = ctx.run("cask load-path")
+    cask_env = {'PATH': '{};{}'.format(execpath.stdout, current_path),
+                'EMACSLOADPATH': '{}'.format(loadpath.stdout)}
+    if curl_path and Path(curl_path).exists():
+        cask_env['PATH'] = "{};{}".format(curl_path, cask_env['PATH'])
+    return cask_env
+
+
+def prepare_ecukes_args(with_doc, with_file, verbose, quiet, help, debug,
+                        tags, script, no_win, win, reporter, timeout,
+                        patterns, anti_patterns, only_failing,
+                        error_log):
+    sarg = ""
+    if with_doc:
+        sarg += " --with-doc"
+    if with_file:
+        sarg += " --with-file"
+    if verbose:
+        sarg += " --verbose"
+    if quiet:
+        sarg += " --quiet"
+    if help:
+        sarg += " --help"
+    if debug:
+        sarg += " --debug"
+    if tags:
+        sarg += f" --tags {tags}"
+    if script:
+        sarg += " --script"
+    if no_win:
+        sarg += " --no-win"
+    if win:
+        sarg += " --win"
+    if reporter:
+        sarg += f" --reporter {reporter}"
+    if timeout:
+        sarg += f" --timeout {timeout}"
+    if patterns:
+        sarg += f" --patterns {patterns}"
+    if anti_patterns:
+        sarg += f" --anti-patterns {anti_patterns}"
+    if only_failing:
+        sarg += " --only-failing"
+    if error_log:
+        sarg += f" --error-log {error_log}"
+    return sarg.strip()
+
+@task
+def ecukes(ctx, curl_path=None, with_doc=False, with_file=False, verbose=False,
+           quiet=False, help=False, debug=False, script=False,
+           no_win=False, win=False, reporter="magnars", timeout=None,
+           patterns=None, anti_patterns=None, tags=None, only_failing=False,
+           error_log=None):
+    # import pdb
+    # pdb.set_trace()
+    cask_env = generate_cask_env(ctx, curl_path)
+    cask_env["ECUKES_ARGS"] = prepare_ecukes_args(with_doc, with_file, verbose,
+                                                  quiet, help, debug, tags,
+                                                  script, no_win, win,
+                                                  reporter, timeout, patterns,
+                                                  anti_patterns, only_failing,
+                                                  error_log)
+    #f"--reporter {reporter} --tags ~@julia,~@ir,~@memory,~@switch"
+    ecukes_bindir = find_file(os.getcwd(), "ecukes")
+    ecukes_cli = ecukes_bindir.parent.parent.joinpath("ecukes-cli.el")
+    if win:
+        ctx.run("emacs --load {} -Q".format(ecukes_cli), env=cask_env)
+    elif no_win:
+        ctx.run("emacs -nw --load {} -Q".format(ecukes_cli), env=cask_env)
+    else:
+        ctx.run("emacs --script {} -Q".format(ecukes_cli), env=cask_env)
+
+
+@task
+def autoloads(ctx):
+    ctx.run("""emacs -Q --batch --eval "(package-initialize)" --eval "(package-generate-autoloads \\"ein\\" \\"./lisp\\")" """)
+
+
+@task
+def clean_elc(ctx):
+    ctx.run("""cask clean-elc""")
+
+
+@task
+def clean(c):
+    clean_elc(c)
+    cwd = Path('.')
+    if Path("test/test-install").exists():
+        Path("test/test-install").unlink()
+    [f.unlink() for f in cwd.glob('log/*websocket*')]
+    [f.unlink() for f in cwd.glob('features/Untitled*.ipynb')]
+    [f.unlink() for f in cwd.glob('test/Untitled*.ipynb')]
+
+
+@task
+def dist_clean(c):
+    clean(c)
+    cwd = Path('.')
+    if Path('dist/').exists():
+        [f.unlink() for f in cwd.glob('dist/*')]
+        Path('dist').rmdir()
+
+
+@task
+def test_compile(c):
+    c.run("cask install")
+
+
+@task
+def test_jupyterhub(c):
+    c.run("cask exec ecukes --tags @jupyterhub --reporter magnars")
+
+
+@task
+def test_poly(ctx, curl_path):
+    updated_env = generate_cask_env(ctx, curl_path)
+    ctx.run("cask exec ert-runner -L ./lisp -L ./test -l test/testfunc.el test/test-poly.el test/test-func.el", env=updated_env)
+    shutil.copy('test/test-poly.el', 'features/support/test-poly.el')
+    ecukes(ctx, curl_path=curl_path, reporter="magnars")
+    Path('features/support/test-poly.el').unlink()
+
+
+@task
+def test_func(ctx, curl_path=None):
+    updated_env = generate_cask_env(ctx, curl_path)
+    ctx.run("cask exec ert-runner -L ./lisp -L ./test -l test/testfunc.el test/test-func.el",
+            env=updated_env)
+
+@task
+def test_int(ctx, curl_path=None):
+    updated_env = generate_cask_env(ctx, curl_path)
+    ecukes(ctx, curl_path=curl_path, reporter="magnars")
+
+
+@task
+def test_unit(c):
+    c.run("cask exec ert-runner -L ./lisp -L ./test -l test/testein-loader.el")
+
+
+@task
+def check_var(c):
+    var = c.run("cask exec-path")
+    print("The cask exec path is = {}".format(var.stdout))
+
+
+@task
+def windowed_testfunc(c):
+    c.run("cask exec ert-runner --win -L ./lisp -L ./test -l test/testfunc.el test/test-func.el")

test/testein-loader.el

@@ -0,0 +1,3 @@
+(load "testein.el" nil t)
+(dolist (file (directory-files (file-name-directory load-file-name) t "test-ein.*\\.el"))
+  (load file nil t))

test/testfunc.el

@@ -27,4 +27,3 @@
 (ein:testing-wait-until (lambda () (ein:notebooklist-list)) nil 15000 1000)
 (defvar *ein:testing-port* (car (ein:jupyter-server-conn-info)))
 (fset 'y-or-n-p (lambda (prompt) nil))
-      

test_requirements.txt

@@ -0,0 +1,6 @@
+jedi <= 0.13.3
+jupyter
+ipython
+numpy
+matplotlib
+invoke

tools/appveyor.init.el

@@ -6,17 +6,17 @@
 (set-terminal-coding-system 'utf-8)
 (set-selection-coding-system
  (if (eq system-type 'windows-nt)
-     'utf-16-le  ;; https://rufflewind.com/2014-07-20/pasting-unicode-in-emacs-on-windows
+     'utf-16le-dos  ;; https://rufflewind.com/2014-07-20/pasting-unicode-in-emacs-on-windows
    'utf-8))
 (prefer-coding-system 'utf-8)
 
-(require 'package)
+;; (require 'package)
-(setq package-enable-at-startup nil)
+;; (setq package-enable-at-startup nil)
-(setq package-archives
+;; (setq package-archives
-      '(("melpa" . "https://raw.githubusercontent.com/d12frosted/elpa-mirror/master/melpa/")
+;;       '(("melpa" . "https://raw.githubusercontent.com/d12frosted/elpa-mirror/master/melpa/")
-        ("org"   . "https://raw.githubusercontent.com/d12frosted/elpa-mirror/master/org/")
+;;         ("org"   . "https://raw.githubusercontent.com/d12frosted/elpa-mirror/master/org/")
-        ("gnu"   . "https://raw.githubusercontent.com/d12frosted/elpa-mirror/master/gnu/")))
+;;         ("gnu"   . "https://raw.githubusercontent.com/d12frosted/elpa-mirror/master/gnu/")))
-(package-initialize)
+;; (package-initialize)
-(unless (package-installed-p 'use-package)
+;; (unless (package-installed-p 'use-package)
-  (package-refresh-contents)
+;;   (package-refresh-contents)
-  (package-install 'use-package))
+;;   (package-install 'use-package))