emacs-ipython-notebook/Enhancements.org

#+STARTUP: indent

* Overview
* Design
** Notebook Format

Version 4.0 [[http://nbformat.readthedocs.org/en/latest/][documented]].

Earlier versions might be documented less formally on the wiki. Can
also look at the IPython source in the json viles.

** Notebook Buffer

Notebook information is stored as a [[file:lisp/ein-notebook.el::ein:$notebook][struct]]. Always associated with a buffer,
[[file:lisp/ein-notebook.el::ein:notebook-buffer][ein:notebook-buffer]] is used to find buffer associated with a notebook.

Notebook does not hold cells, that is delegated to instances of the [[file:lisp/ein-worksheet.el::ein:worksheet][worksheet]]
class. Instances are stored as a list in the `ein:$notebook-worksheets` slot.

Opened notebooks are kept in the ~ein:notebook--opened-map~ hash
table. Keys are cons cells of ~url-or-port~ and ~path~.

There are a number of helper functions for returning the struct for an opened notebook:

 - ~[[file:lisp/ein-notebook.el::(defun%20ein:notebook-get-opened-notebook%20(url-or-port%20path)][ein:notebook-get-opened-notebook]]~ ::
 - ~[[file:lisp/ein-notebook.el::(defun%20ein:notebook-get-opened-buffer%20(url-or-port%20path)][ein:notebook-get-opened-buffer]]~ ::
  
** Notebooklist Buffer
** Kernel communication
** Contents API

Documented at the IPython Github [[https://github.com/ipython/ipython/wiki/IPEP-27%253A-Contents-Service][wiki.]]

There is also [[http://petstore.swagger.io/?url=https://raw.githubusercontent.com/jupyter/jupyter-js-services/master/rest_api.yaml][another]] great online resource for session and kernel
REST API.

** Connecting to a running Kernel
Entry point is ~[[file:lisp/ein-notebook.el::ein:notebook-start-kernel][ein:notebook-start-kernel]]~ which is called from
~ein:notebook-request-open-callback~ after successful call to the notebook
server requesting the contents of a given notebook.

~[[file:lisp/ein-kernel.el::ein:kernel-start][ein:kernel-start]]~ starts/gets a session with a running kernel using the REST API.

On a [[file:lisp/ein-kernel.el::ein:kernel--kernel-s][successful]] return ein [[file:lisp/ein-websocket.el::ein:websocket][creates]] a websocket channel (channels for
IPython 2.x) via a call to ~websocket-open~ in the [[https://github.com/ahyatt/emacs-websocket][emacs-websocket]]
package. The URL request is of the form:

#+BEGIN_QUOTE
ws://{server_address}:{port}/api/kernels/{kernel id from previous REST query}/channels?session_id={session id}
#+END_QUOTE

* Enhancements/Fixes
** Errors on saving new notebook
Happening because metadata slot is borked when creating new notebooks.


** Support company-mode
** Inline latex
See issue [[https://github.com/millejoh/emacs-ipython-notebook/issues/88][#88]].

** Interactive Widgets

For the most part this is a non-starter since in Jupyter this is built on web
and javascript, but maybe with emacs 25's coming integration with [[https://www.emacswiki.org/emacs/EmacsXWidgets][xwidgets]] there
is hope?

** Working with jupyterhub

Jupyterhub requires authentication using username/password, as opposed to just
providing a secret when logging into ipython 3.x and earlier.

On logging in a cookie of form "jupyter-hub-token-<username>" is generated and
propogated with all calls to server. Emacs request should automatically handle
this.

The REST API for this looks like POST http://{host}:{port}/hub/login, username and password
parameters in the POST.

Also looks like the content REST API has been modified so that querys are of the
form: /user/<username>/<command>.

** Imenu/Speedbar Cooperation
Seems to be a couple ways of doing this:

 1. Configuring `[[http://emacswiki.org/emacs/ImenuMode#toc12][imenu-generic-expression]]` regex's.

 2. Redefining imenu-create-index ala python.el.

(2) seems to be the more elegant solution.

EIN currently has minimal support for imenu through
`[[file:lisp/ein-worksheet.el::ein:worksheet-imenu-create-index][ein:worksheet-imenu-create-index]]`, but all it does is look for
headings. Somehow this fails to work with speedbar and also does not handle
indexing Python code (i.e. variables, function, classes, etc.).

To get the speedbar working we will need to define a minor mode per the
following [[http://www.gnu.org/software/emacs/manual/html_node/speedbar/Minor-Display-Modes.html#Minor-Display-Modes][instructions]]. 

For /name/~-speedbar-menu-items~ can I just use ~imenu-generic-expression~?

Maybe the way to do this is for each ~[[file:lisp/ein-cell.el::ein:codecell][codecell]]~ create a temp buffer with the text
of that cell and call ~ein:imenu-create-index~.

#+BEGIN_SRC elisp
  (let ((text (ein:cell-get-text cell)))
    (with-temp-buffer
      (insert text)
      (ein:imenu-create-index)))
#+END_SRC

Still will need way to map temp buffer positions to actual positions in the
notebook buffer (~ein:cell-input-pos-min~ and ~ein:cell-input-pos-max~)

** Live links to other notebooks

 1. Understand how org-mode does it.
 2. Steal???
 3. Profit!!!

** Use polymode

[[https://github.com/vspinu/polymode][Polymode]] uses indirect buffers, which may or may not be a good solution for ein
notebooks. I think this is what nxhtml is doing...

** Use [[https://github.com/magnars/dash.el][dash]]?
Get rid of all those cl compile warnings?

Also look at using [[https://github.com/magnars/s.el][s]] and [[https://github.com/rejeep/f.el][f]].

** Access password protected notebooks (issue [[https://github.com/millejoh/emacs-ipython-notebook/issues/57][#57]])
This is what I have found out so far:

You can authenticate with the IPython/Jupyter notebook server using
ein:notebooklist-login. After calling this a cookie is generated (very easy to
see if you are using curl as the backend for emacs-request) and you can then use
the REST API to list and get notebook data.

Once authenticated REST calls to get notebook json data and create sessions work
fine. After EIN starts a session one can see the kernel is running from the web
interface. The problem starts when ein tries to open a websocket connection to
the kernel. The notebook server generates a 403 forbidden response. I think
because emacs-websocket doesn't know anything about the security cookie
generated during the curl request.

Not sure if that makes sense, but for the moment that is my theory on what's
happening. Somehow we need to provide the security cookie with the websocket
connect request.

<2015-06-09 Tue> SOLVED(?) - issue is that emacs-websocket needs to provide more info with the connection header:

1. Specify the port along with the url.
2. Pass along a security cookie.

** Connect to non-python kernels
** Synergies with pymacs?
** Detect system path of opened notebook
** Jump to notebook code in traceback (issue [[https://github.com/millejoh/emacs-ipython-notebook/issues/42][#42]])

What needs to be done:

1. Carry notebook reference in the ~[[file:lisp/ein-traceback.el::ein:traceback][ein:traceback]]~ structure.
2. Look for ~<ipython-input-3-05c9758a9c21> in <module>()~. The number 3 means
   input #3 in the notebook.
3. Find cell based on input number. Can iterate through list of cells () and look for matching
   ~input-prompt-number~.
4. Call ~ein:cell-goto~ on that cell. May need to swap buffers first.

** The Return of Worksheets

tkf/ein and IPython 2.x allowed for multiple worksheets within an individual
notebook. This feature was removed in 3.0 since multiple worksheets do not make
much sense in the context of a tabbed web browser interface. EIN's legacy code
still supports worksheets, though at the moment that information is lost upon
saving a notebook.

Having multiple worksheet support makes some sense for ein; below is thinking on
how to reimplement this feature.

IPython nbformat 4 specifies a [[http://ipython.org/ipython-doc/3/notebook/nbformat.html#metadata][metadata]] key which can be used to store general
information. Cell metadad has a tag key which is a "A list of string tags on the
cell. Commas are not allowed in a tag."

Best place to set the tag key is when generating [[content]] for saving a notebook.

** Fixing Tests
 - Insert output tests are failing - probably due to how we are making
   the test cell. JSON is per nbformat4, but are we correctly parsing
   mimetypes (i.e. there is an additional call to do this, are we
   making it?). Is [[file:lisp/ein-cell.el::ein:cell-insert-output][ein:cell-insert-output]] getting called?