To me as an org mode and org babel user, notebooks seem to be a very half-assed form or literate programming for the following reasons:
(1) No linking from on cell to another. A sequential execution is assumed or otherwise you need to manually run cells in different order. There might be extensions to help with that. It is not out of the box though.
(2) The documentation language is markdown This does not allow for great technical documents with arbitrary links to arbitrary other parts in the document or inside another notebook. ReStructuredText would have that using link sources and link sinks. org mode can also link to parts in other documents.
(3) I cannot include other notebooks to have it all exported as one whole document or notebook as I can in org mode. In org mode I can even specify at what level some heading should be imported.
(4) The editor in notebooks is not that great compared to any IDE or normal code editor or Emacs.
(5) In a typical JupyterLab environment there is only one programming language per notebook. I know there are other kinds JupyterLab environments, which allow for multiple langs inside the same notebook.
(6) The base format of notebooms is JSON instead of plain text like in org mode. This means diffs are harder to read and version control. Again you need extra tooling for dealing with it well (which does exist).
The literate programming stuff of notebooks has all long been there before notebooks came into existence. It is not like that was the advent of a new form of literate programming. In JupyterLab environments it would take lots of fumbling and tweaking to get something comparable to org + org babel.
Pluto solves¹ many of these problems, but it’s Julia only. Most importantly:
- cells are linked through a dependency graph, so the output you see is independent of the order in which the cells appear;
- each notebook is backed by a plain Julia module, so you can use version control, edit in a real editor instead of only in the browser, and import other notebooks;
- You can create interaction² using HTML inputs, Julia wrappers over those, or go crazy with your own custom JavaScript controls;
- It’s easy to embed LaTeX into the MarkDown, and you can use HTML, so any hyperlinking works.
1 ‘An introduction to pluto’. LWN. Available from: https://lwn.net/Articles/835930/
2 ‘PlutoUI demonstration’. Available from: https://pluton.lee-phillips.org/sliders/uiDemo.html
Org mode and everything Emacs has a crazy steep learning curve. I tried using Org for note-taking with formulas, and I remember having to learn a ton of arcane things like Doom Emacs and endless keyboard bindings (Org + Emacs + Vi). I then just switched back to something easy, where I could fluently get my thoughts down without wasting endless time.
Maybe I should give it another go with Spacemacs because I like the idea of Org mode. That might make the learning curve gentler. I hadn't considered that possibility at the time.
I don't understand why Emacs's features aren't exposed via context menus. You can always switch to the keyboard later as you get good.
The learning curve is steep, that is true. I guess one has to invest some learning to get the good stuff. I myself needed 2 attempts at getting into Emacs. One very short one, where I found myself struggling with even saving a buffer, that ended quickly. The other one more serious, investing my free time out of curiosity, what all the Emacs fuss is about. At some later point I discovered org mode and from then on, there was simply no way back for me to something less. Similar for Magit. Or running shells inside Emacs and being able to simply copy anything from a shell buffer, because it is a buffer and not some readonly thingy in a separate window. Very useful.
Now I use org mode almost every day and definitely for all kinds of documentation and notes. It might not be perfect out of the box for formula rendering, but even that can be arranged/configured, so that you can switch to rendered formulas inside your buffer and I have done so in the past.
I have not tried to use Doom Emacs or Spacemacs or whatever flavour. I started with standard Emacs and went with that and still use that. If Doom Emacs or Spacemacs introduce too many new concepts or make the learning curve too steep, perhaps trying standard Emacs will be a better experience.
Not sure what use context menus would have in my personal workflow. Perhaps if they could be opened via keyboard shortcut and then I could select menu items quickly, it would be useful.
There’s some work in this space, such as Nicolas Rougier’s promising notebook-mode[1]. I’m convinced there would be an audience for an OrgBook app that philosophically treated Emacs as an implementation detail. Give it more familiar keybindings, some out of the box nice looking themes, and configure the new context menu functionality as you suggest. Then package it up as something that can be run and installed with or without an existing Emacs.
It’s hard to imagine experienced Emacsers wanting to lead a project that solves a problem they don’t have, but the community is very friendly so whoever took it on would get plenty of help.
I don't understand why Emacs's features aren't exposed via context menus
Mainly I think it is a matter of efficiency...on the implementation side. I mean the hard part about learning something like org-mode (or emacs itself) is conceptual not key bindings.
It's this way because org-mode (or emacs itself) is a powerful tool that users use for decades. The tooling is intended to reward expertise and experience. No amount of context menus will change the fact that the tools are deep and conceptual mastery will take more than a few hours (never mind the thirty seconds the typical app has to engage the user).
Org-mode was written for it's author's use. It was written to get things done. Easy onboarding and massive investment in graphical UI for non-experts wouldn't do that. It requires work to learn. That's true of anything hard.
It's not that I don't sympathize...though it probably can be interpreted that way. It's that using emacs (and org-mode) is analogous to playing the piano, not playing the stereo.
Finally, I while I get the appeal of things like spacemacs, I think it's mostly a distraction. The documentation and tooling is by it's nature derivative...it is by it's nature extra work. Even worse it's extra work that's driven by ideology. Worser still, it's extra work driven by the ideology of Vim versus Emacs. And worst of all, it's rational is emacs-is-too-hard-to-learn.
Emacs is hard to learn, but not too hard. It's harder to learn than a sixteen week college course. But that's most things in the working world.
You can activate a menu, but it only covers very basic things. If you are confined to using only the menu, you might equally well use notepad.
For base emacs, the learning curve is not as steep as vim. I wonder if your problem was trying to do everything at once?
Regardless, emacs really is more of a long-term investment. It will take you years to be fully proficient. Personally, I have gone away from using emacs (including org-mode and vim emulation), because it took too much time to keep the configuration shiny.
My main issue with org-mode is that it's only really a first class citizen within emacs. I've not found any vim/vscode/etc. plugins that emulates the experience to satisfactory degree, and that lock-in means I've given up on org-mode.
As others have said, a big draw for notebooks is keeping the bar low. Sequential ordering is intuitive and doesn't require a complicated additional ui for managing the execution graph.
In fact, the execution graph logic is handled by the user. When it gets too complex, it's a sign that the notebook itself is too complex, and it's time to move a lot of code into libraries. Notebooks /should/ be conceptually simple.
Then again, I bounced off emacs after deciding the complexity didn't actually justify the endpoint: I was already able to write and maintain code with much simpler tools, so why bother? I can use the party of my brain that would otherwise be used for key bindings for other things. So, mileage varies.
To each their own. I want to question though, whether those simpler tools are truly simpler. Lets say you want to use notebooks instead of org mode. You will need to have the following set up, before you can start:
- python3 (OK, mostly a given, though not on Windows)
- pip and depending on your requirements you might need Poetry or Pipenv or whatever else
- Jupyter ecosystem (JupyterLab, Jupyter/notebook server and all their dependencies)
- some extensions for JupyterLab. Those could be frontend and backend. If frontend, then you will also need to have NodeJS on your machine, to have NPM, to have tsc to be able to build JupyterLab with the extension installed.
Especially for analysis tools (as I tend to use colab/Jupiter), the important part of complexity is cognitive load. Installation is a constant, upfront cost in that regard: I have literally never worried about the NPM or nodejs, as they're handled automatically by standard installers... Or already running in shared environments. (And if I'm deeply worried about binary size and dependency depth, I probably shouldn't be working python in the first place, and have other tools I can go to.)
Cognitive load when using the notebook is very low compared to emacs. As has been remarked on often, emacs is probably fine if you've been using it for thirty years, but having to keep track of different key strokes for copy paste depending on what window you're in (emacs vs the rest of the world) is pointless cognitive load. The idioms are so sprawling and out of sync with everything else that the editor is an obstacle in the way to doing what I want.
And while (historically) that perspective may just paint me as a noob, I'll just say that no amount of memorizing editor functionality or micro optimizing configurations will ever help me prove a theorem. If there's another tool that gets the job done and doesn't want to fight me before/while I use it, then it's the better tool.
R notebooks (which are not jupyter-based) addresses (to varying degrees) issues 3-6. But then you are of course stuck inside the R ecosystem, where only Rstudio and emacs are what I would consider fully mature editors.
As a casual rather than power user, I especially dislike the JSON format of jupyter notebooks.
For all the popularity of Jupyter notebooks, I never did understand the appeal. I agree that R notebooks and Emac's org-mode are far more sensible ways of doing things. Not perfect, by any means, but sensible.
I cannot speak to R, but the Emacs learning curve is the primary reason org-mode is not more popular. There may be a break-even point past which learning Emacs is a net benefit, but few people have the time, skill, and inclination to reach that point.
It is a good thing, that slowly other editors gain support for org files. I've seen them at least highlighted in VS code recently, when someone else shared their screen, even if that is but a tiny fraction of what support for org files can mean.
You might like Starboard Notebook (https://starboard.gg), it's in-browser and mixed multi-language, as well as diffable/version control friendly. (I'm building it).
These are interesting to me as a notebook user, because a lot of these I see as advantages of the notebook design.
(1) This is better to me, because it makes notebooks more consistent and easier to jump into. I read english prose sequentially, why not code?
(2) You can do this in all environments I've used with <a> tags.
(3) this would be nice, but I'm not sure it's worth the tradeoff because the editing environment wouldn't correspond as well to the exported document.
(4) true, but I have a vim plugin that make it better
(5) also true, but this is a good thing for simplicity. I don't want to have to keep track of the lexical scope of multiple language environments
(6) There are tools for this, but yeah it's not great. The main issue is saving outputs
It sounds like the goals of notebooks vs org mode are slightly different, at least the way I use notebooks.
For me, notebooks are a tool for sacrificing control and programming flexibility in exchange for easier collaboration and visualization. It's harder to write structured code, but it's much easier to share with a large number of semi-technical people.
When you can define a cell depending on another cell and can expect that other cell to be run after its dependencies have run, then you can make sure, that a cell will always give the correct result, not a different one, because other cells have already run, which mess up the state. It would allow you to scroll to the end and see the last layer of abstraction, just run that, and implicitly also run all its dependencies (other cells).
In a top to bottom typical notebook, you will have to run the notebook from the top. There is a menu item for that, of course. However, what if you also have some cells in there, which do not have to be run? They will waste time unnecessarily or even worse, will change the result. So you will need to take a look at each cell to make sure you really can run everything sequentially. Either the notebook has to be created to make sure, that running top to bottom always works and gives the same result, or you will have to look at everything. When code segments can define their dependence on other code segments, they can be designed to that it does not matter which one you run, it will always give the intended result.
> (1) This is better to me, because it makes notebooks more consistent and easier to jump into. I read english prose sequentially, why not code?
As a counterexample, some things are not important but have to be mentioned. In English, these things end up in the appendix. In Pluto notebooks, these things can be moved to the bottom.
Another benefit of less strict ordering is when debugging. Sometimes cell A fails because of cell B. It can be useful for debugging to temporarily put these cells next to each other
1) Yeah, the execution count is there, but there isn't a built in way to visualize it other than the default sequential cells. I really wonder if there's a useful way to show a notebook as a nodegraph.
2) I can't understand the desire for RST over Markdown. Between simplicity and adoption, Markdown beats almost all mark-up languages -- maybe even html.
3) You can use the `run` command to link notebook executions and use Markdown linking to link the documentation.
4) This isn't true. I use vim in Jupyter. There's plugins for emacs too. Also, many editors like VSCode can edit notebooks directly.
5) This is a weird complaint. I can imagine a few corner cases where mixing languages in a notebook would be nice...but the concept of a multi-language Jupyter kernel doesn't make much sense from a practical standpoint.
6) There are diff tools such as `nbdime`. Between those and a few git commit hooks to leave your notebooks in a consistent execution state, merging versions notebooks isn't any worse than other files.
(1) No linking from on cell to another. A sequential execution is assumed or otherwise you need to manually run cells in different order. There might be extensions to help with that. It is not out of the box though.
(2) The documentation language is markdown This does not allow for great technical documents with arbitrary links to arbitrary other parts in the document or inside another notebook. ReStructuredText would have that using link sources and link sinks. org mode can also link to parts in other documents.
(3) I cannot include other notebooks to have it all exported as one whole document or notebook as I can in org mode. In org mode I can even specify at what level some heading should be imported.
(4) The editor in notebooks is not that great compared to any IDE or normal code editor or Emacs.
(5) In a typical JupyterLab environment there is only one programming language per notebook. I know there are other kinds JupyterLab environments, which allow for multiple langs inside the same notebook.
(6) The base format of notebooms is JSON instead of plain text like in org mode. This means diffs are harder to read and version control. Again you need extra tooling for dealing with it well (which does exist).
The literate programming stuff of notebooks has all long been there before notebooks came into existence. It is not like that was the advent of a new form of literate programming. In JupyterLab environments it would take lots of fumbling and tweaking to get something comparable to org + org babel.