The case of TeXmacs <-> Jupyter and some other thoughts

#1

Maybe this discussion has already taken place here or elsewhere, but I would like to discuss some of the similarities between Texmacs and the Jupyter ecosystem (notebook, jupyterbook) here and open the discussion on other aspects.

Texmacs is for me the precursor … it offers of course features similar to the kernels of the Jupyter project but with the ease of formatting the text and the equations as well as an unequaled quality of page layout.

The vision of Texmacs and the quality of its realization is already there so what it lacks the most to me is its community size and better visibility … in the order you prefer, the two often go together.

I dream (I’m certainly not the only one) that Texmacs will become one of the key players (just that) in the publishing of interactive (or not: at the discretion and needs of the user) scientific documents and diffusion of these documents (colleagues, students, editors).

I am thinking in particular of several things:

  1. Edition of “static” documents as they are imagined when printed.
  2. Edition/Use of interactive documents: mixture of text, equations, sketches, and code generating computations, tables, figures … possibilities of interacting with them through widgets.
  3. Interaction with other essential existing formats to share, spread and preach the good word (Jupyter notebook, markdown, msoffice or openoffice, … why not org-mode). Pandoc can be an ally here …

If I try to comment these different points:

  1. Texmacs knows how to do it perfectly, it already excels in this area. Some aspects of the GUI can be improved and new style or template can be added (a tufte like layout please :innocent:). Thanks to tm-forge and the growing and very nice Texmacs’s community things are changing.
  2. The number of plugins is constantly growing, see for example the great jupyter and julia plugins. What “lacks” in my opinion is the possibility of interacting with these plugins (code, data, figures) via gui elements like Jupyter widgets can do for example.
  3. Perhaps the most important for improving the visibility and strength of his community. I think this is one of the most important levers for Texmacs to gain all the aura it fully deserves.

For me, the similarities between the Jupyter project and Texmacs are such that I imagine very strong interactions between the two formats (.ipynb and .tm). And if you ask me if I prefer to edit scientific documents (mixing code, equations, figures, …) in Texmacs or a Jupyter Notebook my choice is quickly made. Who doesn’t fall in :heart: with Texmacs.

The problem then comes from sharing and communicating these documents. Clearly the .ipynb of the jupyter project are winning (for the moment).

I took an old class project using the Jupyterbook and editing my files I thought holy shit if only I could do it in Texmacs and then deploy it to the web. I searched the web and saw the very promising tm2md plugin. I thought to myself why not have also a Texmacsbook or a way to export to MyST Markdown. Hello tm2md you look so nice and pretty today lets make babies together …

Texmacs could in my opinion become an (THE ?) essential platform for publishing/sharing scientific documents. We speak more and more of reproducible research, publishers are increasingly interested in it. I think Texmacs can play an essential role in this area with all the advantages it already offers.

And you what are your thoughts ?
Enjoy your day :wink:

#2

For the moment, very short intervention (hopefully the discussion progresses).

  • Linking to the tm2md from either TeXmacs notes or tm-forge could be good (I hope MDBD is ok with that)
  • I have a limited experience with the TeXmacs widget system—I used one widget in a program that I wrote with @jeroen. I do not see how to embed them within the text of a document, and I do not know if one can use, e.g., sliders, but something is already possible.
#3

Hi,

of course I’m ok with mentioning the converter! I will move the repo to the TeXmacs organization in github soon, and maybe include it as a subrepo in tm-forge or mention it as you suggest.

It was a weekend hack for my blog and once it worked for me I didn’t think again about it. Now I’m using it for the company I work for and I have implemented / fixed a few things, so it might be more useful, once I disentangle all the non-standard markdown stuff for the Hugo static site generator.

#4

Supporting MyST looks like quite the endeavour. I will try to unravel the current messy state the converter is in so that implementing extensions other than Hugo’s is possible, but I don’t think I can commit to implementing the whole feature set of MyST. I will think about it!

#5

Hi @fbob! I’m not sure what is the real target of this discussion. There are two orthogonal possible approaches I see in integrating TeXmacs with other formats:

  1. TeXmacs as an editor for other formats: this require both in and out converters which are able to work conservatively. Then one could imagine to be able to use TeXmacs to work on jupyter notebooks, eventually collaborating with other people which do not use TeXmacs. My feeling is this is very hard and would reduce the features to a common denominator.

  2. TeXmacs as the main source of content. Documents are TeXmacs native documents, maybe written with special style file which make conversion to other formats easier or more consistent. In this way one can produce content in TeXmacs and then publish it in any format, be it HTML, HTML+MathML (e.g. with accessibility features), Jupyter notebooks, LaTeX, source code, etc…

This second style is what I think is more meaningful (at least for how I think about TeXmacs and its possibilities in the wider digital ecosystem). But then the question is: why spend resources in developing a converter to jupyter notebook format when we can just as well produce directly the HTML for the web pages? Somehow is not clear to me what is the meaning of having jupyter notebook available from TeXmacs document. In my mind the notebook format is a format designed for editing not for publishing, and if one uses TeXmacs one does not need it because one can pass from the TeXmacs document to the HTML output directly. Note that as far as I know one cannot publish notebooks, i.e. unless you have a jupyter server available you cannot run notebooks in the web. So a published book with jupyter books is as static as a web page and we can just as well produce beautiful HTML books from the documents we already have. Do we really want to produce jupyter notebooks as output? and then, are people allowed to modify these notebooks and expect to be able to see again the result in TeXmacs? If you just want to deploy to the web then jupyter is a middle man that one can cut short, I do not see what is the added value to emit jupyter notebook format. (I do not want to sound harsh, it is a genuine question, I’m asking sometime myself the same kind of questions). The advantage of TeXmacs is that we have full control of the semantics and format of documents, so for us is easy to convert to other formats, while guarantee interoperability is a much harder job and maybe just an illusion given the limited forces we have. It is maybe worth to have some interoperability but the danger is that the user take it for granted and then it remarks that is not perfect and assign the fault to TeXmacs…

#6

Note that I’ve initiated the jupyter plugin just for the reason that I think is important to have support for jupyter-like functionality, not necessarily to emulate what jupyter notebooks are. I think their design is strongly influenced by the limitation of the web platform. We do not have that limitation (maybe we have others) so maybe we can produce different kind of interactive documents. For example Julia has Pluto notebooks in which the state of the computation is never inconsistent and that cells are evaluated according to their logical dependencies (which is not what happens in Jupyter notebooks where the evaluation of cells is sequential). This is a good example that one can do better than Jupyter notebooks. Also we could imagine different way to interact than just a sequence of computational cells, i.e. computations could be attached to other kinds of elements of the document, not only horizontal cells, or hidden, etc…

It would be interesting to write down what people really need in terms of interactivity, and production of interactive content. Then one writes the associated tags and take care of their editing within TeXmacs and also of their conversion to other formats like HTML directly. So I see TeXmacs more as a programmable content editing platform which is capable of a full range of interactive feature (e.g. we can have interactive widgets, the problem is to design them meaningfully and not just as a passive copy of what people do elsewhere, as a copy they will be necessarily less than the original).

#7

This is somewhat off-topic, but I just realized that we can already make a kind of interactive notebook with spreadsheets.

Screenshot_20210427_171734
Screenshot_20210427_171734.png967×1183 34.7 KB

I believe you can reference cells in other spreadsheets in the document, so you can spread these around in your document.

#8

Nice! Well of course TeXmacs is turing complete so you can do anything (see, e.g. the version of tetris that Miguel coded using tables). Part of the work to be done is devise what is that we would like to have and how to implement it properly. If we do not have a well specified interface (or API as nowadays people say) then we go back to TeX-hell and reliable conversion/adaptation to other formats or other publishing media (like HTML or PDF or EPUB or whatever) become impossible.

I think we can even implement sliders with TeXmacs tags by using a bit of scheme to hook into mouse events like the graphics editor does. (which is written in Scheme)

#9

Very interesting ideas and thoughts on the table :+1:

Thanks, @mdbenito, for your kind feedback.
Having tm2md is already great, thanks for that !
Supporting all the features MyST is not mandatory so that this functionality could be already interesting. It was more food for thought to see what’s others are thinking.

I agree with you @mgubi seeing TeXmacs as an editor for other formats is limiting but being able to communicate with other formats is a nice and important feature to have to reach the mass. As you said I think that TeXmacs should be the main source of content.

To me the notebook format “support” or a way to import it or export to it is more for appealing people to use Texmacs as their main editing and publishing platform. Like “Look, don’t be afraid to embrace our tools and join our community because you can use your previous work in Texmacs and in a short manner of time you will do better with it”. And your example with the way Pluto and Jupyter notebook evaluate the cells is indeed very insightful. Thanks to its talented developers (… you’re welcome …), I’m convinced that Texmacs has the capacity to think and do things better.

I am convinced of this because I use Texmacs and to convince more people they have to be attracted and I think that this question of compatibility with other formats is crucial, because it reassures about the sustainability of their work and the ease of communicating it. The example that we all know are the colleagues who work in .docx … Whether we like it or not, it is a standard (like latex, pdf or html) and we cannot de facto do without it except when Texmacs will be the new standard …

Honestly, I am actually frustrated, frustrated that so few people know about this awesome tool. I used Texmacs in its beginnings (after a long passage through latex) … then stopped, because I found it too buggy or I did not know how to use it. I switched to Lyx, then org-mode in parallel to other tools like jupyter-notebooks and each time I said to myself well shit the development of Texmacs was visionary: all these great features that I found elsewhere, most of them already existed in Texmacs.

How is it that so few scientists know about this tool … this is not giving it the honors it deserves. Texmacs isn’t perfect (no software is) but what really sucks is that there aren’t enough people using it yet. That sucks to death …

@jeroen you just made the proof that Texmacs is really that great !

1 Like
#10

All this discussion is very interesting and I will comment extensively again on the above as soon as I get some time. In the meanwhile I found

http://www.numerical-tours.com

which contains a lot of iphython notebooks, it should be a good source of examples if we want to develop a converter to TeXmacs (for the moment). I always prefer the practical approach over too abstract discussions :slight_smile: (in italian one says “si parla del sesso degli angeli”) After some experimentation one can go back with more insights on the theoretical issues.

#11

With the new repo:

It is time for us to make a great community driven GNU TeXmacs plugin. (Not da-driven or max-driven)