Slow editing of large papers

[guraltsev]

Hello everyone,

For quite some time I have been working on a paper for which TeXmacs was a lifesaver. I could not manage to focus and bear everything in mind while editing LaTeX. The proofs were too technical and the issues with them were creaping up much quicker than I could fix them. Thanks to TeXmacs I managed to finish my paper and post it [2205.09851] The full range of uniform bounds for the bilinear Hilbert transform (arxiv.org).

I recently took up the paper back up to implement some fixes to typos, formatting, and exposition that I have found myself, and colleagues have helpfully pointed out. Next I will be submitting it to a journal.

You might remember me asking often for advice on how to make TeXmacs not be unbearably slow. I have since written a smaller paper in TeXmacs where I did not encounter significant slowdowns. But now that I have opened this paper again the situation is quite dire. I checked performance both under Windows and the current snapshot (2.1.2-svn13888) and it is almost unworkable. The bench debug benchmark gives an average of

TeXmacs] std-bench, Task 'typeset' took 96 ms

My laptop is quite powerful:

Processor:	Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz, 2592 Mhz, 6 Core(s), 12 ``
Installed Physical Memory (RAM)	16.0 GB

I wanted to open this post both because I do not want to yet submit a “generic” bug saying “performance is bad” and to gather practical advice on how to deal with slowdowns because of long papers. I am attaching the files to this post and below I’ll review some things I did that partially helped.

Papyrus mode

Apparently page splitting is very computationally costly. Switching to papyrus mode definitely helped but only because with page splitting things were even worse.

Multi-document

When actively writing the paper I actually separated the document into multiple files. However I got confused and clearly did this incorrectly. Referencing from one file to equations from another did not work. Counters started from 0 for every section. I have some macros defined in the preamble. I had to replicate them manually into the preamble of each file.

Could someone explain or link to an explanation of how to split a large file into multiple ones correctly? It would be best if this were to work well with export to LaTeX. When I was exporting to LaTeX I needed to first create a copy, then “expand all inclusions” and then export to LaTeX. Otherwise it was a mess that required a lengthy adjustment in LaTeX.

Equation alignment

I use tables to align pieces of multiline equations. Surely this may contribute to the slow down but I don’t think it is the main culprit. TeXmacs is slow even with simpler typeset equations

Other offenders

I noticed that even in smaller papers the itemize tag brings TeXmacs to a crawl. Creating a new list item by pressing enter usually takes around 1-2 seconds of frozen editor even when the whole file is at most 10-20 pages (not 100 as the paper at hand). I found that the behavior is worst when the itemize list is inside a math environment (theorem or proof).

Editing the authors of the title of a file that is even 10-20 pages long is very slow. It takes around 0.5-1sec per keystroke. The only way I found for editing the title is writing the title somewhere else and copying and pasting it into the correct destination.

Conclusion?

Am I doing something wrong? Did I misconfigure my editor? Am I abusing syntax in some way? Any tips are deeply appreciated.

How to reproduce

Open ubht.tm. Go to Proposition 3.4 with label prop:outer-restricted-interpolation. Maybe enable benchmarking.

Try editing the bulleted list: adding a new list item takes 900ms

Try editing the math (somewhat long equations) at the end of the proof of that Proposition. The input is very sluggish.

EDIT:

1. tried it again after restarting TeXmacs: the latency for the itemized list is still around 900ms. The latency for editing the equation is better: towards 20ms with occasional spikes to 85ms. Better but still suboptimal. Generally, TeXmacs slows down with time.

2. Forgot the attachments: however, I cannot add them here. Could you enable upload of .tm files. The forum tells me it only accepts images!!

3. The mailing list also did not accept an email with the attachments. The only place I found to post them is by opening a bug report here: GNU TeXmacs - Bugs: bug #62891, Slow editing of large papers [Savannah]

[re4zuaFe]

Let me first comment on the “multi-document” part. It seems that this is deprecated. One is recommended to use multi-part mechanism instead. For articles (namely, the document style is an article), the parts seem to be separated by sections. By clicking Document → Part → Show one part or Show several parts, you can specify which part you want to show.

It looks like that you have collaborators. I wonder how you deal with this if the collaborator(s) use LaTeX?

[guraltsev]

Collaboration

My collaborator left academia some time ago so I was working on this by myself. The other paper I coauthored worked as long as one is not constantly changing who is working on the draft. It is actually not that bad to work on LaTeX/Texmacs back and forth as long you agree on a specific subset of LaTeX to use. Currently, the biggest issue for making the TeXmacs->LaTeX be technically the inverse of LaTeX->TeXmacs is the inability to tell TeXmacs how to import alingned math environments (or how to “learn” of new ones). With that implemented, one can use TeXmacs as a Lyx on steroids!

Multi-part

Unfortunately, the multi-part mechanism is very inadequate for some workflows. In particular, I often needed to have several parts of the paper open in different parts of the document e.g. a definition of an object somewhere, and a Lemma what checks all the properties somewhere else.

All windows open to a file visualize the same parts of it. If one would have a

1. buffer-local view configuration (using emacs terminology here)
2. automatic switching between parts when following links (I think this is already there)

Then the multi-part mechanism would be perfect.

[re4zuaFe]

I am not expecting that TeXmacs would ever support any sort of “learning” the imported LaTeX code or “incremental” importations.

Maybe the mechanism that you said is optimal, but I deal with this via loading package preview-ref as in the screenshot:

[guraltsev]

Yeah, preview-ref is a killer feature! I have it always enabled, and it helps me a lot. Sometimes though having two views is useful e.g. when you have to change a definition when a theorem doesn’t quite work

The “learning” is actually not that huge of a deal. All one needs is to write some post-processing filters in scheme. Currently the importer is written in C++ instead of Scheme so this is a bit awkward, however I know that as a long-term project the importer is planned to be migrated to Scheme and will make “postprocessing” import more natural.

Did you by chance check whether the referenced TeXmacs file is slow for you, too?

[guraltsev]

Just in case anyone else is reading this and is not aware: if you follow a link in TeXmacs you can go back to your original location by pressing Alt+<leftarrow>! This allows going to a referenced equation and implementing minor changes before going back.

[spaceman]

Ah, how would one do this on Mac since there is no Alt button. Very interested in this

[re4zuaFe]

On the toolbar, there is a button with the image of a hand pointing to the left. it is also possible to save the current position by Go → Save position.

[re4zuaFe]

Did you by chance check whether the referenced TeXmacs file is slow for you, too?

Your document seems a bit weird. Maybe you first imported from LaTeX then edit. In particular, I don’t know how to use multi-part mechanism in your document. I copy all the contents into an empty file, and also the preamble into preamble, then it works.

By the way, I would not import a bib file in the tm file. Instead, I import the bib file managed by JabRef into the internal bibliography of TeXmacs, enabled by Tools → Database tool, and then Data → Open bibliography and Data → Import — the auto-complete works when the bibliography information is imported this way.

[guraltsev]

The multi-part issue is due to the fact that it is technically still part of a “project” . To be able to use multi-part you should go to Tools->Project and uncheck use as master.

Thanks for the tip on the bibliography. I’ll look into it. For now I just copy paste the keys from Zotero (that I use to manage the bibliography)

[mgubi]

I’ve just tried to reproduce the slowness problems with ubth.tm. I do not notice any permanent changes after opening and closing the preamble. After coming back from the preamble view indeed it seems that typing is slower but if I press some backspace then type typing speed seems back to normal.

With the itemise, indeed it seems that it is a bit slower than an empty document. Maybe it takes up to a second to create a new item. I would not say it slows to a crawl. Unfortunately slowness issue are a bit tricky to establish, unless they are flagrant. Can you create a small test case to make your point? You say that it happens also for small documents. We need a way to reproduce the bug if we want to have a chance to fix it.

I tried to edit the formula at the end of the Prop 3.4 and I do not notice any slowdown.

Also, this thread contain many different observations. It is not obvious they are all related. It would be better to address each single observation in a different thread. Or better file a bug report and then refer here to add some more context and open discussion. Joris is not in the forum, he only looks at the bug reports.

I’m supposed to know it but I realise I don’t : how do you show the timings?

[guraltsev]

Thanks for the follow up. Timing is displayed in the console after selecting: “Menu->Debug->Bench”

[hamorabi]

I’ve noticed slowness on my end and mentioned it here a couple of time, but it seems to have been overlooked.

[guraltsev]

Dear @hamorabi, could you try to launch TeXmacs from the console and enable benchmarking as described above. We have a suspicion that that some lag is due not to rendering but to other factors that are NOT captured by the above benchmark. An important question is: do you see a correlation between the slowdowns and the logging displayed in the console?

[mgubi]

I think that reports on slowness should be more precise and circostanziate. Is not very useful or helpful to the developers to give “general feelings”. I’ve tried to reproduce some of the situations that @guraltsev mentioned. Here my results (on the current svn on MacOSX)

I noticed that even in smaller papers the itemize tag brings TeXmacs to a crawl. Creating a new list item by pressing enter usually takes around 1-2 seconds of frozen editor even when the whole file is at most 10-20 pages (not 100 as the paper at hand). I found that the behavior is worst when the itemize list is inside a math environment ( theorem or proof ).

It is indeed true that itemize inside a theorem seems slower. On my machine I guess it takes a bit less than 1 sec to create a new item. Then editing the itemize has normal speed, until a new line need to be created. It does not seems to depend on the size of the document. I’ve tried @guraltsev file and feeling is the same as a shorter file. But I cannot reproduce this consistently. Now I’ve tried again on a small 15 pages document and creation speed is the same inside and outside a theorem.

Editing the authors of the title of a file that is even 10-20 pages long is very slow. It takes around 0.5-1sec per keystroke. The only way I found for editing the title is writing the title somewhere else and copying and pasting it into the correct destination.

I cannot reproduce this. I’ve created a file with 15 pages and edited title and author and there is no slowness. I however can reproduce that editing the title of ubht.tm is indeed very slow. This is an anomalous situation as far as my experience goes. I do not remember it happens with my files. I would have to understand why it is so. For example if I change the document style from amsart to article, editing the title can be done at normal speed and if I go back the sluggishness reappears. So seems to be a problem processing the macros in amsart. Worth of deeper investigation.

Try editing the math (somewhat long equations) at the end of the proof of that Proposition. The input is very sluggish.

I cannot reproduce this situation. I followed the istructions and I believe edited the right equation and speed seemed normal (i.e. as for other equations in the document).

[amichail]

Do you get garbage collection times?

[guraltsev]

It is indeed true that itemize inside a theorem seems slower. On my machine I guess it takes a bit less than 1 sec to create a new item. Then editing the itemize has normal speed, until a new line need to be created. It does not seems to depend on the size of the document. I’ve tried @guraltsev file and feeling is the same as a shorter file. But I cannot reproduce this consistently. Now I’ve tried again on a small 15 pages document and creation speed is the same inside and outside a theorem.

I tried it in an empty file and I agree: no slowdown independently of being in or out of an environment.

I tried it in a 35 page paper (not ubht.tm) and slowdown is present in all environments (propositions, theorems, proofs). You can find the paper here:
GNU TeXmacs - Bugs: bug #63631, Slow editing of title and author… [Savannah]

Could you check that you observe the slowdowns?

It seems that the slowdown inside environments is generally present for all operations. For example, in ubht.tm I went halfway through the paper to a proof environment. Granted, some proof environments are multiple pages long but I positioned myself into a proof environment that is 5 lines long. I tried typing in some text inside the proof and immediately outside the proof. Here is the bench

Outside proof:

TeXmacs] std-bench, Task 'typeset' took 12 ms
TeXmacs] std-bench, Task 'typeset' took 12 ms
TeXmacs] std-bench, Task 'typeset' took 12 ms
TeXmacs] std-bench, Task 'typeset' took 12 ms
TeXmacs] std-bench, Task 'typeset' took 11 ms
TeXmacs] std-bench, Task 'typeset' took 12 ms
TeXmacs] std-bench, Task 'typeset' took 12 ms
TeXmacs] std-bench, Task 'typeset' took 12 ms
TeXmacs] std-bench, Task 'typeset' took 12 ms
TeXmacs] std-bench, Task 'typeset' took 13 ms
TeXmacs] std-bench, Task 'typeset' took 12 ms
TeXmacs] std-bench, Task 'typeset' took 12 ms
TeXmacs] std-bench, Task 'typeset' took 12 ms
TeXmacs] std-bench, Task 'typeset' took 12 ms
TeXmacs] std-bench, Task 'typeset' took 11 ms
TeXmacs] std-bench, Task 'typeset' took 12 ms
TeXmacs] std-bench, Task 'typeset' took 13 ms

Inside proof:

TeXmacs] std-bench, Task 'typeset' took 26 ms
TeXmacs] std-bench, Task 'typeset' took 26 ms
TeXmacs] std-bench, Task 'typeset' took 25 ms
TeXmacs] std-bench, Task 'typeset' took 25 ms
TeXmacs] std-bench, Task 'typeset' took 24 ms
TeXmacs] std-bench, Task 'typeset' took 25 ms
TeXmacs] std-bench, Task 'typeset' took 24 ms
TeXmacs] std-bench, Task 'typeset' took 37 ms

Note that this happens even when there is no reflow happening (either last line of proof or last characters of a partially filled line.

This surely is a particularly big deal, but do you think there is some optimization to be done in changing the environment rendering logic?

I cannot reproduce this. I’ve created a file with 15 pages and edited title and author and there is no slowness. I however can reproduce that editing the title of ubht.tm is indeed very slow. This is an anomalous situation as far as my experience goes. I do not remember it happens with my files. I would have to understand why it is so. For example if I change the document style from amsart to article , editing the title can be done at normal speed and if I go back the sluggishness reappears. So seems to be a problem processing the macros in amsart . Worth of deeper investigation.

Yes, I tried with another paper I have written and using the article style instead of amsart does help a lot. It is a bit slower but compatible with the idea that you are rendering a title that has non-trivial spacing.

Now that it seems that this is reproducible for me under windows and under WSL I have filed a bug report. I have also attached two files where this can be observed:

GNU TeXmacs - Bugs: bug #63631, Slow editing of title and author… [Savannah]

Thank you for helping to debug this.

I cannot reproduce this situation. I followed the instructions and I believe edited the right equation and speed seemed normal (i.e. as for other equations in the document).

Now I try and it it is by no means as bad as it was before. You can even see it from the std-bench. Currently, at worst I get 40ms and mostly less than 25s. This is noticeable but it does not distract from working on the actual contents.

The lag I reported before:

TeXmacs] std-bench, Task 'typeset' took 96 ms

was almost a dealbreaker. I honestly am completely confused by what is going on. Maybe the snapshot version running on WSL just works better?

[mgubi]

It seems that the slowdown inside environments is generally present for all operations. For example, in ubht.tm I went halfway through the paper to a proof environment. Granted, some proof environments are multiple pages long but I positioned myself into a proof environment that is 5 lines long. I tried typing in some text inside the proof and immediately outside the proof. Here is the bench

I cannot reproduce this. I’m at Lemma 7.5 of utbh.tm and typing single letters without reflow before the lemma I got

TeXmacs] std-bench, Task 'typeset' took 16 ms
TeXmacs] std-bench, Task 'typeset' took 13 ms
TeXmacs] std-bench, Task 'typeset' took 16 ms
TeXmacs] std-bench, Task 'typeset' took 10 ms
TeXmacs] std-bench, Task 'typeset' took 10 ms
TeXmacs] std-bench, Task 'typeset' took 9 ms
TeXmacs] std-bench, Task 'typeset' took 12 ms

typing in the Lemma environement (same conditions)

TeXmacs] std-bench, Task 'typeset' took 16 ms
TeXmacs] std-bench, Task 'typeset' took 10 ms
TeXmacs] std-bench, Task 'typeset' took 16 ms
TeXmacs] std-bench, Task 'typeset' took 12 ms
TeXmacs] std-bench, Task 'typeset' took 15 ms
TeXmacs] std-bench, Task 'typeset' took 9 ms
TeXmacs] std-bench, Task 'typeset' took 14 ms
TeXmacs] std-bench, Task 'typeset' took 8 ms
TeXmacs] std-bench, Task 'typeset' took 9 ms
TeXmacs] std-bench, Task 'typeset' took 6 ms
TeXmacs] std-bench, Task 'typeset' took 12 ms
TeXmacs] std-bench, Task 'typeset' took 13 ms
TeXmacs] std-bench, Task 'typeset' took 14 ms

while typing in the proof

TeXmacs] std-bench, Task 'typeset' took 6 ms
TeXmacs] std-bench, Task 'typeset' took 16 ms
TeXmacs] std-bench, Task 'typeset' took 18 ms
TeXmacs] std-bench, Task 'typeset' took 12 ms
TeXmacs] std-bench, Task 'typeset' took 75 ms
TeXmacs] std-bench, Task 'typeset' took 7 ms
TeXmacs] std-bench, Task 'typeset' took 8 ms
TeXmacs] std-bench, Task 'typeset' took 8 ms
TeXmacs] std-bench, Task 'typeset' took 13 ms
TeXmacs] std-bench, Task 'typeset' took 9 ms
TeXmacs] std-bench, Task 'typeset' took 14 ms
TeXmacs] std-bench, Task 'typeset' took 8 ms

[mgubi]

The lag I reported before:

TeXmacs] std-bench, Task 'typeset' took 96 ms

was almost a dealbreaker. I honestly am completely confused by what is going on. Maybe the snapshot version running on WSL just works better?

In my experience with TeXmacs I never found systematic situations where the program is slow. There were occasions in which particular constructs in a document would make it slow. I think tables are one of this. Sometimes in beamer presentations a lot of decorations made it a bit slower. But I repeat that this is not common and certainly systematic. If it was the case Joris would have met the same conditions (he also uses TeXmacs for his day to day work) and corrected. So what remains are corner cases which are not experienced by the main developers, e.g. I do not use amsart myself and I do not think Joris do. While the bugs or slowness have to be corrected you should expect that they are causes more likely by some bad interaction of some macros with the typesetting, instead of by some general lack of performance. If it was a general lack of performance it would have been known since long and acknowledge by a large part of the community. Joris uses TeXmacs since ~1999 and myself systematically since ~2010 and more sporadically from ~2006. I think we need to pinpoint some reproducible situations. For example the editing of the title in amsart is a good candidate, since seems very reproducible and maybe allow us to understand what are possible sources of slowdown in the typesetter. It is clearly some macro which requires a lot more computation than necessary. Now one would have to isolate which.

[guraltsev]

Dear @mgubi and everyone,

Ok, I spent several hours trying to experiment and pinpoint a culprit, and honestly, I am desperate! My setup is as follows. I wrote a small script that sends keystrokes that write Lorem Ipsum… to TeXmacs and tried to observe the behavior on different files and in different situations. The results were completely chaotic. I tried multiple combinations: switching a fixed document between amsart and article, editing inside the proof environment, editing outside the proof environment, and much more. I tried closing and reopening TeXmacs, tried copying and pasting the entire contents of a file from one to another and editing the new one, etc. etc.

None of the result seemed consistent but periodically I would get very significant slowdowns.

I managed to observe the following happening (I will try to record a screencast to show you).

** Slowdowns without std-bench symptoms **

On several occasions I managed to use my script to type “Lorem ipsum…” inside and outside of the theorem environment. I consistently, in both cases got

I had Texmacs open on the left of my screen and the console on the right.

TeXmacs] std-bench, Task ‘typeset’ took 11 ms
TeXmacs] std-bench, Task ‘typeset’ took 12 ms
TeXmacs] std-bench, Task ‘typeset’ took 12 ms
TeXmacs] std-bench, Task ‘typeset’ took 12 ms
TeXmacs] std-bench, Task ‘typeset’ took 12 ms
TeXmacs] std-bench, Task ‘typeset’ took 11 ms
TeXmacs] std-bench, Task ‘typeset’ took 12 ms
TeXmacs] std-bench, Task ‘typeset’ took 12 ms
TeXmacs] std-bench, Task ‘typeset’ took 11 ms
TeXmacs] std-bench, Task ‘typeset’ took 12 ms
TeXmacs] std-bench, Task ‘typeset’ took 12 ms
TeXmacs] std-bench, Task ‘typeset’ took 12 ms
TeXmacs] std-bench, Task ‘typeset’ took 11 ms
TeXmacs] std-bench, Task ‘typeset’ took 14 ms
TeXmacs] std-bench, Task ‘typeset’ took 12 ms
TeXmacs] std-bench, Task ‘typeset’ took 12 ms
TeXmacs] std-bench, Task ‘typeset’ took 12 ms
TeXmacs] std-bench, Task ‘typeset’ took 12 ms
TeXmacs] std-bench, Task ‘typeset’ took 12 ms
TeXmacs] std-bench, Task ‘typeset’ took 12 ms
TeXmacs] std-bench, Task ‘typeset’ took 11 ms

The typing outside the proof environment was fluid. Inside the proof environment letters were appearing in batches of 10-15 (1 or 2 words) with freezes of 0.5-1 seconds in between. There was NO way to observe this from

TeXmacs] std-bench, Task ‘typeset’ took 11 ms

except that, I think, the lines were just not appearing. It seems that the typesetting was taking 11ms but there was something happening in between that made TeXmacs briefly unresponsive.

** Slowdowns because of references and citations**

I tried typing at the end of this paragraph

The latency was

TeXmacs] std-bench, Task ‘typeset’ took 34 ms

so a bit more than expected. I thought it might be due to the references and citations so I erased all of them. The latency went down to

TeXmacs] std-bench, Task ‘typeset’ took 6 ms

I then retyped all references and citations and the latency went up but only to

TeXmacs] std-bench, Task ‘typeset’ took 10 ms

Clearing the profile

I cleared the profile completely and recreated it fresh. After that I was unable to reproduce any performance issues. However, I have done this in the past. After some time, the performance issues return. Maybe it is a cache issue? This would be plausible since windows might deal differently with FS storage than mac and linux. Maybe that is why I reproduce these issues while others can’t?

** Update **

I managed to reproduce the lag even on a fresh profile, but I cannot reproduce the lags recurrently.