TeXmacs Chez port (BETA)

#1

Hi everyone,

I got some free time and started working on something that has been on my back burner forever. Here’s my attempt to replace the Guile-1.8 scheme engine behind the official TeXmacs with Chez Scheme – a compiler that generates highly performant machine code. This effort was started by @mgubi some years back.

Source is below:

Some short notes on implementation off the top of my head:

  1. Instead of an interpreter, we compile every scheme files first. So the build workflow becomes: build C++ sources first, then bootstraps a headless bare minimal TeXmacs and starts compiling TeXmacs scheme sources.
  2. The Guile-1.8 code is 20+ years old. And it’s not meant to be compiled by Chez R6RS standard. The high level idea for compilation is to translate (compile) the Guile code into R6RS compliant scheme – mostly just partitioning the run-time code from compile-time code and making some macro helper functions available in the compilation environment. There are some other limitations from chez scheme compilers. All addressed by the compile script (I hope); at least enough for it to compile most sources.
  3. The compile time for scheme is around 8 mins on my machine. And the incremental build system is currently broken – since I still haven’t figured out a way to pinpoint module dependencies accurately due to TeXmacs’s dynamic executation nature (so no parallel compilation either).
  4. Performance seems fine. Basic editing works. Though there are bound to be a lot of bugs to fix. At least all plugins are not working. Regarding performance, there’s definitely a lot more we can do with Chez: increasing the optimization level, in-house profiler of chez and packaging all compiled binaries into a texmacs.boot. If done right, we might even skip all the lazy loading.
  5. While I try to make very little changes to the scheme source code, there are still some necessary changes made to make it work. The hope is that I preserve the semantics and logics when doing this. Definitely need some more experienced eyes to check this.
  6. This work is aided heavily by LLM. So there are still some under-reviewed code in there. Left for refactoring in the future.

Hope to hear some feedbacks and gauge interests! Also use this with caution and back up you .tm files!

1 Like
#2

Dear Joy,
this is interesting! I’m still not convinced that Chez will bring a substantial improvement of day-to-day operation of TeXmacs but it is an experiment worth doing. It might be useful for you to check the wip_other_schemes branch since it contains at least other two scheme implementations S7 and femtolisp, both interpreted and this can guide you and the LLM to integrating Chez in TeXmacs. Ideally we want to keep at least 2-3 scheme implementation working so that we can compare the benchmarks more easily. Note that the current texmacs codebase relies on old style macros and Guile specific non-standard choices (e.g. reevaluating macros at every expansion) which do not work well with compiled schemes. And the macro system is not hygienic (and it is not clear we want the hygienic macros in the first place, since they bring a lot of slowness as far as I can see).

1 Like
#3

Hi Max,

Thanks for your interests!

Ignore the benchmark message below! There was some bug with the chez port. The manual wasn’t fully loaded.

Here is one preliminary benchmark (just eye ball timing it with my watch… can do a more comprehensive one in the future; bugs needed iron out before the command-line option works) that I did: Help->Full manuals->User manual from GUI.

s7 42seconds 1.36GB memory;
chez 19seconds 687MB memory.

Actually my main complaints back then with the s7 port was the excessive memory footprint (at least in this regard the chez port is good). Though I did observe that chez is incredibly slow when editing the manual though – probably some hidden bugs again. Something that I need to chase afterwards.

I believe the overall design is quite similar to what you were thinking back in (chez module + define-top-level-value for globally defined function: tm-define): https://groups.google.com/g/chez-scheme/c/zLDWgaZvdXI. If only me and LLM could have found this discussion earlier… We ended up doing more or less what you proposed 5 years ago.

Yes, my current repo is quite messy. As for integration, I’m not entirely sure what you mean. Because I did have to rewrite quite a few places in the kernel modules: specifically the tm-define overloading functionality and quite a few macros (like lazy-define) here and there to stop Chez scheme from complaining. But I believe at least the semantic is preserved. So I don’t think the same code could be used. If by integrating, you mean to merge the repos and branches, I believe I will need to separate quite a few scheme files to be Chez port specific.

Currently I’m writing a small static analysis script to track the tm-define, tm-define-macro, and some macro dependencies across different modules. These are hidden undeclared dependencies TeXmacs rely on implicitly. This should enable an ad-hoc incremental build to speed up the development process – currently, if I edit some kernel files, I need to recompile from scratch, taking at least 6-8 mins…

As for macros, what chez does is expand them right away during compilation. Since there’s a separation between compile-time code (macros) and run-time code (functions, lambdas), these macro could introduce bugs for the chez-port (normal texmacs expects some side effects from evaluating the macro in run-time for example.). And after compilation, there’s no more macros, they are all expanded.

As for hygienic slowness, I’m not familiar with that. Maybe it is the case of interpreter-based schemes. For chez scheme, the macros don’t run in run-time at all. They all become machine code at that point. Hygienic just means that these macros will point you at the right helper function inside the module, i.e., generating the correct machine code for run-time use here, I believe.

#4

every scheme need a compatibility layer, in the wip_other_schemes branch this is made explicit and guile, s7 and femtolisp both share the same scheme files on top of a thin compatibility layer. It would be optimal if you can also have chez fit the frameworks so that we can keep a single branch for all the scheme implementations. But this depends how much you have to change of the existing scheme codebase. Of course things like tm-define and other low-level texmacs infrastructure has to be implemented in a way which depends on the specific Scheme implementation used.

#5

In theory, but TeXmacs C++ code eval at run-time a lot of scheme code, for the menus, for example so you cannot avoid the slowness because no compilation will speed up this (actually compilation make it slower).

compilation will pay out only for things like conversion etc…

#6

That makes sense! Maybe that’s why editing the manual feels so slow? I will take a look to see if there’s way around it.

#7

OK. I can take a look. Though the current compilation probably can’t handle much modifications like adding new core macros – it’s quite ad-hoc, like identifying tm-define and tm-define-macro as special symbols and evaluating them into the compiler environment as you see them. So can’t guarantee that it can be compatible with any downstream modifications.

1 Like
#8

I ran some benchmark numbers. Just for reference. This is chez vs. s7 port built from wip_other_schemes on my machine. And as far as I can tell, the chez port does do better on memory usage vs. s7-port.

I did replace (call (eval “funname”)) with (call (string_to_symbol “funname”)) in C++. Though not sure how much it really helps. Recently, I was struggling with utf8 encoding and chez unicode encoding. Fixed some issues. But probably more bugs behind this.

1 Like
#9

what about femtolisp? Chez could be a great choice if we decide to move to a full scheme implementation of most of the code and could be something great like this https://github.com/paveluv/Schevana but this would be a different app and could not share much UI code or Scheme code with the current TeXmacs. Moreover I’m not sure one can adapt Chez to work fast in the Browser, so this would not have a webasm app.

My original ambition was to replace all C++ apart maybe the typesetter (which would be a too extensive rewrite) with compiled scheme, but before committing too much in this direction one would like to assess really what is gained and lost.

#10

Updated in the same document.

I think full scheme implement is a bit too much commitment for now. I did rewrite some small parts in scheme due to compatibility issues of encoding. Maybe for some foreign function calls we can move to scheme implementation for improved performance – skipping the value copying, and encoding/decoding of string formats (C++ utf8 vs. chez unicode).

#11

Do I read correctly that benchmark right now put Chez on par with S7 and Femtolisp? What is your interpretation?

#12

Yes, for the two benchmarks I run they are more or less on par: chez, s7 and femtolisp.

From what I’m seeing: the typing benchmark is C++ code dominated (manual benchmark is also C++ heavy). I think for scheme part, chez should still be faster most of the time if handled correctly. And chez has to do some extra encoding/decoding to utf8<->unicode from time to time when talking to C++ side (though apparently the overhead is quite small).

Update: there’s a new benchmark results for conversion.scm, which is more scheme heavy. It shows some decent speed improvement.