File locks in TeXmacs

#1

I compiled Mogan under Android. There is a tiny but unpleasant issue of file access. After some discovery, I find that this is due to file locks introduced in an early commit of TeXmacs.

If I understand correctly, according to this answer, flock() does not work on every filesystem of a Unix-like OS. Currently, if the target OS is Unix-like, then a failure of obtaining a lock would lead to a failure of opening the file, which would lead to unnecessary restrictions. In particular, on Android, the share data between termux and internal storage cannot be applied with locks.

I think it more reasonable to use locks as an extra protection, instead of a necessary condition of openning files.

#2

In my opinion, we should not use low-level APIs like fopen and flock and more. We should replace those implementation using the Qt IO APIs.

cc @yufens

Using Qt IO APIs, https://github.com/XmacsLabs/mogan/pull/472 could be simplified a lot.

Here is a related pull request: https://github.com/XmacsLabs/mogan/pull/179

#3

This part (i.e. /src/System) is low-level and re-organizing these codes might be time consuming and error-prone.

I am also not sure whether it is a good idea to depend on Qt on that level ā€” currently, one could still compile and run TeXmacs without Qt, but introducing these IO APIā€™s would make Qt completely integrated.

#4

You are right. But the native impl in src/System/file.cpp is in low quality for cross platform purpose.

#ifdef will help us. In that way, Qt is not required. And for Mogan Editor, Iā€™ve made the decision that just integrate Qt completely!

Here is what I have done for read_directory:

Mogan is built on Qt but GNU TeXmacs could and should work without Qt. But the reality is that few contributors and developers care about GNU TeXmacs without Qt. (Someone cares about it, but they didnā€™t spare time on it)

For Mogan, Iā€™ve removed the other UI implementation because they are not maintained actually. See https://github.com/XmacsLabs/mogan/commit/f6a9b8cf6ccb9b5f8d2b1b7a5496313b7fd466d7

#5

It seems that the CMakeList.txt does not reflect this completely, although I am not familiar with that.

One of difficulties is that, these codes are basically C, not really C++, and it seems that main developers of TeXmacs would like to keep the code pre-standard. Consequently, some important techniques like RAII are possibly unavailable (I am not talking about later fancy techniques, but very basic C++ things.) With RAII, you could define a class and let the compiler deal with fclose() etc., and in fact, I suspect that many of these are available in fstream, even if you want flock().

#6

Weā€™ve migrated from CMake to xmake now.

Thatā€™s true. And for Mogan Editor, we decided to use C++ 17. Using C++ new features is not encouraged until it is necessary. Because we do not want to move too far away from GNU TeXmacs, we still want to contribute back to GNU TeXmacs.

A good idea. Well, if using Qt can free us from learning what is fclose and flock, I think for Mogan Editor, it would be better to use Qt to load a file into a string. I donā€™t think it is a good idea to spend time on low level details while we are working on an app.

Without Qt, there are other mature cross platform IO libraries available like Boost.IO, we should not create a square wheel from scratch.

#7

My impression is that the right approach is to factor out low level details in an interface and the implement it with Plugins. So all the Qt code should be restricted in the Qt Plugin implementation and TeXmacs should rely on abstract interface. The needs of the program are high-level enough that would be easy to implement. The only important decision is how to design the interface API. I remember when Joris abstracted the X11 support into the graphics layer to allow other libraries. This should be done for filesystem and resources like images. At the moment image handling code is scattered in many places and this is not good. The wasm platform would require yet another implementation of all this file handling since it would be done asynchronously. And there is no need to rely on bloated Qt for this. In my mind we need Qt just because nobody wants really to develop for Windows.

1 Like
#8

The point I wanted to make is that, fclose() is very error prone in the sense that, if you forget to do that after closing a file, it would lead to a leak of resource. It is more true for pointers, as memory leakage is a source of bugs (and hard to debug).

#9

Yes, thatā€™s the main reason. I do not use Windows. But TeXmacs users who use Windows are the majority.

@jingkaimori is developing on Windows

#10

Slightly off topic. About the wasm port, given that WebAssembly cannot directly access local file system, I can think of two possible approaches:

  1. Current QT API: call javascript code from c++ code, use html5ā€™s DOM to select and upload local files, then copy the byte array to cppā€™s memory heap. Current Mogan online version uses this approach to open and save simple local tm files: Mogan online. Iā€™m not sure, but it feels inconsistent with TeXmacsā€™s tmfs/url systems.
  2. Use emscriptenā€™s virtual file system: add another button to upload local files to emscriptenā€™s virtual file system, then modify TeXmacsā€™s IO code to make it works on that virtual file system. I personally like this approach more, but then opening the local file becomes a two-step process for users, she needs to upload to the virtual file system first, then load to TeXmacs.
1 Like
Compiling Mogan under Android
#11

I think TeXmacs in the browser cannot be the same application as the TeXmacs app. There are many things which are not possible in the browser, apart from filesystem io there is also subprocess running and therefore no plugin can work. Also some parts of TeXmacs image handling require to run external conversion programs, and the font code looks for system fonts. My idea for TeXmacs on the web is to design a smaller version of TeXmacs in which most of the UI is handled by the browser via javascript and where we can use the browser as a cross platform systems, so we do not really need Qt. While is quite rapid to prototype a running webasm TeXmacs port I guess that many things do not make sense, for example we do not really want to have multiple windows, and different tabs in the browser are usually isolated so they will not be able to act as separate windows in TeXmacs (afaik). Another thing which is not currently working properly are dialogs since the Qt/Wasm code is not very good in handling multiple windows (or anyway we do not account for this). I think that the long term solution is to have an UI implemented in javascript, running in the main thread, with TeXmacs running in a separate thread and communicating with the UI via shared memory or some other inter-thread protocol. Somehow things are already like this since all the event form Qt at queued to be processed by TeXmacs at the right moment, so the architecture is quite decoupled. As inspiration one could look at the MuPDF wasm viewer: https://mupdf.com/wasm/demo/view.html?file=../../docs/mupdf_explored.pdf

which has some lightweight javascript code to implement the UI. For more fancy effect we could get inspired by the Phoenix editor
https://phcode.dev
(or by Overleaf)

2 Likes
#12

Small remark:

and different tabs in the browser are usually isolated so they will not be able to act as separate windows in TeXmacs (afaik)

is not quite true. There is functionality in the browser for inter-tab communication (overleaf allows you to have the PDF in a separate tab from the code). This is not very important, per se. However, the possibility of viewing the same (large) file in multiple windows/tabs and doing a ctrl-click on a link to ā€œopen in new tabā€ would be such a game-changer (at least for me).

#13

Python can run on browser, Scheme can run on browser. It is possbile to make plugin work!

#14

Let me explain more about the decision for Mogan Editor.

If we do not use Qt, we can use Boost Filesystem. Both of them support multiple platform like macOS/Windows/Linux.

Since Mogan Editor is using Qt, I decided that using Qt for Filesystem is a good idea. In that way, we do not need to add a dependency like Boost Filesystem, or create a buggy filesystem wheel.

And because of @yufens 's work on WASM, I realize that another reason to use Qt is that some APIs of Qt works fine on WASM.

1 Like
#15

I agree with the comment by @mgubi that it might be better to have an abstraction (e.g. a virtual class) to encapsulate these, and you could then implement it in Qt, so that it could also be easier to have another implementation in TeXmacs, if whose developers want to have a variant without Qt, or donā€™t want to introduce Qt in that level.

#16

Nothing different (abstraction using class is nice, but it is another topic). The current way of using Qt is OK. Just use #ifdef and put the code in a proper place, thatā€™s all.

And @pikachuhy suggest that besides Boost Filesystem and Qt, we may also use C++17 filesystem:
https://en.cppreference.com/w/cpp/filesystem

#17

I found a better impl for IO.

tbox is created by the author of xmake. I guess it is the best solution for now.

#18

We launched a project called lolly and solved the problem of file:

Now most fs related routines is re-implemented on tbox.

1 Like
#19

Is it right now possible to generate cmake configurations from xmake configurations? I would test this under Termux, but right now I have no time compiling Mogan, and it would be better if a cmake configuration is available (and thus could probably be easily packaged for Termux).

#20

xmake is available on termux: https://github.com/termux/termux-packages/pull/4982