My impression is that we should avoid large dependencies if they are not really needed. Qt comes with its own testing framework, and we depend on it anyway. I do not even understand why we need a large testing framework, given that 80% or more of the code is already written and quite old so that most of the bugs have become features. Also ~1/3 of our code is in scheme, so any C++ testing framework will not cover that. I’m not against tests but we need to understand first why and how we use them, if we want. Currently there is no strategy and it is unlikely there are major unseen bugs in the basic libraries. How we introduce meaningful tests in our legacy codebase? Most of the new code is at the level of the UI. How we test it? How we test the code written in TeXmacs stylesheet language for bugs? How we test the font-handling code?
I don’t see why we would need to test all of the code for tests of part of the code to be useful.
Even if the code is old, some parts may not be so well tested, see for example the Russian hyphenation bug. Besides, bugs may be introduced if ever significant changes are made, for example to the file format encoding.
I wouldn’t advocate spending significant time writing tests, but they could be written when fixing bugs and then integrated into the test suite to ensure we don’t get regressions.
Writing progressively regression tests for features which are either added from scratch or patched to remove bugs makes indeed sense to me. It remains to understand what it the best strategy for a framework for such tests. I think two things are important (to me):
Be wary of adding many or large external dependencies
Be wary that TeXmacs is written in an old-fashioned C++
Some solution which works well wrt. testing also the Scheme code.
It is not clear what a framework like Google’s would give us that we really need (of course it could give us many fancy features, which I think we hardly have the man-power to manage).
Let me also add that what I like about TeXmacs codebase is its independence. This is also a guarantee for the future: libraries “fanciness” rise and fall and we do not want to depend too much on them since we cannot commit man-power to rewrite large portions of the code.
Agreed, Qt Test looks good in that respect. Perhaps a similar approach as with the GUI can be taken, in that TeXmacs specific test macros are defined that are then translated into framework specific code. That way we are even less dependent on the framework.
I don’t think this should be a problem. We can still maintain our style. We only need the framework to verify equality of a computed result to the expected result. How the result is computed is completely up to us.
Qt6 is now also moving to C++17, but such compilers still happily compile TeXmacs’ code.