It seems that Matlab, for some reason try to open a window to do the plot instead of sending it (in some form) to TeXmacs. I do not know about this plugin, but looking at it you should be able to understand why it is so. Usually the plugin is written in matlab itself.
Insert a MATLAB script on TexMacs
You can try to run from a shell:
matlab -nodesktop -nosplash
and then check what happens if you run your code there. -nodesktop should prevent matlab to open the GUI, but I’m not sure what is supposed to do when you try to make a plot.
No, dear @Wiss, I don’t experience these error messages, when running a Matlab session from with in TeXmacs. I have Matlab R2020a Update 6 and the start-up script tm_matlab that ships with TeXmacs 2.1.1 works fine.
Maybe you want to make sure that no previous instance of the Matlab.app is running, i.e. close the GUI app before you launch a Matlab session from within TeXmacs. Give this a try.
Your problem could also be a font issue. If you have TeX installed on your system, you might try to configure Matlab to use TeX fonts. This can be done in startup.m. (Matlab specific)
Cheers
Tilda