This is part of a hack I have done. See this thread (you replied in it as well): http://www.boonex.com/forums/?action=goto&my_threads=1#topic/Files-Edit-Form-Stripping-HTML.htm
I noticed that on any modal window where I have added a tinymce editor to; even third party modules using a modal window for input, that the modal window will close when the Styles, Format, Font Family, Font size, Font colour, and Font background colour are used. All other buttons on the toolbar does not cause the modal window to close; or at least all I have tested. During a test where you told me to watch the console, I did noticed that just before the modal window closed, the font-family selection was applied to the text in the editor window.
I don't know enough about javascript and Dolphin modal windows to be able to gain traction on this one. For my files description I simply left out the above mention TinyMCE selectors but that does not address why it is happening.
I don't have this set up on the test server at the moment. When I have time, I could add it and then create an account for you to test with.