Things I would try tinkering with:
Resolution: That is on the high-side, start by dropping down. Less rendered pixels to push will help.
Bloom: Try turning it off. For some reason Bloom can be a problem in general on Mac game clients.
Anisotropic Filtering: Drop it down
Reflections: depending on how they render them, you could try dropping that down. I'm guessing they don't use a simple reflection map, so that might help.
Max Lights per Object: Try dropping it to 4.
I would say the biggest things to change would be res, filtering and bloom and go from there.
Another option is the Boot Camp option. I normally wind up using Boot Camp for games, even when there is a Mac version via Steam. A lot of ports from DirectX are not well optimized for OpenGL. STO is definitely one I Boot Camp on my desktop, though I do run it MacOS on my laptop. Do not run STO in Parallels, I've gotten some games to perform adequately in it, but STO is not one of them.