My mechanic replaced the port starter on my 496 Mag HO, and did not take the engine out. Port is even harder to get to.
+1
My mechanic had to replace the Starboard side starter on twin 7.4 MPIs.
It took him a LONG time and a LOT of patience because 1) the bolts holding the starter to the bottom of the engine were corroded, and 2) space was so limited that he was essentially using one hand and arm. Formula's starter placement is not ideal.
One word of advice - do not push too hard to have this done for Father's Day, or you may be buying more problems.
If this is a 27PC then chances are that your bolts are also corroded, especially if this is an original starter (unlikely, but possible).
If corroded, these bolts should be treated with the patience of a bonsai tree groomer, because if you strip out or shear one of them by muscling it out, your day's gonna get a lot worse.
Yes, it can likely be done without removing the engine, but you need the right mechanic with the time and patience to work it with penetrating oil and heat until he can get it out safely.
My mechanic was down there at least 5 or 6 times "grooming the bolts" - oil and heat, oil and heat, etc., before he got that pig out.
I hope you have good luck - maybe your starter was replaced recently and the going will be easier for you.
Take care.