There are at least 4 operators providing this tour but they really only operate the buses to Kate Kearney's and from Ross Castle. The jarveys and the boat operators do not really differ and so any such tour operator will work fine. If you book through your accommodations they will have an operator of choice but if you wish to book ahead of time I would suggest http://www.derostours.com/
I am not sure what your questions about the boats refers to? Yes, you can just hire boats on the lakes for short tours. From Ross Castle you can get a boat to Innisfallen Island or for a tour of the lakes. From the boat house by Muckross House you can hire boats for short trips around Muckross Lake. From Lord Brandon's Cottage you can hire boats for short trips along the Upper Lake. You can also hire 'The Gap' boats to take you from Ross Castle to Lord Brandon's Cottage independent of the full Gap of Dunloe Tour but I would book this portion through one of the tour operators. In fact this is what I did in July. We got the boat from Ross to Lord Brandon's and paid E15 each for the journey. We then walked from the Cottage to the top of the Gap - being on the unfit side of things this took us a while and we were pooped by the time we reached the top of the gap. We happened upon a jarvey and he took us through the gap for E10 as far as Kate Kearney's where we had a ride waiting.
If you have 2 cars at your disposal, I would coordinate taking the boat, walking the gap and not paying for the full tour.