If you are not going to recover them, I don't see why removing them will make any difference in treatment. Get a good vinyl cleaner and a good vinyl treatment/protectant. I like the 303 products.
Unfortunately, the sun and water beat up the vinyl and if the outer coating is worn off, any treatments will not last long and may not restore the look to new. That said, I would have expected more than 5 years of service. The trick to longevity is frequent cleaning ( at least rinsing) and application of protectants.
On my bolsters, they also hang on clips but are not bolted or screwed form the back through the gunnel. Instead, there are screws driven from the lower edge front using the vent grommets on the lower edge of the bolster. It can be tricky to find them and unscrew them. There aren't usually a lot either. Once the screws are out, you pull the bolsters straight up and off of the brackets. Sometimes you may need to carefully whack the bottom edge a bit.
