Full disclosure: we didn't actually get to the summit. We used Google Maps to drive from Harrison's Cave to where Google defined "Mount Hillaby", which led us through a grass field. Because it was so muddy and we were in a rental Suzuki Swift, we decided to get out and walk from there.
It's a very beautiful area and a nice walk, but once we got up to the point Maps said the summit was we had difficulty finding the summit. I asked some locals driving by how we can get to the top, and they said that the "shortcut" road to it was closed today. They very kindly told us that we would therefore need to walk down to the bay and the bus stop to the north, and then walk back up in order to access the summit. About an hour walk according to them.
We had a busy day in front of us so we opted out then, but regardless it's a beautiful, bucolic area to explore.