The ~20 miles from the Wilderness Area parking lot to the trail head is just as others describe, a high-clearance vehicle would be best. Lots of ATVs and target shooters along the way. Take the hard right turn onto Pigeon Spring Rd and go a couple hundred yards past Lone Pine trail head to the end of the road at Brown's Peak trail head (#133). The first section to Brown's Saddle is an easy hike through Manzanita, oaks, and rock formations. The last few hundred feet is a scramble up a scree chute over loose rocks and boulders that can be a little hair raising if your not a rock climber, but well worth it.
After 18 miles of off roading, you can follow the Brown's Peak Summit Trail and reach the base of 4 Peaks, and if you dare, scramble almost vertically up a few hundred feet to the very top of the mountain!
By far the best hike I have done to date! The scree chute is both challenging and delightful for beginning climbers and hikers. It takes about 1.5 hours to get to the top, but I usually spend about an hour on the peak taking in the view. Keep an eye out for Humphrey's Peak due north. Well worth the vertigo! DO NOT mark the trail with flagging tape. Cairns work just as well, and produce no litter.
Great spot.