Living in L.A. County and having ridden the coast a lot in this area, I would partly echo what others have already said. The ugly part is the Long Beach and Wilmington area of PCH, about 10 miles' worth, where you're passing refineries and railroad yards, and the roads are in terrible shape, ruined by heavy trucks and never repaired; yet they're not particularly dangerous. There are places where you do have to take the whole lane, but with a mirror, you'll be fine. If you're heading in the direction of San Diego, you'll have a nice tail wind through this area in the afternoon. I recently tried going through there on the next major street north of PCH, and although that one was newly re-paved, it seemed more dangerous for a few different reasons. I have never ridden PCH west of Vermont Ave to know how that might be, but if you go south a mile or two and take Palos Verdes Drive North through there, it's beautiful.
I've ridden from Buellton above Santa Barbara to L.A. a couple of times and enjoyed it immensely. I've ridden from a few miles east of the Long Beach area mentioned above to San Diego probably 15 times, and I enjoy most of that a ton too. There are plenty of places where you're not in town at all, but when you are, it's mostly an easy-going beach atmosphere with everyone enjoying being there whether they're in their convertible, on a bike, on a surfboard, or just walking out there to look and be looked at.