So I own both the 35L1 and 24-70 2.8 II. The 24-70 was a recent pick up, so the 35L1 was one of the main lenses that I used for awhile. While there is crossover, I do feel that there's value in owning the 35L1 even with a zoom, mainly because of the ability to shoot wider than 2.8 and also because I feel that Canon's primes tend to provide a different look and feel that can't be replicated. I would mainly go as wide as 1.8 or 2.0, and having that ability is nice in certain situations. If I could only have one though, I'd probably go with the 24-70. If you can get the version 2 of that lens, go for it. After spending a long weekend with it, it's awesome.
As far as what to look for in buying a used lens, I think someone else covered that and I don't have much to add. The focus should be quick and snappy; if you find that it hunts too much or if there's any type of hesitation, then don't pick it up. My 35L1 didn't require any type of adjustments and was good right out of the box. I purchased mine used from B&H. I didn't pay attention to the serial number either; whatever model I have, it performs great and gave me a real appreciation for primes lenses.
My prime lens line up is the 35L1, 50L, 100L and 135L and I'd like to add the 85L. Even with zooms that cover those focal lengths, I wouldn't get rid of any of those lenses. I could probably do everything I need to do with the 24-70 and 70-200, but the primes are nice to have and like I had mentioned, the "look" you can achieve from those lenses is