Yep, you pretty much answered it already. Lifting one muscle group (especially a major one such as the pec's) should really be left with 2 days of rest between lifting again. Your body just isnt capable of that much repair in 24 hours as it take a considerable amount of protein ingestion, then digestion, then sythesis of new proteins, etc to repair the muscle tissue. I would argue that for those really pounding the weights, taking up to 3 days off between focusing that muscle group again may not even be enough time for full recovery, especially for those beginning or really pressing hard. This case will be situational depending on how quickly you recovery, your amount of protein intake (especially glutamine) and BCAA's taken as these will all help speed recovery. The bottom line however, is that your body "grows" while at rest as this is when the repairing is done. If you do not allow enough time to rest, you will actually slow your growth gains.
If you have one foot stuck in yesterday and one foot stuck in tomorrow, you are peeing on today.