Yes. Not for beginners, definitely worth getting in shape for the hiking involved, you're looking at 3-4 runs per day, probably 500-700 vertical of hiking per run, at 12,000 feet. I spent a few days at 7500 feet beforehand, but still felt pretty tired during the hikes.
Terrain is variable, snow wasn't good when I went, but it's an experience regardless.
The town is very interesting, if you have some additional time, have a look at Ouray as well. Durango is much larger and there's more in the way of services, lodging, etc.