Goforit - I agree; what is more, the practical experience should actually make you stand out in the crowd should you return to consulting - it's a highly respected and laudable route (the CS Fast Track) frankly comparable to the development you would obtain in consulting anyway.
I have myself done backward irrelevant deviation, but only with accomodating ladies on weekends. Kidding :-)
Better to make the move sooner rather than later, and McKinsey et al DO hire people from the civil service at senior levels, as do major industry firms, as specialist consultants and advisors; there are also other interesting routes to follow on - think tanks, corporate advisory roles (which would give you direct access to plc board level), pressure groups, lobbying etc. Fascinating stuff!!!