It's good to know that I'm not the only one in this position.
I'm thinking that if you are thinking of doing a Masters anyway, this is a good opportunity to do a course relevant to the area you want to switch into. Thereafter you can apply to graduate schemes. I do recall from the assessment centres I went to (both for consulting and industry jobs) that there were a fair number of career-changers there, so having a few years experience isn't a complete barrier to getting into a graduate scheme.
I have thought about switching in as an experienced hire, but for a lot of these roles in accounting and banking I wouldn't be comfortable without the experience and qualifications that you pick on the graduate scheme. I also know people who've made that move and feel isolated without the network you get as part of a big intake.