It depends on your contract and on the nature of your relationship with your current employer (and the nature of the relationship with the client). Commonly there will be a clause about not working for competitors or clients for 'x' amount of time (e.g. 6 months) after leaving the firm. Similarly, it's common that the client will have a clause in the contract they hold with your firm, specifying that they won't poach each other's staff during the course of an active engagement without prior agreement.
In reality, it is near-impossible to enforce these clauses and all parties will be pragmatic - if you aren't very senior it is unlikely to be an issue.
Needless to say, it wouldn't be a good idea to break the news or make the move if you're in the middle of a project with that client as that could cause problems in how you manage the transition and the changing nature of your relationship with the project team.