The charges relate to two alleged incidents involving the same woman in 2016, and another alleged assault on a different woman in 2007.
The CPS said the decision had been made “after reviewing a file of evidence from the Metropolitan Police“.
Mr Elphicke is due to appear at Westminster Magistrates’ Court on 6 September.
The Dover and Deal MP was initially suspended by the Conservatives in November 2017 after “serious allegations” were made. He was readmitted last December on the day of a no-confidence vote in Theresa May, despite having been interviewed by police under caution as part of an ongoing investigation.
Mr Elphicke has denied any “criminal wrongdoing” and said he is “completely confident” of clearing his name. His solicitor said he would fight the charges “vigorously”.
If Mr Elphicke is again suspended by the party and the Conservatives lose the Brecon and Radnorshire by-election next month, it would leave the next government with a majority of just two.
That would fall to one if Mr Elphicke was forced to stand down as an MP and Labour captured his seat, which is a marginal constiuency.
A CPS spokesperson said: “The Crown Prosecution Service has today (Monday 22 July) charged Charles Elphicke, MP for Dover, with three charges of sexual assault against two women.
“The CPS made the decision to charge Mr Elphicke after reviewing a file of evidence from the Metropolitan Police.”
In response to the charges, the MP’s solicitor, Ellen Peart, said: “Charlie Elphicke has said from the outset that he denies any wrongdoing. He will defend himself vigorously and is confident that he will clear his name.
“Meanwhile Charlie would like to thank everyone for their continued support of him and his family.
“There will be no further comment at this stage.”
After being readmitted to the party late last year, Mr Elphicke wrote on Twitter: “Important for my constituents to know that, 13 months after having it withdrawn, I have been given back the Conservative whip.
“I remain as confident as I always have been of clearing my name and will continue to work as hard for Dover and Deal as I always have done.”
The decision to restore the Conservative whip to him provoked an angry backlash.
Labour MP Jess Phillips called the move “totally despicable” and Dawn Butler, the shadow equalities minister, said it was “a betrayal of women”.
The Conservative Party has been contacted for comment.