He also alleged that the prime minister "broke the law" and "bypassed" the Chief Justice of India and the Leader of Opposition, who were part of the panel that selected the Central Bureau of Investigation (CBI) chief.
There was no immediate reaction from the Prime Minister's Office (PMO) on the allegations.
Finance Minister Arun Jaitley has, however, dismissed as "rubbish" the allegations that Verma was removed because he wanted to look into the Rafale fighter jet deal.
He said the government's decision to remove Verma and CBI Special Director Rakesh Asthana was based on the Central Vigilance Commission's (CVC) recommendations.
Gandhi also alleged that Verma was forcibly sent on leave as he was collecting documents relating to the Rafale "scam".
Verma and Asthana have been "divested of all powers" amid an ongoing spat between them.
"PM removed the CBI Director to stop him from investigating Rafale. Mr 56 broke the law when he bypassed CJI and LOP.
"Mr Modi, Rafale is a deadly aircraft with a superb radar. You can run, but you can't hide from It," Gandhi said in a tweet.
Earlier, he tweeted, "CBI Chief Alok Verma was collecting documents relating to the Rafale scam. He was forcibly sent on leave."
"The Prime Minister's message is very clear that whoever comes near Rafale (issue) will be removed, wiped out," the Congress chief charged.
The country and the Constitution were in danger, he claimed.
Addressing a poll rally at Jhalawar in Rajasthan, Gandhi targeted the prime minister on the CBI row, alleging that Verma was removed because the agency was raising questions over the Rafale fighter jet deal.
"The CBI director was removed last night because the agency was raising questions over Rafale," Gandhi alleged.
The Congress has been alleging massive irregularities in the Rafale deal.
The government has vehemently rejected all allegations.