He said that although Modi repeatedly attacked him, his father Rajiv Gandhi, grandmother Indira Gandhi and great grandfather Jawaharlal Nehru, he gave him a hug.
The Congress chief's remarks came a week after Modi said in a rally that former PM Rajiv Gandhi "was termed 'Mr Clean' by his courtiers, but his life ended as 'Bhrashtachari No 1' (corrupt number 1)".
Modi had also alleged that Rajiv Gandhi and his family had used the INS Viraat as a "personal taxi" for 10 days while they were vacationing at an island in Lakshadweep in the late 80s.
"BJP, RSS and Prime Minister Narendra Modi has hatred for my family and our job is to remove it. Narendra Modi attacks me. He speaks with hatred about my father, my grandmother and my great grandfather. He speaks in anger, but I go and give him a 'jhappi' (hug)," Rahul Gandhi said.
"You are the prime minister... you must remove hatred and work with love. This will benefit you. I don't hate Narendra Modi, he can give me as much hatred as he can. He can speak about my family whatever he wants," he said.
"Hatred cannot be conquered with hatred in return. Only love can defeat Narendra Modi and not the hatred," he added.
The Congress chief was addressing an election rally here for party candidate and folk singer Prahlad Tipaniya, who is contesting the Lok Sabha election from Dewas seat.
Speaking about his party's proposed NYAY minimum income guarantee scheme, Rahul Gandhi said, "Amazingly, the idea of this scheme (NYAY) came from Narendra Modi. He did not implement it, spoken lie about giving Rs 15 lakh."
"But we took this idea... going to give you Rs 3.60 lakh in five years (under NYAY)," he said.
He added that the scheme would provide purchasing power to people, which would revive the country's economy.
He said demonetisation and poor implementation of the Goods and Services Tax (GST), which he termed as the 'Gabbar Singh Tax', has adversely affected the purchasing power of people.
"When I raised questions about the Rafale scam in Parliament, Modi talked about Jawaharlal Nehru, Indira Gandhi and Rajiv Gandhi but did not reply to my questions," Rahul Gandhi said.
"Narendra Modi ji, I had asked about Rafale and not about my family," he said while accusing the prime minister of carrying out parallel negotiations with France in Rafale fighter jet deal.
He also said that the Congress would provide 22 lakh government jobs in addition to 10 lakh jobs in panchayats to the people after coming to power.
Rahul Gandhi said the agriculture loans of farmers in Madhya Pradesh, including the kin of former chief minister Shivraj Singh Chouhan, have been waived by the Congress government in the state.
The loan waiver was promised by Rahul Gandhi before the Madhya Pradesh Assembly elections last year and it is widely believed to have played a key role in the party's victory in the state after a gap of 15 years.
Madhya Pradesh Chief Minister Kamal Nath and his Chhattisgarh counterpart Bhupesh Baghel also addressed the gathering.