The man, only known as David, was on a call with his sister at Nantes station when an employee from French railway company SNCF approached him.
David told French broadcaster BFM TV that the worker told him he would be fined €150 ($AU 330) if he did not turn off the loudspeaker.
"I think the person was offended. She took out her notebook and fined me," he said.
The man claimed that the fine was later increased to €200 because he did not turn it off on the spot.
The 54-year-old man told BFM TV that he is fighting the penalty.
SNCF said that the man was fined by security staff in a quiet area of the station.
The French state-owned train company disputed some of the details brought forward by David, adding that the interaction between the staff member and the passenger escalated before the fine was issued,
SNCF told BBC News that security staff operate on trains and at stations and are authorised to issue fines.
The company also confirmed that the fine did increase from €150 because "the customer refused to pay."
According to the French Transport Code, "it is forbidden for any person to use, without authorisation, sound devices or instruments, or to disturb the peace of others by noise or disturbances" in "spaces and vehicles assigned to the public transport of passengers or goods".