Riley gave a stand-out performance of Not the Same at the competition in Italy but finished in 15th place, after receiving just two points from the audience vote.
Ukraine's Kalush Orchestra won with their entry Stefania, riding a wave of public support to claim an emotional victory.
Sung in Ukrainian, the winning song fused rap with traditional folk music and was a tribute to band frontman Oleh Psiuk's mother.
After their performance Psiuk made an impassioned plea to free people still trapped in a besieged steel mill in the Ukrainian port city of Mariupol.
Ukraine were in fourth place based on jury voting but claimed victory with a commanding lead in viewer voting. Britain's Sam Ryder finished second.
Australia has never won the Eurovision competition, which has steadily expanded its borders beyond continental Europe, but has been a fan favourite in recent years scoring a top-10 finish four times.
Riley, 23, performed wearing a crystal mask as part of a flamboyant white outfit.
He has said Not The Same was inspired by experiences including being diagnosed with Asperger's syndrome, growing up in public housing and battling with his sexuality in a religious household.
"It is the story I never thought I'd ever feel I'd be able to tell," the singer, from the Gold Coast, told the competition's organisers.
The bookmakers had made Kalush Orchestra the clear favourite for the annual contest, which normally draws a television audience of close to 200 million, based in part on the plight of Ukraine following Russia's invasion in February.
Psiuk had said this week that he was hoping for a win to "lift the spirits" of his compatriots. It is the third time that Ukraine has won the annual competition.