The house on Penny Lane in McCrae collapsed shortly before 9 am on Tuesday.
Local resident Tanina Osborne and her husband had just finished their morning beach swim when they heard a "horrible sound".
"I heard this really, really loud rumbling, and like a crash," Ms Osborne told AAP.
"So my husband and I ran across the road, and I saw that this house had tumbled down the hill - I've just never seen anything like it."
An advice alert for the incident warns the landslide could continue to move for days.
Multiple nearby homes remained a concern and were being monitored by authorities, State Emergency Service operations manager Chris Gregory said.
"A couple of them were vacant holiday homes and others had residents who were working with councils and other agencies to make sure they have a safe place to go," Mr Gregory told ABC Radio Melbourne.
Ambulance Victoria said an injured person had been taken to Frankston Hospital in a stable condition.
The destroyed house was empty after it had been hit by a smaller landslide about a week before and was being monitored by geotechnical engineering experts.
Authorities have disconnected utility services in nearby properties as they assess the area for potential danger, and the Nepean Highway has been closed between McCrae and Dromana.
"We do have a road closure in place, and we'll seek to lift that as soon as we possibly can," Mr Gregory said.
With AAP.
