The changes, including the removal of requirements for Victorians to work or study from home, come into effect from 11.59pm on Friday, Health Minister Martin Foley said on Tuesday.
"Victorians have done such a great job getting vaccinated so we're able to take safe steps to get more people to return to the office," he said.
But masks will be required on public transport, in taxis and rideshares, on planes and in airports and at hospitals and care facilities.
Hospitality, retail, court and corrective services workers will still have to don a mask.
Primary school students in year 3 or above will continue to wear masks, as will teachers, but secondary students will not.
Questioned about the mask requirement for primary schools, Premier Daniel Andrews there were lower vaccination rates among years 3 to 6.
"We've got to draw the line somewhere. That's been a long-standing practice. If you want me to make it right down to prep, that would be to not follow the advice I've been given," he told reporters.
"I look forward to the day when we don't need masks anywhere."
As well, the government has decided remaining restrictions on elective surgery will lift on Monday.
"We're balancing the need to support our health system with the benefits of easing restrictions in a careful and sensible way," Mr Foley said.
Victoria on Tuesday reported 6786 news COVID-19 cases and 14 deaths.
There are 47,464 active cases in the state.
Some 345 infected people are in hospital, with 48 in ICU and eight on ventilators.
The government has also extended the third-dose vaccination mandate for education workers for a month to March 25, assuming they had their second dose on or before October 25.
All fully vaccinated workers who still need their third jab will have three months and two weeks since their second dose to do so.
To date, 55.9 per cent of Victorians aged 18 and over have had their third dose, while 9.8.9 per cent aged over 12 have had their second.