Some vulnerable people and frontline workers are furious they are still waiting for vaccine doses while sportsmen and sportswomen roll up their sleeves.
Senator Colbeck said he could not understand their anger or concern, arguing the vaccine rollout stages were always designed to be blurred.
"No, I can't actually [understand it] because it was always anticipated there would be some overlap," he told the ABC on Tuesday.
"It was never going to be that we would finish 1a before we started 1b or we would finish 1b before we started 2a.
"There was always anticipated overlap in the stages. That is largely driven by the availability of vaccine supplies and that remains a consideration."
More than 2500 athletes and support staff are being prioritised for vaccines ahead of the games in Japan.
"We've never said they are more important," Senator Colbeck said.
"Every aged care centre in Australia has either a date or a two-week window within which their vaccinations will be completed.
"That information has been forwarded to them all and that will be done in the next few weeks."
Senator Colbeck said the Australian Olympic Committee had contracted a private provider to vaccinate athletes and officials, guaranteeing no extra strain or load on the public health system.
Those aged over 50 will be given the AstraZeneca vaccine and people aged under 50 will receive Pfizer jabs.
Athletes are expected to be given daily coronavirus tests in Japan and only allowed to leave their accommodation to train and compete. But the minister conceded Olympic villages were crowded and busy places, and outbreaks were unavoidable.
He is concerned about the virus spreading among athletes, and the effect it could have on the games.
"It is inevitable there will be coronavirus at the Olympics - that would have to be a given," Senator Colbeck said, adding this consideration played into priority access to vaccines.
Japan is facing its fourth wave of COVID-19 infections and Tokyo remains in a state of emergency.
Senator Colbeck acknowledged it was legitimate to question whether the Games should go ahead.
"I am sure the IOC and the Japanese government are giving that matter serious consideration," he said.
"But it is a significant question because quite clearly not all athletes from all nations are going to have the opportunity to be vaccinated."
The minister conceded the Paralympics would be even more dangerous and said Australian athletes would need to make "very conscious" decisions about whether they attended.
The AOC wants a "bespoke" quarantine arrangement for Australian athletes and officials when they return home, similar to resort-style hubs used by the AFL.
But Senator Colbeck expects returning athletes will probably be treated more like tennis players who flew into the country for the Australian Open.
The players were given individual rooms and faced harsher lockdowns if infections were detected among the plane-load of passengers they travelled with.
The AOC remains locked in negotiations with several states over quarantine arrangements.
"My expectation is that it would be a much more Australian Open-style quarantine process than an AFL hub," Senator Colbeck said.