The four were spotted near the tip of Shiretoko Peninsula, although the Japanese coast guard was unable to confirm whether they were rescued alive.
NHK public television said they were unconscious.
Footage on NHK showed one of the rescued people arriving on a helicopter and being transferred to an ambulance on a stretcher, while rescuers held up blue plastic shields for privacy.
The boat carrying 24 passengers and two crew members went missing after sending a distress call on Saturday, saying it had taken in water and was beginning to sink.
Sunday's rescue came after nearly 19 hours of intense searching involving six patrol boats, several aircraft and divers.
The 19-tonne Kazu 1 made an emergency call in the early afternoon, saying the ship's bow had flooded and it was beginning to sink and tilt while travelling off the western coast of Shiretoko Peninsula in the northern island of Hokkaido, the coast guard said.
Average April sea temperatures in Shiretoko National Park are just above freezing.
Prime Minister Fumio Kishida, who was attending a two-day water summit in Kumamoto in southern Japan, cancelled his program for the second day and returned to Tokyo.
He told reporters in the early hours of Sunday he had instructed officials "to do everything they can for the rescue".
The cause of the accident is still under investigation, but experts suspect the boat ran aground and was damaged.
High waves and strong winds were observed in the area around noon, according to a local fisheries co-operative.
Japanese media reports said fishing boats had returned to port before noon because of the bad weather.
NHK said there was a warning for high waves of up to three metres.
A tour boat crew member belonging to another operator told NHK he warned of rough seas when he spotted the Kazu 1 crew and told them not to go.
He also said the same boat ran aground last year and suffered a crack on its bow.
Yoshihiko Yamada, a Tokai University marine science professor, told TBS television the boat was likely to have run aground after it was tossed around in high waves and damaged.
A tour boat of that size usually does not carry a life boat, and passengers could struggle to jump out of a rapidly sinking vessel with its windows probably closed to shield them from strong winds.
The cold temperature and strong wind could also cause hypothermia, Jun Abe, vice chairman of the Society of Water Rescue and Survival Research, said.