Yes, there are no places known to us where the water would not be drinkable. The gas stations have cleaning stations where you can fill up your water tank. So far this service is provided free of charge. We urge all our customers to leave the gas station well cleaned and dispose the lavatory accordingly. http://www.campericeland.is/information/camper-lavatory-disposal/
During the winter time the gas stations offer limited service. We apologize for that inconvenience. However our experience has shown that it's not a big problem to find a water supply during the winter. Simply ask at the gas station, camp grounds (if open) or anywhere where you would expect running water. Icelanders are very friendly and known to help travelers.