Fredericton councillors voted Monday to allow up to four housing units on residential lots across the city with nothing more than a building permit from city hall.