A polo shirt and a pair of slacks qualify as golf apparel, mostly because golf is a low-intensity sport. Compared to basketball and baseball, golf doesn't require rigorous physical activity other than swinging the club. You may break a sweat during a game from the heat or from intense mental concentration, but not necessarily through physical exertion.
The only thing that limits your choice of golf apparel is the dress code. During the early 1900s, country clubs were very particular about proper dress code for guests and members. Knickers, for example, were prevalent at golf courses at the time but were shunned by country clubs. These places obliged players and spectators to come in more formal wear, particularly the collared shirt.