James Taylor had a significant impact on Australian amateur sport during the first half of last century.

Taylor was President of the Australian Swimming Union from 1909 and President of the New South Wales Amateur Swimming Association from 1908. He was the founding Chairman of the Australian Olympic Federation (AOF) from 1915 when the Australian Olympic Movement gained momentum after the war and is one of only two to have held the simultaneous posts of AOF Chairman and IOC member, the other is Kevan Gosper. He was appointed to the IOC in November 1924 as Australia’s second representative. He held all of these posts until his death in 1944.

Taylor attended a meeting on July 14, 1928 in London along with representatives of Canada, South Africa, and New Zealand, at which the Empire Sports Federation was founded. Its main objective was to promote the Empire (later Commonwealth) Games, and it resolved to hold the first in Hamilton, Canada, in 1930.