Jim Rutherford was born at Woodburn on the far north coast of NSW in 1908. He played rugby league for the Tumbulgum and Pirates Rugby League Clubs in the Tweed district with his older brothers, George, and Bert Rutherford.
In 1926, Jim was the second highest point scorer and top try scorer in the Tweed competition. He far exceeded anyone else with sixteen tries to his credit. In 1927 he moved to the town of Lismore and played for the All Blacks Rugby League Club. He was selected to represent Far North Coast and Country Seconds.