Someone will know, but I'm pretty sure it has something to do with an old club called Victorians or something like that. Not sure if it's anywhere near true though.
As far back as 1874, the North Adelaide area had a football club, In that year, a club was formed from a number of people who had moved from Victoria to live in South Australia. They called their club the Victoria Football Club and they played their games at the bottom of Montefiore Hill, behind the Adelaide Oval.
Off the SANFL website