There's many places like this in Australia; I mean a lot! It's a pretty amazing and diverse country, and very large.
Most people who go to Victor Harbour like it and yep, in bad weather, stormy or cold, it has an appeal as well...perfect weather to watch some big waves, do some whale watching (May to October) and have some tasty food in one of the many places to eat.