Both are colloquial phrases, and they are, for the most part, interchangeable. Individual people tend to have their own personal meanings, either in terms of altitude (up at the lodge or down at the crick), positive or negativity (up at the mansion, or down at the factory), or, as I tend to use it, in terms of latitude compared to my current location, North or South (up in Canada, down in Mexico). Some people stick to common phrases they hear a lot without truly thinking as to why they say it (Upta Camp!).
Ultimately, though, those are all tendencies, not rules. There are no strict rules on the subject, and neither are considered formal English.