For me personally, the only case in which I would give up significant time/money to attend a destination wedding would be for one of my own kids getting married. Our vacation time each year is far too limited, and there are too many things we want to do and already have planned out, to give up a week of vacation time (+ several thousand $$) to go to a cousin/friend/niece/etc's wedding. Maybe if we could make it a quick weekend trip, but definitely not if it required more than that.
That said however, I do think that when it is YOUR wedding, you have the right to do absolutely whatever you want. And if going somewhere exotic is what you dream of, then you should go for it. Just don't put any expectation or pressure on anyone else to attend. There's nothing wrong with inviting, as long as it doesn't involve a guilt-trip for saying no. I also don't think anyone should make you feel guilty for having a wedding they can't come to (for whatever reason), as the event is a major event in YOUR life, not theirs. I would not be at all offended or upset to receive an invitation, but would just politely decline, send a gift, and wish them well.