With a few correct words in Japanese, you have a good chance at effectively communicating with a non-English speaker. When I arrived in Japan, I was surprised at how few people spoke a basic level of English. If someone appears flustered when you speak in English, then you should know that they're not being shy or trying to get rid of you - they really can't speak English.
If you only learn one word of Japanese, make it 'daijobu'. This means 'OK' and you can also make it 'Is it okay?' by inflecting your voice at the end. So if someone is telling you something in Japanese and there's no way for you to understand but you think it's just a small detail (i.e., a shopkeeper asking if she should pack everything in one bag or two bags), then just say 'daijobu!' If you're paying for something and you're not sure if you have the right amount then just ask 'daijobu?' when presenting your money and they'll help you. This easy word might even warrant a 'Nihon-go jozou!' which means, basically, 'your Japanese is great!'
You should also know greetings and how to say thank you. When you say hello to someone in the morning, say 'Ohayo gozaimus' (note: this isn't the proper way to write it but it is the correct way to say it). In the afternoon, say 'konnichiwa' and in the evening say 'konbanwa'. Note: 'moshi mosh' is ONLY used for answering the phone. It sounds silly if you say it in any other context. If you want to say 'How are you', say 'Genki des ka?'
With Japanese, the more words you use, the more polite you are. So if you want to thank someone for something trivial, you can say 'domo.' If it's something bigger, say 'domo arrigato.' And for the mother of all thank-yous, say 'domo arrigato gozaimas' (if it is still in the process) or 'domo arrigato gozaimashta' (if it has been completed).
Here's a few more words that you'll hear often and should try out yourself. To say goodbye to someone you probably won't see later the same day, say 'sayonara.' If you are friends, you can say 'ja mata' or 'ja ne.'
If you want to ask if someone speaks English, you can say 'Eigo-ga wakadimas-ka?' ('Do you understand English) and be patient because they may answer in Japanese that they will fetch someone who can speak English. If you want to tell them that you can't speak Japanese, say 'Nihon-go ga wakadimasen.'
If you want to do more in-depth studies then there are countless great textbooks, videos, and CDs out there. ANY amount of effort to speak Japanese is richly rewarded here, especially in tourist areas. So give it a shot!