Soy gringa sí, pero hablo castellano.

Hypothetical situation:
As I'm walking down the street in DC, I see a woman walking in the other direction. I assume she's a Mexican because of her skin color, facial features, and tight clothing. She approaches me and says, in English but with an accent, "Pardon, how do I go to the White House? Is there a tram?" Instead of responding to her in English, I assume her first language is Spanish and start talking to her in Spanish with a US accent.

Of course, this would be wrong of me in many ways:
1a. Assuming nationality based on perceived race/ethnicity and stereotypical style
1b. Even if the woman's nationality or ethnicity (for example, "latina") are correctly guessed, that does not mean she speaks Spanish. Brazilians speak Portuguese; Belize English; Haiti French... And many people in every country do not speak the official or majority language. For instance, there are many latinos who speak only indigenous languages, especially in Bolivia and Guatemala.
2. Responding in a different language than the one in which I was addressed; ignoring the woman's original choice of language
3. Interpreting an accent as the lack of knowledge of a language as opposed to difficulty pronouncing sounds that are not present in her native language
4. Not considering that the woman may have come to the US for the purpose of learning English
5. Implying that my Spanish language skills and accent are better than her English ones without much to back up that claim
6. Implying that she is uneducated and disinterested in other cultures
7. Lack of recognition of the diversity of people; thinking "all Mexicans/latinos don't speak English"
8. Interpreting use of correct yet uncommon words as unfamiliarity with English rather than familiarity with another dialect of English

Fortunately, this type of situation is not common in the US. However, this happens to me every day in reverse while I'm in Latin America. It's extremely frustrating and insulting to me. I imagine it's even more so to Europeans or Canadians who are often assumed to be from the US, especially those who do not speak English at all or at least not as a native language.

And, finally, another note: If you're not being understood by a non-native speaker, consider that it might not have anything at all to do with language. The actual concept may be difficult to understand or remember (ex. complicated directions); the process/product may be unfamiliar to the person due to cultural not lingual differences; it might be difficult to hear you; or you just might not be making any sense.