Switching tenses is something that many great writers do all the time. Tense is a feature of language and grammar, and as such is a tool the writer can exploit. I recall reading a novel written in the past tense by an author who was a slave to tense. He wrote about something he and his brother had done long ago when they were in elementary school. The sentence that jarred my reader's ear was: "My brother was two years older than me."
Now, I have a brother and he is two years older than me. He has been two years older than me for as long as I can remember, and he has never gotten more than two years older than me. Never. There is no good reason for that sentence to be in the past tense.
I received a manuscript of a novel written in the past tense, and there was action in the past of that past, with the result that the word "had" appeared 18 times on the first page. This is not good writing, and it is not good reading either. I call such writers "haddocks."
You can think of writing as having two aspects, which I call articulation and imagery. Tense is a part of articulation. The important part of creative writing is the imagery aspect. You need to be articulate; you need to have some structure that supports your imagery, but you do not want the articulation to call attention to itself. Consider these two sentences:
I would have had to have left earlier if I had wanted to be on time.
AND
I was late again. Mary will kill me.
Some people like the articulation of the first sentence. It flatters their sense of grammatical perfection that they can follow the intricacy of past and past perfect tenses without effort or without getting confused.
There is nothing inherently wrong with switching tenses. You should do it gracefully, which takes some extra work. The reader is not apt to be confused so long as the sequence of images has continuity and the story is developing. If switching tenses jolts the reader in a way that makes him aware of the articulation and he loses contact with the imagery, then you have failed as a writer. But the same is true if your writing becomes awkward by maintaining an arbitrary tense.
In erotic writing, it can be very effective to have a sex scene in the past tense and orgasm in the present tense.