It is a sin to have that thought? Even though I didn't act upon it?
A thought is not a sin. Otherwise there would be no temptation -- only sin. Satan has to get us to think about sin to lure us into it. What should happen is that we immediately follow the thought with "no way," but what sometimes happens is that we think about it over and over. We dwell on it and think "well, maybe; but only if I can do it without being caught." When we cross the line of rejecting an idea to accepting an idea that is sinful, then our thoughts can become sin. That is because all sins first start with thoughts of finding excuses for sinful behavior. "What comes out of a man, that defiles a man. For from within, out of the heart of men, proceed evil thoughts, adulteries, fornications, murders, thefts, covetousness, wickedness, deceit, lewdness, an evil eye, blasphemy, pride, foolishness. All these evil things come from within and defile a man" (Mark 7:20-23).
This is why Jesus said, "But I say to you that whoever looks at a woman to lust for her has already committed adultery with her in his heart" (Matthew 5:28). The only difference between seriously thinking about committing a sin and actually doing it is the opportunity. So when does the sin actually begin? Jesus said it starts when you seriously give thought to breaking God's law.