@Kate: I would definitely speak English with them if I were you, at least as much as conveniently possible, because they will never have this chance again in their lives. Learning languages before puberty is a lot easier than after. People who begin learning a new language as teens or adults can still learn a lot, but they rarely attain the fluency and accuracy, and especially the pronunciation, of those who start younger.
As a kid, I was already disappointed that my parents weren't bilingual and that we never lived in a cool foreign country. (Rabeneltern! *g*) No, seriously, I knew that wasn't their fault -- but I'm not sure I would have forgiven them if they _had_ spoken another language and had failed to teach it to me.
Even if kids protest or resist at the time, it's like piano lessons -- as long as they have any aptitude at all, they will thank you later for having made them stick with it, and regret it themselves later if you wimped out.
Of course some kids' talents lie in other areas than dealing with words and language. But, just as very few people are really tone-deaf, there really aren't that many people who absolutely have a tin ear for words, who simply can't learn languages at all. For the vast majority of the rest of us, only one factor makes a difference: practice.
So as a parent, you're in the position to give them that advantage. Maybe you should consider whether it's fair of you to withhold it from them.
@jay: I agree with nja. Don't pretend you can speak it perfectly yourself, and do be sure they start by hearing native speakers, with children's audio books, videos, and above all, songs. And make sure you choose things with written text for them to read along with while they listen, with your help at first. There's no point in learning a language by ear alone, just as there's no point in learning piano and not learning how to read music.
Then when you have the chance and they know a few words, you could turn 'How do you say that in English?' into a game, something you do together for fun, on car trips, while shopping, etc., or just as a treat at home. If the kids enjoy it (and if you teach them some verbs, not just nouns *g*), maybe they'll want to try speaking it with you some of the time. But even if not, at least they'll have a little head start.
My mom did that with me a little in French. She only knew it from having taken a few years of it in school and listened to operas and so on, so she was by no means fluent, but we sang songs and played games with flash cards when I was only 4 or 5, and it was at least enough to give me the idea, _long_ before language instruction was available in school (the US is backward in that respect), that there were other languages besides the one we spoke at home, and that learning different words for things could be fun.
So IMO it can't hurt, and it could help. (-: