You can politely say no to a rude person like this:
I'll get back to you.
Not this time.
I can't, but maybe another time.
Sorry, but I have something else.
I'm not sure if that's a good idea for me.
Sorry, but it just won't work for me.
My apologies, but I can't right now.