Tell them the truth. Say, "We’re just very selective about who we do business with and chemistry is very important. I didn’t really feel that we connected to well. What did you think?"
Other alternatives are, "We work only with what are called “financial delegators” which means that you trust me completely and do whatever I advise you to do, pretty much no questions asked. I don’t get the sense that this is the kind of relationship you want to have with an advisor. Did I read you wrong about this?"
Also, "We love to help people make smart financial choices and we have a business to run. Our minimum annual fee is $5,000 and based on your financial situation I would not be comfortable charging you that much. Therefore, I think I can give you some good advice on how to find an advisor who will be a better fit for you."
Please click the attachment below for Bill Bachrach's audio version of this answer.