If we are talking about 'after hours calls', they should all be charged. I have people calling at 6am, and then at 9pm... my first comment is, 'I can continue this call for xxx per 15 minutes. Otherwise please call me back during my standard business hours' (which are 9am - 6pm Monday - Friday, or by appointment.)
People need to know that while the internet is on 24 /7, you need your down/personal time. If clients want to interrupt that time, they need to pay for it.