As the URL fragment is not transmitted to the server, you can only use a client side solution. Here’s one using JavaScript:
if (location.href.indexOf("#") > -1) {
location.assign(location.href.replace(/\/?#/, "/"));
}
This simply checks if there is a # in the URL and replaces the first occurrence with /. So /site/#blog would get /site/blog.