The more noise pollution you live around on a daily basis should help dictate how loud your personal audio in your personal space should be. If you live in the country, for example, you can crank it a little louder because your eardrums have not been bombarded by different levels of noise all day.
Your eardrums are not bullet proof, so they should not be "shot at" constantly all day.