If you find that these tiny bugs are getting away from manage and messing with how your web server functions, then you can use robots.txt to acquire them consistent.