robots.txt

robots.txt is a way to supposedly get search engines to leave your site's private resources alone, like a private API or something. But at the same time, you are making valuable resources easily known to attackers. It takes out the time and effort required for directory enumeration with DirBuster.

How can we come up with a better solution to robots.txt that doesn't paint obvious targets?

>>18
It's not about passwords, it's about URI's. Good luck convincing your webadmin to change /users/details to /correcthorsebatterystaple/fuckingxkcd.

>>19
They might as well brute-force your URLs then.

>>20
Not if you make them something that humans will recognize as relevant to the content, but that doesn't use common phraseology. Human brains are very good at filling in patterns.