You’re out of town, staying with your brother-in-law. You’re standing in the middle of the kitchen. Because you and your family will be staying a few days, your goal is to know everything you can about that kitchen and all of its contents in the event that someone asks you where something is. Your first instinct is to start exploring that kitchen so you know where the dishes are and what foods are in the cabinets. Thing is, you can’t move. So you reach in your pocket and you pull out a handful of robot spiders.

You put the spiders on the floor and they start crawling the kitchen. They look in the cabinets and the drawers and in the fridge. All the time they are examining and documenting what they find. When they crawl back to you, they share everything they’ve learned with you and you put it all into your handy dandy… notebook! (Yes, I’m a dad.) Now you know everything you possibly can about that kitchen. When your daughter walks in and says “Where are the cookies?”, you look in your notebook and can tell her that the cookies are in the cupboard on the third shelf. You know this because your spiders crawled that shelf and told you what was on it.

Let’s translate.

The kitchen is the world wide web. You are Google, and you want to know everything about the world wide web, every website, every blog, every address, every phone number, in the event that someone asks you a question (aka a search query – you know – the words you place in the Google box?) So you write these programs and you call them spiders, because it’s like they are crawling the world wide web. These programs look at websites, read all of their content so that they can understand what the website is about. They bring the information back to you (Google) where you place it into an index.

When someone Googles “podiatrist kansas city” you look in that index and you provide the results that you think best match what your spiders found for websites about podiatrists in Kansas City.

Keep in mind, that spiders don’t make logical conclusions the way that people do. They may not know that a podiatrist is a foot doctor. So if the websites they crawled talked about foot doctors, but didn’t specifically mention the word podiatrist, they may not serve up those websites for that particular query. (Note: Spiders are getting smarter and smarter every day. They are often able to make the leap from podiatrist to foot doctor, but still, if the query asked for a podiatrist, that’s what Google wants to serve up as the most relevant result.)

Let’s take it back to the kitchen for just a moment. Your daughter asked for cookies, right? When you checked your notebook you found cookies in the cupboard. Great, your spiders did their job. But what if there were also cookies in the cabinet, and some Girl Scout Thin Mints in the freezer? Now you have three answers for her, but not sure which is the best. If she had a more specific question – aka a long-tail keyword query – like “Where are the chocolate chip cookies?” you can give here a more relevant answer.

The lesson – tell the spiders what your site is about. If you’re a real estate agent make sure your website says “I will sell your home”. If you’re an attorney, be sure to call yourself a lawyer somewhere on that website too. The spiders can be your best friends. And now that you know what their job is, you can help them do it better.

– David McBee