What is the FeastieBot?
The FeastieBot is the web crawling bot (also known as a "spider") for Feastie.com. Crawling food blogs and sites is how we find new recipe pages to add to our search index.
The FeastieBot starts with a list of webpage URLs for known food blogs and sites. When visiting each page, it finds links to other pages to crawl, and detects whether or not the page contains a recipe. If the page contains a recipe, it adds a link to the page to our search index along with a thumbnail image, title, and a set of recipe-specific keywords including the ingredients. The FeastieBot detects recipes, ingredients, and other recipe-specific keywords using rich snippets.
How your site appears in Feastie
Search result snippets from Feastie include:
- The title of the webpage or recipe.
- The name of the source.
- A thumbnail image.
When a user clicks on a snippet, we link directly to the original webpage.
Making sure your site is crawlable
Here are some guidelines for ensuring that the FeastieBot can index your site:
- Don't put the text of your recipe in an image.
- Make sure you haven't inadvertently blocked the FeastieBot by using a script or plugin that blocks bots.
Preventing the FeastieBot from crawling your site
If you would prefer not to have the FeastieBot indexing your site, add the following lines to your robots.txt file:
See www.robotstxt.org for more information on configuring robots.txt.