Instead of sending back the modification time, the server can send back the ETag (fingerprint): Just like last-modifed, ETags solve the problem of comparing file versions, except that “if-none-match” is a bit harder to work into a sentence than “if-modified-since”. It’s like analyzing your milk every time you make cereal to see whether it’s safe to drink. If we know when the milk (logo.png) expires, we keep using it until that date (and maybe a few days longer, if you’re a college student).
Sure, it’s better than buying a new gallon each time, but it’s not exactly wonderful. As soon as it goes expires, we contact the server for a fresh copy, with a new expiration date.
By commenting, you are accepting the DISQUS terms of service.
Search engines may have different crawlers for different properties or purposes. For example, to show a page in Google's web search results, but not in Google News, use the following meta tag: Directives specified without a user-agent are valid for all crawlers.The header looks like this: However, be wary that some cache directives only work on newer HTTP 1.1 browsers.If you are doing special caching of authenticated pages then read more about caching.Here is an example of a robots meta tag that instructs web crawlers to not index the page and to not crawl any of the links on the page: HTTP headers are discovered when a URL is crawled.If a page is disallowed from crawling through the file, then any information about indexing or serving directives will not be found and will therefore be ignored. When you sign in to comment, IBM will provide your email, first name and last name to DISQUS.