What is index.dat file?
If you go poking around in [c]:\Documents and Settings\[user name]\Local Settings\History\, for example, you will find several index.dat files laying around. (Note: several of the directories that I listed in the path are hidden directories.)
Index.dat files are binary files that Internet Explorer uses to save information about URLs. They are basically hash tables that have been optimized to be very fast when dealing with URLs. Given an URL, these files map the URL to a binary blob of data. The binary blob of data is different, depending on what the particular index.dat file is used for. Index.dat files are for IE's internal usage only. They are not designed or tested to be used by others.
There are three index.dat files residing in the following locations.
On Windows 98/Me:
\Windows\Temporary Internet Files\Content.IE5\index.dat
On Windows NT/2000/XP/2003:
\Documents and Settings\[username]\Cookies\index.dat
\Documents and Settings\[username]\Local Settings\History\History.IE5\index.dat
\Documents and Settings\[username]\Local Settings\Temporary Internet Files\Content.IE5\index.dat
Index.dat are files hidden on your computer that contain all of the Web sites that you have ever visited. Every URL, and every Web page is listed there. Not only that but all of the email that has been sent or received through Outlook or Outlook Express is also being logged. The file names and locations depend on what version of Internet Explorer you have. If you are running IE version 4.0 or above, the file name is "index.dat". Microsoft has not supplied an adequate explanation as to what these files are for or why they have been hidden so well.
According to Microsoft, these files are used to cache visited Web sites to help speed up the loading of Web pages in Internet Explorer. Obviously this cannot be the case because when you clear the Temporary Internet Files the "index.dat" files remain behind and continue to grow. If you delete or clear the Temporary Internet Files, there is absolutely no need to index the URL cache because those files no longer exist.
Example: index.dat file before deleting
Example: index.dat file after deleting