The point of using a version control system (VCS) is to track versions of source files in a project, with the idea that building software is complex and it's often difficult to keep track of all of the changes that were made from one revision of a code base to another. If changes that were made cannot be tracked, there is no way to account for unforseen consequences of making a change to source code, such as breaking the build, failing QA regression tests, or negatively affecting performance. A VCS allows you to easily compare one version of a source file to another and figure out what you did to the latest revision, and to revert back to a previous version if necessary. It also allows large teams of developers to work on the same code base at once, by providing them with the ability to merge changes that they have made to the same source artifact, if necessary. For these reasons, the only things that should be checked into a VCS (as far as I can think of, anyways) are the buildable artifacts in a software development project, along with the non-buildable artifacts that the buildable artifacts depend upon.
Binary files that are not part of the build process are generally not a good fit for inclusion in a VCS, because it's difficult to see exactly what changed from one version to another, and nearly impossible to combine changes made to the same binary file by two or more people. This defeats the point of using version control to begin with. Another reason why binary files are not a good fit is that, with most version control systems (Subversion excluded), any minor change to said file results in an entirely new copy of that file being stored in the repository, along with any previous versions that might exist, and there is no way to reliably remove the previous versions of the file without removing all files that changed in that revision. This results in a nearly irreversably huge repository.
I've personally worked with CVS, Subversion, and SourceSafe. Of those, only SourceSafe allowed me to permanently delete individual files. However, this had a number of nasty consequences, mostly having something to do with SourceSafe not properly cleaning up references to deleted files and generally confusing itself, because SourceSafe sucks.
If you're free to choose the version control system you use, I'd highly suggest starting with Subversion. It's an offshoot of the older CVS, and addresses a number of shortcomings with that system. A comparison between CVS and Subversion can be found here. If you're using SourceSafe now, do yourself a favor and stop. Subversion is stable, mature, well-supported, well-documented, and open-source.