Solution provided by Stephane Grobety:

When you download a file from the web using Internet Explorer on XP SP2 or 2003 Server, the browser will mark it as "web" content (it uses an alternate data stream for that).

Now, there have been several attacks against Windows using compiled HTML (*.CHM) files as a vector. So in SP2, Microsoft decided to try to protect users by disabling the execution of CHM files over the network and safeguarding against local use.

As it turns out, you can't properly open a CHM file from a network drive if it has been downloaded from the web. Even locally, you receive a security warning and you MUST check the "do not ask this question again" check box in the dialog before accepting to open them.

My solution was to copy the CHM files to my local machine and open them from there. After that, I had no more "operation canceled" error messages when trying to view the content of a specific topic.

Eugene Mayevski's addition:

Pack the file to a ZIP file and unzip the result to some other folder. This will reset the secondary NTFS streams, where the restriction information is saved. Then you will be able to open the file.