In the next two weeks we have a Disaster Recovery activity for the client. As a part of it the Backup Team was trying to copy some of the backup files from the Production server to the DR server. These backups files several TBs in size. When they were trying to copy one of those files on Windows Server 2003 box, they got an error.
Received Win32 RC 1450 (0x000005aa) from FileRead(): ReadFile '\ServerNamei$SQLBACKUPSFileName.BAK'. Insufficient system resources exist to complete the requested service.
Just because the file they were trying to copy had SQL in its path, they got back to us to fix this issue. This has nothing to do with SQL Server and it is purely related to Memory Management on the server. On Windows servers, if one tries to copy *huge* files in parallel over the network this situation arises. I am no expert on Operating System but I will try to explain the reasons for this with whatever knowledge I have on this topic.
When the system boots the Memory Manager creates two pools namely Paged and Non-paged. The size of these is dependent on the Physical RAM available on the server. The Non-paged pool is used by the Windows kernel and drivers as the memory in this pool cannot be paged out. The memory in the Paged Pool can be paged out. Depending on the Operating System, RAM installed and the architecture (32 or 64 bit) the maximum size of the Paged Pool differs. On a server running Windows Server 2003 (32 bit) the maximum size of the Paged Pool is 650 MB (2GB for 64 bit).
Now coming back to the Copy command, it also makes use of the memory in Paged Pool. The memory that it uses is directly related to the size of the file being copied. As per this article for each MB of file that is being copied it utilizes 2 KB (1KB each for source and destination) of Paged Pool memory. If PAE mode is enabled the memory usage per MB is 4 KB.
On this server the Backup team was trying to copy a file which was 150 GB in size. This is running on Windows Server 2003 and it has PAE mode enabled. To copy this file it requires 600 MB of Paged Pool Memory (150*4). The limit is 650 MB. If we know how much Paged Pool memory is in use, it will be easy to predict if the copy will complete or not. Yet again, Process Explorer comes in handy. To know how much Paged Pool Memory is in use, just launch Process Explorer and click View –> System Information.
As per this information on that server 221 MB of Paged Memory is already in use. To copy this file it requires 600 MB more, which will take the total above the limit of Windows 2003 server. Hence Insufficient system resources exist to complete the requested service error and the copy command failed! Some other process has already occupied this memory and until it is cleaned up, they will not be able to copy the file using the native copy or xcopy commands. What is the fix? Since this memory pool is defined during system boot, the option is to reboot.