Copying large quantity of files taking forever

[Guest SwitchKat]

Hi, I wanted to throw this question out because after losing over a week of

productivity, I'm not sure where else to turn.

 

Here's my issue. We're trying to restore a file repository for a project

that has just under 2 million files. We've tried restoring from tape,

copying from another Windows server and even moving from another Windows

server. The problem we run into in every case is that the file copy to the

new system takes off fast and we can see file transfer rates start out

copying 1000's of files per second, but over time it always slows down to a

crawl and we end up where we are now. We are currently transferring one file

every few minutes.

 

All the files are fairly small (<10MB), gigabit network (although we see the

same performance if we attach the drive locally and just try a copy), and all

are running Windows Server 2000 or 2003 server. The target is always running

2003. We've been monitoring resources, and at this point, both servers

involved show flatlines for their resources being used. My hunch is that it

has something to do with a data block size issue or the file table (maybe

fragmentation?), but I'm not sure. Any ideas?

[Guest Paul Weterings]

Re: Copying large quantity of files taking forever

 

SwitchKat wrote:

> Hi, I wanted to throw this question out because after losing over a week of

> productivity, I'm not sure where else to turn.

>

> Here's my issue. We're trying to restore a file repository for a project

> that has just under 2 million files. We've tried restoring from tape,

> copying from another Windows server and even moving from another Windows

> server. The problem we run into in every case is that the file copy to the

> new system takes off fast and we can see file transfer rates start out

> copying 1000's of files per second, but over time it always slows down to a

> crawl and we end up where we are now. We are currently transferring one file

> every few minutes.

>

> All the files are fairly small (<10MB), gigabit network (although we see the

> same performance if we attach the drive locally and just try a copy), and all

> are running Windows Server 2000 or 2003 server. The target is always running

> 2003. We've been monitoring resources, and at this point, both servers

> involved show flatlines for their resources being used. My hunch is that it

> has something to do with a data block size issue or the file table (maybe

> fragmentation?), but I'm not sure. Any ideas?

 

Give RoboCopy a try

 

--

 

/ ) Regards,

/ /_________

_|__|__) Paul Weterings

/ (O_) http://www.servercare.nl

__/ (O_)

____(O_)

[Guest SwitchKat]

Re: Copying large quantity of files taking forever

 

Ya, Robocopy was the first tool we tried. It works fine up until we cross

the 1.25 million files, and it too starts to slow down to a crawl.

 

"Paul Weterings" wrote:

> SwitchKat wrote:

> > Hi, I wanted to throw this question out because after losing over a week of

> > productivity, I'm not sure where else to turn.

> >

> > Here's my issue. We're trying to restore a file repository for a project

> > that has just under 2 million files. We've tried restoring from tape,

> > copying from another Windows server and even moving from another Windows

> > server. The problem we run into in every case is that the file copy to the

> > new system takes off fast and we can see file transfer rates start out

> > copying 1000's of files per second, but over time it always slows down to a

> > crawl and we end up where we are now. We are currently transferring one file

> > every few minutes.

> >

> > All the files are fairly small (<10MB), gigabit network (although we see the

> > same performance if we attach the drive locally and just try a copy), and all

> > are running Windows Server 2000 or 2003 server. The target is always running

> > 2003. We've been monitoring resources, and at this point, both servers

> > involved show flatlines for their resources being used. My hunch is that it

> > has something to do with a data block size issue or the file table (maybe

> > fragmentation?), but I'm not sure. Any ideas?

>

> Give RoboCopy a try

>

> --

>

> / ) Regards,

> / /_________

> _|__|__) Paul Weterings

> / (O_) http://www.servercare.nl

> __/ (O_)

> ____(O_)

>

[Guest Pegasus \(MVP\)]

Re: Copying large quantity of files taking forever

 

 

"SwitchKat" <SwitchKat@discussions.microsoft.com> wrote in message

news:EFEF754F-402C-438E-9654-0ABC30409346@microsoft.com...

> Hi, I wanted to throw this question out because after losing over a week

> of

> productivity, I'm not sure where else to turn.

>

> Here's my issue. We're trying to restore a file repository for a project

> that has just under 2 million files. We've tried restoring from tape,

> copying from another Windows server and even moving from another Windows

> server. The problem we run into in every case is that the file copy to

> the

> new system takes off fast and we can see file transfer rates start out

> copying 1000's of files per second, but over time it always slows down to

> a

> crawl and we end up where we are now. We are currently transferring one

> file

> every few minutes.

>

> All the files are fairly small (<10MB), gigabit network (although we see

> the

> same performance if we attach the drive locally and just try a copy), and

> all

> are running Windows Server 2000 or 2003 server. The target is always

> running

> 2003. We've been monitoring resources, and at this point, both servers

> involved show flatlines for their resources being used. My hunch is that

> it

> has something to do with a data block size issue or the file table (maybe

> fragmentation?), but I'm not sure. Any ideas?

 

You don't say where these files reside but if they reside in a single

directory then this is your answer. If you want to keep your machine running

at top speed, don't keep more than 10,000 entries in a single folder. 2

million is two hundred times too many.

[Guest SwitchKat]

Re: Copying large quantity of files taking forever

 

The files are spread out through many directories. I originally thought this

might be an issue too since it also takes a long time to open any of the

folders.

 

"Pegasus (MVP)" wrote:

>

> "SwitchKat" <SwitchKat@discussions.microsoft.com> wrote in message

> news:EFEF754F-402C-438E-9654-0ABC30409346@microsoft.com...

> > Hi, I wanted to throw this question out because after losing over a week

> > of

> > productivity, I'm not sure where else to turn.

> >

> > Here's my issue. We're trying to restore a file repository for a project

> > that has just under 2 million files. We've tried restoring from tape,

> > copying from another Windows server and even moving from another Windows

> > server. The problem we run into in every case is that the file copy to

> > the

> > new system takes off fast and we can see file transfer rates start out

> > copying 1000's of files per second, but over time it always slows down to

> > a

> > crawl and we end up where we are now. We are currently transferring one

> > file

> > every few minutes.

> >

> > All the files are fairly small (<10MB), gigabit network (although we see

> > the

> > same performance if we attach the drive locally and just try a copy), and

> > all

> > are running Windows Server 2000 or 2003 server. The target is always

> > running

> > 2003. We've been monitoring resources, and at this point, both servers

> > involved show flatlines for their resources being used. My hunch is that

> > it

> > has something to do with a data block size issue or the file table (maybe

> > fragmentation?), but I'm not sure. Any ideas?

>

> You don't say where these files reside but if they reside in a single

> directory then this is your answer. If you want to keep your machine running

> at top speed, don't keep more than 10,000 entries in a single folder. 2

> million is two hundred times too many.

>

>

>

[Guest Pegasus \(MVP\)]

Re: Copying large quantity of files taking forever

 

The utility xxcopy.exe is known for baulking at large copy jobs. Perhaps

there is a similar but much higher limit for robocopy. Which version do you

use? It should be at least Version XP010.

 

If this is a limitation of robocopy.exe then you may be able to walk around

it by splitting up the job into many smaller jobs, using a batch file. I

would try this:

@echo off

for %%a in (0 1 .. 9 a b .. z) do robocopy /s /.. "D:\SourceFolder"

"E:\DestFolder" %%a*.*

robocopy /s /XO "D:\SourceFolder" "E:\DestFolder" *.*

 

The second line will copy all files starting with 0 .. 9 and a .. z. You

must expand the dots to include the full alphabet - the batch file won't do

it for you! The third line will scoop up any file that was not already

copied by the first line. If the third line slows down to a crawl after a

while then you have to expand the collection of characters in Line 2 to

include all valid ASCII characters. Use charmap.exe to identify them!

 

 

 

"SwitchKat" <SwitchKat@discussions.microsoft.com> wrote in message

news:561323C4-1B08-4034-9BB6-63C4089824C6@microsoft.com...

> The files are spread out through many directories. I originally thought

> this

> might be an issue too since it also takes a long time to open any of the

> folders.

>

> "Pegasus (MVP)" wrote:

>

>>

>> "SwitchKat" <SwitchKat@discussions.microsoft.com> wrote in message

>> news:EFEF754F-402C-438E-9654-0ABC30409346@microsoft.com...

>> > Hi, I wanted to throw this question out because after losing over a

>> > week

>> > of

>> > productivity, I'm not sure where else to turn.

>> >

>> > Here's my issue. We're trying to restore a file repository for a

>> > project

>> > that has just under 2 million files. We've tried restoring from tape,

>> > copying from another Windows server and even moving from another

>> > Windows

>> > server. The problem we run into in every case is that the file copy to

>> > the

>> > new system takes off fast and we can see file transfer rates start out

>> > copying 1000's of files per second, but over time it always slows down

>> > to

>> > a

>> > crawl and we end up where we are now. We are currently transferring

>> > one

>> > file

>> > every few minutes.

>> >

>> > All the files are fairly small (<10MB), gigabit network (although we

>> > see

>> > the

>> > same performance if we attach the drive locally and just try a copy),

>> > and

>> > all

>> > are running Windows Server 2000 or 2003 server. The target is always

>> > running

>> > 2003. We've been monitoring resources, and at this point, both servers

>> > involved show flatlines for their resources being used. My hunch is

>> > that

>> > it

>> > has something to do with a data block size issue or the file table

>> > (maybe

>> > fragmentation?), but I'm not sure. Any ideas?

>>

>> You don't say where these files reside but if they reside in a single

>> directory then this is your answer. If you want to keep your machine

>> running

>> at top speed, don't keep more than 10,000 entries in a single folder. 2

>> million is two hundred times too many.

>>

>>

>>

[Guest SwitchKat]

Re: Copying large quantity of files taking forever

 

Thanks Pegasus! I didn't even think to script it out and break it up. I'll

give it a try.

 

"Pegasus (MVP)" wrote:

> The utility xxcopy.exe is known for baulking at large copy jobs. Perhaps

> there is a similar but much higher limit for robocopy. Which version do you

> use? It should be at least Version XP010.

>

> If this is a limitation of robocopy.exe then you may be able to walk around

> it by splitting up the job into many smaller jobs, using a batch file. I

> would try this:

> @echo off

> for %%a in (0 1 .. 9 a b .. z) do robocopy /s /.. "D:\SourceFolder"

> "E:\DestFolder" %%a*.*

> robocopy /s /XO "D:\SourceFolder" "E:\DestFolder" *.*

>

> The second line will copy all files starting with 0 .. 9 and a .. z. You

> must expand the dots to include the full alphabet - the batch file won't do

> it for you! The third line will scoop up any file that was not already

> copied by the first line. If the third line slows down to a crawl after a

> while then you have to expand the collection of characters in Line 2 to

> include all valid ASCII characters. Use charmap.exe to identify them!

>

>

>

> "SwitchKat" <SwitchKat@discussions.microsoft.com> wrote in message

> news:561323C4-1B08-4034-9BB6-63C4089824C6@microsoft.com...

> > The files are spread out through many directories. I originally thought

> > this

> > might be an issue too since it also takes a long time to open any of the

> > folders.

> >

> > "Pegasus (MVP)" wrote:

> >

> >>

> >> "SwitchKat" <SwitchKat@discussions.microsoft.com> wrote in message

> >> news:EFEF754F-402C-438E-9654-0ABC30409346@microsoft.com...

> >> > Hi, I wanted to throw this question out because after losing over a

> >> > week

> >> > of

> >> > productivity, I'm not sure where else to turn.

> >> >

> >> > Here's my issue. We're trying to restore a file repository for a

> >> > project

> >> > that has just under 2 million files. We've tried restoring from tape,

> >> > copying from another Windows server and even moving from another

> >> > Windows

> >> > server. The problem we run into in every case is that the file copy to

> >> > the

> >> > new system takes off fast and we can see file transfer rates start out

> >> > copying 1000's of files per second, but over time it always slows down

> >> > to

> >> > a

> >> > crawl and we end up where we are now. We are currently transferring

> >> > one

> >> > file

> >> > every few minutes.

> >> >

> >> > All the files are fairly small (<10MB), gigabit network (although we

> >> > see

> >> > the

> >> > same performance if we attach the drive locally and just try a copy),

> >> > and

> >> > all

> >> > are running Windows Server 2000 or 2003 server. The target is always

> >> > running

> >> > 2003. We've been monitoring resources, and at this point, both servers

> >> > involved show flatlines for their resources being used. My hunch is

> >> > that

> >> > it

> >> > has something to do with a data block size issue or the file table

> >> > (maybe

> >> > fragmentation?), but I'm not sure. Any ideas?

> >>

> >> You don't say where these files reside but if they reside in a single

> >> directory then this is your answer. If you want to keep your machine

> >> running

> >> at top speed, don't keep more than 10,000 entries in a single folder. 2

> >> million is two hundred times too many.

> >>

> >>

> >>

>

>

>