VSS Catastrophic Failures

[Guest Max]

Hi,

 

Server 2003 R2 x64 SP2, Exchange Server 2007 SP1. KB940349 has been applied

and resolved issues with our JetDB writers.

 

We're having problems getting VSS to take snapshots for Shadow Copies and

backups. It is now specific to the Exchange VSS Writer.

 

We had BackupExec 12 installed and couldn't do an Exchange backup without

having catasrophic errors. I have since switched to ArcServe 11.5 SP3 and

the last Exchange backup went successfully. ShadowCopy snapshots however are

still failing.

 

After a clean reboot of the server, I can manually take a shadow copy

snapshot and it completes successfully. I took one last night before the

backup without issue.

 

This morning I checked and the 7am scheduled snapshot didn't show up in the

list. I checked the status of the VSS writers, and they're all Online with a

normal status.

 

The following errors were logged in the App log.

--------------------------------------------

Event Type: Error

Event Source: VSS

Event Category: None

Event ID: 12289

Date: 4/30/2008

Time: 7:02:32 AM

User: N/A

Computer: SERVER

Description:

Volume Shadow Copy Service error: Unexpected error

WaitForMultipleObjects(1,000000000032A2B0,1,120000) == 0x00000102. hr =

0x8000ffff.

 

For more information, see Help and Support Center at

http://go.microsoft.com/fwlink/events.asp.

Data:

0000: 2d 20 43 6f 64 65 3a 20 - Code:

0008: 43 4f 52 4c 4f 56 4c 43 CORLOVLC

0010: 30 30 30 30 30 36 39 34 00000694

0018: 2d 20 43 61 6c 6c 3a 20 - Call:

0020: 43 4f 52 4c 4f 56 4c 43 CORLOVLC

0028: 30 30 30 30 30 35 39 39 00000599

0030: 2d 20 50 49 44 3a 20 20 - PID:

0038: 30 30 30 30 37 39 36 30 00007960

0040: 2d 20 54 49 44 3a 20 20 - TID:

0048: 30 30 30 30 36 35 33 36 00006536

0050: 2d 20 43 4d 44 3a 20 20 - CMD:

0058: 43 3a 5c 57 49 4e 44 4f C:\WINDO

0060: 57 53 5c 53 79 73 74 65 WS\Syste

0068: 6d 33 32 5c 76 73 73 76 m32\vssv

0070: 63 2e 65 78 65 20 20 20 c.exe

0078: 2d 20 55 73 65 72 3a 20 - User:

0080: 4e 54 20 41 55 54 48 4f NT AUTHO

0088: 52 49 54 59 5c 53 59 53 RITY\SYS

0090: 54 45 4d 20 20 20 20 20 TEM

0098: 2d 20 53 69 64 3a 20 20 - Sid:

00a0: 53 2d 31 2d 35 2d 31 38 S-1-5-18

00a8: 2d 20 45 72 72 6f 72 3a - Error:

00b0: 57 61 69 74 46 6f 72 4d WaitForM

00b8: 75 6c 74 69 70 6c 65 4f ultipleO

00c0: 62 6a 65 63 74 73 28 31 bjects(1

00c8: 2c 30 30 30 30 30 30 30 ,0000000

00d0: 30 30 30 33 32 41 32 42 00032A2B

00d8: 30 2c 31 2c 31 32 30 30 0,1,1200

00e0: 30 30 29 20 3d 3d 20 30 00) == 0

00e8: 78 30 30 30 30 30 31 30 x0000010

00f0: 32 20 20 20 20 20 20 20 2

------------------------------------------------------

Event Type: Warning

Event Source: MSExchange EdgeSync

Event Category: Topology

Event ID: 1026

Date: 4/30/2008

Time: 7:10:06 AM

User: N/A

Computer: SERVER

Description:

Topology load generated transient exception Active Directory operation

failed on server.hagelaw.local. This error could have been caused by user

input or by the Active Directory server being unavailable. Please retry at a

later time. Additional information: Active directory response: The operation

was aborted because the client side timeout limit was exceeded. .. EdgeSync

must be able to acquire current Exchange topology data from the Active

Directory directory service to properly operate. The Edge Transport server

will attempt to load the topology again.

 

For more information, see Help and Support Center at

http://go.microsoft.com/fwlink/events.asp.

-----------------------------------------------------------------

Event Type: Error

Event Source: MSExchange ADAccess

Event Category: General

Event ID: 2604

Date: 4/30/2008

Time: 7:12:07 AM

User: N/A

Computer: SERVER

Description:

Process MSEXCHANGEADTOPOLOGY (PID=2500). When updating security for a remote

procedure call (RPC) access for the Exchange Active Directory Topology

service, Exchange could not retrieve the security descriptor for Exchange

server object SERVER - Error code=80040955.

The Exchange Active Directory Topology service will continue with limited

permissions.

 

For more information, see Help and Support Center at

http://go.microsoft.com/fwlink/events.asp.

-------------------------------------------------------------------------

Event Type: Error

Event Source: MSExchange System Attendant Mailbox

Event Category: General

Event ID: 4001

Date: 4/30/2008

Time: 7:15:35 AM

User: N/A

Computer: SERVER

Description:

A transient failure has occurred. The problem may resolve itself in awhile.

The service will retry in 56 seconds. Diagnostic information:

 

Active Directory operation failed on server.hagelaw.local. This error could

have been caused by user input or by the Active Directory server being

unavailable. Please retry at a later time. Additional information: Active

directory response: The operation was aborted because the client side timeout

limit was exceeded. .

Microsoft.Exchange.Data.Directory.ADPossibleOperationException: Active

Directory operation failed on server.hagelaw.local. This error could have

been caused by user input or by the Active Directory server being

unavailable. Please retry at a later time. Additional information: Active

directory response: The operation was aborted because the client side timeout

limit was exceeded. . ---> System.DirectoryServices.Protocols.LdapException:

The operation was aborted because the client side timeout limit was exceeded.

at

System.DirectoryServices.Protocols.LdapConnection.ConstructResponse(Int32

messageId, LdapOperation operation, ResultAll resultType, TimeSpan

requestTimeOut, Boolean exceptionOnTimeOut)

at

System.DirectoryServices.Protocols.LdapConnection.SendRequest(DirectoryRequest request, TimeSpan requestTimeout)

at

Microsoft.Exchange.Data.Directory.PooledLdapConnection.SendRequest(DirectoryRequest request, LdapOperation ldapOperation)

at Microsoft.Exchange.Data.Directory.ADSession.Find(ADObjectId rootId,

String optionalBaseDN, ADObjectId readId, QueryScope scope, QueryFilter

filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties,

CreateObjectDelegate objectCreator, CreateObjectsDelegate arrayCreator)

--- End of inner exception stack trace ---

at

Microsoft.Exchange.Data.Directory.ADSession.AnalyzeDirectoryError(PooledLdapConnection

connection, DirectoryRequest request, DirectoryException de, Int32

totalRetries, Int32 retriesOnServer)

at Microsoft.Exchange.Data.Directory.ADSession.Find(ADObjectId rootId,

String optionalBaseDN, ADObjectId readId, QueryScope scope, QueryFilter

filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties,

CreateObjectDelegate objectCreator, CreateObjectsDelegate arrayCreator)

at Microsoft.Exchange.Data.Directory.ADSession.Find(ADObjectId rootId,

QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults,

IEnumerable`1 properties, CreateObjectDelegate objectCtor,

CreateObjectsDelegate arrayCtor)

at Microsoft.Exchange.Data.Directory.ADSession.Find[TResult](ADObjectId

rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32

maxResults, IEnumerable`1 properties)

at

Microsoft.Exchange.Data.Directory.SystemConfiguration.ADSystemConfigurationSession.Find[TResult](ADObjectId

rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults)

at

Microsoft.Exchange.Data.Directory.SystemConfiguration.ADSystemConfigurationSession.FindServerByFqdn(String serverFqdn)

at

Microsoft.Exchange.Data.Directory.SystemConfiguration.ADSystemConfigurationSession.FindLocalServer()

at Microsoft.Exchange.Servicelets.SystemAttendantMailbox.Servicelet.Work()

 

For more information, see Help and Support Center at

http://go.microsoft.com/fwlink/events.asp.

---------------------------------------------------------------------------

Event Type: Error

Event Source: VSS

Event Category: None

Event ID: 7001

Date: 4/30/2008

Time: 7:15:35 AM

User: N/A

Computer: SERVER

Description:

VssAdmin: Unable to create a shadow copy: Catastrophic failure

Command-line: 'C:\WINDOWS\system32\vssadmin.exe Create Shadow /AutoRetry=5

/For=\\?\Volume{f2440b14-f04b-11dc-8efa-806e6f6e6963}\'.

 

For more information, see Help and Support Center at

http://go.microsoft.com/fwlink/events.asp.

Data:

0000: 2d 20 43 6f 64 65 3a 20 - Code:

0008: 41 44 4d 56 41 44 4d 43 ADMVADMC

0010: 30 30 30 30 30 38 33 37 00000837

0018: 2d 20 43 61 6c 6c 3a 20 - Call:

0020: 41 44 4d 56 41 44 4d 43 ADMVADMC

0028: 30 30 30 30 30 37 36 38 00000768

0030: 2d 20 50 49 44 3a 20 20 - PID:

0038: 30 30 30 30 36 37 31 36 00006716

0040: 2d 20 54 49 44 3a 20 20 - TID:

0048: 30 30 30 30 36 39 38 30 00006980

0050: 2d 20 43 4d 44 3a 20 20 - CMD:

0058: 43 3a 5c 57 49 4e 44 4f C:\WINDO

0060: 57 53 5c 73 79 73 74 65 WS\syste

0068: 6d 33 32 5c 76 73 73 61 m32\vssa

0070: 64 6d 69 6e 2e 65 78 65 dmin.exe

0078: 20 43 72 65 61 74 65 20 Create

0080: 53 68 61 64 6f 77 20 2f Shadow /

0088: 41 75 74 6f 52 65 74 72 AutoRetr

0090: 79 3d 35 20 2f 46 6f 72 y=5 /For

0098: 3d 5c 5c 3f 5c 56 6f 6c =\\?\Vol

00a0: 75 6d 65 7b 66 32 34 34 ume{f244

00a8: 30 62 31 34 2d 66 30 34 0b14-f04

00b0: 62 2d 31 31 64 63 2d 38 b-11dc-8

00b8: 65 66 61 2d 38 30 36 65 efa-806e

00c0: 36 66 36 65 36 39 36 33 6f6e6963

00c8: 7d 5c 20 20 20 20 20 20 }\

00d0: 2d 20 55 73 65 72 3a 20 - User:

00d8: 4e 54 20 41 55 54 48 4f NT AUTHO

00e0: 52 49 54 59 5c 53 59 53 RITY\SYS

00e8: 54 45 4d 20 20 20 20 20 TEM

00f0: 2d 20 53 69 64 3a 20 20 - Sid:

00f8: 53 2d 31 2d 35 2d 31 38 S-1-5-18

----------------------------------------------------------------

After this string of events, event logs return to normal.

 

This is the same string of events that are logged after every failed snapshot.

 

After a failed snapshot, I checked the writer status and they're all normal.

None say failed.

 

Any ideas on what to do? Thanks.