Performance Testing Using ACT

I'm using MS ACT to test RS URL access performance but the results
were very confusing...
The parameters I used in ACT:
- 500 simultaneous browser connections
- .dat file containing 500 reference IDs to be append to the URL
- 1 iteration to run the test
My script contains a procedure that will execute 500 times the same as
the available IDs in the .dat file:
============================================================================ Sub SendRequest(strID)
Dim oConnection, oRequest, oResponse, oHeaders, strStatusCode,
fEnableDelays
If fEnableDelays = True then Test.Sleep (0)
Set oConnection = Test.CreateConnection("abc.xyz.com", 80, false)
If (oConnection is Nothing) Then
Test.Trace "Error: Unable to create connection to abc.xyz.com"
Else
Set oRequest = Test.CreateRequest
oRequest.Path = "/ReportServer"+"?/ReportName&rs:Command=Render&rs:format=html4.0&rc:Toolbar=false&ID="+strID
oRequest.Verb = "GET"
oRequest.HTTPVersion = "HTTP/1.1"
set oHeaders = oRequest.Headers
oHeaders.RemoveAll
oHeaders.Add "Accept", "image/gif, image/x-xbitmap,
image/jpeg, image/pjpeg, application/vnd.ms-powerpoint,
application/vnd.ms-excel, application/msword,
application/x-shockwave-flash, */*"
oHeaders.Add "Accept-Language", "en-us"
ASP.NET_SessionId=jdhndgqnhrfgkuzpx53lcbaf;
%2fReportName=5edhrh55zoydow45aeasfj55"
oHeaders.Add "Cookie", "(automatic)"
oHeaders.Add "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0;
Windows NT 5.0; .NET CLR 1.1.4322)"
oHeaders.Add "Host", "abc.xyz.com"
oHeaders.Add "Authorization", "Basic
dG9tbXl0XGFkbWluaXN0cmF0b3I6"
oHeaders.Add "Cookie", "(automatic)"
Set oResponse = oConnection.Send(oRequest)
oConnection.Close
End If
End Sub
============================================================================ The result of the testing shows:
============================================================================ Test Name: ACTSamples: RS Performance Test 3
Test Run Name: report-RS Performance Test 3-Aug 24, 2004 03-48-35
Test Started: 8/24/2004 3:38:57 AM
Test Duration: 00:00:09:37
Test Iterations: 1
Test Notes: -
Properties
Test type: Dynamic
Simultaneous browser connections: 500
Warm up time (secs): 0
Test duration: 00:00:09:37
Test iterations: 1
Detailed test results generated: Yes
Summary
Total number of requests: 500
Total number of connections: 500
Average requests per second: 0.87
Average time to first byte (msecs): 492.48
Average time to last byte (msecs): 1,111.46
Average time to last byte per iteration (msecs): 555,728.00
Number of unique requests made in test: 1
Number of unique response codes: 1
Errors Counts
HTTP: 0
DNS: 0
Socket: 0
Additional Network Statistics
Average bandwidth (bytes/sec): 107,079.37
Number of bytes sent (bytes): 272,940
Number of bytes received (bytes): 61,511,854
Average rate of sent bytes (bytes/sec): 473.03
Average rate of received bytes (bytes/sec): 106,606.33
Number of connection errors: 0
Number of send errors: 0
Number of receive errors: 0
Number of timeout errors: 0
Response Codes
Response Code: 200 - The request completed successfully.
Count: 500
Percent (%): 100.00
============================================================================ Issues:
============================================================================ 1. When I run the script, monitoring the RPS of ACT, it never goes
more than 2 (not multi-thread? flow control?)
2. 500 users took about 10 mins. to complete (same issues as item 1)
3. Checking the RS Web Service Performance monitor, 3 suspicious
counter values were:
a. Total Processing Failure = 5120 (why it is so high?)
b. Report Executed / Sec = 1 (never goes more than 1 during the
entire testing, threading problem?)
c. Request / Sec = = 1 (never goes more than 1 during the entire
testing)
4. All 500 URL requests were logged into ExecutionLog of RS Server and
tagged as "rsSuccess" (ACT was able to send all requests to RS and
have them processed. but it was logged into the table 1 by 1 until all
requests were processed. Meaning you will receive your 500th report
after "n sec/report x 500 reports"?)
5. Why Number of unique requests made in test = 1 while URLs were
distinct because of the ID parameter?
I really wonder why RS behaves this way, it doesn't seems to be
multi-thread. In RSReportServer.config, MaxActiveReqForOneUser was set
to 0 to ensure maximum number of users can be handled but still it
doesn't help.
I don't think DoS attack protection features/setting of RS is a valid
reason for the system to behave like that. If so, then the problem
should be with ACT who issues 500 concurrent connections from the same
workstation? Something must be wrong.
My question:
RS URL access cannot support more than 500 users? If not, why? If yes,
can anyone tell me how to perform a correct performance test? Thanks!
CCCsome answers inline
--
Tudor Trufinescu
Dev Lead
Sql Server Reporting Services
This posting is provided "AS IS" with no warranties, and confers no rights.
"CCC" <tommytan25@.yahoo.com> wrote in message
news:ab9130f9.0408240040.3c25f0d4@.posting.google.com...
> I'm using MS ACT to test RS URL access performance but the results
> were very confusing...
> The parameters I used in ACT:
> - 500 simultaneous browser connections
> - .dat file containing 500 reference IDs to be append to the URL
> - 1 iteration to run the test
> My script contains a procedure that will execute 500 times the same as
> the available IDs in the .dat file:
>
============================================================================> Sub SendRequest(strID)
> Dim oConnection, oRequest, oResponse, oHeaders, strStatusCode,
> fEnableDelays
> If fEnableDelays = True then Test.Sleep (0)
> Set oConnection = Test.CreateConnection("abc.xyz.com", 80, false)
> If (oConnection is Nothing) Then
> Test.Trace "Error: Unable to create connection to abc.xyz.com"
> Else
> Set oRequest = Test.CreateRequest
> oRequest.Path =>
"/ReportServer"+"?/ReportName&rs:Command=Render&rs:format=html4.0&rc:Toolbar
=false&ID="+strID
> oRequest.Verb = "GET"
> oRequest.HTTPVersion = "HTTP/1.1"
> set oHeaders = oRequest.Headers
> oHeaders.RemoveAll
> oHeaders.Add "Accept", "image/gif, image/x-xbitmap,
> image/jpeg, image/pjpeg, application/vnd.ms-powerpoint,
> application/vnd.ms-excel, application/msword,
> application/x-shockwave-flash, */*"
> oHeaders.Add "Accept-Language", "en-us"
> ASP.NET_SessionId=jdhndgqnhrfgkuzpx53lcbaf;
> %2fReportName=5edhrh55zoydow45aeasfj55"
> oHeaders.Add "Cookie", "(automatic)"
> oHeaders.Add "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0;
> Windows NT 5.0; .NET CLR 1.1.4322)"
> oHeaders.Add "Host", "abc.xyz.com"
> oHeaders.Add "Authorization", "Basic
> dG9tbXl0XGFkbWluaXN0cmF0b3I6"
> oHeaders.Add "Cookie", "(automatic)"
> Set oResponse = oConnection.Send(oRequest)
> oConnection.Close
> End If
> End Sub
>
>
============================================================================> The result of the testing shows:
>
============================================================================> Test Name: ACTSamples: RS Performance Test 3
> Test Run Name: report-RS Performance Test 3-Aug 24, 2004 03-48-35
> Test Started: 8/24/2004 3:38:57 AM
> Test Duration: 00:00:09:37
> Test Iterations: 1
> Test Notes: -
>
> Properties
> Test type: Dynamic
> Simultaneous browser connections: 500
> Warm up time (secs): 0
> Test duration: 00:00:09:37
> Test iterations: 1
> Detailed test results generated: Yes
> Summary
> Total number of requests: 500
> Total number of connections: 500
> Average requests per second: 0.87
> Average time to first byte (msecs): 492.48
> Average time to last byte (msecs): 1,111.46
> Average time to last byte per iteration (msecs): 555,728.00
> Number of unique requests made in test: 1
> Number of unique response codes: 1
> Errors Counts
> HTTP: 0
> DNS: 0
> Socket: 0
> Additional Network Statistics
> Average bandwidth (bytes/sec): 107,079.37
> Number of bytes sent (bytes): 272,940
> Number of bytes received (bytes): 61,511,854
> Average rate of sent bytes (bytes/sec): 473.03
> Average rate of received bytes (bytes/sec): 106,606.33
> Number of connection errors: 0
> Number of send errors: 0
> Number of receive errors: 0
> Number of timeout errors: 0
> Response Codes
> Response Code: 200 - The request completed successfully.
> Count: 500
> Percent (%): 100.00
>
============================================================================> Issues:
>
============================================================================
*** Tudor: this shows a report is returned in ~ 1 sec on average, so the
number of request per second match what you see in ACT, assuming your server
has 1 or 2 processors. This is not about flow control, it simply takes that
much to generate the reports in this suite. The number of concurrent users
the system supports depends on how long a report generation takes.
If you would running all 500 requests at the same time, it would probably
trash the system from all the context switches.
***
> 1. When I run the script, monitoring the RPS of ACT, it never goes
> more than 2 (not multi-thread? flow control?)
> 2. 500 users took about 10 mins. to complete (same issues as item 1)
> 3. Checking the RS Web Service Performance monitor, 3 suspicious
> counter values were:
> a. Total Processing Failure = 5120 (why it is so high?)
***
Tudor: I believe this is an additive counter, so its value is the total
since the report server started. There may also be cases in which we wrongly
increment this for warnings. The execution log and ACT should give you the
correct data here.
***
> b. Report Executed / Sec = 1 (never goes more than 1 during the
> entire testing, threading problem?)
> c. Request / Sec = = 1 (never goes more than 1 during the entire
> testing)
> 4. All 500 URL requests were logged into ExecutionLog of RS Server and
> tagged as "rsSuccess" (ACT was able to send all requests to RS and
> have them processed. but it was logged into the table 1 by 1 until all
> requests were processed. Meaning you will receive your 500th report
> after "n sec/report x 500 reports"?)
> 5. Why Number of unique requests made in test = 1 while URLs were
> distinct because of the ID parameter?
> I really wonder why RS behaves this way, it doesn't seems to be
> multi-thread. In RSReportServer.config, MaxActiveReqForOneUser was set
> to 0 to ensure maximum number of users can be handled but still it
> doesn't help.
***
Tudor: You should change MaxActiveReqForOneUser to 500 or a large number I
don't think 0 means unbounded.
***
> I don't think DoS attack protection features/setting of RS is a valid
> reason for the system to behave like that. If so, then the problem
> should be with ACT who issues 500 concurrent connections from the same
> workstation? Something must be wrong.
> My question:
> RS URL access cannot support more than 500 users? If not, why? If yes,
> can anyone tell me how to perform a correct performance test? Thanks!
> CCC|||Don't trust ACT's RPS - instead use ASP.NET's RPS and Requests Executing value. What do they show?
Also, I don't believe ACT factors in the QueryString when determining # of unique requests.
--
Thanks.
Donovan R. Smith
Software Test Lead
This posting is provided "AS IS" with no warranties, and confers no rights.
"CCC" <tommytan25@.yahoo.com> wrote in message
news:ab9130f9.0408240040.3c25f0d4@.posting.google.com...
> I'm using MS ACT to test RS URL access performance but the results
> were very confusing...
> The parameters I used in ACT:
> - 500 simultaneous browser connections
> - .dat file containing 500 reference IDs to be append to the URL
> - 1 iteration to run the test
> My script contains a procedure that will execute 500 times the same as
> the available IDs in the .dat file:
> ============================================================================> Sub SendRequest(strID)
> Dim oConnection, oRequest, oResponse, oHeaders, strStatusCode,
> fEnableDelays
> If fEnableDelays = True then Test.Sleep (0)
> Set oConnection = Test.CreateConnection("abc.xyz.com", 80, false)
> If (oConnection is Nothing) Then
> Test.Trace "Error: Unable to create connection to abc.xyz.com"
> Else
> Set oRequest = Test.CreateRequest
> oRequest.Path => "/ReportServer"+"?/ReportName&rs:Command=Render&rs:format=html4.0&rc:Toolbar=false&ID="+strID
> oRequest.Verb = "GET"
> oRequest.HTTPVersion = "HTTP/1.1"
> set oHeaders = oRequest.Headers
> oHeaders.RemoveAll
> oHeaders.Add "Accept", "image/gif, image/x-xbitmap,
> image/jpeg, image/pjpeg, application/vnd.ms-powerpoint,
> application/vnd.ms-excel, application/msword,
> application/x-shockwave-flash, */*"
> oHeaders.Add "Accept-Language", "en-us"
> ASP.NET_SessionId=jdhndgqnhrfgkuzpx53lcbaf;
> %2fReportName=5edhrh55zoydow45aeasfj55"
> oHeaders.Add "Cookie", "(automatic)"
> oHeaders.Add "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0;
> Windows NT 5.0; .NET CLR 1.1.4322)"
> oHeaders.Add "Host", "abc.xyz.com"
> oHeaders.Add "Authorization", "Basic
> dG9tbXl0XGFkbWluaXN0cmF0b3I6"
> oHeaders.Add "Cookie", "(automatic)"
> Set oResponse = oConnection.Send(oRequest)
> oConnection.Close
> End If
> End Sub
>
> ============================================================================> The result of the testing shows:
> ============================================================================> Test Name: ACTSamples: RS Performance Test 3
> Test Run Name: report-RS Performance Test 3-Aug 24, 2004 03-48-35
> Test Started: 8/24/2004 3:38:57 AM
> Test Duration: 00:00:09:37
> Test Iterations: 1
> Test Notes: -
>
> Properties
> Test type: Dynamic
> Simultaneous browser connections: 500
> Warm up time (secs): 0
> Test duration: 00:00:09:37
> Test iterations: 1
> Detailed test results generated: Yes
> Summary
> Total number of requests: 500
> Total number of connections: 500
> Average requests per second: 0.87
> Average time to first byte (msecs): 492.48
> Average time to last byte (msecs): 1,111.46
> Average time to last byte per iteration (msecs): 555,728.00
> Number of unique requests made in test: 1
> Number of unique response codes: 1
> Errors Counts
> HTTP: 0
> DNS: 0
> Socket: 0
> Additional Network Statistics
> Average bandwidth (bytes/sec): 107,079.37
> Number of bytes sent (bytes): 272,940
> Number of bytes received (bytes): 61,511,854
> Average rate of sent bytes (bytes/sec): 473.03
> Average rate of received bytes (bytes/sec): 106,606.33
> Number of connection errors: 0
> Number of send errors: 0
> Number of receive errors: 0
> Number of timeout errors: 0
> Response Codes
> Response Code: 200 - The request completed successfully.
> Count: 500
> Percent (%): 100.00
> ============================================================================> Issues:
> ============================================================================> 1. When I run the script, monitoring the RPS of ACT, it never goes
> more than 2 (not multi-thread? flow control?)
> 2. 500 users took about 10 mins. to complete (same issues as item 1)
> 3. Checking the RS Web Service Performance monitor, 3 suspicious
> counter values were:
> a. Total Processing Failure = 5120 (why it is so high?)
> b. Report Executed / Sec = 1 (never goes more than 1 during the
> entire testing, threading problem?)
> c. Request / Sec = = 1 (never goes more than 1 during the entire
> testing)
> 4. All 500 URL requests were logged into ExecutionLog of RS Server and
> tagged as "rsSuccess" (ACT was able to send all requests to RS and
> have them processed. but it was logged into the table 1 by 1 until all
> requests were processed. Meaning you will receive your 500th report
> after "n sec/report x 500 reports"?)
> 5. Why Number of unique requests made in test = 1 while URLs were
> distinct because of the ID parameter?
> I really wonder why RS behaves this way, it doesn't seems to be
> multi-thread. In RSReportServer.config, MaxActiveReqForOneUser was set
> to 0 to ensure maximum number of users can be handled but still it
> doesn't help.
> I don't think DoS attack protection features/setting of RS is a valid
> reason for the system to behave like that. If so, then the problem
> should be with ACT who issues 500 concurrent connections from the same
> workstation? Something must be wrong.
> My question:
> RS URL access cannot support more than 500 users? If not, why? If yes,
> can anyone tell me how to perform a correct performance test? Thanks!
> CCC|||Hi Tudor,
Thank you very much for your comments...
Are you saying the RS's URL access cannot handle 500 users accessing
their own report at the same time? I'm not quite clear with the
dependencies of concurrent users and report generation time, mind to
clarify further? Are you saying it is normal for 500th user to get
their report after first 499 reports were generated?
On Total Processing Failure, I restarted all counters before running a
test therefore I'm certain that this counter was generated in 1 test
and not additive of multiple tests. RS Web Service counters is what MS
included in the suite and the counter is unique from the data
presented in ExecutionLog and ACT.
I'm also wondering why give there are a lot of processing failures, RS
can still generate the report correctly (except too slow) and logs
them into ExecutionLog and ACT. What are the other possible cause of
high processing failure aside from "originate from the report
processor or any extension" as stated in RS documentations? Also, what
are the cause of processing failure in report processor and other
extensions?
On MaxActiveReqForOneUser, please see
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/rsadmin/htm/arp_configfiles_v1_9y5i.asp
something must be wrong here, either msdn documentation or... :)
CCC
"Tudor Trufinescu \(MSFT\)" <tudortr@.ms.com> wrote in message news:<Opy2QDiiEHA.1104@.TK2MSFTNGP10.phx.gbl>...
> some answers inline
> --
> Tudor Trufinescu
> Dev Lead
> Sql Server Reporting Services
> This posting is provided "AS IS" with no warranties, and confers no rights.
>
> "CCC" <tommytan25@.yahoo.com> wrote in message
> news:ab9130f9.0408240040.3c25f0d4@.posting.google.com...
> > I'm using MS ACT to test RS URL access performance but the results
> > were very confusing...
> >
> > The parameters I used in ACT:
> > - 500 simultaneous browser connections
> > - .dat file containing 500 reference IDs to be append to the URL
> > - 1 iteration to run the test
> >
> > My script contains a procedure that will execute 500 times the same as
> > the available IDs in the .dat file:
> >
> ============================================================================> > Sub SendRequest(strID)
> >
> > Dim oConnection, oRequest, oResponse, oHeaders, strStatusCode,
> > fEnableDelays
> >
> > If fEnableDelays = True then Test.Sleep (0)
> > Set oConnection = Test.CreateConnection("abc.xyz.com", 80, false)
> > If (oConnection is Nothing) Then
> > Test.Trace "Error: Unable to create connection to abc.xyz.com"
> > Else
> > Set oRequest = Test.CreateRequest
> >
> > oRequest.Path => >
> "/ReportServer"+"?/ReportName&rs:Command=Render&rs:format=html4.0&rc:Toolbar
> =false&ID="+strID
> > oRequest.Verb = "GET"
> > oRequest.HTTPVersion = "HTTP/1.1"
> > set oHeaders = oRequest.Headers
> > oHeaders.RemoveAll
> > oHeaders.Add "Accept", "image/gif, image/x-xbitmap,
> > image/jpeg, image/pjpeg, application/vnd.ms-powerpoint,
> > application/vnd.ms-excel, application/msword,
> > application/x-shockwave-flash, */*"
> > oHeaders.Add "Accept-Language", "en-us"
> > ASP.NET_SessionId=jdhndgqnhrfgkuzpx53lcbaf;
> > %2fReportName=5edhrh55zoydow45aeasfj55"
> > oHeaders.Add "Cookie", "(automatic)"
> > oHeaders.Add "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0;
> > Windows NT 5.0; .NET CLR 1.1.4322)"
> > oHeaders.Add "Host", "abc.xyz.com"
> > oHeaders.Add "Authorization", "Basic
> > dG9tbXl0XGFkbWluaXN0cmF0b3I6"
> > oHeaders.Add "Cookie", "(automatic)"
> > Set oResponse = oConnection.Send(oRequest)
> > oConnection.Close
> > End If
> >
> > End Sub
> >
> >
> >
> ============================================================================> > The result of the testing shows:
> >
> ============================================================================> > Test Name: ACTSamples: RS Performance Test 3
> > Test Run Name: report-RS Performance Test 3-Aug 24, 2004 03-48-35
> > Test Started: 8/24/2004 3:38:57 AM
> > Test Duration: 00:00:09:37
> > Test Iterations: 1
> > Test Notes: -
> >
> >
> > Properties
> >
> > Test type: Dynamic
> > Simultaneous browser connections: 500
> > Warm up time (secs): 0
> > Test duration: 00:00:09:37
> > Test iterations: 1
> > Detailed test results generated: Yes
> >
> > Summary
> >
> > Total number of requests: 500
> > Total number of connections: 500
> >
> > Average requests per second: 0.87
> > Average time to first byte (msecs): 492.48
> > Average time to last byte (msecs): 1,111.46
> > Average time to last byte per iteration (msecs): 555,728.00
> >
> > Number of unique requests made in test: 1
> > Number of unique response codes: 1
> >
> > Errors Counts
> >
> > HTTP: 0
> > DNS: 0
> > Socket: 0
> >
> > Additional Network Statistics
> >
> > Average bandwidth (bytes/sec): 107,079.37
> >
> > Number of bytes sent (bytes): 272,940
> > Number of bytes received (bytes): 61,511,854
> >
> > Average rate of sent bytes (bytes/sec): 473.03
> > Average rate of received bytes (bytes/sec): 106,606.33
> >
> > Number of connection errors: 0
> > Number of send errors: 0
> > Number of receive errors: 0
> > Number of timeout errors: 0
> >
> > Response Codes
> >
> > Response Code: 200 - The request completed successfully.
> > Count: 500
> > Percent (%): 100.00
> >
> >
> ============================================================================> > Issues:
> >
> ============================================================================> *** Tudor: this shows a report is returned in ~ 1 sec on average, so the
> number of request per second match what you see in ACT, assuming your server
> has 1 or 2 processors. This is not about flow control, it simply takes that
> much to generate the reports in this suite. The number of concurrent users
> the system supports depends on how long a report generation takes.
> If you would running all 500 requests at the same time, it would probably
> trash the system from all the context switches.
> ***
> > 1. When I run the script, monitoring the RPS of ACT, it never goes
> > more than 2 (not multi-thread? flow control?)
> > 2. 500 users took about 10 mins. to complete (same issues as item 1)
> > 3. Checking the RS Web Service Performance monitor, 3 suspicious
> > counter values were:
> > a. Total Processing Failure = 5120 (why it is so high?)
> ***
> Tudor: I believe this is an additive counter, so its value is the total
> since the report server started. There may also be cases in which we wrongly
> increment this for warnings. The execution log and ACT should give you the
> correct data here.
> ***
> > b. Report Executed / Sec = 1 (never goes more than 1 during the
> > entire testing, threading problem?)
> > c. Request / Sec = = 1 (never goes more than 1 during the entire
> > testing)
> > 4. All 500 URL requests were logged into ExecutionLog of RS Server and
> > tagged as "rsSuccess" (ACT was able to send all requests to RS and
> > have them processed. but it was logged into the table 1 by 1 until all
> > requests were processed. Meaning you will receive your 500th report
> > after "n sec/report x 500 reports"?)
> > 5. Why Number of unique requests made in test = 1 while URLs were
> > distinct because of the ID parameter?
> >
> > I really wonder why RS behaves this way, it doesn't seems to be
> > multi-thread. In RSReportServer.config, MaxActiveReqForOneUser was set
> > to 0 to ensure maximum number of users can be handled but still it
> > doesn't help.
> ***
> Tudor: You should change MaxActiveReqForOneUser to 500 or a large number I
> don't think 0 means unbounded.
> ***
> >
> > I don't think DoS attack protection features/setting of RS is a valid
> > reason for the system to behave like that. If so, then the problem
> > should be with ACT who issues 500 concurrent connections from the same
> > workstation? Something must be wrong.
> >
> > My question:
> > RS URL access cannot support more than 500 users? If not, why? If yes,
> > can anyone tell me how to perform a correct performance test? Thanks!
> >
> > CCC|||> Are you saying the RS's URL access cannot handle 500 users accessing
> their own report at the same time?
No, I am just saying that the number of requests/sec the server supports
depends on the resources available on the server (processors, memory, disk)
and the report complexity. From your results it looks like the server can
sustain 1 reports/sec in this configuration and this type of load. Also from
ACT results, your 500 users will get their report back in ~ 1 sec on average
which is a relatively good response time. If you want to support a higher
load, try simpler reports or use RS caching your number of reports per
second will increase accordingly.
I would not worry about the report errors counter for now, the execution log
has the right data in it.
The documentation for MaxRequestsForOneUser seems wrong, thanks for pointing
that out.
--
Tudor Trufinescu
Dev Lead
Sql Server Reporting Services
This posting is provided "AS IS" with no warranties, and confers no rights.
"CCC" <tommytan25@.yahoo.com> wrote in message
news:ab9130f9.0408241848.3bf408e2@.posting.google.com...
> Hi Tudor,
> Thank you very much for your comments...
> Are you saying the RS's URL access cannot handle 500 users accessing
> their own report at the same time? I'm not quite clear with the
> dependencies of concurrent users and report generation time, mind to
> clarify further? Are you saying it is normal for 500th user to get
> their report after first 499 reports were generated?
> On Total Processing Failure, I restarted all counters before running a
> test therefore I'm certain that this counter was generated in 1 test
> and not additive of multiple tests. RS Web Service counters is what MS
> included in the suite and the counter is unique from the data
> presented in ExecutionLog and ACT.
> I'm also wondering why give there are a lot of processing failures, RS
> can still generate the report correctly (except too slow) and logs
> them into ExecutionLog and ACT. What are the other possible cause of
> high processing failure aside from "originate from the report
> processor or any extension" as stated in RS documentations? Also, what
> are the cause of processing failure in report processor and other
> extensions?
> On MaxActiveReqForOneUser, please see
>
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/rsadmin/htm/arp_configfiles_v1_9y5i.asp
> something must be wrong here, either msdn documentation or... :)
> CCC
> "Tudor Trufinescu \(MSFT\)" <tudortr@.ms.com> wrote in message
news:<Opy2QDiiEHA.1104@.TK2MSFTNGP10.phx.gbl>...
> > some answers inline
> >
> > --
> > Tudor Trufinescu
> > Dev Lead
> > Sql Server Reporting Services
> > This posting is provided "AS IS" with no warranties, and confers no
rights.
> >
> >
> > "CCC" <tommytan25@.yahoo.com> wrote in message
> > news:ab9130f9.0408240040.3c25f0d4@.posting.google.com...
> > > I'm using MS ACT to test RS URL access performance but the results
> > > were very confusing...
> > >
> > > The parameters I used in ACT:
> > > - 500 simultaneous browser connections
> > > - .dat file containing 500 reference IDs to be append to the URL
> > > - 1 iteration to run the test
> > >
> > > My script contains a procedure that will execute 500 times the same as
> > > the available IDs in the .dat file:
> > >
> >
============================================================================> > > Sub SendRequest(strID)
> > >
> > > Dim oConnection, oRequest, oResponse, oHeaders, strStatusCode,
> > > fEnableDelays
> > >
> > > If fEnableDelays = True then Test.Sleep (0)
> > > Set oConnection = Test.CreateConnection("abc.xyz.com", 80, false)
> > > If (oConnection is Nothing) Then
> > > Test.Trace "Error: Unable to create connection to abc.xyz.com"
> > > Else
> > > Set oRequest = Test.CreateRequest
> > >
> > > oRequest.Path => > >
> >
"/ReportServer"+"?/ReportName&rs:Command=Render&rs:format=html4.0&rc:Toolbar
> > =false&ID="+strID
> > > oRequest.Verb = "GET"
> > > oRequest.HTTPVersion = "HTTP/1.1"
> > > set oHeaders = oRequest.Headers
> > > oHeaders.RemoveAll
> > > oHeaders.Add "Accept", "image/gif, image/x-xbitmap,
> > > image/jpeg, image/pjpeg, application/vnd.ms-powerpoint,
> > > application/vnd.ms-excel, application/msword,
> > > application/x-shockwave-flash, */*"
> > > oHeaders.Add "Accept-Language", "en-us"
> > > ASP.NET_SessionId=jdhndgqnhrfgkuzpx53lcbaf;
> > > %2fReportName=5edhrh55zoydow45aeasfj55"
> > > oHeaders.Add "Cookie", "(automatic)"
> > > oHeaders.Add "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0;
> > > Windows NT 5.0; .NET CLR 1.1.4322)"
> > > oHeaders.Add "Host", "abc.xyz.com"
> > > oHeaders.Add "Authorization", "Basic
> > > dG9tbXl0XGFkbWluaXN0cmF0b3I6"
> > > oHeaders.Add "Cookie", "(automatic)"
> > > Set oResponse = oConnection.Send(oRequest)
> > > oConnection.Close
> > > End If
> > >
> > > End Sub
> > >
> > >
> > >
> >
============================================================================> > > The result of the testing shows:
> > >
> >
============================================================================> > > Test Name: ACTSamples: RS Performance Test 3
> > > Test Run Name: report-RS Performance Test 3-Aug 24, 2004 03-48-35
> > > Test Started: 8/24/2004 3:38:57 AM
> > > Test Duration: 00:00:09:37
> > > Test Iterations: 1
> > > Test Notes: -
> > >
> > >
> > > Properties
> > >
> > > Test type: Dynamic
> > > Simultaneous browser connections: 500
> > > Warm up time (secs): 0
> > > Test duration: 00:00:09:37
> > > Test iterations: 1
> > > Detailed test results generated: Yes
> > >
> > > Summary
> > >
> > > Total number of requests: 500
> > > Total number of connections: 500
> > >
> > > Average requests per second: 0.87
> > > Average time to first byte (msecs): 492.48
> > > Average time to last byte (msecs): 1,111.46
> > > Average time to last byte per iteration (msecs): 555,728.00
> > >
> > > Number of unique requests made in test: 1
> > > Number of unique response codes: 1
> > >
> > > Errors Counts
> > >
> > > HTTP: 0
> > > DNS: 0
> > > Socket: 0
> > >
> > > Additional Network Statistics
> > >
> > > Average bandwidth (bytes/sec): 107,079.37
> > >
> > > Number of bytes sent (bytes): 272,940
> > > Number of bytes received (bytes): 61,511,854
> > >
> > > Average rate of sent bytes (bytes/sec): 473.03
> > > Average rate of received bytes (bytes/sec): 106,606.33
> > >
> > > Number of connection errors: 0
> > > Number of send errors: 0
> > > Number of receive errors: 0
> > > Number of timeout errors: 0
> > >
> > > Response Codes
> > >
> > > Response Code: 200 - The request completed successfully.
> > > Count: 500
> > > Percent (%): 100.00
> > >
> > >
> >
============================================================================> > > Issues:
> > >
> >
============================================================================> >
> > *** Tudor: this shows a report is returned in ~ 1 sec on average, so the
> > number of request per second match what you see in ACT, assuming your
server
> > has 1 or 2 processors. This is not about flow control, it simply takes
that
> > much to generate the reports in this suite. The number of concurrent
users
> > the system supports depends on how long a report generation takes.
> > If you would running all 500 requests at the same time, it would
probably
> > trash the system from all the context switches.
> > ***
> >
> > > 1. When I run the script, monitoring the RPS of ACT, it never goes
> > > more than 2 (not multi-thread? flow control?)
> > > 2. 500 users took about 10 mins. to complete (same issues as item 1)
> > > 3. Checking the RS Web Service Performance monitor, 3 suspicious
> > > counter values were:
> > > a. Total Processing Failure = 5120 (why it is so high?)
> >
> > ***
> > Tudor: I believe this is an additive counter, so its value is the total
> > since the report server started. There may also be cases in which we
wrongly
> > increment this for warnings. The execution log and ACT should give you
the
> > correct data here.
> > ***
> >
> > > b. Report Executed / Sec = 1 (never goes more than 1 during the
> > > entire testing, threading problem?)
> > > c. Request / Sec = = 1 (never goes more than 1 during the entire
> > > testing)
> > > 4. All 500 URL requests were logged into ExecutionLog of RS Server and
> > > tagged as "rsSuccess" (ACT was able to send all requests to RS and
> > > have them processed. but it was logged into the table 1 by 1 until all
> > > requests were processed. Meaning you will receive your 500th report
> > > after "n sec/report x 500 reports"?)
> > > 5. Why Number of unique requests made in test = 1 while URLs were
> > > distinct because of the ID parameter?
> > >
> > > I really wonder why RS behaves this way, it doesn't seems to be
> > > multi-thread. In RSReportServer.config, MaxActiveReqForOneUser was set
> > > to 0 to ensure maximum number of users can be handled but still it
> > > doesn't help.
> >
> > ***
> > Tudor: You should change MaxActiveReqForOneUser to 500 or a large number
I
> > don't think 0 means unbounded.
> > ***
> > >
> > > I don't think DoS attack protection features/setting of RS is a valid
> > > reason for the system to behave like that. If so, then the problem
> > > should be with ACT who issues 500 concurrent connections from the same
> > > workstation? Something must be wrong.
> > >
> > > My question:
> > > RS URL access cannot support more than 500 users? If not, why? If yes,
> > > can anyone tell me how to perform a correct performance test? Thanks!
> > >
> > > CCCsql