Performance tuning issues

I have built a solution which runs for two hours on a server with 4CPU 2GHz each and 2GB of RAM on windows 2000 server (CPU utilization almost 70% and almost out of RAM). I moved the two source databases and the solution to a new box runing 8 xeon's at 3GHz each and 16GB of RAM running widows 2003 server 32bit and it still runs for 2 hours (CPU utilization 10% and ample RAM left).

I was expecting it to run much faster. So I started exploring the performance tuning features in SSIS and started tweaking the following:

Control Flow:

DefaultBufferMaxRows: Based on row size and buffer size, calculated the max rows.

DefaultBufferSize: Set this to max 100MB

DataFlow Destination:

Rows Per Batch: Set this to equal to the numbe of rows expected from the source.

Maximum Insert Commit Size: Set this to zero since memory was not an issue

I took the recommendations from other threads on similar issues here including the excellent recommendations at http://www.microsoft.com/technet/prodtechnol/sql/2005/ssisperf.mspx but now the job is running for 6 hours.

Can anyone explain what I am doing wrong here? I have tried each of the above one by one and all together. No matter what combination I try it does not work any faster and both source and destination database are on the same server. Even selects from the same database also slowed down from 10 minutes to one hour.

Any assistance is appreciated, I need to get this job run in an hour.

Thanks!

- Philips.

How complex is your solution? I would recommend you to look at a lower grain;take a look a the log execution and compare it againt previous logs in the old server to see if you can identify a specifc part of the process as the bottleneck...|||

I'd also recommend watching the OVAL webcast that I talk about here:

Donald Farmer's Technet webcast
(http://blogs.conchango.com/jamiethomson/archive/2006/06/14/4076.aspx)

-Jamie

|||Did you setup the server to access more than 4gb of memory?

You need to configure Windows and then SQL server.|||Yes, SQL server is using around 14GB of memory and awe is turned on. Thanks!|||

It is a financial warehouse job which collects data from an ERP system loads a staging area and then the datamart. It also creates aggreagate tables. There are around 8 packages called from the master package.

My problem is I cannot find any way of using those four performance tuning settings accurately.

Any changes I make to the default setting is slowing the job down.

|||

Thanks! I went through it.

I am at a point where I am willing to drop all the control flow task and use plain old insert into statements and use the power of the DBMS engine than try to get the SSIS engine to do it. Unless I figure out what is wrong and why.

|||

Philips-HCR wrote:

Thanks! I went through it.

I am at a point where I am willing to drop all the control flow task and use plain old insert into statements and use the power of the DBMS engine than try to get the SSIS engine to do it. Unless I figure out what is wrong and why.

There's nothing wrong with doing that. The use of SSIS does not dictate that you should use data-flows to move your data about. If SQL is an option then invariably it will be the best option. it depends on your requirements and your preferences. You can issue SQL from an Execute SQL Task and still leverage all the other good stuff in SSIS like logging, workflow, portability etc... if you so wish.

-Jamie

Performance problems

Hi.
We have a system with approximately 1000 users running on a dual Pentium
with hyperthreading and 2Gb memory. The server is ultra fast and is
performing great - most of the time. The problem is that sometimes a query
that usually takes 2 seconds to run, suddenly will take 40 seconds or more.
It's like the system clutters up and everything becomes very slow.
We have been running a profiler on the system, and found several stored
procedures that needed optimizing and have done so - but we are still
experiencing the same problem. We have looked at sp_lock and found that most
of the time there are about 200 locks but suddenly the number of locks will
increase to as much as 200.000 mostly on one table and for one spid.
This leads us to believe that we have a query, an application or some other
unknown that prevents sql server from escalating a row level lock to table
level.
How do we find out if this is the case - is profiler the only way to go or
are we missing something here? Any ideas or similar experiences?
Your help will be greatly appreciated.When this happens, did you try running the query manually from Query
Analyzer and see if the execution plan is any different, compared to the one
when everything was okay? It could be that the statistics changed or not
up-to-date. Do you recall any data changes that correspond with the times of
bad performance?
--
HTH,
Vyas, MVP (SQL Server)
SQL Server Articles and Code Samples @. http://vyaskn.tripod.com/
"Thomas Jespersen" <tje@.nospam.mentum.dk> wrote in message
news:uLzi1GObFHA.464@.TK2MSFTNGP15.phx.gbl...
Hi.
We have a system with approximately 1000 users running on a dual Pentium
with hyperthreading and 2Gb memory. The server is ultra fast and is
performing great - most of the time. The problem is that sometimes a query
that usually takes 2 seconds to run, suddenly will take 40 seconds or more.
It's like the system clutters up and everything becomes very slow.
We have been running a profiler on the system, and found several stored
procedures that needed optimizing and have done so - but we are still
experiencing the same problem. We have looked at sp_lock and found that most
of the time there are about 200 locks but suddenly the number of locks will
increase to as much as 200.000 mostly on one table and for one spid.
This leads us to believe that we have a query, an application or some other
unknown that prevents sql server from escalating a row level lock to table
level.
How do we find out if this is the case - is profiler the only way to go or
are we missing something here? Any ideas or similar experiences?
Your help will be greatly appreciated.|||"Thomas Jespersen" <tje@.nospam.mentum.dk> wrote in message
news:uLzi1GObFHA.464@.TK2MSFTNGP15.phx.gbl...
> This leads us to believe that we have a query, an application or some
> other unknown that prevents sql server from escalating a row level lock to
> table level.
I had this ince, by a simple DTS table export, got _millions_ of locks.
Defragmented the database in question, and it helped, operation done exactly
the same way suddenly went without any problems.
Seems like some hard-to-reproduce SQL Server problem. I monitor the lock
count closely since then, and defragmenting is my first resort in such a
case, but it never happened again since then.
Hope it helps.
Regards
Wojtek|||Hi
Locks are managed by memory. Have you checked how much memory SQL Server
consumes?
Also make sure that you don't have along running transactions. Have you
checked that your tables have propely defined indexes and moreover an
optimizer is able to use them when it creates an execution plan.
Try to identify by DBCC INPUTBUFFER (spid) what query is running by this
spid.
"Thomas Jespersen" <tje@.nospam.mentum.dk> wrote in message
news:uLzi1GObFHA.464@.TK2MSFTNGP15.phx.gbl...
> Hi.
> We have a system with approximately 1000 users running on a dual Pentium
> with hyperthreading and 2Gb memory. The server is ultra fast and is
> performing great - most of the time. The problem is that sometimes a query
> that usually takes 2 seconds to run, suddenly will take 40 seconds or
more.
> It's like the system clutters up and everything becomes very slow.
> We have been running a profiler on the system, and found several stored
> procedures that needed optimizing and have done so - but we are still
> experiencing the same problem. We have looked at sp_lock and found that
most
> of the time there are about 200 locks but suddenly the number of locks
will
> increase to as much as 200.000 mostly on one table and for one spid.
> This leads us to believe that we have a query, an application or some
other
> unknown that prevents sql server from escalating a row level lock to table
> level.
> How do we find out if this is the case - is profiler the only way to go or
> are we missing something here? Any ideas or similar experiences?
> Your help will be greatly appreciated.
>

Performance problems

Hi.
We have a system with approximately 1000 users running on a dual Pentium
with hyperthreading and 2Gb memory. The server is ultra fast and is
performing great - most of the time. The problem is that sometimes a query
that usually takes 2 seconds to run, suddenly will take 40 seconds or more.
It's like the system clutters up and everything becomes very slow.
We have been running a profiler on the system, and found several stored
procedures that needed optimizing and have done so - but we are still
experiencing the same problem. We have looked at sp_lock and found that most
of the time there are about 200 locks but suddenly the number of locks will
increase to as much as 200.000 mostly on one table and for one spid.
This leads us to believe that we have a query, an application or some other
unknown that prevents sql server from escalating a row level lock to table
level.
How do we find out if this is the case - is profiler the only way to go or
are we missing something here? Any ideas or similar experiences?
Your help will be greatly appreciated.
When this happens, did you try running the query manually from Query
Analyzer and see if the execution plan is any different, compared to the one
when everything was okay? It could be that the statistics changed or not
up-to-date. Do you recall any data changes that correspond with the times of
bad performance?
HTH,
Vyas, MVP (SQL Server)
SQL Server Articles and Code Samples @. http://vyaskn.tripod.com/
"Thomas Jespersen" <tje@.nospam.mentum.dk> wrote in message
news:uLzi1GObFHA.464@.TK2MSFTNGP15.phx.gbl...
Hi.
We have a system with approximately 1000 users running on a dual Pentium
with hyperthreading and 2Gb memory. The server is ultra fast and is
performing great - most of the time. The problem is that sometimes a query
that usually takes 2 seconds to run, suddenly will take 40 seconds or more.
It's like the system clutters up and everything becomes very slow.
We have been running a profiler on the system, and found several stored
procedures that needed optimizing and have done so - but we are still
experiencing the same problem. We have looked at sp_lock and found that most
of the time there are about 200 locks but suddenly the number of locks will
increase to as much as 200.000 mostly on one table and for one spid.
This leads us to believe that we have a query, an application or some other
unknown that prevents sql server from escalating a row level lock to table
level.
How do we find out if this is the case - is profiler the only way to go or
are we missing something here? Any ideas or similar experiences?
Your help will be greatly appreciated.
|||"Thomas Jespersen" <tje@.nospam.mentum.dk> wrote in message
news:uLzi1GObFHA.464@.TK2MSFTNGP15.phx.gbl...
> This leads us to believe that we have a query, an application or some
> other unknown that prevents sql server from escalating a row level lock to
> table level.
I had this ince, by a simple DTS table export, got _millions_ of locks.
Defragmented the database in question, and it helped, operation done exactly
the same way suddenly went without any problems.
Seems like some hard-to-reproduce SQL Server problem. I monitor the lock
count closely since then, and defragmenting is my first resort in such a
case, but it never happened again since then.
Hope it helps.
Regards
Wojtek
|||Hi
Locks are managed by memory. Have you checked how much memory SQL Server
consumes?
Also make sure that you don't have along running transactions. Have you
checked that your tables have propely defined indexes and moreover an
optimizer is able to use them when it creates an execution plan.
Try to identify by DBCC INPUTBUFFER (spid) what query is running by this
spid.
"Thomas Jespersen" <tje@.nospam.mentum.dk> wrote in message
news:uLzi1GObFHA.464@.TK2MSFTNGP15.phx.gbl...
> Hi.
> We have a system with approximately 1000 users running on a dual Pentium
> with hyperthreading and 2Gb memory. The server is ultra fast and is
> performing great - most of the time. The problem is that sometimes a query
> that usually takes 2 seconds to run, suddenly will take 40 seconds or
more.
> It's like the system clutters up and everything becomes very slow.
> We have been running a profiler on the system, and found several stored
> procedures that needed optimizing and have done so - but we are still
> experiencing the same problem. We have looked at sp_lock and found that
most
> of the time there are about 200 locks but suddenly the number of locks
will
> increase to as much as 200.000 mostly on one table and for one spid.
> This leads us to believe that we have a query, an application or some
other
> unknown that prevents sql server from escalating a row level lock to table
> level.
> How do we find out if this is the case - is profiler the only way to go or
> are we missing something here? Any ideas or similar experiences?
> Your help will be greatly appreciated.
>

Performance problems

Hi.
We have a system with approximately 1000 users running on a dual Pentium
with hyperthreading and 2Gb memory. The server is ultra fast and is
performing great - most of the time. The problem is that sometimes a query
that usually takes 2 seconds to run, suddenly will take 40 seconds or more.
It's like the system clutters up and everything becomes very slow.
We have been running a profiler on the system, and found several stored
procedures that needed optimizing and have done so - but we are still
experiencing the same problem. We have looked at sp_lock and found that most
of the time there are about 200 locks but suddenly the number of locks will
increase to as much as 200.000 mostly on one table and for one spid.
This leads us to believe that we have a query, an application or some other
unknown that prevents sql server from escalating a row level lock to table
level.
How do we find out if this is the case - is profiler the only way to go or
are we missing something here? Any ideas or similar experiences?
Your help will be greatly appreciated.When this happens, did you try running the query manually from Query
Analyzer and see if the execution plan is any different, compared to the one
when everything was okay? It could be that the statistics changed or not
up-to-date. Do you recall any data changes that correspond with the times of
bad performance?
--
HTH,
Vyas, MVP (SQL Server)
SQL Server Articles and Code Samples @. http://vyaskn.tripod.com/
"Thomas Jespersen" <tje@.nospam.mentum.dk> wrote in message
news:uLzi1GObFHA.464@.TK2MSFTNGP15.phx.gbl...
Hi.
We have a system with approximately 1000 users running on a dual Pentium
with hyperthreading and 2Gb memory. The server is ultra fast and is
performing great - most of the time. The problem is that sometimes a query
that usually takes 2 seconds to run, suddenly will take 40 seconds or more.
It's like the system clutters up and everything becomes very slow.
We have been running a profiler on the system, and found several stored
procedures that needed optimizing and have done so - but we are still
experiencing the same problem. We have looked at sp_lock and found that most
of the time there are about 200 locks but suddenly the number of locks will
increase to as much as 200.000 mostly on one table and for one spid.
This leads us to believe that we have a query, an application or some other
unknown that prevents sql server from escalating a row level lock to table
level.
How do we find out if this is the case - is profiler the only way to go or
are we missing something here? Any ideas or similar experiences?
Your help will be greatly appreciated.|||"Thomas Jespersen" <tje@.nospam.mentum.dk> wrote in message
news:uLzi1GObFHA.464@.TK2MSFTNGP15.phx.gbl...
> This leads us to believe that we have a query, an application or some
> other unknown that prevents sql server from escalating a row level lock to
> table level.
I had this ince, by a simple DTS table export, got _millions_ of locks.
Defragmented the database in question, and it helped, operation done exactly
the same way suddenly went without any problems.
Seems like some hard-to-reproduce SQL Server problem. I monitor the lock
count closely since then, and defragmenting is my first resort in such a
case, but it never happened again since then.
Hope it helps.
Regards
Wojtek|||Hi
Locks are managed by memory. Have you checked how much memory SQL Server
consumes?
Also make sure that you don't have along running transactions. Have you
checked that your tables have propely defined indexes and moreover an
optimizer is able to use them when it creates an execution plan.
Try to identify by DBCC INPUTBUFFER (spid) what query is running by this
spid.
"Thomas Jespersen" <tje@.nospam.mentum.dk> wrote in message
news:uLzi1GObFHA.464@.TK2MSFTNGP15.phx.gbl...
> Hi.
> We have a system with approximately 1000 users running on a dual Pentium
> with hyperthreading and 2Gb memory. The server is ultra fast and is
> performing great - most of the time. The problem is that sometimes a query
> that usually takes 2 seconds to run, suddenly will take 40 seconds or
more.
> It's like the system clutters up and everything becomes very slow.
> We have been running a profiler on the system, and found several stored
> procedures that needed optimizing and have done so - but we are still
> experiencing the same problem. We have looked at sp_lock and found that
most
> of the time there are about 200 locks but suddenly the number of locks
will
> increase to as much as 200.000 mostly on one table and for one spid.
> This leads us to believe that we have a query, an application or some
other
> unknown that prevents sql server from escalating a row level lock to table
> level.
> How do we find out if this is the case - is profiler the only way to go or
> are we missing something here? Any ideas or similar experiences?
> Your help will be greatly appreciated.
>

Performance monitoring

Hi
We are running a Windows 2003 Std. and a MSSQL std. on a 2xPIII having 2Gb
RAM and SCSI disks in RAID1.
The server handles 80 Databases and has some 80-100 connections from
webapplications.
I guess most of you are used to 4xXeon and 128Gb Ram ;-) anyway - the
performance is at the moment very good, but I am of cause a bit concernt
about the future. Does et actually perfom OK as we think it does? Will it
keep performing if we add 10 databases for it to handle? And so on. When
will "it break" performance wise.
I was looking at windows "performance monitor" adding and deleting all sorts
of performance counters - but which one should I look at and what is "in
range"? Any general ideas?
And of cause - any handson books/tutorials that describes this?
Best regards
JanHere are some links to have a look at:
http://www.microsoft.com/sql/techinfo/administration/2000/perftuning.asp
Performance WP's
http://www.swynk.com/friends/vandenberg/perfmonitor.asp Perfmon counters
http://www.sql-server-performance.com/sql_server_performance_audit.asp
Hardware Performance CheckList
http://www.sql-server-performance.com/best_sql_server_performance_tips.asp
SQL 2000 Performance tuning tips
http://www.support.microsoft.com/?id=q224587 Troubleshooting App
Performance
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/adminsql/ad_perfmon_24u1.asp
Disk Monitoring
Andrew J. Kelly SQL MVP
<msdn@.csite.com> wrote in message
news:OrxgP391FHA.1028@.TK2MSFTNGP12.phx.gbl...
> Hi
> We are running a Windows 2003 Std. and a MSSQL std. on a 2xPIII having 2Gb
> RAM and SCSI disks in RAID1.
> The server handles 80 Databases and has some 80-100 connections from
> webapplications.
> I guess most of you are used to 4xXeon and 128Gb Ram ;-) anyway - the
> performance is at the moment very good, but I am of cause a bit concernt
> about the future. Does et actually perfom OK as we think it does? Will it
> keep performing if we add 10 databases for it to handle? And so on. When
> will "it break" performance wise.
> I was looking at windows "performance monitor" adding and deleting all
> sorts of performance counters - but which one should I look at and what is
> "in range"? Any general ideas?
> And of cause - any handson books/tutorials that describes this?
> Best regards
> Jan
>