SCOM Console Performance

Tuning the Consoles in Operations Manager 2007

I have been working on creating new dashboards recently  and  I have spent allot of time looking at the console in a 6000 agent managed MG. It is not very snappy at the minute so the following are a few recommendations to improve speed

1)      Avoid making too many changes during peak usage as in office hours. I know we are making minor tweaks and changes in PRD which is fine but we need to manage those activities between ourselves. This will reduce the Config churn that is caused as for every small change even modifying a view will trigger agents to contact their respective MS to get updates.

2)       The Console appears to be running slow but the RMS/SQL is well resourced albeit I will be running some tests on the Disk utilisation.  I ran 3 tests to just open the Console using the Clear Cache switch and timed how long it took to display the data so the view appeared complete as in not wait for the Green bar to complete. Each test I ran three times. Test performed between 09:00 & 10:00

TEST 1 -Open Monitoring/Overview page:

RUN 1: 40 Secs RUN 2: 30 Secs. RUN 3: 34Secs

TEST 2 – Open Active Alerts Dashboard Page and wait for all views to populate approx. 3500 open alerts. (Note I closed the console first then selected the new view on each test)

RUN 1: 20 Secs RUN 2: 1 Min & 30 Secs. RUN 3: 16 Secs

3)      I ran the same 3 tests to just open the Console not using the Clear Cache switch and timed how long it took.

TEST 1 -Open Monitoring/Overview page:

RUN 1: 12 Secs RUN 2: 10 Secs. RUN 3: 15 Secs

TEST 2 – Open Active Alerts Dashboard Page and wait for all views to populate approx. 3500 open alerts. (Note I closed the console first then selected the new view on each test)

RUN 1: 6 Secs RUN 2: 6 Secs. RUN 3: 5 Secs

 

                In a nutshell the clear cache switch will slow you down by three. Use the Clear Cache if you are getting the Data Mismatch error only.

4)      Modify your Refresh Rate for the console from default 60 secs up to 120 or higher Secs. I will also be ensuring for end user suuport staff and Citrix Thin SCOM Console I will be setting to 5 Mins to reduce the queries on the OpsDB amongst some other Console Tweaks.

 

Console refresh rate

By default, the Operations Console refreshes its cache after 1 minute of inactivity. The refresh occurs only if there has been no user activity in the given console instance for the duration of the refresh period. Default value is one minute, with a maximum of 10. Raising this value on workstations running the console is another way to reduce calls to the operations database. This can be adjusted through the following registry key: Path: HKCU\Software\Microsoft\Microsoft OperationsManager\3.0\Console\CacheParameters\ Keyword: PollingInterval (DWORD)

5)      On the server/desktop that you open the console on frequently. You views build up over time. So Delete the console key from the following path ‘HKEY_CURRENT_USER\Software\Microsoft\Microsoft Operations Manager\3.0\Console’. This does not need to be done all the time but I advise you all do this on your console. Take care not to delete the wrong key especially on the MS/RM. Also make sure you close the console first before deleting the key. (Note when you delete this key your personalised views will be removed and the default View will be displayed)

After I performed these steps I re-ran the Console Test

TEST 1 -Open Monitoring/Overview page:

RUN 1: 9 Secs RUN 2: 10 Secs. RUN 3: 14 Secs

TEST 2 – Open Active Alerts Dashboard Page and wait for all views to populate approx. 3500 open alerts. (Note I closed the console first then selected the new view on each test)

RUN 1: 5 Secs RUN 2: 4 Secs. RUN 3: 5 Secs

6)      Keep Alerts Open and Closed down to 2000 benchmark. Currently  averaging around 3500 open alerts… The Cluster MP is the main culprit as is un-tuned and very noisy. An exercise to report on noisy alerts is due but reports should be Monthly to continually reduce noise in an established environemnt. An Exercise to report on High Repeat count alerts needs attention either by Override or troubleshoot and correct. This is not a task the End users will be bothered about.

7)      Console Errors. The Following Error Is Logged in Console sometime.

You may find if you have created an Active Alerts Dashboard View with mul;tiple panels open for a period of time it will occur most often if you have not witnessed already. I have not experimented adjusting the refresh rate to see if this improves.

Link to possible cause http://www.ureader.com/msg/1682572.aspx

“In the SQL Server error logs at about the same time as the console exception happened, you should see an errorwith a description of a read operation on a large object failed while sending  data to the client. A common cause for this is if the application is running in READ UNCOMMITTED isolation level. This connection will be terminated. A user has the Console open on an Alerts View.  While the Alerts View is opening or refreshing, there are Alerts coming in that are repeated Alerts which updates the Repeat Count and Context fields for that Alert.  Since the Context field is a large object type field, and the Alert View queries have NOLOCK hints in them, The Alert View has already read this changing Alert.  That gives us a dirty read and SQL Server disconnects that SQL Connection since we’ve tried to do a dirty read on a large object field which is not allowed.  So, since that connection is disconnected the next line of SDK code that tries to use this connection that it thinks is still open will cause exception ‘Execute Scalar requires an open and available

Connection. The connection’s current state is closed”.

So to help get rid of this problem we need to reduce the High Repeat Count Alerts that are updating the alerts frequently.

Ruin this SQL query against your OpsMgr DB

 SELECT RepeatCount, COUNT(*) AS NumAlertsWithRepeatCount
FROM Alert
WHERE ResolutionState = 0
AND RepeatCount > 0
GROUP BY RepeatCount
ORDER BY COUNT(*) DESC
Advertisements
This entry was posted in Console. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s