SQL Server Support DBA
SQL Server Support DBA
Quick Reference for Support Questions............................................................................1 Our SQL Servers & Their Service Levels..........................................................................3 Differences Between Tiers.............................................................................................3 Database Server Restrictions............................................................................................6 Things That Require DBA Signoff..................................................................................6 Things Not Available on Production Servers.................................................................7 Things Not Available On Any Server.............................................................................8 Load Testing....................................................................................................................10 Backups & Restores........................................................................................................11 Network Backups.........................................................................................................11 Restoring Databases....................................................................................................12 High Availability & Disaster Recovery.............................................................................13 Clustering.....................................................................................................................13 Database Mirroring.......................................................................................................13 Change Control................................................................................................................17 Deployment Practices..................................................................................................17 Maintenance Windows & Patches...................................................................................19 QA & Development Servers.........................................................................................19 Mission-Critical & Production Servers..........................................................................19 When to Get Help from the DBA......................................................................................20 SQL Coding Standards....................................................................................................21 Design for Database Mirroring.....................................................................................21 Table Design................................................................................................................21 Stored Procedures & Functions...................................................................................22 SQL Server Setup Standards..........................................................................................23 SQL Server 2005 Setup Checklist...............................................................................23
Company Help Desk 1-800-555-1000. Ask to open a Severity 1 help desk ticket, and they can reach the next available member of the IT team.
Production
QA/Testing
Dev
Restrictions
Users can change tables/views Yes Yes Yes Yes Users can restore dbs, kill procs No No* No* Yes Users can be sysadmins No No* No* Yes Users can be dbowners Yes Yes Yes Yes Users can remote desktop into server No No* No* No Performance level Very High High High Medium * Note these features are available if the database is installed in its own dedicated virtual machine (VM).
Servers
General Shared Use (SQL 2008) General Shared Use Legacy (No new deployments allowed) (SQL 2000) Data Warehouse (BI) Data Marts (Reporting) Sharepoint Point of Sale Systems CXSQLPRVS1 (New York) CXMEMSQL1 (Memphis mirror) CXSQL1 CXDWPROD1 CXDMSQLPR1 CXSPPRVS1 CXPOSSQLPR1 (New York) CXPOSSQLDR1 (Memphis mirror) CXSQLQA1 CXSQLDV1 (32-bit)
CXSQL2 (Uses shared one) CXDMSQLQA1 (Uses shared one) CXDMSQLDV1 CXPOSSQLDV1
Ideally, we wont need this time but on occasion, we do, and on the development servers, well actually do it during business hours. This practice means that we dont have to incur as much downtime on the production servers. We can say with confidence that installing a particular patch will only take X minutes, and that it wont have any negative effects on the production environment.
CXDWPROD1. They have a standard process theyve put into place with extensive monitoring and testing, and we can reuse a lot of their work easily. For questions about it, meet with Jim Smith.
DTS Packages
We do not use DTS packages on the centralized database servers. DTS packages are deprecated with SQL Server 2005. If an application still needs DTS packages, we can install them on an existing SQL Server 2000 machine, or put the standalone Express Edition on the application server. Express is limited to small amounts of data, but for an installation that strictly does DTS packages, that shouldnt be a problem since were not storing data locally anyway.
Load Testing
Email Your Friendly Database Administrator
When scheduling load tests, send the database administrators an email to let them know that youll be doing load testing. They can run a SQL profile and a Windows performance monitor during that time range to let you know about any particular bottlenecks. They can also tell you if anyone else has scheduled testing runs during that same time span that might slow your testing down. The nightly backup jobs might make your load testing look artificially slow, or other developers may also be load testing their own applications on these servers as well.
Restoring Databases
With Litespeed
Remote desktop into CXSQLDV1 (a SQL utility machine, not an actual database server) and click Start, Programs, Imceda, LiteSpeed, Enterprise Console. If Litespeed asks for the central repository name, choose CXSQLPRVS1. If it shows a Welcome to LiteSpeed introductory window, click Close. Click Connections, Registered Servers to see the list of servers, and add any server that you need to work with. Drill into the server where you want to restore, then right-click and click Restore Database Wizard. Pay attention to the file names dialogue box because you may be restoring the data files to other drive locations if youre restoring onto another server. Click Restore, and you can watch the status as each job goes through.
Without Litespeed
Servers without Litespeed use normal SQL Server backup/restore methods. Its a pain to restore the transaction logs, though.
Database Mirroring
We use SQL Servers built-in Database Mirroring to keep a live copy in Memphis of some critical production databases. We use Asynchronous (High-Performance) mirroring with manual failover. That means when we do a failover from the primary server to the mirrored server, we have to manually make sure that all transactions on the primary server finish before we fail over. We dont use Synchronous (High-Safety) mirroring because our WAN connection between Memphis and New York isnt fast enough. Before deciding to change to Synchronous, read through the Database Mirroring Best Practices, which suggest that SQL transaction response times in Synchronous mode degrade pretty badly over 20 milliseconds:
http://www.microsoft.com/technet/prodtechnol/sql/2005/technologies/dbm_best_pract.mspx
In the examples below, Im going to focus on DeliveryApp. SalesApp is basically the same, but because it relies on multiple databases instead of just one, its more complicated to role swap. All of the databases must be manually managed to make sure they stay in sync during the role swap we cant have some databases live in Memphis and others in New York, because the applications will fail.
Planned Failovers
This can be tested on the DeliveryApp servers using the MirrorTraining database. That database is mirrored between New York & Memphis just like the DeliveryApp database
(DeliveryDB) and can be failed over back & forth between sites at any time. No change control is required. To fail over from one server to another: 1. Make sure all of the users are out of DeliveryApp. When testing this with the MirrorTraining database, you can skip this step it doesnt affect DeliveryApp at all. 2. Connect to the principal server with SSMS. Open SQL Server Management Studio (on your machine or remote desktop into a SQL Server but never production) and connect to the server currently acting as the principal (primary) server. Drill into Databases, and your screen should look like the below. Notice that the databases say Principal, Synchronized. If they say anything else, then you connected to the mirrored server by mistake. Connect to the principal one and pray your luck gets better, because its early and youve already goofed this up. 3. Change the database to Synchronous mirroring. Right-click on the database you want to fail over and click Tasks, Mirror. Change the Operating Mode from high performance to high safety. This forces it to get caught up, and makes sure that any transactions are committed at both servers simultaneously. Click OK. 4. Fail over to the mirror. Go back into Tasks, Mirror and click the Failover button. You will get a warning that basically means if any DeliveryApp users are active, theyre going to lose their lunch. Now is your chance to double-check Citrix, and if everybodys out, click Yes. 5. Verify that the database failed over. Right-click anywhere in the right side window (Databases) and click Refresh. The database should now say Mirror, Synchronized / Restoring . . . ) If it says anything else, the failover did not succeed. Contact a database administrator. 6. Disable the complete backup jobs on the old primary server. Drill into SQL Server Agent, Jobs, and right-click on the Complete Backup Job for DB Maintenance Plan Core DBA User DBs - Mirrored and click Disable. There may also be a job for Core DBA System DBs, but do not disable that job. Do the same for any other jobs like Transaction Log or Optimizations, as long as they end with Core DBA User DBs - Mirrored. There may also be jobs for Core DBA System DBs or Core DBA User DBs NonMirrored, but do not disable those jobs. 7. Enable those same jobs on the new principal server. Connect to the new principal server with SQL Server Management Studio by clicking Connect, Database Engine and type in the server name. Drill into SQL Server Agent, Jobs, and then right-click on each of the Core DBA User DBs - Mirrored jobs and click Enable. 8. Take the new principal out of synchronous mode. On the new principal, right-click on the database name and click Tasks, Mirror. Change the operating mode from High Safety to High Performance, and click OK. 9. Reconfigure the applications and start them again. Change the database connection string in DeliveryApp to point to the new principal server. This can be
done ahead of time by setting up both servers in the connection string, with CXSQLDR1 as the mirror. To fail back, just repeat the same process after connecting to the new principal server.
Unplanned Failovers
If the principal server has suffered catastrophic failure and must be brought online, you can force the mirror server online for service. 1. Connect to the mirror server with SSMS. Open SQL Server Management Studio (on your machine or remote desktop into a SQL Server but never production) and connect to the server currently acting as the mirror (backup) server. 2. Force the mirror database into service. Click the New Query button at the top left, which will open a query window on the right side. Copy/paste this into the query window and click the Execute button:
ALTER DATABASE DELIVERYDB SET PARTNER FORCE_SERVICE_ALLOW_DATA_LOSS
That command will force the DeliveryDB database into principal mode. If any errors come up, consult with a DBA. That command will only work when the principal server is completely down. If its powered on, especially if SQL Server is running, SQL Server will not allow this command to work. If you really want to force it but the principal SQL Server is still up and running, youll have to power down the principal SQL Server, then run the command on the mirror. 3. Verify that the database failed over. Right-click anywhere in the right side window (Databases) and click Refresh. The database should now say Principal, Synchronized) If it says anything else, the failover did not succeed. Contact a database administrator. 4. Enable the complete backup jobs. Drill into SQL Server Agent, Jobs, and right-click on the Complete Backup Job for DB Maintenance Plan Core DBA User DBs - Mirrored and click Enable. There may also be a job for Core DBA System DBs, but do not disable that job. Do the same for any other jobs like Transaction Log or Optimizations, as long as they end with Core DBA User DBs - Mirrored. There may also be jobs for Core DBA System DBs or Core DBA User DBs NonMirrored, but do not disable those jobs. 5. Reconfigure the applications and start them again. Change the database connection string in Roadnet to point to the new principal server. This can be done ahead of time by setting up both servers in the connection string, with CXSQLDR1 as the mirror. If the old principal servers outage lasts too long, the new principals log files may fill up, causing SQL Server to crash. Within a few hours of the outage, the team needs to make a decision about whether to break mirroring. Theres no hard and fast rule the DBA should be involved to gauge the rate at which the log files are filling up.
To break mirroring, right-click on the database name and click Tasks, Mirroring and click the Remove Mirroring button. After the old principal server is repaired, database mirroring will need to be re-established. This should only be done by a database administrator, but the short version is: Disable all backup jobs for this database on the principal server Do one complete backup on the principal Do one transaction log backup on the principal Restore both the complete and transaction log on the mirror server, but leave the database in recovery mode Enable database mirroring on the principal Alter the database partner timeout to 90 seconds instead of 10 (because our WAN link is slow and we get constant false alarms) Fail back over to New York if necessary Enable/disable backup jobs on the appropriate servers
Change Control
Deployment Practices
For a stable production environment, we follow this deployment practice: Developers script every change on the development servers first The manager tests the script on the QA box and signs off that it works The manager puts in a change control request The DBA either watches that test take place, or tests it again on QA The DBA applies the script on the production boxes
Table Design
All tables should have a primary key.
In any table, we should be able to identify a set of fields that make each row unique. They may not even be a natural set of keys they might be a surrogate key field like an autonumbered identity field but there does have to be a unique thing on each row. Ideally, it should not be a GUID (uniqueidentifier) field. Those fields are rather large, and they take a lot more to store. This has negative impacts on index speed. Before using a GUID as your unique key, check with your database administrator to see what other options are available.
A programmer or DBA should be able to open the stored procedure and see right away what it does, who calls it, and whos been changing it lately. Notice that the description block is BELOW the create procedure statement. That way, the code stays intact inside the stored proc.
5. Install the SMS client. 6. Install SNMP and WMI. Configure SNMP for a read-only community named snmpn0w, send traps to flmirwhatsup1.companyx.com, and accept SNMP packets from any host. (This is for WhatsUp.) 7. For 32-bit servers, enable the /PAE switch in boot.ini. 8. Set the page file on the C drive to 1.5x physical memory. 9. Go into Local Security Settings, Local Policies, User Rights Assignment. a. In the permission Perform Volume Maintenance Tasks, add the domain group SQL Service Service Admins. This enables SQL 2005s fast file initialization, so multi-gig file increases can be done instantly. b. Set up the Lock pages in memory permission, which enables better memory management. 10. Install SQL Server 2005. a. The services should start under the domain accounts set up earlier. This lets us back up to network shares. b. Do not install Reporting Services or Notification Services on production servers well implement separate servers for those if necessary. 11. If doing a cluster setup, install the client tools on the second server. 12. If the server uses iSCSI, set up the cluster & SQL services as dependent on iSCSI. Do this at the command prompt:
sc config clussvc depend= clusnet/netman/Rpcss/w32time/msiscsi sc config mssqlserver depend= msiscsi
13. Install SQL Service Pack 2 and latest cumulative hotfixes. 14. Move the TempDB to a non-system drive. a. Configure one TempDB file per core. The data file creation should take a few seconds or less if it takes longer, then instant file initialization isnt configured correctly. 15. Configure SQL Server to use the appropriate amount of memory.
16. Run the SQL Server 2005 Surface Area Configuration for Services & Connections wizard and enable remote connections over TCP/IP. 17. Run the SQL Server 2005 Surface Area Configuration for Features wizard and enable database mail. 18. Go into Management Studio and configure Database Mail for mail.companyx.com. 19. After mail setup, right-click on SQL Agent, configure the alerting system, and restart SQL Agent. 20. Go into SQL Server Agent and configure operators for each sysadmin. 21. Create alerts for each severity level 16 through 25, and send responses to the operators. 22. Set the default SQL data & log file paths. 23. Run the SQL Server 2005 Performance Dashboard Reports setup.sql script. 24. Install Quest Litespeed under the service account CXDOMAIN\svcQuestLitespeed and point the central repository at CXSQLPRVS1. a. Set the default backup directory to \\cxbackups\sql\(servername). 25. Set up backups: a. For production servers - using Lightspeed, set up complete backups daily and transaction log backups every 15 minutes, both to the designated network backup share. b. For QA servers full backups once a week. We rarely have to restore these, and we actually tell the users these arent backed up at all. We dont really have the backup window to do fulls on these since theyre the same size as production. c. For development servers full backups daily. 26. Configure WhatsUp to monitor the server including performance monitors. 27. Set up database mirroring. a. If the principal server is a cluster, alter the database timeout so that the database doesnt fail over when the cluster fails over. This should also be done if the mirroring is being transferred over a slow WAN link, which can time out quickly. ALTER DATABASE (mydatabasename) SET PARTNER TIMEOUT 90