Ever since I have started writing about Error Messages a number of you ping me from time to time to understand why you are getting an error. As I always say, Error messages are a great way to learn concepts and we become better if we understand the reason behind every error one gets. As I scan through almost every single mail that comes in, some of them catch my attention and I tend to write this experience back as a blog post.

Last week, one fresher who was working for an European client wrote back to me with an error mentioned below:

Msg 468, Level 16, State 9, Line 15
Cannot resolve the collation conflict between “SQL_Latin1_General_CP1_CI_AS” and “French_CS_AI” in the equal to operation.

Though the error looked simple and self-explanatory, this kid wanted to mimic this error in his environment. He also added that he was using temp tables while this error surfaced. This statement got me interested and I wanted to show him with a simple script to how this error can be achieved.

Let me start by knowing the server collation to start with:

-- Server Collation
SELECT SERVERPROPERTY('Collation')

——————————————————————–

SQL_Latin1_General_CP1_CI_AS

As you can see, I am using the default collation on my server at this moment. Now let us create an DB with a different collation (French) for example. We will also create two tables post that. First will be a normal table while the second one will be an Temp table.

-- Create database with different collation
CREATE DATABASE [DB_Not_Contained]
CONTAINMENT = NONE
COLLATE French_CS_AI
GO


-- Create 2 tables with 1 being a temp table so it goes to tempdb and uses the server collation and not the database collation
USE [DB_Not_Contained]
GO
CREATE TABLE [DemoCollation]
(DemoCollationNM VARCHAR(100))
GO
CREATE TABLE #DemoCollation
(DemoCollationNM VARCHAR(100))


-- Insert records into both tables
INSERT dbo.DemoCollation
(DemoCollationNM)
VALUES ('Test Join');
INSERT #DemoCollation
(DemoCollationNM)
VALUES ('Test Join');


Now that the basic setup of data is over. Let us make a simple join between these two tables.

-- Now query and try and join both tables having 2 different collations
SELECT p.DemoCollationNM
FROM dbo.DemoCollation p
INNER JOIN #DemoCollation d ON
p.DemoCollationNM = d.DemoCollationNM


Woot, we will receive the error message:

Msg 468, Level 16, State 9, Line 35
Cannot resolve the collation conflict between “SQL_Latin1_General_CP1_CI_AS” and “French_CS_AI” in the equal to operation.

Whenever you encounter this error message, please go ahead and check the collation of tables under question to be clear they are not different.

So what are the learnings from these simple steps?

• We cannot join tables, columns when there is conflict in collation between the objects
• Temp table uses the same collation as our Server by default, so are the objects that are created.

I know I have given you a hint on how you might potentially mitigate this, but let me see if anyone can give me the solution to this problem and how have you ever been able to solve these? Have you ever encountered this error?

Reference: Pinal Dave (http://blog.sqlauthority.com)

There are parts of SQL Server where I always like to take expert advice. Sometimes a quick call to a friend can save you tons of time. When I see topics around AlwaysOn, though I know what the basic steps are to building and working with AlwaysOn, the hardcore troubleshooting is not the forte perse. But when I encounter questions around it, I put my learning hat to explore the possible reasons. There was a mail from one of the readers which has been on my Inbox for couple of weeks and I wanted to find the answer. His question goes like this:

While working with AlwaysOn availability group and cleaning it up, I am encountering below error while dropping a login.

DROP LOGIN Pinal
GO


Msg 15173, Level 16, State 1, Line 1
Server principal ‘Pinal’ has granted one or more permission(s). Revoke the permission(s) before dropping the server principal.

Even trying from UI also gives same error

Next task was to find what is causing the error, so I made below query which can tell the exact objects that are dependent and making our DROP command to fail.

SELECT class_desc,*
FROM sys.server_permissions
WHERE grantor_principal_id = (
SELECT principal_id
FROM sys.server_principals
WHERE NAME = N'Pinal')

SELECT NAME
,type_desc
FROM sys.server_principals
WHERE principal_id IN (
SELECT grantee_principal_id
FROM sys.server_permissions
WHERE grantor_principal_id = (
SELECT principal_id
FROM sys.server_principals
WHERE NAME = N'Pinal'))


Based on output, it means there is an endpoint on which Pinal has been granted permission. Second result shows that I have used my account i.e. Pinal to create endpoint and granted permission to AppUser account using below command:

CREATE ENDPOINT [hadr_endpoint]
STATE=STARTED
AS TCP (LISTENER_PORT = 5023, LISTENER_IP = ALL)
FOR DATA_MIRRORING (ROLE = ALL, AUTHENTICATION = WINDOWS NEGOTIATE
, ENCRYPTION = REQUIRED ALGORITHM AES)
GO
GRANT CONNECT ON ENDPOINT::[hadr_endpoint] TO AppUser
GO


As the error message suggested, I revoked permission using the below command:

REVOKE CONNECT ON ENDPOINT::[hadr_endpoint] TO AppUser
GO


After the above step, I was able to delete login. Yes, this was a great learning for me today. Do let me know if you ever encountered similar errors in your environments?

Reference: Pinal Dave (http://blog.sqlauthority.com)

Using CASE statements is something I see a lot of developers use. They use it in complex environments and I have been fortunate to troubleshoot and look at code blocks that run for pages using the standard CASE statements. This blog post is inspired by one such errors using CASE statements. This is one of the common error which is seen by many developers while using case/when combination. To understand the reason for such error, here is one of the simplified repro of the error:

SET NOCOUNT ON
GO
DECLARE @TempTable TABLE (id INT PRIMARY KEY, STATUS INT, Score INT)
INSERT INTO @TempTable VALUES (1,1,200)
INSERT INTO @TempTable VALUES (2,2,200)
INSERT INTO @TempTable VALUES (3,1,99)
INSERT INTO @TempTable VALUES (4,1,10)
SELECT id
,CASE
WHEN STATUS = 2
THEN 'Inactive'
ELSE score
END 'Score'
FROM @TempTable


If we execute above in management studio, we would get below error

id          Score

----------- -----------

1           200

Msg 245, Level 16, State 1, Line 8

Conversion failed when converting the varchar value 'Inactive' to data type int.

Always remember the golden rule of case/when: “All branches – same type”. It is very important that all return values have the same data type. If we don’t do this, we would end up in conversion problems, like the one shown above. In our sample, score column is integer whereas value “Inactive’ is string. Due to datatype precedence rule, Data Type Precedence SQL Server is trying to implicitly convert Inactive to Integer data type. SQL Server will always return the same data type regardless of which branch of the case/when expression is actually returned. For CASE expressions all expressions should be of the same type or implicitly convertible types.

Here is another simple example which would fail randomly.

SELECT CASE WHEN RAND() < 0.5 THEN 200 ELSE 'SQlAuthority' END AS SomeColumnName;

Based on output of RAND() it would fail with error

Msg 245, Level 16, State 1, Line 1

Conversion failed when converting the varchar value 'SQlAuthority' to data type int.

Here is the fix for the original problem.

SET NOCOUNT ON
GO
DECLARE @TempTable TABLE (    id INT PRIMARY KEY, STATUS INT, Score INT)
INSERT INTO @TempTable VALUES (1,1,200)
INSERT INTO @TempTable VALUES (2,2,200)
INSERT INTO @TempTable VALUES (3,1,99)
INSERT INTO @TempTable VALUES (4,1,10)
SELECT id
,CASE
WHEN STATUS = 2
THEN 'Inactive'
ELSE CAST (score AS VARCHAR(10))
END 'Score'
FROM @TempTable

Here is the output:

Have you been following this practice while coding? What are your workarounds for these sort of case statements? Let me know over comments.

Reference : Pinal Dave (http://blog.SQLAuthority.com)

I have a tendency to try out various settings on a typical SQL Server just to see how the behavior changes from time to time. While playing with affinity settings in SQL Server – I shot myself in the foot. It was interesting to reach how I came out of the situation. I was not able to reproduce this error on the enterprise edition though.

Here is what I found in Errorlog:

2015-07-28 17:12:11.31 Server    Processor affinity turned on: node 0, processor mask 0x00000006. Threads will execute on CPUs per affinity settings.

2015-07-28 17:12:11.31 Server    I/O affinity turned on, processor mask 0x00000001. Disk I/Os will execute on CPUs per affinity I/O mask/affinity64 mask config option.

2015-07-28 17:12:17.44 Server    Node configuration: node 0: CPU mask: 0x00000006:0 Active CPU mask: 0x00000006:0.

2015-07-28 17:12:17.45 Server    Error: 17120, Severity: 16, State: 1.

2015-07-28 17:12:17.45 Server    SQL Server could not spawn lazy writer thread. Check the SQL Server error log and the Windows event logs for information about possible related problems.
If we convert highlighted number to binary, we can decode the affinity which I have set.

0x00000006 (in hex) = 0110 (in binary) in Processor affinity
0x00000001 (in hex) = 0001 (in binary) in I/O affinity

You can treat 1s and 0s as checkboxes for processors. Below picture should make it more clear.

As you can see there is no overlap so I don’t see any problem, but still SQL Server didn’t start. I checked various online places, but few articles asked to repair the instance, which will not work in this situation. So finally, I sat down calmly and recalled that there is an option to start SQL with minimal configuration. I have used that earlier with one of my clients for SQL startup due to tempdb misconfiguration.

Here are the steps which worked:

1. Start SQL in minimal configuration using –f parameter.

NET START MSSQLSERVER /f

If you are having named instance called Inst1 then you need to use below
NET START MSSQL$INST1 /f

2. Connect to SQL using any client tool (SQLCMD or SSMS)
3. Change the affinity back to default.

T-SQL

ALTER SERVER CONFIGURATION SET PROCESS AFFINITY CPU = AUTO;
GO
EXEC sys.sp_configure N'show advanced options', N'1';
GO
RECONFIGURE WITH OVERRIDE;
GO
EXEC sys.sp_configure N'affinity I/O mask', N'0';
GO
RECONFIGURE;
GO
EXEC sys.sp_configure N'show advanced options', N'0';
GO
RECONFIGURE WITH OVERRIDE;
GO


UI

4. Stop SQL Service

Default Instance – NET STOP MSSQLSERVER

Named Instance – NET STOP MSSQL$INST1

5. Start it normally.

In some situations, you might run into issue where you would get this error while following above steps.

Login failed for user ‘LoginName’. Reason: Server is in single user mode. Only one administrator can connect at this time. (Microsoft SQL Server, Error: 18461)

You can refer my friend Balmukund’s blog to make a connection in single user mode via start up parameter “m”

Help: How to fix error – Reason: Server is in single user mode. Only one administrator can connect at this time

Once you are able to connect using SQLCMD, you need to use T-SQL to fix the affinity value. Hope this blog helps you to solve one of the things I discovered by my experiments. Do let me know.

Reference: Pinal Dave (http://blog.sqlauthority.com)

While working with transaction log backup restore or log shipping, you must have seen below error message.

Msg 4305, Level 16, State 1, Line 47
The log in this backup set begins at LSN 33000000048000001, which is too recent to apply to the database. An earlier log backup that includes LSN 33000000044800001 can be restored.
Msg 3013, Level 16, State 1, Line 47
RESTORE LOG is terminating abnormally.

Let’s have a look at the cause of this message. Before that let’s have our sample database ready with backups so that we can play with them.

--Create an empty database
CREATE DATABASE SQLAuthority
GO
-- use the database
USE SQLAuthority
GO
--create first test table.
CREATE TABLE MyTestTable1 (iCol INT)
-- perform full backup F1
BACKUP DATABASE SQLAuthority TO DISK = 'c:\SQLAuthority_F1.bak' WITH FORMAT
--create second test table.
CREATE TABLE MyTestTable2 (a INT)
-- perform transaction log backup T1
BACKUP LOG SQLAuthority TO DISK = 'c:\SQLAuthority_T1.trn' WITH FORMAT
--create third test table.
CREATE TABLE MyTestTable3 (a INT)
-- perform transaction log backup T2
BACKUP LOG SQLAuthority TO DISK =  'c:\SQLAuthority_T2.trn' WITH FORMAT
--create forth test table.
CREATE TABLE MyTestTable4 (a INT)
-- perform full backup F2
BACKUP DATABASE SQLAuthority TO DISK = 'c:\SQLAuthority_F2.bak' WITH FORMAT


Now, we have backups taken in below order:

1. Full back up ‘c:\SQLAuthority_F1.bak’
2. First Transaction log backup ‘c:\SQLAuthority_T1.trn’
3. Second Transaction log backup ‘c:\SQLAuthority_T2.trn’
4. Full back up ‘c:\SQLAuthority_F2.bak’

Reproduce the Error

-- Let’s perform restore now but before that, lets drop the database
USE MASTER
GO
ALTER DATABASE SQLAuthority SET single_user WITH ROLLBACK IMMEDIATE
GO
DROP DATABASE SQLAuthority
GO
RESTORE DATABASE SQLAuthority FROM DISK = 'c:\SQLAuthority_F1.bak' WITH NORECOVERY
GO
RESTORE LOG SQLAuthority FROM DISK = 'c:\SQLAuthority_T2.trn' WITH NORECOVERY
GO


Here is the error.

Processed 280 pages for database ‘SQLAuthority’, file ‘SQLAuthority’ on file 1.
Processed 6 pages for database ‘SQLAuthority’, file ‘SQLAuthority_log’ on file 1.
RESTORE DATABASE successfully processed 286 pages in 0.148 seconds (15.097 MB/sec).
Msg 4305, Level 16, State 1, Line 43
The log in this backup set begins at LSN 33000000026400001, which is too recent to apply to the database. An earlier log backup that includes LSN 33000000023200001 can be restored.
Msg 3013, Level 16, State 1, Line 43
RESTORE LOG is terminating abnormally.

The error appeared because we have skipped T1. As the message says “An earlier log backup that includes … can be restored”. We can get list of backups taken from MSDB database on source server.

DECLARE @db_name VARCHAR(100)
SELECT @db_name = 'SQLAuthority'
--Get Backup History for required database
SELECT TOP (30) s.database_name
,m.physical_device_name
,CAST(CAST(s.backup_size / 1000000 AS INT) AS VARCHAR(14)) + ' ' + 'MB' AS bkSize
,CAST(DATEDIFF(second, s.backup_start_date, s.backup_finish_date) AS VARCHAR(4)) + ' ' + 'Seconds' TimeTaken
,s.backup_start_date
,CAST(s.first_lsn AS VARCHAR(50)) AS first_lsn
,CAST(s.last_lsn AS VARCHAR(50)) AS last_lsn
,CASE s.[type]
WHEN 'D'
THEN 'Full'
WHEN 'I'
THEN 'Differential'
WHEN 'L'
THEN 'Transaction Log'
END AS BackupType
,s.server_name
,s.recovery_model
FROM msdb.dbo.backupset s
INNER JOIN msdb.dbo.backupmediafamily m ON s.media_set_id = m.media_set_id
WHERE s.database_name = @db_name
ORDER BY backup_start_date DESC
,backup_finish_date


We can also get the same information from ERRORLOG as well.

SQL SERVER – Where is ERRORLOG? Various Ways to Find its Location

Here are the sample messages. I have trimmed them for clarity.

2015-08-08 17:49:06.340 Backup Database backed up. Database: SQLAuthority, creation date(time): 2015/08/08(17:49:05), pages dumped: 291, first LSN: 33:136:179, last LSN: 33:232:1, number of dump devices: 1, device information: (FILE=1, TYPE=DISK: {‘c:\SQLAuthority_F1.bak’}).

2015-08-08 17:49:06.550 Backup Log was backed up. Database: SQLAuthority, creation date(time): 2015/08/08(17:49:05), first LSN: 33:136:179, last LSN: 33:264:1, number of dump devices: 1, device information: (FILE=1, TYPE=DISK: {‘c:\SQLAuthority_T1.trn’}).

2015-08-08 17:49:06.680 Backup Log was backed up. Database: SQLAuthority, creation date(time): 2015/08/08(17:49:05), first LSN: 33:264:1, last LSN: 33:280:1, number of dump devices: 1, device information: (FILE=1, TYPE=DISK: {‘c:\SQLAuthority_T2.trn’}).

2015-08-08 17:49:07.110 Backup Database backed up. Database: SQLAuthority, creation date(time): 2015/08/08(17:49:05), pages dumped: 291, first LSN: 33:312:143, last LSN: 33:392:1, number of dump devices: 1, device information: (FILE=1, TYPE=DISK: {‘c:\SQLAuthority_F2.bak’}).

So, once we find missing transaction log backup, we need to restore them in sequence. Do you have any other trick to solve this issue? If you can share, it would help others as well.
Reference: Pinal Dave (http://blog.sqlauthority.com)

Recently I was playing with tempdb database in SQL Server and made some mistake. Due to that I was then not able to start SQL Service itself. Since it was an interesting problem so worth blogging about it.

Let’s have a look at problem. Whenever there is a SQL startup trouble, always start from SQL ERRORLOG. If you are

SQL SERVER – Where is ERRORLOG? Various Ways to Find its Location

2015-08-08 15:07:16.67 spid10s Clearing tempdb database.
…
2015-08-08 15:07:16.99 spid10s Error: 17066, Severity: 16, State: 1.
2015-08-08 15:07:16.99 <span style="color: #ff0000;">spid10s SQL Server Assertion: File: <"logmgr.cpp"</span>>,<span style="color: #ff0000;"> line=14870 Failed Assertion = '</span>logFileSize<span style="color: #ff0000;"> <= BlkToByteOffset ((ULONG</span>)<span style="color: #ff0000;">-1)'</span>. This error may be timing-related. If the error persists after rerunning the statement, use DBCC CHECKDB to check the database for structural integrity, or restart the server to ensure in-memory data structures are not corrupted.

2015-08-08 15:07:16.99 spid10s Error: 3624, Severity: 20, State: 1.
2015-08-08 15:07:16.99 spid10s A system assertion check has failed. Check the SQL Server error log for details. Typically, an assertion failure is caused by a software bug or data corruption. To check for database corruption, consider running DBCC CHECKDB. If you agreed to send dumps to Microsoft during setup, a mini dump will be sent to Microsoft. An update might be available from Microsoft in the latest Service Pack or in a Hotfix from Technical Support.

2015-08-08 15:07:17.00 spid10s Error: 5173, Severity: 16, State: 1.
2015-08-08 15:07:17.00 spid10s One or more files do not match the primary file of the database. If you are attempting to attach a database, retry the operation with the correct files. If this is an existing database, the file may be corrupted and should be restored from a backup.

2015-08-08 15:07:17.00 spid10s Error: 1802, Severity: 16, State: 4.
2015-08-08 15:07:17.00 spid10s CREATE DATABASE failed. Some file names listed could not be created. Check related errors.
2015-08-08 15:07:17.00 spid10s Could not create tempdb. You may not have enough disk space available. Free additional disk space by deleting other files on the tempdb drive and then restart SQL Server. Check for additional errors in the event log that may indicate why the tempdb files could not be initialized.

Above is snippet of ERRORLOG where I have removed many things to make this clear in the blog. Here is something interesting. If we look at the file size of the tempdb transaction log file, its 0 KB (yes zero KB). I thought its TempDB so it would be recreated anyways, so I deleted them but still no joy. I was having another SQL instance with the same build of SQL Server so I picked files from there and dumped to desired location but SQL Startup automatically replaced those files.

Solution:

A) Start SQL Server with startup parameter /f which stands for minimal configuration. You need to change instance name/server name. For me it is named instance of SQL Server called SQL2014.

net start MSSQL$SQL2014 /f

If you have default instance, then it would be called as MSSQLServer.

B) Connect to SQL via SQLCMD. You need to change instance name/server name

SQLCMD -S.\SQL2014

C) After connection is made, check the file sizes of tempdb database

SELECT size FROM sys.master_files WHERE database_id = 2


In my case I got 0 for LDF file and that was the cause of the whole problem.

D) If you get zero, then alter the database and change the size again

ALTER DATABASE [tempdb] MODIFY FILE(NAME=N'templog',SIZE= 10240KB);

E) Check size again by running the same command as step 3. Then exit from SQLCMD

F) Stop SQL Service

net stop MSSQL$SQL2014

G) Start SQL normally and if you were hitting the same issue, then it should be fixed. The same command as step 1 but without /f parameter.

net start MSSQL$SQL2014

Here is the screenshot of the solution

What caused the problem?

As I said in the beginning, I was playing with TempDB and by mistake I ran below command

ALTER DATABASE [tempdb] MODIFY FILE(NAME=N'templog',SIZE= 0);

That zero in size has caused the whole problem.

Note: Please do not run above command on production server. 

Reference: Pinal Dave (http://blog.sqlauthority.com)

This is one of the most common error searched on my blog search (http://search.sqlauthority.com) and lately I realized that I have not written any blog about the cause of such error and fixing that.

If you ever talk to an SQL Expert about login failed for user, he/she might ask for the state of the message or complete error message. All login failed for user message would have error number 18456 but the state of the message in the ERRORLOG would tell the exact cause of login failure. It is important to note that SQL Server does not tell the exact state to the client and it would be a state 1 always.

Login failed for user ‘sa’. (.Net SqlClient Data Provider)
——————————
Server Name: BIGPINAL
Error Number: 18456
Severity: 14
State: 1
Line Number: 65536

If you are new to SQL Server then use below to find ERRORLOG

SQL SERVER – Where is ERRORLOG? Various Ways to Find its Location

Here is the message in ERRORLOG file

Error: 18456, Severity: 14, State: 58.
Login failed for user ‘sa’. Reason: An attempt to login using SQL authentication failed. Server is configured for Windows authentication only. [CLIENT: <local machine>]

As we can see, the message in ERRORLOG file is having state 58 and exact reason.

Coming back to message in title, here is the complete message

Error: 18456, Severity: 14, State: 11.
Login failed for user ‘domain\user$’. Reason: Token-based server access validation failed with an infrastructure error. Check for previous errors. [CLIENT: 10.10.10.10]

There are few things to note about this message.

1. It would come for windows accounts only.
2. If you are seeing $ in the user name, then it is a machine account / computer account. This mostly happens when a “service” running on the remote machine is connecting to SQL.
3. If you are not seeing $ in the user name, then it is a user account.

Possible causes and solutions for State 11

1. If you are seeing login failure for machine account, then you may want to give permission to machine account. You need to run T-SQL like below (replace domain and machine name)

CREATE LOGIN [<Domain>\<Machine account>$] FROM WINDOWS


2. If it’s a windows domain user account, then it needs to have connect permission

GRANT CONNECT SQL TO [DOMAIN\User]

3. If few cases, account is part of a group (or many groups) then you need to make sure there is no “DENY” permission inherited via group membership.

SELECT sp.[name],sp.type_desc
FROM sys.server_principals sp
INNER JOIN sys.server_permissions PERM ON sp.principal_id = PERM.grantee_principal_id
WHERE PERM.state_desc = 'DENY'

4. If the message only comes with local connectivity and same account works fine remotely then it could be a UAC issue. This means that Elevation is required to connect properly, but you do not see any of the normal prompts to elevate. To use elevation (while launching the SSMS to connect to a locally running instance of SQL) Right click->Select “Run as administrator”.

Depending on the situation, out of four, any option might work. If this didn’t work for you, please comment and let me know. I would love to learn from you too.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

Some of the simplest recommendations can lead you to some bizarre output that we never anticipated. I always recommend running DBCC CHECKDB on mission critical databases because we need to be 200% sure that the database disk system is healthy from time to time. During one of my sessions, I gave this as a generic recommendation and guess what, I got a strange message from one of the attendees. They then claimed that they were getting errors of mag 2520. One further probing and requesting for the actual error, I understood the problem. They were running CHECKDB on the ResourceDB. I asked – “Why are you doing this on ResourceDB?”. They answered, “Pinal, isn’t ResourceDB a database? Wouldn’t it not get corrupt? What is wrong in doing a CHECKDB on it? If I cant, how can I do it?”

This was a fully loaded question and I had to answer it elaborately. So I took this blog post to explain some of the basics.

Resource database is a system database in SQL Server 2005. Since this database has some special attributes and differs from regular databases in some respects, these indirectly affect the ability to do some operations on this database. One of the areas is the ability to perform DBCC checks and repair on the resource database. The following are the key points to keep in mind when you plan to run any kind of check against the resource database:

When the server is running in multi-user mode, running the command
DBCC CHECKDB ( mssqlsystemresource )


will give the following output:

Msg 2520, Level 16, State 11, Line 1
Could not find database ‘mssqlsystemresource‘. The database either does not exist, or was dropped before a statement tried to use it. Verify if the database exists by querying the sys.databases catalog view.

This is applicable for commands like DBCC CHECKALLOC and DBCC CHECKCATALOG.

When the server is running in multi-user mode, running DBCC CHECKDB on the master database will run DBCC CHECKDB on resource database as well. In the DBCC CHECKDB output result set, we will notice the following sections:

DBCC CHECKDB ( MASTER )

CHECKDB found 0 allocation errors and 0 consistency errors in database ‘master’.
DBCC execution completed. If DBCC printed error messages, contact your system administrator.
CHECKDB found 0 allocation errors and 0 consistency errors in database ‘mssqlsystemresource’.
DBCC execution completed. If DBCC printed error messages, contact your system administrator.

DBCC execution completed. If DBCC printed error messages, contact your system administrator.

Errorlog will show the following entries:

2015-08-25 14:44:22.650                spid63   DBCC CHECKDB (master) executed by sa found 0 errors and repaired 0 errors. Elapsed time: 0 hours 0 minutes 0 seconds.  Internal database snapshot has split point LSN = 000000e4:0000017c:0001 and first LSN = 000000e4:00000179:0001.
2015-08-25 14:44:22.840                spid63   DBCC CHECKDB (mssqlsystemresource) executed by sa found 0 errors and repaired 0 errors. Elapsed time: 0 hours 0 minutes 0 seconds.

When the above checks are run, for the master database checks alone the internal transient replica is created. For resource database, no replica is created for the checks.

When the server is started in single_user mode to perform repairs in master, a special mechanism is used. Only for master database, the check command is run with the repair option. When it comes to checking the resource database, the normal check is done with repair disabled. You can see the effect of that from the errorlog entries:

2015-08-25 15:44:22.650 spid61      DBCC CHECKDB (master, repair_allow_data_loss) executed by sa found 0 errors and repaired 0 errors. Elapsed time: 0 hours 0 minutes 0 seconds.
2015-08-25 15:44:22.650 spid61      DBCC CHECKDB (mssqlsystemresource) executed by sa found 0 errors and repaired 0 errors. Elapsed time: 0 hours 0 minutes 0 seconds.

If there is a corruption problem in the resource database and there is a need to get it repaired for the efficient functioning of the SQL Server, then you will need to replace the resource database if there is any physical corruption introduced in the database. Normally users will not have the ability to write to this database. So unless there is a hardware problem, there is less chance that this database should get corrupted ever.

Though this blog started for an unusual error message, I thought it was important to talk some of these fine prints when working with ResourceDB. Hope you have never encountered any error with ResourceDB, if you have – do let me know via comments to what you did in that situation. Would love to learn from you.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Our software systems are funny and sometimes give us strange values. I don’t quite venture out trying to fiddle around with system level properties on my servers, but having said that, this query had come from one of my colleagues who was working on his SQL Server instance and he found something strange.

When working on the server and when an error occurs, I generally ask for the @@Version. This is a standard practice for me to do it because I don’t have to ask about the version installed, what service pack, CU update has been done on the server etc. This output would give me all this information in black and white.

As I was so used to this kind of output, my friend said that the value that is returned by his @@Servername was NULL. This was such a surprise and I wanted to see it in real life. He was kind enough to show me via Skype that this was indeed the case. This got me thinking and I needed to solve this trivial problem. I had gone to bed early as it was a lazy Sunday and I couldn’t put myself to sleep as this was plaguing my mind – I needed a solution. Finally, after an hour and getting some help from my SQL buddies, I found the deal to crack this problem. Strange but I am sure many of you have encountered this somewhere and were looking for a solution.

The @@ServerName refers to the local server that SQL Server is running on.  This property is set during installation, but there are instances where @@ServerName may be NULL or may not be correct.

Running the following will show the current TSQL value for this property:

SELECT @@SERVERNAME;
GO

To Fix the Problem

If the @@ServerName property is NULL, running the following will fix the issue- but only after you restart the SQL Server instance. Yep, fixing this requires an outage.

EXEC sp_addserver '<LocalServerName>', local;
GO

If the @@ServerName property is incorrect, run the following to correct the issue:

EXEC sp_dropserver 'old_name';
GO
EXEC sp_addserver 'new_name', 'local';
GO


Though the solution to this was so simple, I did find it strange about this behavior. I am yet to come to terms this can every happen. If you ever encountered this error on your server, can you please let us know how you landed to this situation. I am eager to learn the cause for this rather than just looking at solution to the problem.

Reference: Pinal Dave (http://blog.sqlauthority.com)

The best way I have learnt about building SQL Server AlwaysOn Availability Group is by contacting my close friends who have written the only book on this topic. It is not common that I ping them for such technical queries but that is what friendship is made off. Recently, while preparing a demo for AlwaysOn Availability Group, I ran into interesting issue and was able to fix it as well.

First, let us look at the configuration.

SERVER1              –              Primary Replica
SERVER2              –              Secondary Replica

They are synchronous and configured for automatic failover. I have shutdown SERVER1 and as expected, SERVER2 became primary. As soon as SERVER1 came back, it became secondary but databases were NOT synchronizing. That was a surprise to me as I was expecting to see the data movement from SERVER2 to SERVER1 now.

As always, my troubleshooting checklist has ERRORLOG is my first point of investigation. If you have not seen earlier, you can use below blog to know how to find ERRORLOG

SQL SERVER – Where is ERRORLOG? Various Ways to Find its Location

On SERVER1, I found below messages multiple times.

2015-08-24 01:24:18.480 Logon        Database Mirroring login attempt failed with error: 'Connection handshake failed. An OS call failed: (8009030c) 0x8009030c(The logon attempt failed). State 67.'.  [CLIENT: 10.0.0.2]

There are some interesting learning/finding from above messages.

1. Database Mirroring and AlwaysOn uses same endpoint and that is why we are seeing Database Mirroring in error message even if I am using AlwaysOn Availability Group.
2. IP Address in the message belong to SERVER2 (which is primary as of now).

This means that SERVER2 is not able to talk to SERVER1 via Mirroring port. I searched on hex code 8009030c is not helping much because it means – The logon attempt failed, which is already there is the same message.

Steps which I tried after searching on internet (and they didn’t help)

1. I logged in to SERVER2 via remote desktop and tried making connection to SERVER1 via Management Studio and it was working fine.
2. Restart endpoints on both replicas. No joy!

USE MASTER
GO
ALTER ENDPOINT hadr_endpoint STATE = STOPPED;
GO
ALTER ENDPOINT hadr_endpoint STATE = STARTED;
GO


3. PING from and to each other. This also worked well which means DNS was not an issue.
4. Verified that SQL was running under domain account and account was not locked.

None of the above helped. Taking last hint, Then I started backtracking to recall what I did with my SQL Service account. It did not take long to realize that I had change service account domain password in recent past. I rarely use AlwaysOn Availability Group so I changed the service account password on SERVER1 but forgot on SERVER2.

When I was looking at errorlog, I also found below

SQL Server failed to communicate with filter daemon launch service  (Windows error: The service did not start due to a logon failure.). Full-Text filter daemon process failed to start. Full-text search functionality will not be available.

Above message came when SQL was trying to start full-text service automatically. Since I didn’t change the password for full-text service, I was seeing above message.

Solution (which worked for me): Correct the Service account password for SQL Server related service via SQL Server Configuration Manager.

When I corrected service account password, the databases were synchronized and I was happy again.

Reference: Pinal Dave (http://blog.sqlauthority.com)

Recently I was getting ready for an upcoming demo session and found that my maintenance plan for CHECKDB were failing. When I executed manually, I got below error

Execution failed. See the maintenance plan and SQL Server Agent job history logs for details.

I followed error message and checked the history of maintenance plan, here was the error.

Failed: (0) Alter failed for server ‘ComputerName\InstanceName’

My research yielded https://support.microsoft.com/en-us/kb/945067 which was talking about incorrect setting of Allow Updates in sp_configure. In my case it was just fine. I have put profiler and found below statement failing.

EXEC sys.sp_configure N'user options', 0 RECONFIGURE


When I executed below statement from SQL Server Management Studio, I got below error

Configuration option ‘user options’ changed from 0 to 0. Run the RECONFIGURE statement to install.

Msg 5807, Level 16, State 1, Line 1

Recovery intervals above 60 minutes not recommended.

Use the RECONFIGURE WITH OVERRIDE statement to force this configuration.

In my case, it was recovery interval value which I as 10000, was the problem. Once I bought it down to 60, I was able to run the maintenance plan without any problem.

sp_configure 'recovery interval (min)', 60
GO
RECONFIGURE WITH override
GO

As shown above, the maintenance plan is using the “reconfigure” command and getting an error. Due to that it wasn’t able to proceed further and failing with “Alter failed for server” error. Hope it’s clear that if you are getting the same error, you should run the RECONFIGURE command from SQL Server Management studio and check the any error. Here are the various errors which I saw.

1. Make sure that “Allow updates” is 0
2. Make sure that the “recovery interval (min0” is less than 60
3. Make sure that the service account is having “Lock Pages in Memory” when AWE is enabled.

Later, when I tried to reproduce the same on SQL 2012 and SQL Server 2014, I was not able to. This is a good news that the issue is fixed by Microsoft in later versions.

Have you ever found the same error and the solution was other than three which I mentioned? Please comment and let me know so that it can help others also.

Reference: Pinal Dave (http://blog.sqlauthority.com)

One of the most difficult and haunted issue for any DBA is to do a root cause analysis or RCA of a performance issue with SQL Server. That is why monitoring of SQL Server is very important and companies spend a lot of time in buying tools for that. Sometimes, when there is no monitoring then we have to look at all possible logs available. I have been a big believer of checking SQL Server ERRORLOG for any unexpected behavior. Sometimes we do get good piece of information from there. If you don’t know the location of ERRORLOG file, then you should read below blog.

SQL SERVER – Where is ERRORLOG? Various Ways to Find its Location

Below is one of the error message that I have seen in my one of my client’s ERRORLOG file.

Autogrow of file ‘<File Name>’ in database ‘<Database Name>’ was cancelled by user or timed out after %d milliseconds.  Use ALTER DATABASE to set a smaller FILEGROWTH value for this file or to explicitly set a new file size.

The error indicates that the transaction/session cannot wait until the file growth completes and then the session is been cancelled. Due to that, the file growth is also cancelled. The reason is that the file growth cannot complete in time. In above message, we need to note few thing: a) which file, b) which database and c) how much time before timeout. When the database file (MDF or LDF) is full and it is set to grow automatically, the growth would kick in and transaction would be halted until growth is complete. The time taken to grow would be dependent on two factors. 1) size of growth 2) Disk speed.

Let’s assume that database transaction log size is 50 GB and the file growth is 10%, it will grow by 5 GB. The time needed for this size increase exceeds the time of the session which it can wait. Most of the application have timeout of 30 seconds, which is more than enough for auto-grow to complete. If the disk can response much faster, the issue may not appear at all.

A seasoned DBA performs the transaction log backup regularly and avoid such a large size of transaction log. With a large transaction log, sometimes it can also cause a long database startup time after a restart if some problem happens previously. To avoid the issue in our case, we can try to set the file growth to a smaller value such as 1GB or 500 MB. Microsoft has always recommended doing manual growth so that size is in predictable range. Autogrow is like a band-aid. As per SQL industry best practices, we should avoid file auto-growth and initialize the file to a reasonable size according to the application requirement.  Then we can leave the file growth to 1GB or a large size according to the disk response time. This applies to both data file (MDF or NDF) and transaction log file (LDF).

Since SQL Server 2005, SQL Server has a feature called “instant file initialization” which can accelerate the file growth. Its important to keep in mind that only data files can be initialized instantaneously by this feature. Instant file initialization would be activated when the SQL Server (MSSQLSERVER) service account has been granted SE_MANAGE_VOLUME (Perform Volume Maintenance Tasks). https://technet.microsoft.com/en-us/library/ms175935.aspx

To summarize:

• Increase the file size in a smaller value. Prefer manual growth or fixed size auto-grow.
• Consider providing “Perform Volume Maintenance Tasks” permission to SQL Service account. This would allow “instant file initialization” for faster growth of data files.
• Perform transaction log backup regularly so avoid huge size transaction log in full recovery model.
• Make sure disk performance is good.

Have you ever seen such errors on your production environment?

Reference: Pinal Dave (http://blog.sqlauthority.com)

Replication as a topic is quite old and proven technology. After I saw the introduction of SQL Server AlwaysOn from SQL Server 2012, I hardly see folks talking about replication. For me replication will always be a technology which allows for partial (selected tables/articles) synchronization technology, which a lot of people in a large DB world still require.

Recently, I was trying to configure replication for one of my upcoming sessions internally playing around. I was getting the same error for “Configure Distribution” and “New Publication”. Here is the text of the error message:

TITLE: Configure Distribution Wizard
——————————
SQL Server is unable to connect to server ‘SQL16NODEB\SQL2014’.
——————————
ADDITIONAL INFORMATION:
SQL Server replication requires the actual server name to make a connection to the server. Specify the actual server name, ‘MyServer‘. (Replication.Utilities)
——————————
BUTTONS:
OK
——————————

On my machine, here are the steps to reproduce this problem:

1. Open Microsoft SQL Server Management Studio and connect to Server
2. Expand the server node.
3. Expand Replication
4. Right click “Local Publications” and select “New Publication …”

OR

1. Open Microsoft SQL Server Management Studio and connect to Server
2. Expand the server node.
3. Right click “Replication” and choose “Configure Distribution”

First, notice the error message, it talks about two names here: SQL16NODEB\SQL2014 and MyServer. I have captured profiler on the SQL instance to find out what is being done by SQL Server to raise this error. Here are the two meaningful queries:

SELECT SERVERPROPERTY(N'servername')
SELECT @@SERVERNAME


When I run them individually in SSMS, I get below output:

After seeing this I realized that I changed my server name recently for some other demo and that is causing a problem.

DECLARE @actualname NVARCHAR(100)
DECLARE @currentname NVARCHAR(100)
SELECT @actualname = CONVERT(NVARCHAR(100), SERVERPROPERTY(N'servername'))
SELECT @currentname = @@SERVERNAME
EXEC sp_dropserver @currentname
EXEC sp_addserver @actualname, local

After running above, we need to restart the SQL Server Service. Then, make sure that you are getting the same output for below queries:

SELECT SERVERPROPERTY(N'servername')
SELECT @@SERVERNAME

Have you ever encountered the same issue and found some other solution? Please comment and let me know so that we can learn from each other.

Reference: Pinal Dave (http://blog.sqlauthority.com)

SQL Server supports Error handling efficiency. You can do this with the help of TRY and CATCH blocks. If there is an error you can get the exact error message using the system function ERROR_MESSAGE().

I will demonstrate it using the following code

BEGIN TRY
SELECT 1+'SQL' AS RESULT
END TRY
BEGIN CATCH
SELECT ERROR_MESSAGE() AS ERROR
END CATCH


The result you get is

ERROR
——————————————————————————
Conversion failed when converting the varchar value ‘SQL’ to data type int.

It is because number 1 cannot be added to a string

But ERROR_MESSAGE will work only inside the CATCH block. If you use it outside of it, you will get nothing

BEGIN TRY
SELECT 1+'SQL' AS RESULT
END TRY
BEGIN CATCH
SELECT ERROR_MESSAGE() AS ERROR
END CATCH
SELECT ERROR_MESSAGE() AS ERROR

If you execute the above, you get the following two result sets

Resultset 1:

ERROR
——————————————————————————-
Conversion failed when converting the varchar value ‘SQL’ to data type int.

Resultset 2:

ERROR
——————————————————————————-
NULL

As you see ERROR_MESSAGE() returns NULL when used outside of CATCH block.

Reference: Pinal Dave (http://blog.sqlauthority.com)

I always try to keep myself up-to-date with latest release of SQL Server. As a part of that I wanted to upgrade my SQL Server 2016 CTP2.2 to CTP2.3 version. While doing in-place upgrade I was blocked with few error and I was not able to proceed. Here is the screenshot of upgrade screen.

When we click on those messages, here is the text.

Error # 1

—————————
Rule Check Result
—————————

Rule “Valid DSN” failed.
The report server configuration is not complete or is invalid. Use Reporting Services Configuration Manager to verify the report server configuration.
—————————
OK
—————————

Error # 2

—————————

Rule Check Result
—————————
Rule “Valid Database compatibility level and successful connection” failed.

The report server database is not a supported compatibility level or a connection cannot be established. Use Reporting Services Configuration Manager to verify the report server configuration and SQL Server management tools to verify the compatibility level.

—————————
OK
—————————

Then I looked into the setup log to see what they have to say and found below. Below is what is also shown in the pop-up message.

(08) 2015-09-18 10:46:10 Slp: Initializing rule      : Valid DSN
(08) 2015-09-18 10:46:10 Slp: Rule is will be executed  : True
(08) 2015-09-18 10:46:10 Slp: Init rule target object: Microsoft.SqlServer.Configuration.RSExtension.DsnUpgradeBlockers
(08) 2015-09-18 10:46:10 RS: Error validating the dsn: No DSN server name found.
(08) 2015-09-18 10:46:10 Slp: Evaluating rule        : RS_ValidDSN
(08) 2015-09-18 10:46:10 Slp: Rule running on machine: SQL16NODEB
(08) 2015-09-18 10:46:10 Slp: Rule evaluation done   : Failed
(08) 2015-09-18 10:46:10 Slp: Rule evaluation message: The report server configuration is not complete or is invalid. Use Reporting Services Configuration Manager to verify the report server configuration.
(08) 2015-09-18 10:46:10 Slp: Send result to channel : RulesEngineNotificationChannel
…
(08) 2015-09-18 10:46:10 Slp: Initializing rule      : Valid Database compatibility level and successful connection
(08) 2015-09-18 10:46:10 Slp: Rule is will be executed  : True
(08) 2015-09-18 10:46:10 Slp: Init rule target object: Microsoft.SqlServer.Configuration.RSExtension.DsnUpgradeBlockers
(08) 2015-09-18 10:46:10 RS: Error validating the dsn: No DSN server name found.
(08) 2015-09-18 10:46:10 Slp: Evaluating rule        : RS_ValidDatabaseVersion
(08) 2015-09-18 10:46:10 Slp: Rule running on machine: SQL16NODEB
(08) 2015-09-18 10:46:10 Slp: Rule evaluation done   : Failed
(08) 2015-09-18 10:46:10 Slp: Rule evaluation message: The report server database is not a supported compatibility level or a connection cannot be established. Use Reporting Services Configuration Manager to verify the report server configuration and SQL Server management tools to verify the compatibility level.

The log clearly mentioned the action I need to take.

Solution

1. Open Reporting Service Configuration Manager and go to Database tab and we see the “Current Report Server database” as empty. That is THE problem.

2. To fix that, click on “Change database” and below Wizard would open

3. We can easily follow the wizard and provide details. I have used default settings. Database Server:

4. On Database Page:

5. Credentials:

6. Summary

7. Progress and Finish

7. Finish

Once above steps are complete, we can come back to screen which was giving error and choose “Re Run” and then we should be able to proceed next with setup. I was able to upgrade to next version without above error:

Have you even seen such upgrade blockers?

Reference: Pinal Dave (http://blog.sqlauthority.com)

Another email from one of my blog reader, which made me think about the incorrect information which is available on the internet.

Hi Pinal,

I have corruption in model database and due to that SQL Service is not able to start. Here are the messages in SQL ERRORLOG

2015-09-23 21:45:07.00 spid11s     Srting up database ‘model’.

2015-09-23 21:45:07.45 spid11s     Error: 5125, Severity: 24, State: 2.

2015-09-23 21:45:07.45 spid11s     File ‘E:\Program Files\Microsoft SQL Server\MSSQL12.SQL2014\MSSQL\DATA\model.mdf’ appears to have been truncated by the operating system.  Expected size is 4288 KB but actual size is 4280 KB.

2015-09-23 21:45:07.49 spid11s     SQL Trace was stopped due to server shutdown. Trace ID = ‘1’. This is an informational message only; no user action is required.

What should I do? I have backup of the model database and want to restore it. I have spoken to many folks and I was getting different answers. Can you please provide me right direction?

Thanks,
<Name Hidden>

In this case problem is straight forward, restore backup of model when current model is corrupted and SQL is not starting due to that. If we search on internet, we get below incorrect hints:

1. Start SQL with trace flag 3608. To do that Use this from command prompt: sqlservr.exe -T3608
2. Start SQL Server via command prompt using sqlservr.exe -c –m

To test above, I renamed the database files of model database and then tried restore after started via trace flag 3608. Here is the message which I received during restore.

Shared Memory Provider: The pipe has been ended.
Communication link failure

Restoring model is not a big deal, it doesn’t need any special switch or startup parameter. We can use T-SQL query or Management Studio UI to restore model.

Our problem above is that SQL is not getting started because model is in bad state. Below are the working solutions.

Hopefully, this flow chart would help in getting exact steps needed.  Model can be restored directly via T-SQL or SSMS.

Have you ever found incorrect information on the internet? Never trust information from untrusted sources.

Reference: Pinal Dave (http://blog.sqlauthority.com)

One of the blog reader sent me the below mail. I always find that using linked server have been of concern and trouble at a number of places. So when the mail landed – well I was not surprised that there was a problem. It is common and hence the resolution for the same is also common yet not discussed. So I thought this blog will bring the nuances that I thought was the resolution and which helped our blog reader.

Hi Pinal,
I have two SQL Server instances on same machine and I want to fetch data from each other. So I went ahead and created linked server. In SQL Server Management Studio, Server Objects > Right-click, Linked Servers, and then selected New linked server. I gave the remote server name as SQL16NODEB\SQL2014 and “Server type” as SQL Server as shown below.

While saving I got error as below

TITLE: Microsoft SQL Server Management Studio

——————————
The linked server has been created but failed a connection test. Do you want to keep the linked server?
——————————

The OLE DB provider “SQLNCLI11” for linked server “SQL16NODEB\SQL2014” reported an error. Authentication failed.

Cannot initialize the data source object of OLE DB provider “SQLNCLI11” for linked server “SQL16NODEB\SQL2014”.

OLE DB provider “SQLNCLI11” for linked server “SQL16NODEB\SQL2014” returned message “Invalid authorization specification”. (Microsoft SQL Server, Error: 7399)

If I hit No, the linked server is not created. If I created Yes, linked server is created, but any query is failing with the same error.

What should I do?

Thanks,
<name hidden>

I asked to check and run

sp_testlinkedserver N'SQL16NODEB\SQL2014'


and as expected, it failed with the same error.

OLE DB provider “SQLNCLI11” for linked server “SQL16NODEB\SQL2014” returned message “Invalid authorization specification”.

Msg 7399, Level 16, State 1, Procedure sp_testlinkedserver, Line 1

The OLE DB provider “SQLNCLI11” for linked server “SQL16NODEB\SQL2014” reported an error. Authentication failed.

Msg 7303, Level 16, State 1, Procedure sp_testlinkedserver, Line 1

Cannot initialize the data source object of OLE DB provider “SQLNCLI11” for linked server “SQL16NODEB\SQL2014”.

The message “Invalid authorization specification” means that linked server settings to connect to the server are not correct.  To fix that, we need to go back to linked server properties and go to “security” tab and choose the proper authentication method from last two.

If there is a SQL Login to be used, then we need to provide account and its password.

Here is the T-SQL command for 3rd option where @useself is set to true

USE [master]
GO
EXEC MASTER.dbo.sp_addlinkedsrvlogin @rmtsrvname = N'SQL16NODEB\SQL2014', @locallogin = NULL , @useself = N'True', @rmtuser = N''
GO
'

Here is the T-SQL command for 4th option where @useself is set to false so we need to provide @rmtuser and @rmtpassword

USE [master]
GO
EXEC MASTER.dbo.sp_addlinkedsrvlogin @rmtsrvname = N'SQL16NODEB\SQL2014', @locallogin = NULL , @useself = N'False', @rmtuser = N'sa', @rmtpassword = N'sa'
GO


Hope this will helps and do let me know if you have ever got this error in your environments.

Reference: Pinal Dave (http://blog.sqlauthority.com)

I travel quite a bit and the month of October and November seem to be filled with half my life is going to be spent hopping from one airport to another. Not to mention the amount of screening that go through every single time in every flight. However strange as it might sound, in one of such airport transits I was waiting for my bag to be screened through the machine and I was watching the screen on the other side where the security officer sits. It was interesting because it was a very skillful task as he was finding a needle in a haystack (in the literal sense). He was able to figure out what items, sharp items and toys which are restricted were getting passed through. I was amused and was transported into a world of wonderland because it was quite a challenging task.

With these memories not subsiding, I landed to my desired destination and started checking my mails. There was one mail that caught my attention because suddenly it was a junior DBA who was trying to give some permissions and was getting an error. I was quick to get interested because it was a topic around security and I wanted to crack it. The message stated like:

Msg 300, Level 14, State 1, Line 1
VIEW SERVER STATE permission was denied on object ‘server’, database ‘master’.
Msg 297, Level 16, State 1, Line 1
The user does not have permission to perform this action.

As I always mention, most of the SQL Server error messages these days are well documented and were self-explanatory. To make sure I got the error message right, I requested the person to send me the query that was being executed that caused the error. The reply was:

SELECT wait_type, wait_time_ms
FROM sys.dm_os_wait_stats

This made complete sense and the solution for this was also simple. From a repro point of view, I created a user Pinal and tried to run the above command. And guess what, as luck might turn out – the same error message as described in the start. To mitigate this error, I was to give the following GRANT and the error message disappeared:

USE MASTER
GO
GRANT VIEW SERVER STATE TO Pinal

Just curious to know have you as a DBA ever granted this right to users? What was the scenario in your case? Would love to see why you don’t want to give to specific users? Will be a great learning for all of us. Let me know via the comments, please.

Reference: Pinal Dave (http://blog.sqlauthority.com)

If you are a DBA and handled Log-shipping as high availability solution, there are a number of common errors that come that you would over a period of time become pro on resolving. Here is one of the common error which you must have seen:

Message
2015-10-13 21:09:05.13     *** Error: The file ‘C:\LS_S\LSDemo_20151013153827.trn’ is too recent to apply to the secondary database ‘LSDemo’.(Microsoft.SqlServer.Management.LogShipping) ***
2015-10-13 21:09:05.13     *** Error: The log in this backup set begins at LSN 32000000047300001, which is too recent to apply to the database. An earlier log backup that includes LSN 32000000047000001 can be restored.
RESTORE LOG is terminating abnormally.(.Net SqlClient Data Provider) ***
Above error is a shown in failure of the history of restore job. If the failure is more than configured thresholds, then we would start seen below error in SQL ERRORLOG on secondary also:
2015-10-14 06:22:00.240 spid60       Error: 14421, Severity: 16, State: 1.
2015-10-14 06:22:00.240 spid60       The log shipping secondary database PinalServer.LSDemo has restore threshold of 45 minutes and is out of sync. No restore was performed for 553 minutes. Restored latency is 4 minutes. Check agent log and logshipping monitor information.

To start troubleshooting, we can look at Job activity monitor on secondary which would fail with the below state:

If you know SQL transaction log backup basics, you might be able to guess the cause. If we look closely to the error, it talks about LSN mismatch. Most of the cases, a manual transaction log backup was taken. I remember few scenarios where a 3^rd party tool would have taken transaction log backup of database which was also part of a log shipping configuration.

Since we know the cause now, what we need to figure out is – where is that “out of band” backup? Here is the query which I have written on my earlier blog.

-- Assign the database name to variable below
DECLARE @db_name VARCHAR(100)
SELECT @db_name = 'LSDemo'
-- query
SELECT TOP (30) s.database_name
,m.physical_device_name
,CAST(CAST(s.backup_size / 1000000 AS INT) AS VARCHAR(14)) + ' ' + 'MB' AS bkSize
,CAST(DATEDIFF(second, s.backup_start_date, s.backup_finish_date) AS VARCHAR(4)) + ' ' + 'Seconds' TimeTaken
,s.backup_start_date
,CAST(s.first_lsn AS VARCHAR(50)) AS first_lsn
,CAST(s.last_lsn AS VARCHAR(50)) AS last_lsn
,CASE s.[type]
WHEN 'D'
THEN 'Full'
WHEN 'I'
THEN 'Differential'
WHEN 'L'
THEN 'Transaction Log'
END AS BackupType
,s.server_name
,s.recovery_model
FROM msdb.dbo.backupset s
INNER JOIN msdb.dbo.backupmediafamily m ON s.media_set_id = m.media_set_id
WHERE s.database_name = @db_name
ORDER BY backup_start_date DESC
,backup_finish_date


Once we run the query, we would get list of backups happened on the database. This information is picked from MSDB database.

Below picture is self-explanatory.

Once we found the “problematic” backup, we need to restore it manually on secondary database. Make sure that we are using either norecovery or standby option so that other logs can be restored. Once file is restored, the restore job would be able to pick-up from the same place and would catch up automatically.

What are the other problems you have seen with Log-shipping? If you can share some of the common errors, it would be of great help for others and I will try to blog about them too with your help.

Reference: Pinal Dave (http://blog.sqlauthority.com)

SQL Server troubleshooting topics seem to be the most searched for in this blog. In continuation to this, blog for today is also inspired with one such error message. Some time back I have written below blog which talks about upgrade script mode of SQL Server:

SQL SERVER – Login Failed For User – Reason Server is in Script Upgrade Mode

One of the reader reached out to me with below error in ERRORLOG

Setting object permissions…
Error: 2714, Severity: 16, State: 6.
There is already an object named ‘TargetServersRole’ in the database.
Error: 2759, Severity: 16, State: 0.
CREATE SCHEMA failed due to previous errors.
Error: 912, Severity: 21, State: 2.
Script level upgrade for database ‘master’ failed because upgrade step ‘sqlagent100_msdb_upgrade.sql’ encountered error 2714, state 6, severity 25. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the ‘master’ database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.
Error: 3417, Severity: 21, State: 3.
Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online.

Above error is truly scary. “Restore master from a full backup, repair it, or rebuild it”. So I asked my friend to share the script ‘sqlagent100_msdb_upgrade.sql’ to check which query is failing. It was not very difficult to find (I searched for “Setting object permissions…” which was printed in ERRORLOG before error) and I found that below is the piece present in script, which was failing.

IF (EXISTS (SELECT *
FROM msdb.dbo.sysusers
WHERE (name = N'TargetServersRole')
AND (issqlrole = 1)))
BEGIN
-- If there are no members in the role, then drop and re-create it
IF ((SELECT COUNT(*)
FROM msdb.dbo.sysusers   su,
msdb.dbo.sysmembers sm
WHERE (su.uid = sm.groupuid)
AND (su.name = N'TargetServersRole')
AND (su.issqlrole = 1)) = 0)
BEGIN
EXECUTE msdb.dbo.sp_droprole @rolename = N'TargetServersRole'
EXECUTE msdb.dbo.sp_addrole @rolename = N'TargetServersRole'
END
END
ELSE
EXECUTE msdb.dbo.sp_addrole @rolename = N'TargetServersRole'

To be very precise, adding the role was failing on MSDB database on his SQL Instance. I asked him to run below and got an error.

Now, Since SQL we were not able to connect to SQL, we need to use trace flag 902 to bypass the script. Here are the steps followed.

1. Applied the trace flag -T902 on SQL Server configuration manager
2. Start SQL Services and it should allow us to connect because the trace flag would bypass upgrade script mode.
3. Find the schema/role TargetServersRole in MSDB and make a note of members, if any.
4. In his case, the schema was having someone else as owner. Asked him to modify using below T-SQL
USE [msdb]
GO
ALTER AUTHORIZATION ON ROLE::[TargetServersRole] TO [TargetServersRole]
GO

5. After fixing the cause, we stopped SQL, removed trace flag and started again.
6. This time script got executed and SQL was out of upgrade mode.

Have you also encountered similar issue?

Reference: Pinal Dave (http://blog.sqlauthority.com)