DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Related

  • Advanced Maintenance of a Multi-Database Citus Cluster With Flyway
  • Beyond Partitioning and Z-Order: A Deep Dive into Liquid Clustering for Unity Catalog Managed Tables
  • Modernizing Cloud Data Automation for Faster Insights
  • Reconciling Privacy Preferences Across Two Datastores With Snowflake and Airflow

Trending

  • Architecting an Embedded Efficiency Layer: A Platform Deep Dive into Day-Two Operational Tuning
  • From APIs to Actions: Rethinking Back-End Design for Agents
  • Why Your DLP Policies Fall Short the Moment AI Agents Enter the Picture
  • Product-Led Software Delivery: Intelligent Platforms for DevOps at Scale
  1. DZone
  2. Data Engineering
  3. Data
  4. Stop Adding Indexes: What's Actually Slowing Your SQL Server Queries When SSIS Loads Data

Stop Adding Indexes: What's Actually Slowing Your SQL Server Queries When SSIS Loads Data

The default response to a slow query is "add an index." Sometimes that's correct. More often, the index isn't the problem.

By 
Abhilash Rao Mesala user avatar
Abhilash Rao Mesala
·
Apr. 22, 26 · Analysis
Likes (0)
Comment
Save
Tweet
Share
2.2K Views

Join the DZone community and get the full member experience.

Join For Free

The Ticket That Started It

A query was taking 12 seconds pulling from a staging table that the morning SSIS package loads. Someone opened the execution plan, spotted a clustered index scan, and added a non-clustered index. The query dropped to 400ms. Ticket closed.

Three weeks later, the SSIS package started timing out. The ETL window that used to finish in 40 minutes was now running 90. Nobody connected the two events as they happened weeks apart and the symptoms looked completely unrelated. Different team members, different Jira boards, different oncall rotations.

But it was the same problem, just showing up on the other end.

Your SSIS Load Window Is Shrinking, Here's Why

Every non-clustered index you add to a SQL Server table is a write tax. Not just on your application inserts, it's on every INSERT, UPDATE, and DELETE, including the ones your SSIS package is doing at 6 AM. SQL Server has to maintain every index B-tree on every write. That's not a quirk, it's by design but it compounds fast when you're loading bulk data.

Take a staging table that an OLE DB Destination writes 500,000 rows into every morning:

MS SQL
 
CREATE TABLE dbo.OrderStaging (
    OrderID        INT            NOT NULL,
    CustomerID     INT            NOT NULL,
    OrderDate      DATETIME2      NOT NULL,
    ProductCode    VARCHAR(50)    NOT NULL,
    Quantity       INT            NOT NULL,
    UnitPrice      DECIMAL(18,4)  NOT NULL,
    Status         VARCHAR(20)    NOT NULL,
    LoadedAt       DATETIME2      DEFAULT GETUTCDATE()
);


Then over six months, four indexes get added, each one justified by a legitimate query complaint:

MS SQL
 
-- Daily sales report needed this
CREATE NONCLUSTERED INDEX IX_OrderStaging_CustomerID
ON dbo.OrderStaging (CustomerID);

-- Product summary query
CREATE NONCLUSTERED INDEX IX_OrderStaging_ProductCode
ON dbo.OrderStaging (ProductCode, OrderDate);

-- Status dashboard
CREATE NONCLUSTERED INDEX IX_OrderStaging_Status
ON dbo.OrderStaging (Status) INCLUDE (OrderID, UnitPrice);

-- Date range filter
CREATE NONCLUSTERED INDEX IX_OrderStaging_OrderDate
ON dbo.OrderStaging (OrderDate);


Every one of those indexes was added for good reason. And every one of them is now being maintained on all 500,000 rows your SSIS package writes each morning. You're not doing 500,000 inserts you're doing closer to 2.5 million index page operations. The ETL window didn't shrink because the data got bigger. It shrunk because the index maintenance overhead grew every time someone fixed a slow query the "obvious" way.

What Actually Helps

The approach that works is treating the load window and the query window as separate concerns. Disable the non-clustered indexes before SSIS runs, let the data load against a bare table, then rebuild afterward:

MS SQL
 
-- Step 1: Disable indexes before SSIS loads
ALTER INDEX IX_OrderStaging_CustomerID  ON dbo.OrderStaging DISABLE;
ALTER INDEX IX_OrderStaging_ProductCode ON dbo.OrderStaging DISABLE;
ALTER INDEX IX_OrderStaging_Status      ON dbo.OrderStaging DISABLE;
ALTER INDEX IX_OrderStaging_OrderDate   ON dbo.OrderStaging DISABLE;

-- Step 2: SSIS loads the data here (bare table, no index maintenance overhead)

-- Step 3: Rebuild indexes after load completes
ALTER INDEX IX_OrderStaging_CustomerID  ON dbo.OrderStaging REBUILD;
ALTER INDEX IX_OrderStaging_ProductCode ON dbo.OrderStaging REBUILD;
ALTER INDEX IX_OrderStaging_Status      ON dbo.OrderStaging REBUILD;
ALTER INDEX IX_OrderStaging_OrderDate   ON dbo.OrderStaging REBUILD;


Wrap these in Execute SQL Tasks in your SSIS control flow one task before the Data Flow Task and one after. The rebuild produces a clean, unfragmented index, which is actually better than the incrementally maintained version you'd have gotten from live insert-time maintenance.

For tables that get truncated and fully reloaded each cycle, this is essentially free TRUNCATE TABLE already deallocates the index pages, so you're rebuilding from scratch regardless. Making it explicit just ensures it's always done, not assumed.

The rebuild also produces a clean, unfragmented index. That's better than what you'd have gotten from six months of live insert-time maintenance anyway.

That said, index sprawl only explains part of why queries stay slow after a load. The more invisible problem  and the one that trips up even experienced SQL Server developers is when the index is there, it's not fragmented, and the optimizer still won't use it.

The Index That SQL Server Ignores

The scenario: you've built an index on a column, statistics are current, the query filters on exactly that column. The execution plan shows a scan anyway.

The most common reason is an implicit conversion and it's almost always introduced by an SSIS pipeline with a type mismatch between the source and the destination table.

When SQL Server receives a value of a different type than the column it's filtering on, it casts every value in the column to match the incoming type before comparing which means reading the entire column, index or not. Here's the exact scenario that causes it:

MS SQL
 
CREATE TABLE dbo.CustomerAccounts (
    AccountNumber  VARCHAR(20)    NOT NULL,  -- VARCHAR in SQL Server
    AccountName    NVARCHAR(100)  NOT NULL,
    Region         CHAR(3)        NOT NULL
);

CREATE NONCLUSTERED INDEX IX_CustomerAccounts_AccountNumber
ON dbo.CustomerAccounts (AccountNumber);


And this query, run after the SSIS load:

MS SQL
 
-- Application passes AccountNumber as NVARCHAR
DECLARE @AccountNum NVARCHAR(20) = N'ACC-00042';

SELECT AccountName, Region
FROM dbo.CustomerAccounts
WHERE AccountNumber = @AccountNum;


The column is VARCHAR. The parameter is NVARCHAR. SQL Server silently converts every VARCHAR value in the index to NVARCHAR before comparing the index becomes useless. In the execution plan, you'll see a warning triangle on the scan node with the message: "Type conversion in expression may affect CardinalityEstimate."

You can also catch implicit conversions across your entire server by querying the plan cache:

MS SQL
 
SELECT
    qs.execution_count,
    qs.total_logical_reads / qs.execution_count AS avg_logical_reads,
    SUBSTRING(qt.text,
        (qs.statement_start_offset / 2) + 1,
        ((CASE qs.statement_end_offset
              WHEN -1 THEN DATALENGTH(qt.text)
              ELSE qs.statement_end_offset
          END - qs.statement_start_offset) / 2) + 1
    ) AS query_text,
    qp.query_plan
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle)  qt
CROSS APPLY sys.dm_exec_query_plan(qs.plan_handle) qp
WHERE CAST(qp.query_plan AS NVARCHAR(MAX)) LIKE '%ImplicitConvert%'
ORDER BY avg_logical_reads DESC;


Run that on any production instance that's been running for more than a few months. You'll find something. It's a useful audit to do after any major SSIS pipeline deployment.

Why SSIS Is Particularly Prone to This

SSIS has its own internal type system that maps to SQL Server types when writing data. The mappings are mostly correct, but there are sharp edges worth knowing:

SSIS Type SQL Server Type
DT_STR VARCHAR
DT_WSTR NVARCHAR
DT_NUMERIC DECIMAL / NUMERIC
DT_DBTIMESTAMP2 DATETIME2
DT_I4 INT


If your source column in the SSIS Data Flow arrives as DT_WSTR and your destination SQL Server column is VARCHAR, every write involves an implicit narrowing conversion. It doesn't fail it silently converts. And every query filtering on that column afterward hits the scan-instead-of-seek problem.

The discipline here is: audit your SSIS Data Flow column types against your destination table DDL before you build any indexes. Put a Data Conversion Transformation before the OLE DB Destination to enforce the right types at the pipeline level. Not at query time, when the data is already there and the damage is done.

Both of those issues index sprawl and implicit conversions are configuration problems. You can find them with the right queries and fix them once. The third issue is harder because it's not a configuration problem at all. It's a behavior that SQL Server is doing correctly by design, and it still breaks things.

The Query That Works Perfectly in Dev

SQL Server compiles an execution plan the first time a stored procedure runs, using the parameter values from that first call to estimate row counts. It caches that plan and reuses it for every subsequent execution even when the parameters are completely different in terms of the data they return.

When your data distribution is even across values, that's fine. When it's skewed which is nearly always the case in real staging tables loaded by SSIS you end up with a cached plan that was built for a tiny test case and gets hammered by full production data volumes.

A stored procedure querying the same OrderStaging table from the earlier example:

MS SQL
 
CREATE PROCEDURE dbo.GetOrdersByRegion
    @Region     CHAR(3),
    @StartDate  DATETIME2,
    @EndDate    DATETIME2
AS
BEGIN
    SELECT
        o.OrderID,
        o.CustomerID,
        o.ProductCode,
        o.Quantity,
        o.UnitPrice
    FROM dbo.OrderStaging o
    WHERE o.Region    = @Region
      AND o.OrderDate BETWEEN @StartDate AND @EndDate;
END


In development, someone tests with @Region = 'TST' a test region with 50 rows. SQL Server compiles a nested loop join with an index seek. Makes sense for 50 rows.

First week in production, the application calls this with @Region = 'USA' 400,000 rows. SQL Server pulls the cached plan from dev. The nested loop that was efficient for 50 rows now iterates 400,000 times. The procedure that ran in 30ms in dev runs in 45 seconds in production, and nobody immediately knows why because the code hasn't changed.

How to Confirm It

Pull the actual execution plan and compare estimated rows to actual rows. When estimated is dramatically lower than actual, that's the sniffing signature the plan was compiled for a different world than the one it's running in. Force a recompile on the specific call to confirm:

MS SQL
 
EXEC dbo.GetOrdersByRegion
    @Region    = 'USA',
    @StartDate = '2025-01-01',
    @EndDate   = '2025-03-31'
WITH RECOMPILE;


If that runs significantly faster, you have your answer. Now pick the right fix for the situation:

Three Ways to Handle It

Option 1: OPTIMIZE FOR UNKNOWN

Tells the optimizer to compile based on average statistics rather than the sniffed value. Right choice when data distribution is reasonably even:

MS SQL
 
CREATE PROCEDURE dbo.GetOrdersByRegion
    @Region     CHAR(3),
    @StartDate  DATETIME2,
    @EndDate    DATETIME2
AS
BEGIN
    SELECT o.OrderID, o.CustomerID, o.ProductCode, o.Quantity, o.UnitPrice
    FROM dbo.OrderStaging o
    WHERE o.Region    = @Region
      AND o.OrderDate BETWEEN @StartDate AND @EndDate
    OPTION (OPTIMIZE FOR UNKNOWN);
END


Option 2: Local Variable Masking

Assigning parameters to local variables prevents SQL Server from sniffing the values at all it compiles based on statistics alone. No hint required:

MS SQL
 
CREATE PROCEDURE dbo.GetOrdersByRegion
    @Region     CHAR(3),
    @StartDate  DATETIME2,
    @EndDate    DATETIME2
AS
BEGIN
    DECLARE @LocalRegion    CHAR(3)    = @Region;
    DECLARE @LocalStartDate DATETIME2  = @StartDate;
    DECLARE @LocalEndDate   DATETIME2  = @EndDate;

    SELECT o.OrderID, o.CustomerID, o.ProductCode, o.Quantity, o.UnitPrice
    FROM dbo.OrderStaging o
    WHERE o.Region    = @LocalRegion
      AND o.OrderDate BETWEEN @LocalStartDate AND @LocalEndDate;
END


Option 3: WITH RECOMPILE at the Procedure Level

For procedures called infrequently with highly variable parameter ranges exactly the post-load report scenario common in SSIS workflows recompiling every execution is acceptable:

MS SQL
 
CREATE PROCEDURE dbo.GetOrdersByRegion
    @Region     CHAR(3),
    @StartDate  DATETIME2,
    @EndDate    DATETIME2
WITH RECOMPILE
AS
BEGIN
    ...
END


The recompile overhead per call is real don't use this on a procedure running thousands of times per minute. But for a procedure that runs twice a day after an SSIS load completes, it's completely appropriate and eliminates the problem permanently.

What I Actually Do Before and After an SSIS Load

The three problems above sound disconnected, but they have a natural order in a real pipeline. Before the Data Flow Task runs, I disable the non-clustered indexes on any destination table that's getting a full reload. That eliminates the write amplification completely. Inside the Data Flow itself, I've learned to always put a Data Conversion Transformation before OLE DB Destinations on any column that came from an external source, source systems rarely match your SQL Server DDL types exactly, and finding that mismatch at load time is far better than diagnosing it from a plan cache query at 9 AM.

After the Data Flow completes, I rebuild the indexes with ALTER INDEX ... REBUILD rather than just re-enabling them. A rebuild gives you a clean, unfragmented index. Then I run UPDATE STATISTICS dbo.OrderStaging WITH FULLSCAN before any downstream stored procedures execute. Fresh statistics mean the optimizer is working with accurate row counts from today's load not carryover estimates from last week. That one step reduces the sniffing risk significantly on the first post-load queries.

For stored procedures that query those tables with parameters that can vary widely by region, date range, or customer segment, I add OPTION (OPTIMIZE FOR UNKNOWN) during development rather than waiting to discover the sniffing problem in production. It takes 30 seconds to add and avoids a pattern that's otherwise reliably going to bite you on the first week of a production deployment.

Closing Thought

The slow query is almost never caused by the thing that looks like the obvious fix. Indexes help reads but they have a write cost that compounds over months in SSIS-heavy systems. Implicit conversions are invisible until you know what the warning triangle in the execution plan is telling you. Parameter sniffing is SQL Server doing exactly what it's designed to do, in a way that breaks production.

What these three have in common is that the execution plan shows you all of it. Estimated rows vs. actual rows, implicit convert warnings, seek vs. scan it's all there. The tool isn't the limitation. Knowing what to look for is.

Extract, transform, load Data (computing) sql

Opinions expressed by DZone contributors are their own.

Related

  • Advanced Maintenance of a Multi-Database Citus Cluster With Flyway
  • Beyond Partitioning and Z-Order: A Deep Dive into Liquid Clustering for Unity Catalog Managed Tables
  • Modernizing Cloud Data Automation for Faster Insights
  • Reconciling Privacy Preferences Across Two Datastores With Snowflake and Airflow

Partner Resources

×

Comments

The likes didn't load as expected. Please refresh the page and try again.

  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook