Transcription
Migrating to Snowflake?Here’s What You Need to Test
Migrating to Snowflake? 1Migrating legacy data (e.g., from IBM Netezza, Oracle, MSSQL, PostgreSQL ) to Snowflake is not a simple “liftand shift.” It can’t happen all at once, and it must be tested to ensure the tens of thousands of data reportscontinue to operate properly on the new model. Organizations must ensure the data is moved efficiently byperforming extensive validation and reconciliation across the old and new worlds.The current process surrounding the migration and ongoing testing of Snowflake is to utilize manual teststhat create a dependency matrix. This process is error prone, so many different iterations are usuallyrequired. As a result, timelines can easily shift from days/weeks to months—adding considerable delays andcosts to the migration project.This paper outlines the top challenges that enterprise organizations typically encounter during a Snowflakemigration. For each challenge, we briefly explain how Tricentis Data Integrity has been used to address thechallenge. We conclude with a proactive approach to eliminating data integrity issues—before, during, andafter Snowflake migration.Challenges of Snowflake MigrationNo database migration is simple. Gartner reported that 83% of data migration projects either fail to meetbudgets and schedule expectations or fail altogether. Migrating workloads from on-prem solutions to clouddatabases is even more complex. Forrester estimates that an average Snowflake migration requires full-timeinvolvement from 3 DBAs/IT staff for 6 months—plus considerable consulting time.Here are the top Snowflake migration challenges we’ve encountered at customer and prospect sites, alongwith ways to address them.Data must be migrated incrementallySnowflake is so popular that they limit the transactions to their system. You can be limited to transferring only10GB a day unless you get special permission. For a 5 terabyte system, that could mean 500 days.if you do itperfectly. To avoid drawing out an already lengthy process, automatically reconcile and validate each transferas it happens.Tricentis Data Integrity verifies that the data moves efficiently and accurately. Automated reconciliation testsprovide instant insight into which transformation requirements have been tested and whether those testssucceeded or failed.Organizations don’t want to move and store bad dataSince Snowflake charges per terabyte, it’s in your best interest to clean “garbage data” and duplicate recordsbefore moving it over. However, few organizations have the time or resources to do this at any s.comv
Migrating to Snowflake? 2less when they’re preparing for a massive migration project. Automatically-generated tests that expose dataerrors will not only save time, but also enable a much more thorough and accurate inspection than manualefforts ever could.Tricentis Data Integrity’s “pre-screening” tests exposes data that isn’t fit for migration. For instance, it findsmissing values, duplicates, data formats issues, data beyond the acceptable range, etc.Migrating workloads is tedious and error-proneMoving workloads from legacy environments to Snowflake is an error-prone activity with a high risk ofdisrupting business as usual. Migrating code, business logic, and analytics jobs all have their own set ofunique challenges. For example, workloads must have their exact target equivalent matching the productionperformance SLAs. To achieve this, enterprises must perform all the following steps before putting newworkloads into production:1.2.3.4.Thoroughly assess the existing inventory of workloads to identify the chain of workloads to be moved.Match the source and target data.Convert scripts, business logic, reporting logic, etc.Validate the migrated logic.Tricentis Data Integrity is used to validate the migrated logic. Typically, we find that 60% of the legacy dataworkloads can be migrated as-is, 20% workloads might require some additional optimization, and 20%workloads require total re-engineering. In all cases, testing can be automated with our end-to-end suite ofdata integrity tests. These tests span from pre-screening, to vital checks for consistency and correctness,through any data transformations, and finally to the analytics and report checks that verify the process wascompleted correctly.Data processes are deeply embeddedWith RDBMS, existing ETL pipelines push data to legacy warehouses, customized visualization tools pull dataout of their warehouses, and custom applications also depend closely on data from their warehouse. Whenyou move to Snowflake, all these processes must be re-engineered and tested.Tricentis Data Integrity can effectively deal with the reconciliation and validation required to make themigration comv
Migrating to Snowflake? 3Rethink Data Testing: Before, During, andAfter Your Snowflake MigrationRather than tackle these challenges with a “whack-a-mole” approach, use Snowflake migration as anopportunity to modernize and transform your overall approach to data integrity—just like you’re modernizingand transforming your approach to data management.Tricentis Data Integrity’s end-to-end data reconciliation and validation has helped top organizations unleashthe full power and speed of Snowflake. Before the integration, take the opportunity to assess the data, identify issues, and fix them so yourSnowflake data is streamlined and accurate from the start. During the migration, automatically detect unintentional changes from the old data to the newSnowflake stores and processes. This automated regression testing can run throughout the migrationperiod to expose change impacts the moment they are introduced—which is when they are 10X fasterto find and fix. Once you’re up and running on Snowflake, reuse the same tests to identify when ongoing systemmodifications compromise your processes and data. These extensible, reusable, and resilient testsand embed them into the DevOps toolchain of your choice. With this baseline, you can exposeunintentional data impacts as soon as they occur.For a deeper dive into what’s involved in this strategy—including a look at how we approach each step—watch our webinar.About Tricentis Data IntegrityTricentis Data Integrity is the industry’s top end-to-end data testing solution for enterprise organizations. Ourend-to-end automation covers everything from the integrity of the data fed into your system, to the accuracyof integrations, transformations, and migrations, to verification of report logic and presentation.Tricentis Data Integrity takes advantage of the unique capabilities of Snowflake. For example, for Time Travel,we create tests that profile and monitor changes in the data as it enters the process—not at the end whenthis bad data impacts end users in the business units.What sets Tricentis Data Integrity apart?www.tricentis.comTricentiswww.tricentis.comv
Migrating to Snowflake? 4 End-to-end: Automates end-to-end data testing covering all reconciliation and validation tasks fromsources to stores to reporting and visualizations. Any technology: Sits on top of any data landscape, covering structured, unstructured, and messagedata from any source or technology as well as reports in any analytics tool via UI, API, and PDF. Snowflake enrichment: Allows you to create tests utilizing Snowflake’s unique capabilities (such asTime Travel) to pinpoint data regression issues as they happen at the source. Accessible automation: Enables Business Analysts, Data Stewards, Data Engineers, etc. to automatetesting, replacing spotty “state and compare” checking as well as complex, unscalable SQL scripting. CI/CD integration: Integrates into CI/CD pipelines to ensure frequent application changes don’tinadvertently alter ETL processes and compromise data quality. Enterprise grade: Delivers a mature enterprise-grade solution with highly-scalable performance andenterprise-grade global support to help you achieve your goals, fast. Risk-based: Guides teams to focus limited testing resources on top business risks; reveals whether arelease candidate is sufficiently tested and fit for release.Next StepsLearn more about how Tricentis can help your organization simplify your migration to Snowflake and ensurethat ongoing system modifications in Snowflake don’t compromise data integrity. Contact your organization’sTricentis representative to schedule a briefing with our data integrity s.comv
Migrating legacy data (e.g., from IBM Netezza, Oracle, MSSQL, PostgreSQL ) to Snowflake is not a simple "lift . The current process surrounding the migration and ongoing testing of Snowflake is to utilize manual tests . With RDBMS, existing ETL pipelines push data to legacy warehouses, customized visualization tools pull data out of .
